[
https://issues.apache.org/jira/browse/TRAFODION-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067604#comment-16067604
]
ASF GitHub Bot commented on TRAFODION-2664:
-------------------------------------------
GitHub user mkby opened a pull request:
https://github.com/apache/incubator-trafodion/pull/1155
[TRAFODION-2664] Instance will be down when the zookeeper on name node has
been down
do retry on every 'zoo_exists' call when we get a ZCONNECTIONLOSS failure.
@zcorrea , could you have to review the changes?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mkby/incubator-trafodion 2664
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-trafodion/pull/1155.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1155
----
commit 7e1408c5584ae587e4d1198fa0bcf95a4c43c2cc
Author: Eason <[email protected]>
Date: 2017-06-29T02:25:07Z
[TRAFODION-2664] Instance will be down when the zookeeper on name node has
been down
----
> Instance will be down when the zookeeper on name node has been down
> -------------------------------------------------------------------
>
> Key: TRAFODION-2664
> URL: https://issues.apache.org/jira/browse/TRAFODION-2664
> Project: Apache Trafodion
> Issue Type: Bug
> Components: foundation
> Affects Versions: 2.2-incubating
> Environment: Test Environment:
> CDH5.4.8: 10.10.23.19:7180, total 6 nodes.
> HDFS-HA and DCS-HA: enabled
> OS: Centos6.8, physic machine.
> SW Build: R2.2.3 (EsgynDB_Enterprise Release 2.2.3 (Build release [sbroeder],
> branch 1ce8d39-xdc_nari, date 11Jun17)
> Reporter: Jarek
> Assignee: Gonzalo E Correa
> Priority: Critical
> Labels: build
> Fix For: 2.2-incubating
>
>
> Description: Instance will be down when the zookeeper on name node has been
> down
> Test Steps:
> Step 1. Start OE and 4 long queries with trafci on the first node
> esggy-clu-n010
> Step 2. Wait several minutes and stop zookeeper on name node of node
> esggy-clu-n010 in Cloudera Manager page.
> Step 3. With trafci, run a basic query and 4 long queries again.
> In the above Step 3, we will see the whole instance as down after a while.
> For this test scenario, I tried it several times, always found instance as
> down.
> Timestamp:
> Test Start Time: 20170616132939
> Test End Time: 20170616134350
> Stop zookeeper on name node of node esggy-clu-n010: 20170616133344
> Check logs:
> 1) Each node displays the following error:
> 2017-06-16 13:33:46,276, ERROR, MON, Node Number: 0,, PIN: 5017 , Process
> Name: $MONITOR,,, TID: 5429, Message ID: 101371801,
> [CZClient::IsZNodeExpired], zoo_exists() for
> /trafodion/instance/cluster/esggy-clu-n010.esgyn.cn failed with error
> ZCONNECTIONLOSS
> 2) Zookeeper displays:
> ls /trafodion/instance/cluster
> []
> So, It seems zclient has been lost on each node.
> Location of logs:
> esggy-clu-n010:
> /data4/jarek/ha.interactive/trafodion_and_cluster_logs/cluster_logs.20170616134816.tar.gz
> and trafodion_logs.20170616134816.tar.gz
> By the way, because the size of the logs is out of the limited value, so i
> cannot upload it as the attachment in this JIRA ID.
> How many zookeeper quorum servers in the cluster? total 3.
> <property>
> <name>dcs.zookeeper.quorum</name>
>
> <value>esggy-clu-n010.esgyn.cn,esggy-clu-n011.esgyn.cn,esggy-clu-n012.esgyn.cn</value>
> </property>
> How to access the cluster?
> 1. Login 10.10.10.8 from US machine: trafodion/traf123
> 2. Login 10.10.23.19 from 10.10.10.8: trafodion/traf123
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
