[
https://issues.apache.org/jira/browse/HADOOP-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14583024#comment-14583024
]
lvchuanwen commented on HADOOP-10251:
-------------------------------------
Answer 2.when start only one ZKFC and NN ，the NN can be staying in ACTIVE for
long time.
############################hdfs-nn1-zkfc-host195.log################################
2015-06-12 07:18:11,471 INFO
org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Failover controller
configured for NameNode NameNode at zdh195/10.43.156.195:9000
2015-06-12 07:18:11,608 INFO org.apache.zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.4.5-cdh5.3.2--1, built on 05/15/2015 03:44 GMT
2015-06-12 07:18:11,608 INFO org.apache.zookeeper.ZooKeeper: Client
environment:host.name=zdh195
2015-06-12 07:18:11,608 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.version=1.7.0_55
2015-06-12 07:18:11,608 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.vendor=Oracle Corporation
2015-06-12 07:18:11,608 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.home=/usr/java/jdk/jre
2015-06-12 07:18:11,608 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.class.path=/home/hdfs/hdfs/etc/hadoop:/home/hdfs/hdfs/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-cli-1.2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/curator-client-2.6.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jsch-0.1.42.jar:/home/hdfs/hdfs/share/hadoop/common/lib/avro-1.7.6-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/hdfs/hdfs/share/hadoop/common/lib/curator-framework-2.6.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/hdfs/hdfs/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jsp-api-2.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/httpclient-4.2.5.jar:/home/hdfs/hdfs/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/hdfs/hdfs/share/hadoop/common/lib/hadoop-auth-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jetty-6.1.26.cloudera.4.jar:/home/hdfs/hdfs/share/hadoop/common/lib/xz-1.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-net-3.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-io-2.4.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jersey-core-1.9.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/hdfs/hdfs/share/hadoop/common/lib/guava-11.0.2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/asm-3.2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-digester-1.8.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-lang-2.6.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/common/lib/curator-recipes-2.6.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/activation-1.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/hdfs/hdfs/share/hadoop/common/lib/cdh-commons-1.04.03.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/servlet-api-2.5.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-codec-1.4.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jersey-json-1.9.jar:/home/hdfs/hdfs/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hdfs/hdfs/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/common/lib/zookeeper-3.4.5-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.4.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-el-1.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/httpcore-4.2.5.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/hdfs/hdfs/share/hadoop/common/lib/gson-2.2.4.jar:/home/hdfs/hdfs/share/hadoop/common/lib/log4j-1.2.17.jar:/home/hdfs/hdfs/share/hadoop/common/lib/hadoop-annotations-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jettison-1.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/hdfs/hdfs/share/hadoop/common/lib/zdh-restclient-1.04.01.jar:/home/hdfs/hdfs/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/cooma-0.4.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/xmlenc-0.52.jar:/home/hdfs/hdfs/share/hadoop/common/lib/junit-4.11.jar:/home/hdfs/hdfs/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/home/hdfs/hdfs/share/hadoop/common/lib/security-checkclients-1.04.03.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jersey-server-1.9.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/common/lib/paranamer-2.3.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/hdfs/hdfs/share/hadoop/common/hadoop-nfs-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/common/hadoop-common-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/common/hadoop-common-2.5.0-cdh5.3.2-tests.jar:/home/hdfs/hdfs/share/hadoop/hdfs:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-mapreduce-client-shuffle-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-aws-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jackson-databind-2.2.3.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jersey-client-1.9.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/httpclient-4.2.5.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-lzo-0.4.20.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.4.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jackson-jaxrs-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-client-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-mapreduce-client-common-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-mapreduce-client-core-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-mapreduce-client-app-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/asm-3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jackson-xc-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-yarn-common-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/activation-1.1.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jackson-core-2.2.3.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-yarn-api-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/stax-api-1.0-2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-yarn-server-common-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-hdfs-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.4.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jaxb-api-2.2.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jackson-annotations-2.2.3.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/httpcore-4.2.5.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-mapreduce-client-jobclient-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/aws-java-sdk-1.7.4.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-yarn-client-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/hdfs/hadoop-hdfs-2.5.0-cdh5.3.2-tests.jar:/home/hdfs/hdfs/share/hadoop/hdfs/hadoop-hdfs-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/hadoop-hdfs-nfs-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/hadoop-concurrent-copy-2.5.0-cdh5.3.2-tests.jar:/home/hdfs/hdfs/share/hadoop/hdfs/hadoop-concurrent-copy-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/javax.inject-1.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jline-0.9.94.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jetty-6.1.26.cloudera.4.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/xz-1.0.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jackson-jaxrs-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/asm-3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jackson-xc-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/guice-3.0.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/activation-1.1.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/zookeeper-3.4.5-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jetty-util-6.1.26.cloudera.4.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jettison-1.1.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-common-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-api-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-server-tests-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-server-common-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-client-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/avro-1.7.6-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/hadoop-annotations-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0-cdh5.3.2-tests.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/contrib/capacity-scheduler/*.jar:/home/hdfs/hdfs/contrib/capacity-scheduler/*.jar:/home/hdfs/hdfs/contrib/capacity-scheduler/*.jar
2015-06-12 07:18:11,608 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.library.path=/home/hdfs/hdfs/lib/native
2015-06-12 07:18:11,608 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
2015-06-12 07:18:11,608 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2015-06-12 07:18:11,608 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.name=Linux
2015-06-12 07:18:11,609 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2015-06-12 07:18:11,609 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.version=2.6.32-431.el6.x86_64
2015-06-12 07:18:11,609 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.name=hdfs
2015-06-12 07:18:11,609 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.home=/home/hdfs
2015-06-12 07:18:11,609 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.dir=/home/hdfs/hdfs
2015-06-12 07:18:11,609 INFO org.apache.zookeeper.ZooKeeper: Initiating client
connection, connectString=zdh196:2181,zdh195:2181,zdh197:2181
sessionTimeout=10000
watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@3c1e1fd3
2015-06-12 07:18:11,624 INFO org.apache.zookeeper.ClientCnxn: Opening socket
connection to server zdh195/10.43.156.195:2181. Will not attempt to
authenticate using SASL (unknown error)
2015-06-12 07:18:11,626 INFO org.apache.zookeeper.ClientCnxn: Socket connection
established to zdh195/10.43.156.195:2181, initiating session
2015-06-12 07:18:11,632 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server zdh195/10.43.156.195:2181, sessionid =
0x24d91acb5cb1c57, negotiated timeout = 10000
2015-06-12 07:18:11,636 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session
connected.
2015-06-12 07:18:11,662 INFO org.apache.hadoop.ipc.CallQueueManager: Using
callQueue class java.util.concurrent.LinkedBlockingQueue
2015-06-12 07:18:11,684 INFO org.apache.hadoop.ipc.Server: Starting Socket
Reader #1 for port 8019
2015-06-12 07:18:11,712 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2015-06-12 07:18:11,713 INFO org.apache.hadoop.ipc.Server: IPC Server listener
on 8019: starting
2015-06-12 07:18:11,831 INFO org.apache.hadoop.ha.HealthMonitor: Entering state
SERVICE_HEALTHY
2015-06-12 07:18:11,831 INFO org.apache.hadoop.ha.ZKFailoverController: Local
service NameNode at zdh195/10.43.156.195:9000 entered state: SERVICE_HEALTHY
2015-06-12 07:18:11,854 INFO org.apache.hadoop.ha.ZKFailoverController: ZK
Election indicated that NameNode at zdh195/10.43.156.195:9000 should become
standby
2015-06-12 07:18:11,863 INFO org.apache.hadoop.ha.ZKFailoverController:
Successfully transitioned NameNode at zdh195/10.43.156.195:9000 to standby state
2015-06-12 07:20:02,104 INFO
org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Failover controller
configured for NameNode NameNode at zdh195/10.43.156.195:9000
2015-06-12 07:20:02,250 INFO org.apache.zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.4.5-cdh5.3.2--1, built on 05/15/2015 03:44 GMT
2015-06-12 07:20:02,250 INFO org.apache.zookeeper.ZooKeeper: Client
environment:host.name=zdh195
2015-06-12 07:20:02,250 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.version=1.7.0_55
2015-06-12 07:20:02,250 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.vendor=Oracle Corporation
2015-06-12 07:20:02,250 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.home=/usr/java/jdk/jre
2015-06-12 07:20:02,250 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.class.path=/home/hdfs/hdfs/etc/hadoop:/home/hdfs/hdfs/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-cli-1.2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/curator-client-2.6.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jsch-0.1.42.jar:/home/hdfs/hdfs/share/hadoop/common/lib/avro-1.7.6-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/hdfs/hdfs/share/hadoop/common/lib/curator-framework-2.6.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/hdfs/hdfs/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jsp-api-2.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/httpclient-4.2.5.jar:/home/hdfs/hdfs/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/hdfs/hdfs/share/hadoop/common/lib/hadoop-auth-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jetty-6.1.26.cloudera.4.jar:/home/hdfs/hdfs/share/hadoop/common/lib/xz-1.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-net-3.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-io-2.4.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jersey-core-1.9.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/hdfs/hdfs/share/hadoop/common/lib/guava-11.0.2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/asm-3.2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-digester-1.8.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-lang-2.6.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/common/lib/curator-recipes-2.6.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/activation-1.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/hdfs/hdfs/share/hadoop/common/lib/cdh-commons-1.04.03.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/servlet-api-2.5.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-codec-1.4.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jersey-json-1.9.jar:/home/hdfs/hdfs/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hdfs/hdfs/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/common/lib/zookeeper-3.4.5-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.4.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-el-1.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/httpcore-4.2.5.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/hdfs/hdfs/share/hadoop/common/lib/gson-2.2.4.jar:/home/hdfs/hdfs/share/hadoop/common/lib/log4j-1.2.17.jar:/home/hdfs/hdfs/share/hadoop/common/lib/hadoop-annotations-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jettison-1.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/hdfs/hdfs/share/hadoop/common/lib/zdh-restclient-1.04.01.jar:/home/hdfs/hdfs/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/cooma-0.4.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/xmlenc-0.52.jar:/home/hdfs/hdfs/share/hadoop/common/lib/junit-4.11.jar:/home/hdfs/hdfs/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/home/hdfs/hdfs/share/hadoop/common/lib/security-checkclients-1.04.03.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jersey-server-1.9.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/common/lib/paranamer-2.3.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/hdfs/hdfs/share/hadoop/common/hadoop-nfs-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/common/hadoop-common-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/common/hadoop-common-2.5.0-cdh5.3.2-tests.jar:/home/hdfs/hdfs/share/hadoop/hdfs:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-mapreduce-client-shuffle-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-aws-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jackson-databind-2.2.3.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jersey-client-1.9.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/httpclient-4.2.5.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-lzo-0.4.20.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.4.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jackson-jaxrs-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-client-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-mapreduce-client-common-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-mapreduce-client-core-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-mapreduce-client-app-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/asm-3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jackson-xc-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-yarn-common-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/activation-1.1.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jackson-core-2.2.3.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-yarn-api-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/stax-api-1.0-2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-yarn-server-common-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-hdfs-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.4.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jaxb-api-2.2.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jackson-annotations-2.2.3.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/httpcore-4.2.5.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-mapreduce-client-jobclient-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/aws-java-sdk-1.7.4.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-yarn-client-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/hdfs/hadoop-hdfs-2.5.0-cdh5.3.2-tests.jar:/home/hdfs/hdfs/share/hadoop/hdfs/hadoop-hdfs-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/hadoop-hdfs-nfs-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/hadoop-concurrent-copy-2.5.0-cdh5.3.2-tests.jar:/home/hdfs/hdfs/share/hadoop/hdfs/hadoop-concurrent-copy-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/javax.inject-1.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jline-0.9.94.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jetty-6.1.26.cloudera.4.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/xz-1.0.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jackson-jaxrs-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/asm-3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jackson-xc-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/guice-3.0.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/activation-1.1.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/zookeeper-3.4.5-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jetty-util-6.1.26.cloudera.4.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jettison-1.1.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-common-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-api-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-server-tests-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-server-common-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-client-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/avro-1.7.6-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/hadoop-annotations-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0-cdh5.3.2-tests.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/contrib/capacity-scheduler/*.jar:/home/hdfs/hdfs/contrib/capacity-scheduler/*.jar:/home/hdfs/hdfs/contrib/capacity-scheduler/*.jar
2015-06-12 07:20:02,250 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.library.path=/home/hdfs/hdfs/lib/native
2015-06-12 07:20:02,251 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
2015-06-12 07:20:02,251 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2015-06-12 07:20:02,251 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.name=Linux
2015-06-12 07:20:02,251 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2015-06-12 07:20:02,251 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.version=2.6.32-431.el6.x86_64
2015-06-12 07:20:02,251 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.name=hdfs
2015-06-12 07:20:02,251 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.home=/home/hdfs
2015-06-12 07:20:02,251 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.dir=/home/hdfs/hdfs
2015-06-12 07:20:02,252 INFO org.apache.zookeeper.ZooKeeper: Initiating client
connection, connectString=zdh196:2181,zdh195:2181,zdh197:2181
sessionTimeout=10000
watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@3ddc25a9
2015-06-12 07:20:02,269 INFO org.apache.zookeeper.ClientCnxn: Opening socket
connection to server zdh196/10.43.156.196:2181. Will not attempt to
authenticate using SASL (unknown error)
2015-06-12 07:20:02,273 INFO org.apache.zookeeper.ClientCnxn: Socket connection
established to zdh196/10.43.156.196:2181, initiating session
2015-06-12 07:20:02,285 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server zdh196/10.43.156.196:2181, sessionid =
0x14d91acb8921ca0, negotiated timeout = 10000
2015-06-12 07:20:02,290 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session
connected.
2015-06-12 07:20:02,319 INFO org.apache.hadoop.ipc.CallQueueManager: Using
callQueue class java.util.concurrent.LinkedBlockingQueue
2015-06-12 07:20:02,342 INFO org.apache.hadoop.ipc.Server: Starting Socket
Reader #1 for port 8019
2015-06-12 07:20:02,373 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2015-06-12 07:20:02,373 INFO org.apache.hadoop.ipc.Server: IPC Server listener
on 8019: starting
2015-06-12 07:20:03,442 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: zdh195/10.43.156.195:9000. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
2015-06-12 07:20:03,446 WARN org.apache.hadoop.ha.HealthMonitor:
Transport-level exception trying to monitor health of NameNode at
zdh195/10.43.156.195:9000: Call From zdh195/10.43.156.195 to zdh195:9000 failed
on connection exception: java.net.ConnectException: 拒绝连接; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
2015-06-12 07:20:03,447 INFO org.apache.hadoop.ha.HealthMonitor: Entering state
SERVICE_NOT_RESPONDING
2015-06-12 07:20:03,447 INFO org.apache.hadoop.ha.ZKFailoverController: Local
service NameNode at zdh195/10.43.156.195:9000 entered state:
SERVICE_NOT_RESPONDING
2015-06-12 07:20:03,448 INFO org.apache.hadoop.ha.ZKFailoverController:
Quitting master election for NameNode at zdh195/10.43.156.195:9000 and marking
that fencing is necessary
2015-06-12 07:20:03,448 INFO org.apache.hadoop.ha.ActiveStandbyElector:
Yielding from election
2015-06-12 07:20:03,451 INFO org.apache.zookeeper.ZooKeeper: Session:
0x14d91acb8921ca0 closed
2015-06-12 07:20:03,451 INFO org.apache.zookeeper.ClientCnxn: EventThread shut
down
2015-06-12 07:20:05,454 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: zdh195/10.43.156.195:9000. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
2015-06-12 07:20:05,834 INFO org.apache.hadoop.ha.HealthMonitor: Entering state
SERVICE_HEALTHY
2015-06-12 07:20:05,834 INFO org.apache.hadoop.ha.ZKFailoverController: Local
service NameNode at zdh195/10.43.156.195:9000 entered state: SERVICE_HEALTHY
2015-06-12 07:20:05,844 INFO org.apache.zookeeper.ZooKeeper: Initiating client
connection, connectString=zdh196:2181,zdh195:2181,zdh197:2181
sessionTimeout=10000
watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@14b867db
2015-06-12 07:20:05,845 INFO org.apache.zookeeper.ClientCnxn: Opening socket
connection to server zdh197/10.43.156.197:2181. Will not attempt to
authenticate using SASL (unknown error)
2015-06-12 07:20:05,846 INFO org.apache.zookeeper.ClientCnxn: Socket connection
established to zdh197/10.43.156.197:2181, initiating session
2015-06-12 07:20:05,849 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server zdh197/10.43.156.197:2181, sessionid =
0x34dcf74b50a0896, negotiated timeout = 10000
2015-06-12 07:20:05,854 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session
connected.
2015-06-12 07:20:05,860 INFO org.apache.hadoop.ha.ActiveStandbyElector:
Checking for any old active which needs to be fenced...
2015-06-12 07:20:05,865 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old
node exists: 0a0a636c757374657231393512036e6e321a067a646831393620a84628d33e
2015-06-12 07:20:05,868 INFO org.apache.hadoop.ha.ZKFailoverController: Should
fence: NameNode at zdh196/10.43.156.196:9000
2015-06-12 07:20:06,873 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: zdh196/10.43.156.196:9000. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
2015-06-12 07:20:06,877 WARN org.apache.hadoop.ha.FailoverController: Unable to
gracefully make NameNode at zdh196/10.43.156.196:9000 standby (unable to
connect)
java.net.ConnectException: Call From zdh195/10.43.156.195 to zdh196:9000 failed
on connection exception: java.net.ConnectException: 拒绝连接; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
at org.apache.hadoop.ipc.Client.call(Client.java:1415)
at org.apache.hadoop.ipc.Client.call(Client.java:1364)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy12.transitionToStandby(Unknown Source)
at
org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.transitionToStandby(HAServiceProtocolClientSideTranslatorPB.java:112)
at
org.apache.hadoop.ha.FailoverController.tryGracefulFence(FailoverController.java:172)
at
org.apache.hadoop.ha.ZKFailoverController.doFence(ZKFailoverController.java:516)
at
org.apache.hadoop.ha.ZKFailoverController.fenceOldActive(ZKFailoverController.java:507)
at
org.apache.hadoop.ha.ZKFailoverController.access$1100(ZKFailoverController.java:61)
at
org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.fenceOldActive(ZKFailoverController.java:894)
at
org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:901)
at
org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:800)
at
org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:415)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:605)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:499)
Caused by: java.net.ConnectException: 拒绝连接
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
at
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463)
at org.apache.hadoop.ipc.Client.call(Client.java:1382)
... 14 more
2015-06-12 07:20:06,881 INFO org.apache.hadoop.ha.NodeFencer: ====== Beginning
Service Fencing Process... ======
2015-06-12 07:20:06,881 INFO org.apache.hadoop.ha.NodeFencer: Trying method
1/2: org.apache.hadoop.ha.SshFenceByTcpPort(null)
2015-06-12 07:20:06,899 INFO org.apache.hadoop.ha.SshFenceByTcpPort: Connecting
to zdh196...
2015-06-12 07:20:06,900 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
Connecting to zdh196 port 22
2015-06-12 07:20:06,907 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
Connection established
2015-06-12 07:20:06,931 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
Remote version string: SSH-2.0-OpenSSH_5.3
2015-06-12 07:20:06,931 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Local
version string: SSH-2.0-JSCH-0.1.42
2015-06-12 07:20:06,931 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
CheckCiphers:
aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256
2015-06-12 07:20:07,105 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
aes256-ctr is not available.
2015-06-12 07:20:07,106 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
aes192-ctr is not available.
2015-06-12 07:20:07,106 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
aes256-cbc is not available.
2015-06-12 07:20:07,106 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
aes192-cbc is not available.
2015-06-12 07:20:07,106 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
arcfour256 is not available.
2015-06-12 07:20:07,106 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
SSH_MSG_KEXINIT sent
2015-06-12 07:20:07,106 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
SSH_MSG_KEXINIT received
2015-06-12 07:20:07,107 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex:
server->client aes128-ctr hmac-md5 none
2015-06-12 07:20:07,107 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex:
client->server aes128-ctr hmac-md5 none
2015-06-12 07:20:07,115 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
SSH_MSG_KEXDH_INIT sent
2015-06-12 07:20:07,115 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
expecting SSH_MSG_KEXDH_REPLY
2015-06-12 07:20:07,125 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
ssh_rsa_verify: signature true
2015-06-12 07:20:07,128 WARN org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
Permanently added 'zdh196' (RSA) to the list of known hosts.
2015-06-12 07:20:07,128 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
SSH_MSG_NEWKEYS sent
2015-06-12 07:20:07,128 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
SSH_MSG_NEWKEYS received
2015-06-12 07:20:07,132 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
SSH_MSG_SERVICE_REQUEST sent
2015-06-12 07:20:07,133 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
SSH_MSG_SERVICE_ACCEPT received
2015-06-12 07:20:07,135 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
Authentications that can continue:
gssapi-with-mic,publickey,keyboard-interactive,password
2015-06-12 07:20:07,135 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Next
authentication method: gssapi-with-mic
2015-06-12 07:20:07,142 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
Authentications that can continue: publickey,keyboard-interactive,password
2015-06-12 07:20:07,142 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Next
authentication method: publickey
2015-06-12 07:20:07,188 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
Authentication succeeded (publickey).
2015-06-12 07:20:07,189 INFO org.apache.hadoop.ha.SshFenceByTcpPort: Connected
to zdh196
2015-06-12 07:20:07,189 INFO org.apache.hadoop.ha.SshFenceByTcpPort: Looking
for process running on port 9000
2015-06-12 07:20:09,289 INFO org.apache.hadoop.ha.SshFenceByTcpPort:
Indeterminate response from trying to kill service. Verifying whether it is
running using nc...
2015-06-12 07:20:11,049 INFO org.apache.hadoop.ha.SshFenceByTcpPort: Verified
that the service is down.
2015-06-12 07:20:11,050 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
Disconnecting from zdh196 port 22
2015-06-12 07:20:11,052 INFO org.apache.hadoop.ha.NodeFencer: ====== Fencing
successful by method org.apache.hadoop.ha.SshFenceByTcpPort(null) ======
2015-06-12 07:20:11,052 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing
znode /hadoop-ha/cluster195/ActiveBreadCrumb to indicate that the local node is
the most recent active...
2015-06-12 07:20:11,053 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch:
Caught an exception, leaving main loop due to Socket closed
2015-06-12 07:20:11,061 INFO org.apache.hadoop.ha.ZKFailoverController: Trying
to make NameNode at zdh195/10.43.156.195:9000 active...
2015-06-12 07:20:12,333 INFO org.apache.hadoop.ha.ZKFailoverController:
Successfully transitioned NameNode at zdh195/10.43.156.195:9000 to active state
############################hdfs-nn2-zkfc-host196.log################################
2015-06-12 07:18:26,905 INFO
org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Failover controller
configured for NameNode NameNode at zdh196/10.43.156.196:9000
2015-06-12 07:18:27,042 INFO org.apache.zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.4.5-cdh5.3.2--1, built on 05/15/2015 03:44 GMT
2015-06-12 07:18:27,042 INFO org.apache.zookeeper.ZooKeeper: Client
environment:host.name=zdh196
2015-06-12 07:18:27,042 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.version=1.7.0_55
2015-06-12 07:18:27,042 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.vendor=Oracle Corporation
2015-06-12 07:18:27,042 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.home=/usr/java/jdk/jre
2015-06-12 07:18:27,042 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.class.path=/home/hdfs/hdfs/etc/hadoop:/home/hdfs/hdfs/share/hadoop/common/lib/xz-1.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/hdfs/hdfs/share/hadoop/common/lib/curator-framework-2.6.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/activation-1.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/cooma-0.4.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/zookeeper-3.4.5-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/paranamer-2.3.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/curator-recipes-2.6.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/cdh-commons-1.04.03.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-cli-1.2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jsp-api-2.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/servlet-api-2.5.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/hdfs/hdfs/share/hadoop/common/lib/asm-3.2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.4.jar:/home/hdfs/hdfs/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/hdfs/hdfs/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-net-3.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/log4j-1.2.17.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jsch-0.1.42.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-el-1.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/common/lib/xmlenc-0.52.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/junit-4.11.jar:/home/hdfs/hdfs/share/hadoop/common/lib/gson-2.2.4.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-digester-1.8.jar:/home/hdfs/hdfs/share/hadoop/common/lib/httpclient-4.2.5.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-codec-1.4.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jersey-server-1.9.jar:/home/hdfs/hdfs/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jersey-json-1.9.jar:/home/hdfs/hdfs/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/httpcore-4.2.5.jar:/home/hdfs/hdfs/share/hadoop/common/lib/hadoop-auth-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-io-2.4.jar:/home/hdfs/hdfs/share/hadoop/common/lib/hadoop-annotations-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jetty-6.1.26.cloudera.4.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jettison-1.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hdfs/hdfs/share/hadoop/common/lib/zdh-restclient-1.04.01.jar:/home/hdfs/hdfs/share/hadoop/common/lib/avro-1.7.6-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-lang-2.6.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jersey-core-1.9.jar:/home/hdfs/hdfs/share/hadoop/common/lib/curator-client-2.6.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/hdfs/hdfs/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/hdfs/hdfs/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/hdfs/hdfs/share/hadoop/common/lib/security-checkclients-1.04.03.jar:/home/hdfs/hdfs/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/hdfs/hdfs/share/hadoop/common/lib/guava-11.0.2.jar:/home/hdfs/hdfs/share/hadoop/common/hadoop-common-2.5.0-cdh5.3.2-tests.jar:/home/hdfs/hdfs/share/hadoop/common/hadoop-common-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/common/hadoop-nfs-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs:/home/hdfs/hdfs/share/hadoop/hdfs/lib/activation-1.1.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-client-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/aws-java-sdk-1.7.4.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-mapreduce-client-shuffle-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/asm-3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-mapreduce-client-app-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.4.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-hdfs-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-yarn-client-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-yarn-common-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-lzo-0.4.20.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jackson-jaxrs-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-mapreduce-client-jobclient-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jackson-xc-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/stax-api-1.0-2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-yarn-server-common-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jackson-core-2.2.3.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-yarn-api-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/httpclient-4.2.5.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jackson-annotations-2.2.3.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/httpcore-4.2.5.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-mapreduce-client-common-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jackson-databind-2.2.3.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-mapreduce-client-core-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.4.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jaxb-api-2.2.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/jersey-client-1.9.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/hadoop-aws-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/hadoop-hdfs-nfs-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/hadoop-concurrent-copy-2.5.0-cdh5.3.2-tests.jar:/home/hdfs/hdfs/share/hadoop/hdfs/hadoop-hdfs-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/hdfs/hadoop-hdfs-2.5.0-cdh5.3.2-tests.jar:/home/hdfs/hdfs/share/hadoop/hdfs/hadoop-concurrent-copy-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/xz-1.0.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/guice-3.0.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/activation-1.1.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/zookeeper-3.4.5-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/asm-3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jetty-util-6.1.26.cloudera.4.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jline-0.9.94.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jackson-jaxrs-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jackson-xc-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/javax.inject-1.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jetty-6.1.26.cloudera.4.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jettison-1.1.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/hdfs/hdfs/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-client-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-common-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-server-tests-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-server-common-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-api-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/hadoop-annotations-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/avro-1.7.6-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0-cdh5.3.2.jar:/home/hdfs/hdfs/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0-cdh5.3.2-tests.jar:/home/hdfs/hdfs/contrib/capacity-scheduler/*.jar:/home/hdfs/hdfs/contrib/capacity-scheduler/*.jar:/home/hdfs/hdfs/contrib/capacity-scheduler/*.jar
2015-06-12 07:18:27,043 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.library.path=/home/hdfs/hdfs/lib/native
2015-06-12 07:18:27,043 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
2015-06-12 07:18:27,043 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2015-06-12 07:18:27,043 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.name=Linux
2015-06-12 07:18:27,043 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2015-06-12 07:18:27,043 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.version=2.6.32-431.el6.x86_64
2015-06-12 07:18:27,043 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.name=hdfs
2015-06-12 07:18:27,044 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.home=/home/hdfs
2015-06-12 07:18:27,044 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.dir=/home/hdfs/hdfs
2015-06-12 07:18:27,044 INFO org.apache.zookeeper.ZooKeeper: Initiating client
connection, connectString=zdh196:2181,zdh195:2181,zdh197:2181
sessionTimeout=10000
watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@1a722605
2015-06-12 07:18:27,060 INFO org.apache.zookeeper.ClientCnxn: Opening socket
connection to server zdh196/10.43.156.196:2181. Will not attempt to
authenticate using SASL (unknown error)
2015-06-12 07:18:27,065 INFO org.apache.zookeeper.ClientCnxn: Socket connection
established to zdh196/10.43.156.196:2181, initiating session
2015-06-12 07:18:27,073 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server zdh196/10.43.156.196:2181, sessionid =
0x14d91acb8921c9f, negotiated timeout = 10000
2015-06-12 07:18:27,077 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session
connected.
2015-06-12 07:18:27,104 INFO org.apache.hadoop.ipc.CallQueueManager: Using
callQueue class java.util.concurrent.LinkedBlockingQueue
2015-06-12 07:18:27,125 INFO org.apache.hadoop.ipc.Server: Starting Socket
Reader #1 for port 8019
2015-06-12 07:18:27,155 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2015-06-12 07:18:27,155 INFO org.apache.hadoop.ipc.Server: IPC Server listener
on 8019: starting
2015-06-12 07:18:27,277 INFO org.apache.hadoop.ha.HealthMonitor: Entering state
SERVICE_HEALTHY
2015-06-12 07:18:27,277 INFO org.apache.hadoop.ha.ZKFailoverController: Local
service NameNode at zdh196/10.43.156.196:9000 entered state: SERVICE_HEALTHY
2015-06-12 07:18:27,301 INFO org.apache.hadoop.ha.ActiveStandbyElector:
Checking for any old active which needs to be fenced...
2015-06-12 07:18:27,309 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old
node exists: 0a0a636c757374657231393512036e6e311a067a646831393520a84628d33e
2015-06-12 07:18:27,312 INFO org.apache.hadoop.ha.ZKFailoverController: Should
fence: NameNode at zdh195/10.43.156.195:9000
2015-06-12 07:18:27,338 INFO org.apache.hadoop.ha.ZKFailoverController:
Successfully transitioned NameNode at zdh195/10.43.156.195:9000 to standby
state without fencing
2015-06-12 07:18:27,338 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing
znode /hadoop-ha/cluster195/ActiveBreadCrumb to indicate that the local node is
the most recent active...
2015-06-12 07:18:27,348 INFO org.apache.hadoop.ha.ZKFailoverController: Trying
to make NameNode at zdh196/10.43.156.196:9000 active...
2015-06-12 07:18:39,494 INFO org.apache.hadoop.ha.ZKFailoverController:
Successfully transitioned NameNode at zdh196/10.43.156.196:9000 to active state
> Both NameNodes could be in STANDBY State if SNN network is unstable
> -------------------------------------------------------------------
>
> Key: HADOOP-10251
> URL: https://issues.apache.org/jira/browse/HADOOP-10251
> Project: Hadoop Common
> Issue Type: Bug
> Components: ha
> Affects Versions: 2.2.0
> Reporter: Vinayakumar B
> Assignee: Vinayakumar B
> Priority: Critical
> Fix For: 2.5.0
>
> Attachments: HADOOP-10251.patch, HADOOP-10251.patch,
> HADOOP-10251.patch, HADOOP-10251.patch, HADOOP-10251.patch
>
>
> Following corner scenario happened in one of our cluster.
> 1. NN1 was Active and NN2 was Standby
> 2. NN2 machine's network was slow
> 3. NN1 got shutdown.
> 4. NN2 ZKFC got the notification and trying to check for old active for
> fencing. (This took little more time, again due to slow network)
> 5. In between, NN1 got restarted by our automatic monitoring, and ZKFC made
> it Active.
> 6. Now NN2 ZKFC got Old Active as NN1 and it did graceful fencing of NN1 to
> STANBY.
> 7. Before writing ActiveBreadCrumb to ZK, NN2 ZKFC got session timeout and
> got shutdown before making NN2 Active.
> *Now cluster having both NameNodes as STANDBY.*
> NN1 ZKFC still thinks that its nameNode is in Active state.
> NN2 ZKFC waiting for election.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
