Good eye, that could be a problem. hbase-site.xml doesn't seem to be included in the classpath... It should appear in this line of the map log, right?
2017-01-25 17:50:33,951 INFO [main] org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/tmp/hadoop-hdadmin/nm-local-dir/usercache/idstest/appcache/application_1485362948532_0001/container_1485362948532_0001_01_000002:/opt/hadoop-2.7.2/etc/hadoop:/opt/hadoop-2.7.2/share/hadoop/common/hadoop-nfs-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/common/hadoop-common-2.7.2-tests.jar:/opt/hadoop-2.7.2/share/hadoop/common/hadoop-common-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-collections-3.2.2.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/hadoop-annotations-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/hamcrest-core-1.3.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jsr305-3.0.0.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/junit-4.11.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-net-3.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-logging-1.1.3.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/xz-1.0.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/gson-2.2.4.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-io-2.4.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/paranamer-2.3.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/avro-1.7.4.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/asm-3.2.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/guava-11.0.2.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/zookeeper-3.4.6.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/curator-framework-2.7.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jsch-0.1.42.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/httpcore-4.2.5.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jettison-1.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/curator-client-2.7.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jetty-6.1.26.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/httpclient-4.2.5.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jets3t-0.9.0.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jersey-core-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/activation-1.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-lang-2.6.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/hadoop-auth-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/hadoop-hdfs-2.7.2-tests.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/hadoop-hdfs-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/commons-io-2.4.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-common-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-server-common-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-registry-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-api-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-client-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/guice-3.0.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/commons-cli-1.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jersey-json-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/xz-1.0.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/commons-io-2.4.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jersey-client-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/servlet-api-2.5.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/asm-3.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/guava-11.0.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jettison-1.1.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jetty-6.1.26.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/commons-codec-1.4.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/activation-1.1.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/commons-lang-2.6.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.2-tests.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/guice-3.0.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/junit-4.11.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/xz-1.0.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/asm-3.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/javax.inject-1.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:job.jar/job.jar:job.jar/classes/:job.jar/lib/*:/tmp/hadoop-hdadmin/nm-local-dir/usercache/idstest/appcache/application_1485362948532_0001/container_1485362948532_0001_01_000002/job.jar:/tmp/hadoop-hdadmin/nm-local-dir/usercache/idstest/appcache/application_1485362948532_0001/container_1485362948532_0001_01_000002/hbase-common-1.2.4.jar:/tmp/hadoop-hdadmin/nm-local-dir/usercache/idstest/appcache/application_1485362948532_0001/container_1485362948532_0001_01_000002/hbase-protocol-1.2.4.jar:/tmp/hadoop-hdadmin/nm-local-dir/usercache/idstest/appcache/application_1485362948532_0001/container_1485362948532_0001_01_000002/hbase-client-1.2.4.jar Or perhaps I'm confused with the classpaths now... I guess I need to have the hbase-site.xml path written in the map log under the property On Wed, 2017-01-25 at 07:52 -0800, Ted Yu wrote: > From the new log snippet you posted, the hbase client tried to connect > to 192.168.0.24 whose node is not listed in the quorum. > > Can you check the classpath for the maptask ? Looks like the effective > hbase-site.xml wasn't on the classpath. > > On Wed, Jan 25, 2017 at 7:21 AM, Hernán Blanco <[email protected]> > wrote: > > > Hello Ted, > > > > Thank you for your reply. Indeed, I wasn't sure if that property > > existed, and in fact hbase-default.xml doesn't include it. I just > > followed the advise from some webpage (I couldn't recall which) that > > stated that you can include *any* property from the zoo.cfg native > > config file for ZooKeeper standalone into hbase-site.xml by just adding > > the "hbase.zookeeper.property." prefix. Even the comment in lines > > 397-398 of > > https://github.com/apache/hbase/blob/master/hbase- > > common/src/main/resources/hbase-default.xml mentioned this, but I > > probably misinterpreted it. > > > > I removed those incorrect properties from hbase-site.xml and reduced my > > ZooKeeper quorum to only 3 nodes -the main and 2 others: > > > > > > ... > > <property> > > <name>hbase.zookeeper.quorum</name> > > <value>bwcloud-fip26.rz.uni-ulm.de,bwcloud-fip27.rz.uni-ulm.de, > > bwcloud-fip144.rz.uni-ulm.de</value> > > </property> > > ... > > > > > > But the problem remains. The MapReduce job keeps stuck, and same kind of > > logs, both on HBase startup and during job execution, with an added > > exception which is that those map tasks that are deployed on the nodes > > that aren't running a ZooKeeper generate the following log: > > > > > > ... > > 2017-01-25 15:57:50,295 INFO [main-SendThread(bwcloud- > > fip150.rz.uni-ulm.de:2181)] org.apache.zookeeper.ClientCnxn: Opening > > socket connection to server bwcloud-fip150.rz.uni-ulm.de/192.168.0.24:2181. > > Will not attempt to authenticate using SASL (unknown error) > > 2017-01-25 15:57:50,296 WARN [main-SendThread(bwcloud- > > fip150.rz.uni-ulm.de:2181)] org.apache.zookeeper.ClientCnxn: Session 0x0 > > for server null, unexpected error, closing socket connection and attempting > > reconnect > > java.net.ConnectException: Connection refused > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > at sun.nio.ch.SocketChannelImpl.finishConnect( > > SocketChannelImpl.java:717) > > at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport( > > ClientCnxnSocketNIO.java:361) > > at org.apache.zookeeper.ClientCnxn$SendThread.run( > > ClientCnxn.java:1081) > > 2017-01-25 15:57:51,400 INFO [main-SendThread(bwcloud- > > fip150.rz.uni-ulm.de:2181)] org.apache.zookeeper.ClientCnxn: Opening > > socket connection to server bwcloud-fip150.rz.uni-ulm.de/192.168.0.24:2181. > > Will not attempt to authenticate using SASL (unknown error) > > 2017-01-25 15:57:51,401 WARN [main-SendThread(bwcloud- > > fip150.rz.uni-ulm.de:2181)] org.apache.zookeeper.ClientCnxn: Session 0x0 > > for server null, unexpected error, closing socket connection and attempting > > reconnect > > java.net.ConnectException: Connection refused > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > at sun.nio.ch.SocketChannelImpl.finishConnect( > > SocketChannelImpl.java:717) > > at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport( > > ClientCnxnSocketNIO.java:361) > > at org.apache.zookeeper.ClientCnxn$SendThread.run( > > ClientCnxn.java:1081) > > ... > > > > > > I actually tried this before, but my conclusion was that all nodes that > > are running a YARN NodeManager need to have ZooKeeper running. > > > > Best, > > Hernán. > > > > > > On Wed, 2017-01-25 at 06:23 -0800, Ted Yu wrote: > > > > > bq. <name>hbase.zookeeper.property.server.7</name> > > > > > > I searched 1.2 codebase but didn't find config parameter in the above > > form. > > > http://hbase.apache.org/book.html didn't mention it either. > > > > > > May I ask where you obtained such config ? > > > > > > For hbase.zookeeper.quorum, do you have zookeeper running on the 12 > > nodes ? > > > Normally 3 zookeeper nodes should be enough. > > > > > > Cheers > > > > > > On Wed, Jan 25, 2017 at 2:08 AM, Hernán Blanco < > > [email protected]> > > > wrote: > > > > > > > Hi all, > > > > > > > > I'm running HBase 1.2.4 on a distributed setup with 12 virtual machines > > > > on the same local network. The "main" node (node26.example.com) runs > > the > > > > HMaster, while the other 11 machines run RegionServers. No backup > > > > HMaster. This cluster also runs Hadoop 2.7.2 smoothly. > > > > > > > > Both HBase shell and accessing through the HBase client API run > > > > properly, and even importing a TSV file into a table with > > > > org.apache.hadoop.hbase.mapreduce.ImportTsv succeeds, remaining > > > > registered in the YARN history. > > > > > > > > But the problem appears when trying to access from a MapReduce job to > > an > > > > HBase table (using Hadoop 2.7.2). Here a minimal code that produces the > > > > issue by connecting and scanning an HBase table: > > > > http://pastebin.com/pm8tbbTq. The maps hang until timeout and then > > > > retries deploying new maps until failing, each map showing the > > following > > > > messages in the syslog: > > > > > > > > ---------- > > > > > > > > 2017-01-24 20:04:07,904 INFO [main] org.apache.hadoop.metrics2. > > impl.MetricsConfig: > > > > loaded properties from hadoop-metrics2.properties > > > > 2017-01-24 20:04:08,186 INFO [main] org.apache.hadoop.metrics2. > > impl.MetricsSystemImpl: > > > > Scheduled snapshot period at 10 second(s). > > > > 2017-01-24 20:04:08,186 INFO [main] org.apache.hadoop.metrics2. > > impl.MetricsSystemImpl: > > > > MapTask metrics system started > > > > 2017-01-24 20:04:08,193 INFO [main] org.apache.hadoop.mapred. > > YarnChild: > > > > Executing with tokens: > > > > 2017-01-24 20:04:08,193 INFO [main] org.apache.hadoop.mapred. > > YarnChild: > > > > Kind: mapreduce.job, Service: job_1485182272940_0336, Ident: > > > > (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@6f03482 > > ) > > > > 2017-01-24 20:04:08,450 INFO [main] org.apache.hadoop.mapred. > > YarnChild: > > > > Sleeping for 0ms before retrying again. Got null now. > > > > 2017-01-24 20:04:08,994 INFO [main] org.apache.hadoop.mapred. > > YarnChild: > > > > mapreduce.cluster.local.dir for child: /tmp/hadoop-hdadmin/nm-local- > > > > dir/usercache/idstest/appcache/application_1485182272940_0336 > > > > 2017-01-24 20:04:09,617 INFO [main] org.apache.hadoop.conf. > > Configuration.deprecation: > > > > session.id is deprecated. Instead, use dfs.metrics.session-id > > > > 2017-01-24 20:04:10,968 INFO [main] org.apache.hadoop.mapreduce. > > > > lib.output.FileOutputCommitter: File Output Committer Algorithm > > version > > > > is 1 > > > > 2017-01-24 20:04:11,044 INFO [main] org.apache.hadoop.mapred.Task: > > Using > > > > ResourceCalculatorProcessTree : [ ] > > > > 2017-01-24 20:04:11,455 INFO [main] org.apache.hadoop.mapred.MapTask: > > > > Processing split: hdfs://node26.example.com/ > > user/idstest/data/articles-50/ > > > > 99322.txt:0+3757 > > > > 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTask: > > > > (EQUATOR) 0 kvi 26214396(104857584) > > > > 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTask: > > > > mapreduce.task.io.sort.mb: 100 > > > > 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTask: > > > > soft limit at 83886080 > > > > 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTask: > > > > bufstart = 0; bufvoid = 104857600 > > > > 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTask: > > > > kvstart = 26214396; length = 6553600 > > > > 2017-01-24 20:04:11,612 INFO [main] org.apache.hadoop.mapred.MapTask: > > Map > > > > output collector class = org.apache.hadoop.mapred. > > MapTask$MapOutputBuffer > > > > 2017-01-24 20:04:12,211 INFO [main] org.apache.hadoop.hbase.zookeeper. > > RecoverableZooKeeper: > > > > Process identifier=hconnection-0x650eab8 connecting to ZooKeeper > > > > ensemble=localhost:2181 > > > > 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:zookeeper.version=3.4.6-1569965, built on > > 02/20/2014 > > > > 09:09 GMT > > > > 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:host.name=node27.example.com > > > > 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:java.version=1.8.0_77-Debian > > > > 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:java.vendor=Oracle Corporation > > > > 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre > > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:java.class.path= /* ... */ > > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:java.library.path=/tmp/hadoop-hdadmin/nm-local- > > > > dir/usercache/idstest/appcache/application_ > > 1485182272940_0336/container_ > > > > 1485182272940_0336_01_000048:/opt/hadoop-2.7.2/lib/native:/ > > > > usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/ > > > > lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni: > > /lib:/usr/lib > > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:java.io.tmpdir=/tmp/hadoop-hdadmin/nm-local- > > > > dir/usercache/idstest/appcache/application_ > > 1485182272940_0336/container_ > > > > 1485182272940_0336_01_000048/tmp > > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:java.compiler=<NA> > > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:os.name=Linux > > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:os.arch=amd64 > > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:os.version=3.16.0-4-amd64 > > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:user.name=hdadmin > > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:user.home=/home/hdadmin > > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:user.dir=/tmp/hadoop-hdadmin/nm-local-dir/ > > > > usercache/idstest/appcache/application_1485182272940_ > > > > 0336/container_1485182272940_0336_01_000048 > > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Initiating client connection, connectString=localhost:2181 > > > > sessionTimeout=90000 watcher=hconnection-0x650eab80x0, > > > > quorum=localhost:2181, baseZNode=/hbase > > > > 2017-01-24 20:04:12,322 INFO [main-SendThread(node27.example.com:2181 > > )] > > > > org.apache.zookeeper.ClientCnxn: Opening socket connection to server > > > > node27.example.com/192.168.0.17:2181. Will not attempt to authenticate > > > > using SASL (unknown error) > > > > 2017-01-24 20:04:12,323 INFO [main-SendThread(node27.example.com:2181 > > )] > > > > org.apache.zookeeper.ClientCnxn: Socket connection established to > > > > node27.example.com/192.168.0.17:2181, initiating session > > > > 2017-01-24 20:04:12,331 INFO [main-SendThread(node27.example.com:2181 > > )] > > > > org.apache.zookeeper.ClientCnxn: Session establishment complete on > > server > > > > node27.example.com/192.168.0.17:2181, sessionid = 0x159d07980b50189, > > > > negotiated timeout = 90000 > > > > 2017-01-24 20:04:51,657 INFO [hconnection-0x650eab8- > > metaLookup-shared--pool2-t1] > > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, > > > > tries=10, retries=35, started=38852 ms ago, cancelled=false, msg=row > > > > 'idstest1,,99999999999999' on table 'hbase:meta' at > > region=hbase:meta,,1.1588230740, > > > > hostname=node149.example.com,16020,1485261342943, seqNum=0 > > > > 2017-01-24 20:05:01,664 INFO [hconnection-0x650eab8- > > metaLookup-shared--pool2-t1] > > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, > > > > tries=11, retries=35, started=48860 ms ago, cancelled=false, msg=row > > > > 'idstest1,,99999999999999' on table 'hbase:meta' at > > region=hbase:meta,,1.1588230740, > > > > hostname=node149.example.com,16020,1485261342943, seqNum=0 > > > > 2017-01-24 20:05:40,044 INFO [hconnection-0x650eab8- > > metaLookup-shared--pool2-t2] > > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, > > > > tries=10, retries=35, started=38273 ms ago, cancelled=false, msg=row > > > > 'idstest1,,99999999999999' on table 'hbase:meta' at > > region=hbase:meta,,1.1588230740, > > > > hostname=node149.example.com,16020,1485261342943, seqNum=0 > > > > 2017-01-24 20:05:50,059 INFO [hconnection-0x650eab8- > > metaLookup-shared--pool2-t2] > > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, > > > > tries=11, retries=35, started=48288 ms ago, cancelled=false, msg=row > > > > 'idstest1,,99999999999999' on table 'hbase:meta' at > > region=hbase:meta,,1.1588230740, > > > > hostname=node149.example.com,16020,1485261342943, seqNum=0 > > > > 2017-01-24 20:06:28,543 INFO [hconnection-0x650eab8- > > metaLookup-shared--pool2-t3] > > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, > > > > tries=10, retries=35, started=38273 ms ago, cancelled=false, msg=row > > > > 'idstest1,,99999999999999' on table 'hbase:meta' at > > region=hbase:meta,,1.1588230740, > > > > hostname=node149.example.com,16020,1485261342943, seqNum=0 > > > > 2017-01-24 20:06:38,632 INFO [hconnection-0x650eab8- > > metaLookup-shared--pool2-t3] > > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, > > > > tries=11, retries=35, started=48362 ms ago, cancelled=false, msg=row > > > > 'idstest1,,99999999999999' on table 'hbase:meta' at > > region=hbase:meta,,1.1588230740, > > > > hostname=node149.example.com,16020,1485261342943, seqNum=0 > > > > ... > > > > > > > > ---------- > > > > > > > > My hbase-site.xml in the Master node node26.example.com (almost same > > for > > > > the other nodes, just referring as 0.0.0.0 to themselves): > > > > > > > > ---------- > > > > > > > > <configuration> > > > > <property> > > > > <name>hbase.rootdir</name> > > > > <value>hdfs://node26.example.com:8020/hbase</value> > > > > </property> > > > > <property> > > > > <name>hbase.zookeeper.property.dataDir</name> > > > > <value>/home/hdadmin/zookeeper</value> > > > > </property> > > > > <property> > > > > <name>hbase.cluster.distributed</name> > > > > <value>true</value> > > > > </property> > > > > <property> > > > > <name>hbase.zookeeper.quorum</name> > > > > <value>node26.example.com,node27.example.com,node144.example.com, > > > > node145.example.com,node146.example.com,node147.example.com, > > > > node148.example.com,node149.example.com,node150.example.com, > > > > node151.example.com,node152.example.com,node153.example.com</value> > > > > </property> > > > > <property> > > > > <name>hbase.zookeeper.property.server.0</name> > > > > <value>0.0.0.0:2888:3888</value> > > > > </property> > > > > <property> > > > > <name>hbase.zookeeper.property.server.1</name> > > > > <value>node27.example.com:2888:3888</value> > > > > </property> > > > > <property> > > > > <name>hbase.zookeeper.property.server.2</name> > > > > <value>node144.example.com:2888:3888</value> > > > > </property> > > > > <property> > > > > <name>hbase.zookeeper.property.server.3</name> > > > > <value>node145.example.com:2888:3888</value> > > > > </property> > > > > <property> > > > > <name>hbase.zookeeper.property.server.4</name> > > > > <value>node146.example.com:2888:3888</value> > > > > </property> > > > > <property> > > > > <name>hbase.zookeeper.property.server.5</name> > > > > <value>node147.example.com:2888:3888</value> > > > > </property> > > > > <property> > > > > <name>hbase.zookeeper.property.server.6</name> > > > > <value>node148.example.com:2888:3888</value> > > > > </property> > > > > <property> > > > > <name>hbase.zookeeper.property.server.7</name> > > > > <value>node149.example.com:2888:3888</value> > > > > </property> > > > > <property> > > > > <name>hbase.zookeeper.property.server.8</name> > > > > <value>node150.example.com:2888:3888</value> > > > > </property> > > > > <property> > > > > <name>hbase.zookeeper.property.server.9</name> > > > > <value>node151.example.com:2888:3888</value> > > > > </property> > > > > <property> > > > > <name>hbase.zookeeper.property.server.10</name> > > > > <value>node152.example.com:2888:3888</value> > > > > </property> > > > > <property> > > > > <name>hbase.zookeeper.property.server.11</name> > > > > <value>node153.example.com:2888:3888</value> > > > > </property> > > > > </configuration> > > > > > > > > ---------- > > > > > > > > After starting HBase using the start-hbase.sh script, the RegionServer > > > > logs have no warnings or errors, but the different ZooKeeper instances > > > > generate diverse outputs. 3 examples: > > > > > > > > > > > > - The master node has an empty log: > > > > > > > > ---------- > > > > > > > > Tue Jan 24 20:52:38 CET 2017 Starting zookeeper on node26.example.com > > > > core file size (blocks, -c) 0 > > > > data seg size (kbytes, -d) unlimited > > > > scheduling priority (-e) 0 > > > > file size (blocks, -f) unlimited > > > > pending signals (-i) 128914 > > > > max locked memory (kbytes, -l) 64 > > > > max memory size (kbytes, -m) unlimited > > > > open files (-n) 65536 > > > > pipe size (512 bytes, -p) 8 > > > > POSIX message queues (bytes, -q) 819200 > > > > real-time priority (-r) 0 > > > > stack size (kbytes, -s) 8192 > > > > cpu time (seconds, -t) unlimited > > > > max user processes (-u) 128914 > > > > virtual memory (kbytes, -v) unlimited > > > > file locks (-x) unlimited > > > > > > > > ---------- > > > > > > > > > > > > - node153.example.com has the following small ZooKeeper log: > > > > > > > > ---------- > > > > > > > > Tue Jan 24 20:52:38 CET 2017 Starting zookeeper on node153.example.com > > > > core file size (blocks, -c) 0 > > > > data seg size (kbytes, -d) unlimited > > > > scheduling priority (-e) 0 > > > > file size (blocks, -f) unlimited > > > > pending signals (-i) 128914 > > > > max locked memory (kbytes, -l) 64 > > > > max memory size (kbytes, -m) unlimited > > > > open files (-n) 65536 > > > > pipe size (512 bytes, -p) 8 > > > > POSIX message queues (bytes, -q) 819200 > > > > real-time priority (-r) 0 > > > > stack size (kbytes, -s) 8192 > > > > cpu time (seconds, -t) unlimited > > > > max user processes (-u) 128914 > > > > virtual memory (kbytes, -v) unlimited > > > > file locks (-x) unlimited > > > > 2017-01-24 20:52:39,816 WARN [main] quorum.QuorumPeerConfig: > > Non-optimial > > > > configuration, consider an odd number of servers. > > > > 2017-01-24 20:52:39,817 INFO [main] quorum.QuorumPeerConfig: > > Defaulting > > > > to majority quorums > > > > 2017-01-24 20:52:39,984 INFO [main] quorum.QuorumPeerMain: Starting > > > > quorum peer > > > > 2017-01-24 20:52:39,994 INFO [main] server.NIOServerCnxnFactory: > > binding > > > > to port 0.0.0.0/0.0.0.0:2181 > > > > > > > > ---------- > > > > > > > > As an extra hint, this node is not generating > > > > the /tmp/hbase-hdadmin-zookeeper.pid, so it shows an error when > > running > > > > the stop-hbase.sh script. > > > > > > > > > > > > - And node144.example.com (as other nodes) has the following ZooKeeper > > > > log with "connection refused" errors: > > > > > > > > ---------- > > > > > > > > Tue Jan 24 20:52:38 CET 2017 Starting zookeeper on node144.example.com > > > > core file size (blocks, -c) 0 > > > > data seg size (kbytes, -d) unlimited > > > > scheduling priority (-e) 0 > > > > file size (blocks, -f) unlimited > > > > pending signals (-i) 128914 > > > > max locked memory (kbytes, -l) 64 > > > > max memory size (kbytes, -m) unlimited > > > > open files (-n) 65536 > > > > pipe size (512 bytes, -p) 8 > > > > POSIX message queues (bytes, -q) 819200 > > > > real-time priority (-r) 0 > > > > stack size (kbytes, -s) 8192 > > > > cpu time (seconds, -t) unlimited > > > > max user processes (-u) 128914 > > > > virtual memory (kbytes, -v) unlimited > > > > file locks (-x) unlimited > > > > 2017-01-24 20:52:39,473 WARN [main] quorum.QuorumPeerConfig: > > Non-optimial > > > > configuration, consider an odd number of servers. > > > > 2017-01-24 20:52:39,473 INFO [main] quorum.QuorumPeerConfig: > > Defaulting > > > > to majority quorums > > > > 2017-01-24 20:52:39,567 INFO [main] quorum.QuorumPeerMain: Starting > > > > quorum peer > > > > 2017-01-24 20:52:39,581 INFO [main] server.NIOServerCnxnFactory: > > binding > > > > to port 0.0.0.0/0.0.0.0:2181 > > > > 2017-01-24 20:52:39,601 INFO [main] quorum.QuorumPeer: tickTime set to > > > > 3000 > > > > 2017-01-24 20:52:39,601 INFO [main] quorum.QuorumPeer: > > minSessionTimeout > > > > set to -1 > > > > 2017-01-24 20:52:39,601 INFO [main] quorum.QuorumPeer: > > maxSessionTimeout > > > > set to 90000 > > > > 2017-01-24 20:52:39,601 INFO [main] quorum.QuorumPeer: initLimit set > > to 10 > > > > 2017-01-24 20:52:39,613 INFO [main] persistence.FileSnap: Reading > > > > snapshot /home/hdadmin/zookeeper2/version-2/snapshot.1d0000023f > > > > 2017-01-24 20:52:39,714 INFO [Thread-2] quorum.QuorumCnxManager: My > > > > election bind port: node144.example.com/192.168.0.19:3888 > > > > 2017-01-24 20:52:39,729 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > quorum.QuorumPeer: LOOKING > > > > 2017-01-24 20:52:39,731 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > quorum.FastLeaderElection: New election. My id = 2, proposed > > > > zxid=0x1e0000258d > > > > 2017-01-24 20:52:39,740 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 2 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,740 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 0 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,742 WARN [WorkerSender[myid=2]] > > > > quorum.QuorumCnxManager: Cannot open channel to 3 at election address > > > > node145.example.com/192.168.0.18:3888 > > > > java.net.ConnectException: Connection refused > > > > at java.net.PlainSocketImpl.socketConnect(Native Method) > > > > at java.net.AbstractPlainSocketImpl.doConnect( > > > > AbstractPlainSocketImpl.java:350) > > > > at java.net.AbstractPlainSocketImpl.connectToAddress( > > > > AbstractPlainSocketImpl.java:206) > > > > at java.net.AbstractPlainSocketImpl.connect( > > > > AbstractPlainSocketImpl.java:188) > > > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > > > > at java.net.Socket.connect(Socket.java:589) > > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > > connectOne(QuorumCnxManager.java:368) > > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > > toSend(QuorumCnxManager.java:341) > > > > at org.apache.zookeeper.server.quorum.FastLeaderElection$ > > > > Messenger$WorkerSender.process(FastLeaderElection.java:449) > > > > at org.apache.zookeeper.server.quorum.FastLeaderElection$ > > > > Messenger$WorkerSender.run(FastLeaderElection.java:430) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 20:52:39,744 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 1 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,745 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 1 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,745 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 2 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,746 INFO [WorkerSender[myid=2]] > > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping > > the > > > > connection: (4, 2) > > > > 2017-01-24 20:52:39,747 INFO [WorkerSender[myid=2]] > > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping > > the > > > > connection: (5, 2) > > > > 2017-01-24 20:52:39,747 INFO [node144.example.com/192.168.0.19:3888] > > > > quorum.QuorumCnxManager: Received connection request / > > 192.168.0.21:40064 > > > > 2017-01-24 20:52:39,747 INFO [node144.example.com/192.168.0.19:3888] > > > > quorum.QuorumCnxManager: Received connection request / > > 192.168.0.22:54362 > > > > 2017-01-24 20:52:39,747 INFO [WorkerSender[myid=2]] > > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping > > the > > > > connection: (6, 2) > > > > 2017-01-24 20:52:39,748 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 4 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,748 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 5 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,748 INFO [node144.example.com/192.168.0.19:3888] > > > > quorum.QuorumCnxManager: Received connection request / > > 192.168.0.23:53410 > > > > 2017-01-24 20:52:39,749 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 5 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,749 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 0 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,749 INFO [WorkerSender[myid=2]] > > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping > > the > > > > connection: (7, 2) > > > > 2017-01-24 20:52:39,749 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 4 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,749 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 6 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,750 INFO [node144.example.com/192.168.0.19:3888] > > > > quorum.QuorumCnxManager: Received connection request / > > 192.168.0.25:33085 > > > > 2017-01-24 20:52:39,750 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 6 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,751 WARN [WorkerSender[myid=2]] > > > > quorum.QuorumCnxManager: Cannot open channel to 8 at election address > > > > node150.example.com/192.168.0.24:3888 > > > > java.net.ConnectException: Connection refused > > > > at java.net.PlainSocketImpl.socketConnect(Native Method) > > > > at java.net.AbstractPlainSocketImpl.doConnect( > > > > AbstractPlainSocketImpl.java:350) > > > > at java.net.AbstractPlainSocketImpl.connectToAddress( > > > > AbstractPlainSocketImpl.java:206) > > > > at java.net.AbstractPlainSocketImpl.connect( > > > > AbstractPlainSocketImpl.java:188) > > > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > > > > at java.net.Socket.connect(Socket.java:589) > > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > > connectOne(QuorumCnxManager.java:368) > > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > > toSend(QuorumCnxManager.java:341) > > > > at org.apache.zookeeper.server.quorum.FastLeaderElection$ > > > > Messenger$WorkerSender.process(FastLeaderElection.java:449) > > > > at org.apache.zookeeper.server.quorum.FastLeaderElection$ > > > > Messenger$WorkerSender.run(FastLeaderElection.java:430) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 20:52:39,751 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 7 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,752 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 7 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,753 INFO [WorkerSender[myid=2]] > > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping > > the > > > > connection: (9, 2) > > > > 2017-01-24 20:52:39,753 INFO [WorkerSender[myid=2]] > > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping > > the > > > > connection: (10, 2) > > > > 2017-01-24 20:52:39,753 INFO [node144.example.com/192.168.0.19:3888] > > > > quorum.QuorumCnxManager: Received connection request / > > 192.168.0.26:35997 > > > > 2017-01-24 20:52:39,754 INFO [WorkerSender[myid=2]] > > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping > > the > > > > connection: (11, 2) > > > > 2017-01-24 20:52:39,755 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 2 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,755 INFO [WorkerSender[myid=2]] > > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping > > the > > > > connection: (3, 2) > > > > 2017-01-24 20:52:39,756 WARN [WorkerSender[myid=2]] > > > > quorum.QuorumCnxManager: Cannot open channel to 8 at election address > > > > node150.example.com/192.168.0.24:3888 > > > > java.net.ConnectException: Connection refused > > > > at java.net.PlainSocketImpl.socketConnect(Native Method) > > > > at java.net.AbstractPlainSocketImpl.doConnect( > > > > AbstractPlainSocketImpl.java:350) > > > > at java.net.AbstractPlainSocketImpl.connectToAddress( > > > > AbstractPlainSocketImpl.java:206) > > > > at java.net.AbstractPlainSocketImpl.connect( > > > > AbstractPlainSocketImpl.java:188) > > > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > > > > at java.net.Socket.connect(Socket.java:589) > > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > > connectOne(QuorumCnxManager.java:368) > > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > > toSend(QuorumCnxManager.java:341) > > > > at org.apache.zookeeper.server.quorum.FastLeaderElection$ > > > > Messenger$WorkerSender.process(FastLeaderElection.java:449) > > > > at org.apache.zookeeper.server.quorum.FastLeaderElection$ > > > > Messenger$WorkerSender.run(FastLeaderElection.java:430) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 20:52:39,757 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 9 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,757 INFO [WorkerSender[myid=2]] > > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping > > the > > > > connection: (10, 2) > > > > 2017-01-24 20:52:39,757 INFO [WorkerSender[myid=2]] > > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping > > the > > > > connection: (11, 2) > > > > 2017-01-24 20:52:39,758 INFO [node144.example.com/192.168.0.19:3888] > > > > quorum.QuorumCnxManager: Received connection request / > > 192.168.0.27:34404 > > > > 2017-01-24 20:52:39,759 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 9 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,768 INFO [node144.example.com/192.168.0.19:3888] > > > > quorum.QuorumCnxManager: Received connection request / > > 192.168.0.18:59032 > > > > 2017-01-24 20:52:39,769 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 11 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,769 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 3 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,776 INFO [node144.example.com/192.168.0.19:3888] > > > > quorum.QuorumCnxManager: Received connection request / > > 192.168.0.27:34405 > > > > 2017-01-24 20:52:39,776 WARN [RecvWorker:11] quorum.QuorumCnxManager: > > > > Connection broken for id 11, my id = 2, error = > > > > java.io.EOFException > > > > at java.io.DataInputStream.readInt(DataInputStream.java:392) > > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager$ > > > > RecvWorker.run(QuorumCnxManager.java:765) > > > > 2017-01-24 20:52:39,777 WARN [RecvWorker:11] quorum.QuorumCnxManager: > > > > Interrupting SendWorker > > > > 2017-01-24 20:52:39,777 WARN [SendWorker:11] quorum.QuorumCnxManager: > > > > Interrupted while waiting for message on queue > > > > java.lang.InterruptedException > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ > > > > ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer. > > > > java:2014) > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ > > > > ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) > > > > at java.util.concurrent.ArrayBlockingQueue.poll( > > > > ArrayBlockingQueue.java:418) > > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > > pollSendQueue(QuorumCnxManager.java:849) > > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > > access$500(QuorumCnxManager.java:64) > > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager$ > > > > SendWorker.run(QuorumCnxManager.java:685) > > > > 2017-01-24 20:52:39,777 INFO [node144.example.com/192.168.0.19:3888] > > > > quorum.QuorumCnxManager: Received connection request / > > 192.168.0.28:57974 > > > > 2017-01-24 20:52:39,778 WARN [SendWorker:11] quorum.QuorumCnxManager: > > > > Send worker leaving thread > > > > 2017-01-24 20:52:39,778 INFO [node144.example.com/192.168.0.19:3888] > > > > quorum.QuorumCnxManager: Received connection request / > > 192.168.0.28:57975 > > > > 2017-01-24 20:52:39,778 WARN [RecvWorker:10] quorum.QuorumCnxManager: > > > > Connection broken for id 10, my id = 2, error = > > > > java.io.EOFException > > > > at java.io.DataInputStream.readInt(DataInputStream.java:392) > > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager$ > > > > RecvWorker.run(QuorumCnxManager.java:765) > > > > 2017-01-24 20:52:39,778 WARN [RecvWorker:10] quorum.QuorumCnxManager: > > > > Interrupting SendWorker > > > > 2017-01-24 20:52:39,778 WARN [SendWorker:10] quorum.QuorumCnxManager: > > > > Exception when using channel: for id 10 my id = 2 error = > > > > java.net.SocketException: Broken pipe > > > > 2017-01-24 20:52:39,778 WARN [SendWorker:10] quorum.QuorumCnxManager: > > > > Send worker leaving thread > > > > 2017-01-24 20:52:39,779 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 3 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,782 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 10 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 10 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,788 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 11 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,789 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 10 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,815 INFO [node144.example.com/192.168.0.19:3888] > > > > quorum.QuorumCnxManager: Received connection request / > > 192.168.0.24:56503 > > > > 2017-01-24 20:52:39,829 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 8 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 8 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,829 INFO [WorkerReceiver[myid=2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 8 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:40,030 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > quorum.QuorumPeer: FOLLOWING > > > > 2017-01-24 20:52:40,035 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > quorum.Learner: TCP NoDelay set to: true > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:zookeeper.version= > > 3.4.6-1569965, > > > > built on 02/20/2014 09:09 GMT > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:host.name=node144. > > example.com > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:java.version=1.8. > > 0_77-Debian > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:java.vendor=Oracle > > Corporation > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:java.home=/usr/ > > > > lib/jvm/java-8-openjdk-amd64/jre > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:java.class.path= /* ... */ > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:java.library.path= > > > > /usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/ > > > > lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni: > > /lib:/usr/lib > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:java.io.tmpdir=/tmp > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:java.compiler=<NA> > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:os.name=Linux > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:os.arch=amd64 > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:os.version=3.16.0-4-amd64 > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:user.name=hdadmin > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:user.home=/home/hdadmin > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:user.dir=/opt/hbase-1.2.4 > > > > 2017-01-24 20:52:40,042 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Created server with tickTime 3000 > > minSessionTimeout > > > > 6000 maxSessionTimeout 90000 datadir /home/hdadmin/zookeeper2/ > > version-2 > > > > snapdir /home/hdadmin/zookeeper2/version-2 > > > > 2017-01-24 20:52:40,043 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > quorum.Learner: FOLLOWING - LEADER ELECTION TOOK - 311 > > > > 2017-01-24 20:52:40,046 WARN [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > quorum.Learner: Unexpected exception, tries=0, connecting to > > > > node153.example.com/192.168.0.27:2888 > > > > java.net.ConnectException: Connection refused > > > > at java.net.PlainSocketImpl.socketConnect(Native Method) > > > > at java.net.AbstractPlainSocketImpl.doConnect( > > > > AbstractPlainSocketImpl.java:350) > > > > at java.net.AbstractPlainSocketImpl.connectToAddress( > > > > AbstractPlainSocketImpl.java:206) > > > > at java.net.AbstractPlainSocketImpl.connect( > > > > AbstractPlainSocketImpl.java:188) > > > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > > > > at java.net.Socket.connect(Socket.java:589) > > > > at org.apache.zookeeper.server.quorum.Learner. > > > > connectToLeader(Learner.java:225) > > > > at org.apache.zookeeper.server.quorum.Follower.followLeader( > > > > Follower.java:71) > > > > at org.apache.zookeeper.server.quorum.QuorumPeer.run( > > > > QuorumPeer.java:786) > > > > 2017-01-24 20:52:41,073 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > quorum.Learner: Getting a diff from the leader 0x1e0000258d > > > > 2017-01-24 20:52:41,116 INFO [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > persistence.FileTxnSnapLog: Snapshotting: 0x1e0000258d to > > > > /home/hdadmin/zookeeper2/version-2/snapshot.1e0000258d > > > > 2017-01-24 20:52:42,408 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.29:44336 > > > > 2017-01-24 20:52:42,418 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.29:44336 > > > > 2017-01-24 20:52:42,421 WARN [QuorumPeer[myid=2]/0:0:0:0:0: > > 0:0:0:2181] > > > > quorum.Learner: Got zxid 0x1f00000001 expected 0x1 > > > > 2017-01-24 20:52:42,421 INFO [SyncThread:2] persistence.FileTxnLog: > > > > Creating new log file: log.1f00000001 > > > > 2017-01-24 20:52:42,440 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750000 with negotiated timeout 90000 for > > > > client /192.168.0.29:44336 > > > > 2017-01-24 20:52:42,734 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.22:43377 > > > > 2017-01-24 20:52:42,737 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.22:43377 > > > > 2017-01-24 20:52:42,742 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750001 with negotiated timeout 90000 for > > > > client /192.168.0.22:43377 > > > > 2017-01-24 20:52:43,257 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.23:59005 > > > > 2017-01-24 20:52:43,258 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.23:59005 > > > > 2017-01-24 20:52:43,262 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750002 with negotiated timeout 90000 for > > > > client /192.168.0.23:59005 > > > > > > > > ---------- > > > > > > > > > > > > I've manually checked connecting to the most relevant ports of HBase > > and > > > > ZooKeeper (using nc or telnet) and always succeeded. > > > > > > > > During a job execution, these are the type of errors that I'm getting: > > > > > > > > ---------- > > > > > > > > 2017-01-24 21:10:37,187 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.29:44750 > > > > 2017-01-24 21:10:37,193 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.29:44750 > > > > 2017-01-24 21:10:37,198 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750003 with negotiated timeout 90000 for > > > > client /192.168.0.29:44750 > > > > 2017-01-24 21:10:38,136 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750003, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:10:38,137 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.29:44750 which had sessionid 0x259d20997750003 > > > > 2017-01-24 21:18:45,469 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49706 > > > > 2017-01-24 21:18:45,479 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.19:49706 > > > > 2017-01-24 21:18:45,543 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750004 with negotiated timeout 90000 for > > > > client /192.168.0.19:49706 > > > > 2017-01-24 21:18:45,684 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49708 > > > > 2017-01-24 21:18:45,689 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.19:49708 > > > > 2017-01-24 21:18:45,820 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49709 > > > > 2017-01-24 21:18:45,824 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.19:49709 > > > > 2017-01-24 21:18:45,825 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750005 with negotiated timeout 90000 for > > > > client /192.168.0.19:49708 > > > > 2017-01-24 21:18:45,840 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750006 with negotiated timeout 90000 for > > > > client /192.168.0.19:49709 > > > > 2017-01-24 21:18:45,892 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49710 > > > > 2017-01-24 21:18:45,896 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.19:49710 > > > > 2017-01-24 21:18:45,965 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750007 with negotiated timeout 90000 for > > > > client /192.168.0.19:49710 > > > > 2017-01-24 21:18:46,010 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49711 > > > > 2017-01-24 21:18:46,012 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.19:49711 > > > > 2017-01-24 21:18:46,065 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49712 > > > > 2017-01-24 21:18:46,067 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.19:49712 > > > > 2017-01-24 21:18:46,133 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750008 with negotiated timeout 90000 for > > > > client /192.168.0.19:49711 > > > > 2017-01-24 21:18:46,169 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750009 with negotiated timeout 90000 for > > > > client /192.168.0.19:49712 > > > > 2017-01-24 21:18:46,571 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49713 > > > > 2017-01-24 21:18:46,578 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.19:49713 > > > > 2017-01-24 21:18:46,655 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d2099775000a with negotiated timeout 90000 for > > > > client /192.168.0.19:49713 > > > > 2017-01-24 21:18:46,875 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49714 > > > > 2017-01-24 21:18:46,877 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.19:49714 > > > > 2017-01-24 21:18:46,894 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d2099775000b with negotiated timeout 90000 for > > > > client /192.168.0.19:49714 > > > > 2017-01-24 21:29:04,319 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750006, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:29:04,320 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49709 which had sessionid 0x259d20997750006 > > > > 2017-01-24 21:29:04,344 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d2099775000a, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:29:04,345 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49713 which had sessionid 0x259d2099775000a > > > > 2017-01-24 21:29:04,361 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750004, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:29:04,361 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49706 which had sessionid 0x259d20997750004 > > > > 2017-01-24 21:29:04,381 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750008, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:29:04,381 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49711 which had sessionid 0x259d20997750008 > > > > 2017-01-24 21:29:04,392 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d2099775000b, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:29:04,393 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49714 which had sessionid 0x259d2099775000b > > > > 2017-01-24 21:29:04,412 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750005, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:29:04,413 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49708 which had sessionid 0x259d20997750005 > > > > 2017-01-24 21:29:04,425 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750007, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:29:04,426 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49710 which had sessionid 0x259d20997750007 > > > > 2017-01-24 21:29:04,445 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750009, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:29:04,446 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49712 which had sessionid 0x259d20997750009 > > > > 2017-01-24 21:29:10,042 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49761 > > > > 2017-01-24 21:29:10,044 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.19:49761 > > > > 2017-01-24 21:29:10,051 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d2099775000c with negotiated timeout 90000 for > > > > client /192.168.0.19:49761 > > > > 2017-01-24 21:29:12,150 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49765 > > > > 2017-01-24 21:29:12,152 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.19:49765 > > > > 2017-01-24 21:29:12,165 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d2099775000d with negotiated timeout 90000 for > > > > client /192.168.0.19:49765 > > > > 2017-01-24 21:29:13,332 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49769 > > > > 2017-01-24 21:29:13,334 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.19:49769 > > > > 2017-01-24 21:29:13,343 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d2099775000e with negotiated timeout 90000 for > > > > client /192.168.0.19:49769 > > > > 2017-01-24 21:29:13,933 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49772 > > > > 2017-01-24 21:29:13,935 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.19:49772 > > > > 2017-01-24 21:29:13,939 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d2099775000f with negotiated timeout 90000 for > > > > client /192.168.0.19:49772 > > > > 2017-01-24 21:39:34,294 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d2099775000e, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:39:34,295 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49769 which had sessionid 0x259d2099775000e > > > > 2017-01-24 21:39:34,314 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d2099775000d, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:39:34,315 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49765 which had sessionid 0x259d2099775000d > > > > 2017-01-24 21:39:34,343 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d2099775000f, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:39:34,343 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49772 which had sessionid 0x259d2099775000f > > > > 2017-01-24 21:39:34,366 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d2099775000c, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:39:34,367 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49761 which had sessionid 0x259d2099775000c > > > > 2017-01-24 21:39:38,640 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49817 > > > > 2017-01-24 21:39:38,642 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.19:49817 > > > > 2017-01-24 21:39:38,647 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750010 with negotiated timeout 90000 for > > > > client /192.168.0.19:49817 > > > > 2017-01-24 21:39:40,386 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49821 > > > > 2017-01-24 21:39:40,392 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.19:49821 > > > > 2017-01-24 21:39:40,399 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750011 with negotiated timeout 90000 for > > > > client /192.168.0.19:49821 > > > > 2017-01-24 21:39:41,708 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49825 > > > > 2017-01-24 21:39:41,712 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.19:49825 > > > > 2017-01-24 21:39:41,716 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750012 with negotiated timeout 90000 for > > > > client /192.168.0.19:49825 > > > > 2017-01-24 21:39:42,420 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49828 > > > > 2017-01-24 21:39:42,421 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.19:49828 > > > > 2017-01-24 21:39:42,426 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750013 with negotiated timeout 90000 for > > > > client /192.168.0.19:49828 > > > > 2017-01-24 21:50:04,309 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750013, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:50:04,309 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49828 which had sessionid 0x259d20997750013 > > > > 2017-01-24 21:50:04,328 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750012, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:50:04,329 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49825 which had sessionid 0x259d20997750012 > > > > 2017-01-24 21:50:04,384 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750010, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:50:04,387 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49817 which had sessionid 0x259d20997750010 > > > > 2017-01-24 21:50:04,396 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750011, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:50:04,397 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49821 which had sessionid 0x259d20997750011 > > > > 2017-01-24 21:50:08,380 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49857 > > > > 2017-01-24 21:50:08,382 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.19:49857 > > > > 2017-01-24 21:50:08,387 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750014 with negotiated timeout 90000 for > > > > client /192.168.0.19:49857 > > > > 2017-01-24 21:50:09,791 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49862 > > > > 2017-01-24 21:50:09,792 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.19:49862 > > > > 2017-01-24 21:50:09,799 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750015 with negotiated timeout 90000 for > > > > client /192.168.0.19:49862 > > > > 2017-01-24 21:50:11,381 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49866 > > > > 2017-01-24 21:50:11,383 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.19:49866 > > > > 2017-01-24 21:50:11,390 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750016 with negotiated timeout 90000 for > > > > client /192.168.0.19:49866 > > > > 2017-01-24 21:50:12,189 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49869 > > > > 2017-01-24 21:50:12,191 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session at / > > > > 192.168.0.19:49869 > > > > 2017-01-24 21:50:12,196 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750017 with negotiated timeout 90000 for > > > > client /192.168.0.19:49869 > > > > 2017-01-24 22:00:34,306 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750015, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 22:00:34,307 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49862 which had sessionid 0x259d20997750015 > > > > 2017-01-24 22:00:34,330 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750017, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 22:00:34,337 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49869 which had sessionid 0x259d20997750017 > > > > 2017-01-24 22:00:34,355 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750016, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 22:00:34,356 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49866 which had sessionid 0x259d20997750016 > > > > 2017-01-24 22:00:34,369 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750014, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 22:00:34,369 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49857 which had sessionid 0x259d20997750014 > > > > > > > > ---------- > > > > > > > > The RegionServer logs are clean, no warnings. I assume this is a > > > > ZooKeeper problem more than HBase, but I tried many different > > > > configurations and nothing worked. Does anyone have an idea what could > > > > be happening? > > > > > > > > Best, > > > > Hernán. > > > > > > > > > >
