We're having some trouble with the Flume & the HbaseSink working. Seems we cannot hang on to zookeeper sessions. The flume list recommended I cross post here as this seems to be something up with the hbaseclient &/or zookeeper. In all honesty I am probably just doing something dumb and not seeing it.
Using latest flume-ng 1.3.0, hbase 0.94.0. Hbase & ZK both look fine and are in normal prod use. I don't think we are hitting our hbase.zookeeper.property.maxClientCnxns as we only have about 95 sessions in ZK DUMP and I believe it defaults to 2k. I can establish new sessions using the CLI from the same server and idle there for 10 minutes without getting dropped. When I start up flume I see the session show up in the ZK DUMP, see active TCP connection on the port etc., it just keeps dropping/reconnecting. Flume & zookeeper logs below, using rsync'd copies of our hadoop & hbase directories from one of our datanode+regionserver's. *Searched the flume list, this user appeared to have the same problem, not sure how he fixed it though:* http://mail-archives.apache.org/mod_mbox/flume-user/201207.mbox/%3ccadgprhcgztwhuuji1izmrx2uokeawwhyokkskkhymkopakw...@mail.gmail.com%3E *startup cmd */app/apache-flume-1.3.0-SNAPSHOT/bin/flume-ng agent -n agent1 -c ./conf -f conf/brian.properties -Dflume.root.logger=INFO,console* * *flume-env.sh* $ cat conf/flume-env.sh # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be sourced # during Flume startup. # Enviroment variables can be set here. #JAVA_HOME=/usr/lib/jvm/java-6-sun # Give Flume more memory and pre-allocate, enable remote monitoring via JMX #JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote" # Note that the Flume conf directory is always included in the classpath. #FLUME_CLASSPATH="/app/flume/lib" FLUME_CLASSPATH="/app/apache-flume-1.3.0-SNAPSHOT/lib" HBASE_HOME="/app/hbase-0.94.0" HADOOP_HOME="/app/hadoop-1.0.1" *config* $ cat conf/brian.properties #example.conf: A single-node Flume configuration # Name the components on this agent agent1.sources = source1 agent1.sinks = sink1 agent1.channels = channel1 # Describe/configure source1 agent1.sources.source1.type = exec agent1.sources.source1.command = tail -F /tank/log_dev.log agent1.sources.source1.batchSize = 1 # Describe sink1 #agent1.sinks.sink1.type = logger agent1.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink agent1.sinks.sink1.table = brian_test agent1.sinks.sink1.columnFamily = f1 #agent1.sinks.sink1.serializer = org.apache.flume.sink.hbase.SimpleHBaseEventSerializer #agent1.sinks.sink1.serializer = org.apache.flume.sink.hbase.SimpleHbaseEventSerializer # Use a channel which buffers events in memory agent1.channels.channel1.type = memory agent1.channels.channel1.capacity = 1000 agent1.channels.channel1.transactionCapactiy = 100 # Bind the source and sink to the channel agent1.sources.source1.channels = channel1 agent1.sinks.sink1.channel = channel1 *flume console log* 2012-10-08 12:06:19,007 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.library.path=:/app/hadoop-1.0.1/libexec/../lib/native/FreeBSD-amd64-64:/app/hbase-0.94.0/bin/../lib/native/FreeBSD-amd64-64 2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.io.tmpdir=/var/tmp/ 2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.compiler=<NA> 2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.name=FreeBSD 2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.arch=amd64 2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.version=9.0-RELEASE 2012-10-08 12:06:19,009 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.name=czhang 2012-10-08 12:06:19,009 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.home=/users/czhang 2012-10-08 12:06:19,009 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.dir=/app/apache-flume-1.3.0-SNAPSHOT 2012-10-08 12:06:19,010 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating client connection, connectString=hbasemaster0.hadoop.domain.com:2181sessionTimeout=180000 watcher=hconnection 2012-10-08 12:06:19,036 (lifecycleSupervisor-1-1) [INFO - org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:97)] The identifier of this process is [email protected] 2012-10-08 12:06:19,041 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)] Opening socket connection to server hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to authenticate using SASL (unknown error) 2012-10-08 12:06:19,049 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)] Socket connection established to hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session 2012-10-08 12:06:19,072 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)] Session establishment complete on server hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid = 0x138a0620045bbce, negotiated timeout = 180000 2012-10-08 12:08:19,070 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)] Client session timed out, have not heard from server in 120001ms for sessionid 0x138a0620045bbce, closing socket connection and attempting reconnect 2012-10-08 12:08:19,233 (lifecycleSupervisor-1-1) [WARN - org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:195)] Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master 2012-10-08 12:08:19,234 (lifecycleSupervisor-1-1) [INFO - org.apache.hadoop.hbase.util.RetryCounter.sleepUntilNextRetry(RetryCounter.java:53)] Sleeping 2000ms before retry #1... 2012-10-08 12:08:20,691 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)] Opening socket connection to server hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to authenticate using SASL (unknown error) 2012-10-08 12:08:20,692 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)] Socket connection established to hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session 2012-10-08 12:08:20,695 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)] Session establishment complete on server hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid = 0x138a0620045bbce, negotiated timeout = 180000 2012-10-08 12:10:20,695 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)] Client session timed out, have not heard from server in 120000ms for sessionid 0x138a0620045bbce, closing socket connection and attempting reconnect 2012-10-08 12:10:20,796 (lifecycleSupervisor-1-1) [WARN - org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:195)] Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master 2012-10-08 12:10:20,796 (lifecycleSupervisor-1-1) [INFO - org.apache.hadoop.hbase.util.RetryCounter.sleepUntilNextRetry(RetryCounter.java:53)] Sleeping 4000ms before retry #2... 2012-10-08 12:10:22,792 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)] Opening socket connection to server hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to authenticate using SASL (unknown error) 2012-10-08 12:10:22,794 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)] Socket connection established to hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session 2012-10-08 12:10:22,799 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)] Session establishment complete on server hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid = 0x138a0620045bbce, negotiated timeout = 180000 2012-10-08 12:12:22,800 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)] Client session timed out, have not heard from server in 120001ms for sessionid 0x138a0620045bbce, closing socket connection and attempting reconnect 2012-10-08 12:12:22,902 (lifecycleSupervisor-1-1) [WARN - org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:195)] Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master 2012-10-08 12:12:22,903 (lifecycleSupervisor-1-1) [INFO - org.apache.hadoop.hbase.util.RetryCounter.sleepUntilNextRetry(RetryCounter.java:53)] Sleeping 8000ms before retry #3... 2012-10-08 12:12:24,289 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)] Opening socket connection to server hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to authenticate using SASL (unknown error) 2012-10-08 12:12:24,291 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)] Socket connection established to hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session 2012-10-08 12:12:24,294 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)] Session establishment complete on server hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid = 0x138a0620045bbce, negotiated timeout = 180000 2012-10-08 12:14:24,294 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)] Client session timed out, have not heard from server in 120000ms for sessionid 0x138a0620045bbce, closing socket connection and attempting reconnect 2012-10-08 12:14:24,395 (lifecycleSupervisor-1-1) [WARN - org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:195)] Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master 2012-10-08 12:14:24,395 (lifecycleSupervisor-1-1) [ERROR - org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:197)] ZooKeeper exists failed after 3 retries 2012-10-08 12:14:24,398 (lifecycleSupervisor-1-1) [WARN - org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:239)] hconnection-0x138a0620045bbce-0x138a0620045bbce-0x138a0620045bbce-0x138a0620045bbce Unable to set watcher on znode /hbase/master org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:230) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:590) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:818) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133) at org.apache.flume.sink.hbase.HBaseSink.start(HBaseSink.java:107) at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46) at org.apache.flume.SinkRunner.start(SinkRunner.java:79) at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) 2012-10-08 12:14:24,405 (lifecycleSupervisor-1-1) [ERROR - org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.keeperException(ZooKeeperWatcher.java:408)] hconnection-0x138a0620045bbce-0x138a0620045bbce-0x138a0620045bbce-0x138a0620045bbce Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:230) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:590) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:818) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133) at org.apache.flume.sink.hbase.HBaseSink.start(HBaseSink.java:107) at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46) at org.apache.flume.SinkRunner.start(SinkRunner.java:79) at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) 2012-10-08 12:14:24,407 (lifecycleSupervisor-1-1) [INFO - org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1654)] This client just lost it's session with ZooKeeper, will automatically reconnect when needed. 2012-10-08 12:14:26,166 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)] Opening socket connection to server hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to authenticate using SASL (unknown error) 2012-10-08 12:14:26,167 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)] Socket connection established to hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session 2012-10-08 12:14:26,170 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)] Session establishment complete on server hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid = 0x138a0620045bbce, negotiated timeout = 180000 *zookeeper log* 2012-10-08 19:06:19,049 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.10.10.64:16755 2012-10-08 19:06:19,053 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.10.10.64:16755 2012-10-08 19:06:19,069 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x138a0620045bbce with negotiated timeout 180000 for client /10.10.10.64:16755 2012-10-08 19:08:19,072 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.10.10.64:16755 which had sessionid 0x138a0620045bbce 2012-10-08 19:08:20,692 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.10.10.64:33245 2012-10-08 19:08:20,693 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to renew session 0x138a0620045bbce at /10.10.10.64:33245 2012-10-08 19:08:20,694 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x138a0620045bbce with negotiated timeout 180000 for client /10.10.10.64:33245 2012-10-08 19:10:20,695 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x138a0620045bbce, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224) at java.lang.Thread.run(Thread.java:619) 2012-10-08 19:10:20,698 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.10.10.64:33245 which had sessionid 0x138a0620045bbce 2012-10-08 19:10:22,794 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.10.10.64:44993 2012-10-08 19:10:22,796 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to renew session 0x138a0620045bbce at /10.10.10.64:44993 2012-10-08 19:10:22,798 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x138a0620045bbce with negotiated timeout 180000 for client /10.10.10.64:44993 *We also tried the hbase async sink as that is recommended over in the flume group, and got similar ** results (but seemingly more serious CONNECTIONLOSS errors)* 2012-10-08 14:38:29,640 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:106)] Starting new configuration:{ sourceRunners:{source1=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:source1,state:IDLE} }} sinkRunners:{sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@3968a6fe counterGroup:{ name:null counters:{} } }} channels:{channel1=org.apache.flume.channel.MemoryChannel{name: channel1}} } 2012-10-08 14:38:29,640 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:113)] Starting Channel channel1 2012-10-08 14:38:29,641 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:82)] Component type: CHANNEL, name: channel1 started 2012-10-08 14:38:29,642 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:141)] Starting Sink sink1 2012-10-08 14:38:29,642 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:82)] Component type: SINK, name: sink1 started 2012-10-08 14:38:29,643 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:152)] Starting Source source1 2012-10-08 14:38:29,643 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.ExecSource.start(ExecSource.java:155)] Exec source starting with command:tail -F /tank/clients/telescope/users/czhang/connecttwitter_dev.log 2012-10-08 14:38:29,816 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:zookeeper.version=3.4.4-1386507, built on 09/17/2012 08:33 GMT 2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:host.name=app14.app.domain.com 2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.version=1.7.0_02 2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.vendor=Oracle Corporation 2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.home=/usr/local/openjdk7/jre 2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.class.path=/app/flume/conf:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-avro-source-1.3.0-20120929.012214-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-node-1.3.0-SNAPSHOT.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-log4jappender-1.3.0-20120929.012213-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-lang-2.5.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/netty-3.2.7.Final.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-scribe-source-1.3.0-20120929.012216-17.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-pool-1.5.4.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/avro-1.7.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/avro-ipc-1.7.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-thrift-source-1.3.0-20120929.012214-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/httpclient-4.0.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-core-1.3.0-SNAPSHOT.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-core-1.3.0-20120929.012205-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/mina-core-2.0.4.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jackson-mapper-asl-1.9.3.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/servlet-api-2.5-20110124.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jcl-over-slf4j-1.7.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jsr305-1.3.9.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-sdk-1.3.0-20120929.012209-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/netty-3.4.0.Final.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jetty-util-6.1.26.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-hdfs-sink-1.3.0-20120929.012214-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-configuration-1.3.0-SNAPSHOT.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-collections-3.2.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/async-1.2.0.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/paranamer-2.3.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/velocity-1.7.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-codec-1.3.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-hbase-sink-1.3.0-20120929.012215-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/irclib-1.10.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-jdbc-channel-1.3.0-20120929.012212-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-dbcp-1.4.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/slf4j-api-1.6.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/protobuf-java-2.4.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-irc-sink-1.3.0-20120929.012215-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-configuration-1.3.0-20120929.012205-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/joda-time-2.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-io-2.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-file-channel-1.3.0-20120929.012212-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-node-1.3.0-20120929.012209-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-recoverable-memory-channel-1.3.0-20120929.012213-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-cli-1.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jetty-6.1.26.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/asynchbase-1.2.0.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/gson-2.2.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/derby-10.8.2.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-logging-1.1.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/guava-10.0.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-sdk-1.3.0-SNAPSHOT.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/slf4j-log4j12-1.6.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/httpcore-4.0.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/libthrift-0.6.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/log4j-1.2.16.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/zookeeper-3.4.4.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/snappy-java-1.0.4.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jackson-core-asl-1.9.3.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib:/app/hadoop-1.0.1/libexec/../conf:/usr/local/lib/tools.jar:/app/hadoop-1.0.1/libexec/..:/app/hadoop-1.0.1/libexec/../hadoop-core-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/asm-3.2.jar:/app/hadoop-1.0.1/libexec/../lib/aspectjrt-1.6.5.jar:/app/hadoop-1.0.1/libexec/../lib/aspectjtools-1.6.5.jar:/app/hadoop-1.0.1/libexec/../lib/commons-beanutils-1.7.0.jar:/app/hadoop-1.0.1/libexec/../lib/commons-beanutils-core-1.8.0.jar:/app/hadoop-1.0.1/libexec/../lib/commons-cli-1.2.jar:/app/hadoop-1.0.1/libexec/../lib/commons-codec-1.4.jar:/app/hadoop-1.0.1/libexec/../lib/commons-collections-3.2.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-configuration-1.6.jar:/app/hadoop-1.0.1/libexec/../lib/commons-daemon-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-digester-1.8.jar:/app/hadoop-1.0.1/libexec/../lib/commons-el-1.0.jar:/app/hadoop-1.0.1/libexec/../lib/commons-httpclient-3.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-lang-2.4.jar:/app/hadoop-1.0.1/libexec/../lib/commons-logging-1.1.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-logging-api-1.0.4.jar:/app/hadoop-1.0.1/libexec/../lib/commons-math-2.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-net-1.4.1.jar:/app/hadoop-1.0.1/libexec/../lib/core-3.1.1.jar:/app/hadoop-1.0.1/libexec/../lib/hadoop-capacity-scheduler-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/hadoop-fairscheduler-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/hadoop-thriftfs-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/hsqldb-1.8.0.10.jar:/app/hadoop-1.0.1/libexec/../lib/jackson-core-asl-1.8.8.jar:/app/hadoop-1.0.1/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/app/hadoop-1.0.1/libexec/../lib/jasper-compiler-5.5.12.jar:/app/hadoop-1.0.1/libexec/../lib/jasper-runtime-5.5.12.jar:/app/hadoop-1.0.1/libexec/../lib/jdeb-0.8.jar:/app/hadoop-1.0.1/libexec/../lib/jersey-core-1.8.jar:/app/hadoop-1.0.1/libexec/../lib/jersey-json-1.8.jar:/app/hadoop-1.0.1/libexec/../lib/jersey-server-1.8.jar:/app/hadoop-1.0.1/libexec/../lib/jets3t-0.6.1.jar:/app/hadoop-1.0.1/libexec/../lib/jetty-6.1.26.jar:/app/hadoop-1.0.1/libexec/../lib/jetty-util-6.1.26.jar:/app/hadoop-1.0.1/libexec/../lib/jsch-0.1.42.jar:/app/hadoop-1.0.1/libexec/../lib/junit-4.5.jar:/app/hadoop-1.0.1/libexec/../lib/kfs-0.2.2.jar:/app/hadoop-1.0.1/libexec/../lib/log4j-1.2.15.jar:/app/hadoop-1.0.1/libexec/../lib/mockito-all-1.8.5.jar:/app/hadoop-1.0.1/libexec/../lib/oro-2.0.8.jar:/app/hadoop-1.0.1/libexec/../lib/servlet-api-2.5-20081211.jar:/app/hadoop-1.0.1/libexec/../lib/xmlenc-0.52.jar:/app/hadoop-1.0.1/libexec/../lib/jsp-2.1/jsp-2.1.jar:/app/hadoop-1.0.1/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/app/hbase-0.94.0/bin/../conf:/usr/local/lib/tools.jar:/app/hbase-0.94.0/bin/..:/app/hbase-0.94.0/bin/../hbase-0.94.0-tests.jar:/app/hbase-0.94.0/bin/../hbase-0.94.0.jar:/app/hbase-0.94.0/bin/../lib/activation-1.1.jar:/app/hbase-0.94.0/bin/../lib/asm-3.1.jar:/app/hbase-0.94.0/bin/../lib/avro-1.5.3.jar:/app/hbase-0.94.0/bin/../lib/avro-ipc-1.5.3.jar:/app/hbase-0.94.0/bin/../lib/commons-beanutils-1.7.0.jar:/app/hbase-0.94.0/bin/../lib/commons-beanutils-core-1.8.0.jar:/app/hbase-0.94.0/bin/../lib/commons-cli-1.2.jar:/app/hbase-0.94.0/bin/../lib/commons-codec-1.4.jar:/app/hbase-0.94.0/bin/../lib/commons-collections-3.2.1.jar:/app/hbase-0.94.0/bin/../lib/commons-configuration-1.6.jar:/app/hbase-0.94.0/bin/../lib/commons-digester-1.8.jar:/app/hbase-0.94.0/bin/../lib/commons-el-1.0.jar:/app/hbase-0.94.0/bin/../lib/commons-httpclient-3.1.jar:/app/hbase-0.94.0/bin/../lib/commons-io-2.1.jar:/app/hbase-0.94.0/bin/../lib/commons-lang-2.5.jar:/app/hbase-0.94.0/bin/../lib/commons-logging-1.1.1.jar:/app/hbase-0.94.0/bin/../lib/commons-math-2.1.jar:/app/hbase-0.94.0/bin/../lib/commons-net-1.4.1.jar:/app/hbase-0.94.0/bin/../lib/core-3.1.1.jar:/app/hbase-0.94.0/bin/../lib/guava-r09.jar:/app/hbase-0.94.0/bin/../lib/hadoop-core-1.0.1.jar:/app/hbase-0.94.0/bin/../lib/high-scale-lib-1.1.1.jar:/app/hbase-0.94.0/bin/../lib/httpclient-4.1.2.jar:/app/hbase-0.94.0/bin/../lib/httpcore-4.1.3.jar:/app/hbase-0.94.0/bin/../lib/jackson-core-asl-1.5.5.jar:/app/hbase-0.94.0/bin/../lib/jackson-jaxrs-1.5.5.jar:/app/hbase-0.94.0/bin/../lib/jackson-mapper-asl-1.5.5.jar:/app/hbase-0.94.0/bin/../lib/jackson-xc-1.5.5.jar:/app/hbase-0.94.0/bin/../lib/jamon-runtime-2.3.1.jar:/app/hbase-0.94.0/bin/../lib/jasper-compiler-5.5.23.jar:/app/hbase-0.94.0/bin/../lib/jasper-runtime-5.5.23.jar:/app/hbase-0.94.0/bin/../lib/jaxb-api-2.1.jar:/app/hbase-0.94.0/bin/../lib/jaxb-impl-2.1.12.jar:/app/hbase-0.94.0/bin/../lib/jersey-core-1.4.jar:/app/hbase-0.94.0/bin/../lib/jersey-json-1.4.jar:/app/hbase-0.94.0/bin/../lib/jersey-server-1.4.jar:/app/hbase-0.94.0/bin/../lib/jettison-1.1.jar:/app/hbase-0.94.0/bin/../lib/jetty-6.1.26.jar:/app/hbase-0.94.0/bin/../lib/jetty-util-6.1.26.jar:/app/hbase-0.94.0/bin/../lib/jruby-complete-1.6.5.jar:/app/hbase-0.94.0/bin/../lib/jsp-2.1-6.1.14.jar:/app/hbase-0.94.0/bin/../lib/jsp-api-2.1-6.1.14.jar:/app/hbase-0.94.0/bin/../lib/libthrift-0.8.0.jar:/app/hbase-0.94.0/bin/../lib/log4j-1.2.16.jar:/app/hbase-0.94.0/bin/../lib/netty-3.2.4.Final.jar:/app/hbase-0.94.0/bin/../lib/protobuf-java-2.4.0a.jar:/app/hbase-0.94.0/bin/../lib/servlet-api-2.5-6.1.14.jar:/app/hbase-0.94.0/bin/../lib/snappy-java-1.0.3.2.jar:/app/hbase-0.94.0/bin/../lib/stax-api-1.0.1.jar:/app/hbase-0.94.0/bin/../lib/velocity-1.7.jar:/app/hbase-0.94.0/bin/../lib/xmlenc-0.52.jar:/app/hbase-0.94.0/bin/../lib/zookeeper-3.4.3.jar:.:/app/hbase-0.94.0/conf 2012-10-08 14:38:29,819 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.library.path=:/app/hadoop-1.0.1/libexec/../lib/native/FreeBSD-amd64-64:/app/hbase-0.94.0/bin/../lib/native/FreeBSD-amd64-64 2012-10-08 14:38:29,819 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.io.tmpdir=/var/tmp/ 2012-10-08 14:38:29,819 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.compiler=<NA> 2012-10-08 14:38:29,819 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.name=FreeBSD 2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.arch=amd64 2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.version=9.0-RELEASE 2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.name=czhang 2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.home=/tank/clients/telescope/users/czhang 2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.dir=/app/apache-flume-1.3.0-SNAPSHOT 2012-10-08 14:38:29,822 (lifecycleSupervisor-1-1) [INFO - org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating client connection, connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914 2012-10-08 14:38:29,863 (lifecycleSupervisor-1-1) [INFO - org.hbase.async.HBaseClient$ZKClient.getDeferredRoot(HBaseClient.java:2238)] Need to find the -ROOT- region 2012-10-08 14:38:29,870 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)] Opening socket connection to server hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to authenticate using SASL (unknown error) 2012-10-08 14:38:29,879 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)] Socket connection established to hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session 2012-10-08 14:38:29,908 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)] Session establishment complete on server hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid = 0x138a0620045bdb5, negotiated timeout = 6000 2012-10-08 14:38:33,906 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)] Client session timed out, have not heard from server in 4001ms for sessionid 0x138a0620045bdb5, closing socket connection and attempting reconnect 2012-10-08 14:38:34,013 (lifecycleSupervisor-1-1-EventThread) [ERROR - org.hbase.async.HBaseClient$ZKClient$2.processResult(HBaseClient.java:2407)] Looks like our ZK session expired or is broken, rc=-4: CONNECTIONLOSS 2012-10-08 14:38:34,969 (lifecycleSupervisor-1-1-EventThread) [INFO - org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session: 0x138a0620045bdb5 closed 2012-10-08 14:38:34,969 (lifecycleSupervisor-1-1-EventThread) [INFO - org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating client connection, connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914 2012-10-08 14:38:34,971 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)] Opening socket connection to server hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to authenticate using SASL (unknown error) 2012-10-08 14:38:34,972 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)] Socket connection established to hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session 2012-10-08 14:38:34,986 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)] Session establishment complete on server hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid = 0x138a0620045bdba, negotiated timeout = 6000 2012-10-08 14:38:39,088 (lifecycleSupervisor-1-1-EventThread) [INFO - org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session: 0x138a0620045bdba closed 2012-10-08 14:38:39,089 (lifecycleSupervisor-1-1-EventThread) [WARN - org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2304)] No longer connected to ZooKeeper, event=WatchedEvent state:Disconnected type:None path:null 2012-10-08 14:38:39,089 (lifecycleSupervisor-1-1-EventThread) [INFO - org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating client connection, connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914 2012-10-08 14:38:39,090 (lifecycleSupervisor-1-1-EventThread) [INFO - org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:509)] EventThread shut down 2012-10-08 14:38:39,090 (lifecycleSupervisor-1-1-EventThread) [INFO - org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:509)] EventThread shut down 2012-10-08 14:38:39,091 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)] Opening socket connection to server hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to authenticate using SASL (unknown error) 2012-10-08 14:38:39,092 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)] Socket connection established to hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session 2012-10-08 14:38:39,163 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)] Session establishment complete on server hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid = 0x138a0620045bdbb, negotiated timeout = 6000 2012-10-08 14:38:43,164 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)] Client session timed out, have not heard from server in 4001ms for sessionid 0x138a0620045bdbb, closing socket connection and attempting reconnect 2012-10-08 14:38:43,265 (lifecycleSupervisor-1-1-EventThread) [ERROR - org.hbase.async.HBaseClient$ZKClient$2.processResult(HBaseClient.java:2407)] Looks like our ZK session expired or is broken, rc=-4: CONNECTIONLOSS 2012-10-08 14:38:43,683 (lifecycleSupervisor-1-1-EventThread) [INFO - org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session: 0x138a0620045bdbb closed 2012-10-08 14:38:43,684 (lifecycleSupervisor-1-1-EventThread) [INFO - org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating client connection, connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914 2012-10-08 14:38:43,686 (lifecycleSupervisor-1-1-EventThread) [ERROR - org.hbase.async.HBaseClient$ZKClient$2.processResult(HBaseClient.java:2407)] Looks like our ZK session expired or is broken, rc=-4: CONNECTIONLOSS 2012-10-08 14:38:43,687 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)] Opening socket connection to server hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to authenticate using SASL (unknown error) 2012-10-08 14:38:43,688 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)] Socket connection established to hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session 2012-10-08 14:38:43,701 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)] Session establishment complete on server hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid = 0x138a0620045bdbc, negotiated timeout = 6000 2012-10-08 14:38:47,802 (lifecycleSupervisor-1-1-EventThread) [INFO - org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session: 0x138a0620045bdbc closed 2012-10-08 14:38:47,804 (lifecycleSupervisor-1-1-EventThread) [INFO - org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating client connection, connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914 2012-10-08 14:38:47,807 (lifecycleSupervisor-1-1-EventThread) [INFO - org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:509)] EventThread shut down 2012-10-08 14:38:47,810 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)] Opening socket connection to server hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to authenticate using SASL (unknown error) 2012-10-08 14:38:47,812 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)] Socket connection established to hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session 2012-10-08 14:38:47,821 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)] Session establishment complete on server hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid = 0x138a0620045bdbd, negotiated timeout = 6000 2012-10-08 14:38:51,922 (lifecycleSupervisor-1-1-EventThread) [INFO - org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session: 0x138a0620045bdbd closed 2012-10-08 14:38:51,922 (lifecycleSupervisor-1-1-EventThread) [WARN - org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2304)] No longer connected to ZooKeeper, event=WatchedEvent state:Disconnected type:None path:null 2012-10-08 14:38:51,923 (lifecycleSupervisor-1-1-EventThread) [INFO - org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating client connection, connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914 2012-10-08 14:38:51,924 (lifecycleSupervisor-1-1-EventThread) [INFO - org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:509)] EventThread shut down 2012-10-08 14:38:51,924 (lifecycleSupervisor-1-1-EventThread) [ERROR - org.hbase.async.HBaseClient$ZKClient$2.processResult(HBaseClient.java:2407)] Looks like our ZK session expired or is broken, rc=-4: CONNECTIONLOSS 2012-10-08 14:38:51,925 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)] Opening socket connection to server hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to authenticate using SASL (unknown error) 2012-10-08 14:38:51,926 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)] Socket connection established to hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session 2012-10-08 14:38:51,945 (lifecycleSupervisor-1-1-SendThread( hbasemaster0.hadoop.domain.com:2181)) [INFO - org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)] Session establishment complete on server hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid = 0x138a0620045bdbe, negotiated timeout = 6000 Thanks for the continued help from the community on getting us going w/flume Cheers, -- Iain Wright Cell: (562) 852-5916 <http://www.labctsi.org/> This email message is confidential, intended only for the recipient(s) named above and may contain information that is privileged, exempt from disclosure under applicable law. If you are not the intended recipient, do not disclose or disseminate the message to anyone except the intended recipient. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by return email, and delete all copies of this message.
