Thanks St.Ack! I must have lost that config after my last hadoop upgrade. Much appreciated!
Travis -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Stack Sent: Thursday, July 15, 2010 11:42 AM To: [email protected] Subject: Re: Client Connection to Zookeeper times out during mapreduce Is this intentional? Attempting connection to server localhost/127.0.0.1:2181 I'd guess not. Would seem to indicate then that your conf dir is not on the CLASSPATH when the child task runs? You've seen http://hbase.apache.org/docs/r0.20.5/api/org/apache/hadoop/hbase/mapreduce/package-summary.html#classpath? St.Ack On Thu, Jul 15, 2010 at 7:16 AM, Hegner, Travis <[email protected]> wrote: > Afterthought: > > hadoop: 0.20.2, r911707 > hbase: 0.20.5, r950246 > > Thanks Again! > > > > -----Original Message----- > From: Hegner, Travis [mailto:[email protected]] > Sent: Thursday, July 15, 2010 10:15 AM > To: [email protected] > Subject: Client Connection to Zookeeper times out during mapreduce > > Hi All! > > I apologize in advance for what is probably a rookie mistake, but it's been a > while since I've done a map reduce with Hbase as input or output. I'm > assuming that this is a cluster config problem because even the built-in > "rowcounter" mapreduce job is failing. The hbase shell is operating normally, > and I can even access Hbase from thrift. > > Cluster: > 1 master: NN, JT, ZK, HM > 5 slaves: DN, TT, RS > > When attempting to run a mapreduce job with Hbase as a source, my map tasks > are failing with the following logged exceptions: > > 2010-07-15 09:16:22,383 INFO org.apache.zookeeper.ZooKeeper: Client > environment:user.dir=/tmp/hadoop-hadoop/mapred/local/taskTracker/jobcache/job_201007150924_0003/attempt_201007150924_0003_m_000000_0/work > 2010-07-15 09:16:22,386 INFO org.apache.zookeeper.ZooKeeper: Initiating > client connection, connectString=localhost:2181 sessionTimeout=60000 > watcher=org.apache.hadoop.hbase.client.hconnectionmanager$clientzkwatc...@80fa6f > 2010-07-15 09:16:22,389 INFO org.apache.zookeeper.ClientCnxn: > zookeeper.disableAutoWatchReset is false > 2010-07-15 09:16:22,402 INFO org.apache.zookeeper.ClientCnxn: Attempting > connection to server localhost/127.0.0.1:2181 > 2010-07-15 09:16:22,407 WARN org.apache.zookeeper.ClientCnxn: Exception > closing session 0x0 to sun.nio.ch.selectionkeyi...@1d85f79 > java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933) > 2010-07-15 09:16:22,416 WARN org.apache.zookeeper.ClientCnxn: Ignoring > exception during shutdown input > java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) > at > sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) > at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) > 2010-07-15 09:16:22,416 WARN org.apache.zookeeper.ClientCnxn: Ignoring > exception during shutdown output > java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) > at > sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) > at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) > 2010-07-15 09:16:22,532 WARN > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create /hbase > -- check quorum servers, currently=localhost:2181 > org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode > = ConnectionLoss for /hbase > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:90) > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:42) > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780) > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:974) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:625) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:670) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:630) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:670) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:634) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601) > at > org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:134) > at > org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:112) > at > org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:87) > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) > at > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:573) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > at org.apache.hadoop.mapred.Child.main(Child.java:170) > 2010-07-15 09:16:22,689 INFO org.apache.zookeeper.ClientCnxn: Attempting > connection to server localhost/0:0:0:0:0:0:0:1:2181 > 2010-07-15 09:16:22,690 WARN org.apache.zookeeper.ClientCnxn: Exception > closing session 0x0 to sun.nio.ch.selectionkeyi...@17ce4e7 > java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933) > 2010-07-15 09:16:22,690 WARN org.apache.zookeeper.ClientCnxn: Ignoring > exception during shutdown input > java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) > at > sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) > at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) > 2010-07-15 09:16:22,690 WARN org.apache.zookeeper.ClientCnxn: Ignoring > exception during shutdown output > java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) > at > sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) > at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) > 2010-07-15 09:16:24,056 INFO org.apache.zookeeper.ClientCnxn: Attempting > connection to server localhost/127.0.0.1:2181 > 2010-07-15 09:16:24,056 WARN org.apache.zookeeper.ClientCnxn: Exception > closing session 0x0 to sun.nio.ch.selectionkeyi...@982589 > java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933) > 2010-07-15 09:16:24,057 WARN org.apache.zookeeper.ClientCnxn: Ignoring > exception during shutdown input > java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) > at > sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) > at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) > 2010-07-15 09:16:24,057 WARN org.apache.zookeeper.ClientCnxn: Ignoring > exception during shutdown output > java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) > at > sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) > at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) > > Where I'm confused is that the log says it's connecting to the quorum at > 'localhost', but the quorum is configured only on the master node. All DNS, > rDNS is configured properly. The hadoop-env.sh file has the zookeeper.jar in > the classpath on all of the slaves. > > Can anyone see what I'm missing? > > Thanks, > Travis Hegner > http://www.travishegner.com/ > > > ________________________________ > The information contained in this communication is confidential and is > intended only for the use of the named recipient. Unauthorized use, > disclosure, or copying is strictly prohibited and may be unlawful. If you > have received this communication in error, you should know that you are bound > to confidentiality, and should please immediately notify the sender or our IT > Department at 866.459.4599. > > The information contained in this communication is confidential and is > intended only for the use of the named recipient. Unauthorized use, > disclosure, or copying is strictly prohibited and may be unlawful. If you > have received this communication in error, you should know that you are bound > to confidentiality, and should please immediately notify the sender or our IT > Department at 866.459.4599. > The information contained in this communication is confidential and is intended only for the use of the named recipient. Unauthorized use, disclosure, or copying is strictly prohibited and may be unlawful. If you have received this communication in error, you should know that you are bound to confidentiality, and should please immediately notify the sender or our IT Department at 866.459.4599.
