Is this intentional? Attempting connection to server localhost/127.0.0.1:2181
I'd guess not. Would seem to indicate then that your conf dir is not on the CLASSPATH when the child task runs? You've seen http://hbase.apache.org/docs/r0.20.5/api/org/apache/hadoop/hbase/mapreduce/package-summary.html#classpath? St.Ack On Thu, Jul 15, 2010 at 7:16 AM, Hegner, Travis <[email protected]> wrote: > Afterthought: > > hadoop: 0.20.2, r911707 > hbase: 0.20.5, r950246 > > Thanks Again! > > > > -----Original Message----- > From: Hegner, Travis [mailto:[email protected]] > Sent: Thursday, July 15, 2010 10:15 AM > To: [email protected] > Subject: Client Connection to Zookeeper times out during mapreduce > > Hi All! > > I apologize in advance for what is probably a rookie mistake, but it's been a > while since I've done a map reduce with Hbase as input or output. I'm > assuming that this is a cluster config problem because even the built-in > "rowcounter" mapreduce job is failing. The hbase shell is operating normally, > and I can even access Hbase from thrift. > > Cluster: > 1 master: NN, JT, ZK, HM > 5 slaves: DN, TT, RS > > When attempting to run a mapreduce job with Hbase as a source, my map tasks > are failing with the following logged exceptions: > > 2010-07-15 09:16:22,383 INFO org.apache.zookeeper.ZooKeeper: Client > environment:user.dir=/tmp/hadoop-hadoop/mapred/local/taskTracker/jobcache/job_201007150924_0003/attempt_201007150924_0003_m_000000_0/work > 2010-07-15 09:16:22,386 INFO org.apache.zookeeper.ZooKeeper: Initiating > client connection, connectString=localhost:2181 sessionTimeout=60000 > watcher=org.apache.hadoop.hbase.client.hconnectionmanager$clientzkwatc...@80fa6f > 2010-07-15 09:16:22,389 INFO org.apache.zookeeper.ClientCnxn: > zookeeper.disableAutoWatchReset is false > 2010-07-15 09:16:22,402 INFO org.apache.zookeeper.ClientCnxn: Attempting > connection to server localhost/127.0.0.1:2181 > 2010-07-15 09:16:22,407 WARN org.apache.zookeeper.ClientCnxn: Exception > closing session 0x0 to sun.nio.ch.selectionkeyi...@1d85f79 > java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933) > 2010-07-15 09:16:22,416 WARN org.apache.zookeeper.ClientCnxn: Ignoring > exception during shutdown input > java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) > at > sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) > at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) > 2010-07-15 09:16:22,416 WARN org.apache.zookeeper.ClientCnxn: Ignoring > exception during shutdown output > java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) > at > sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) > at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) > 2010-07-15 09:16:22,532 WARN > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create /hbase > -- check quorum servers, currently=localhost:2181 > org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode > = ConnectionLoss for /hbase > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:90) > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:42) > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780) > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:974) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:625) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:670) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:630) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:670) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:634) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601) > at > org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:134) > at > org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:112) > at > org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:87) > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) > at > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:573) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > at org.apache.hadoop.mapred.Child.main(Child.java:170) > 2010-07-15 09:16:22,689 INFO org.apache.zookeeper.ClientCnxn: Attempting > connection to server localhost/0:0:0:0:0:0:0:1:2181 > 2010-07-15 09:16:22,690 WARN org.apache.zookeeper.ClientCnxn: Exception > closing session 0x0 to sun.nio.ch.selectionkeyi...@17ce4e7 > java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933) > 2010-07-15 09:16:22,690 WARN org.apache.zookeeper.ClientCnxn: Ignoring > exception during shutdown input > java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) > at > sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) > at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) > 2010-07-15 09:16:22,690 WARN org.apache.zookeeper.ClientCnxn: Ignoring > exception during shutdown output > java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) > at > sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) > at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) > 2010-07-15 09:16:24,056 INFO org.apache.zookeeper.ClientCnxn: Attempting > connection to server localhost/127.0.0.1:2181 > 2010-07-15 09:16:24,056 WARN org.apache.zookeeper.ClientCnxn: Exception > closing session 0x0 to sun.nio.ch.selectionkeyi...@982589 > java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933) > 2010-07-15 09:16:24,057 WARN org.apache.zookeeper.ClientCnxn: Ignoring > exception during shutdown input > java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) > at > sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) > at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) > 2010-07-15 09:16:24,057 WARN org.apache.zookeeper.ClientCnxn: Ignoring > exception during shutdown output > java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) > at > sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) > at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) > > Where I'm confused is that the log says it's connecting to the quorum at > 'localhost', but the quorum is configured only on the master node. All DNS, > rDNS is configured properly. The hadoop-env.sh file has the zookeeper.jar in > the classpath on all of the slaves. > > Can anyone see what I'm missing? > > Thanks, > Travis Hegner > http://www.travishegner.com/ > > > ________________________________ > The information contained in this communication is confidential and is > intended only for the use of the named recipient. Unauthorized use, > disclosure, or copying is strictly prohibited and may be unlawful. If you > have received this communication in error, you should know that you are bound > to confidentiality, and should please immediately notify the sender or our IT > Department at 866.459.4599. > > The information contained in this communication is confidential and is > intended only for the use of the named recipient. Unauthorized use, > disclosure, or copying is strictly prohibited and may be unlawful. If you > have received this communication in error, you should know that you are bound > to confidentiality, and should please immediately notify the sender or our IT > Department at 866.459.4599. >
