Hi All!
I apologize in advance for what is probably a rookie mistake, but it's been a
while since I've done a map reduce with Hbase as input or output. I'm assuming
that this is a cluster config problem because even the built-in "rowcounter"
mapreduce job is failing. The hbase shell is operating normally, and I can even
access Hbase from thrift.
Cluster:
1 master: NN, JT, ZK, HM
5 slaves: DN, TT, RS
When attempting to run a mapreduce job with Hbase as a source, my map tasks are
failing with the following logged exceptions:
2010-07-15 09:16:22,383 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.dir=/tmp/hadoop-hadoop/mapred/local/taskTracker/jobcache/job_201007150924_0003/attempt_201007150924_0003_m_000000_0/work
2010-07-15 09:16:22,386 INFO org.apache.zookeeper.ZooKeeper: Initiating client
connection, connectString=localhost:2181 sessionTimeout=60000
watcher=org.apache.hadoop.hbase.client.hconnectionmanager$clientzkwatc...@80fa6f
2010-07-15 09:16:22,389 INFO org.apache.zookeeper.ClientCnxn:
zookeeper.disableAutoWatchReset is false
2010-07-15 09:16:22,402 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server localhost/127.0.0.1:2181
2010-07-15 09:16:22,407 WARN org.apache.zookeeper.ClientCnxn: Exception closing
session 0x0 to sun.nio.ch.selectionkeyi...@1d85f79
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
2010-07-15 09:16:22,416 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at
sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
2010-07-15 09:16:22,416 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at
sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
2010-07-15 09:16:22,532 WARN
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create /hbase --
check quorum servers, currently=localhost:2181
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for /hbase
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:974)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:625)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:670)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:630)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:670)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:634)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:134)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:112)
at
org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:87)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:573)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
2010-07-15 09:16:22,689 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server localhost/0:0:0:0:0:0:0:1:2181
2010-07-15 09:16:22,690 WARN org.apache.zookeeper.ClientCnxn: Exception closing
session 0x0 to sun.nio.ch.selectionkeyi...@17ce4e7
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
2010-07-15 09:16:22,690 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at
sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
2010-07-15 09:16:22,690 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at
sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
2010-07-15 09:16:24,056 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server localhost/127.0.0.1:2181
2010-07-15 09:16:24,056 WARN org.apache.zookeeper.ClientCnxn: Exception closing
session 0x0 to sun.nio.ch.selectionkeyi...@982589
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
2010-07-15 09:16:24,057 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at
sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
2010-07-15 09:16:24,057 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at
sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
Where I'm confused is that the log says it's connecting to the quorum at
'localhost', but the quorum is configured only on the master node. All DNS,
rDNS is configured properly. The hadoop-env.sh file has the zookeeper.jar in
the classpath on all of the slaves.
Can anyone see what I'm missing?
Thanks,
Travis Hegner
http://www.travishegner.com/
________________________________
The information contained in this communication is confidential and is intended
only for the use of the named recipient. Unauthorized use, disclosure, or
copying is strictly prohibited and may be unlawful. If you have received this
communication in error, you should know that you are bound to confidentiality,
and should please immediately notify the sender or our IT Department at
866.459.4599.
