Looks like your zookeeper went away. You should ping the hbase user list... getting close to out of my depth for email troubleshooting :)
On Thu, Jan 27, 2011 at 3:24 PM, Jacob Perkins <[email protected]>wrote: > So here's what I can see in the thrift logs: > > java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) > 2011-01-25 23:05:16,399 INFO org.apache.zookeeper.ClientCnxn: Opening > socket connection to server ip-10-113-50-207.ec2.internal/ > 10.113.50.207:2181 > 2011-01-25 23:05:16,410 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 > for server null, unexpected error, closing socket connection and attempting > reconnect > java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) > 2011-01-25 23:05:16,510 INFO > org.apache.hadoop.hbase.client.HConnectionManager$TableServers: getMaster > attempt 9 of 10 failed; no more retrying. > java.io.IOException: > org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss for /hbase/master > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:481) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:377) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:381) > at > org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:78) > at > org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.<init>(ThriftServer.java:191) > at > org.apache.hadoop.hbase.thrift.ThriftServer.doMain(ThriftServer.java:817) > at > org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:874) > Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss for /hbase/master > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:90) > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:42) > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:921) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:477) > ... 6 more > > Even more confusing... > > --jacob > @thedatachef > > > On Jan 27, 2011, at 5:19 PM, Dmitriy Ryaboy wrote: > > > Looks like it's trying to create the HTable object (to get the splits) > and > > failing. > > Check the hbase logs.. it's making the call to hbase ok, and failing > there > > > > On Thu, Jan 27, 2011 at 2:58 PM, Jacob Perkins < > [email protected]>wrote: > > > >> Running this script: > >> > >> data = LOAD '$TABLE' USING HBaseStorage('$CF:field_1'); > >> DUMP data; > >> > >> fails with the following error: > >> > >> Failed Jobs: > >> JobId Alias Feature Message Outputs > >> N/A data MAP_ONLY Message: > >> org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Call > to / > >> 10.117.86.81:60020 failed on local exception: java.io.EOFException > >> at > >> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:280) > >> at > >> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:899) > >> at > >> org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:916) > >> at > org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170) > >> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:834) > >> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:793) > >> at java.security.AccessController.doPrivileged(Native Method) > >> at javax.security.auth.Subject.doAs(Subject.java:396) > >> at > >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063) > >> at > >> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:793) > >> at > org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:767) > >> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378) > >> at > >> > org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247) > >> at > >> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279) > >> at java.lang.Thread.run(Thread.java:662) > >> Caused by: java.io.IOException: Call to /10.117.86.81:60020 failed on > >> local exception: java.io.EOFException > >> at > >> > org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:788) > >> at > >> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:757) > >> at > >> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) > >> at $Proxy6.getProtocolVersion(Unknown Source) > >> at > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419) > >> at > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393) > >> at > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444) > >> at > >> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349) > >> at > >> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:952) > >> at > >> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:969) > >> at > >> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:685) > >> at > >> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:584) > >> at > >> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:559) > >> at > >> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:782) > >> at > >> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:588) > >> at > >> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:553) > >> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171) > >> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145) > >> at > >> com.infochimps.hbase.pig.HBaseStorage.setLocation(HBaseStorage.java:279) > >> at > >> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:261) > >> ... 14 more > >> Caused by: java.io.EOFException > >> at java.io.DataInputStream.readInt(DataInputStream.java:375) > >> at > >> > org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:521) > >> at > >> > org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:459) > >> > >> hdfs://ip-10-113-50-207.ec2.internal/tmp/temp-1162684483/tmp278494015 > >> > >> > >> > >> I'm not sure what resource, exactly, it's looking for and not finding. > Any > >> ideas? Thanks, > >> > >> --jacob > >> @thedatachef > >
