So here's what I can see in the thrift logs:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
2011-01-25 23:05:16,399 INFO org.apache.zookeeper.ClientCnxn: Opening socket
connection to server ip-10-113-50-207.ec2.internal/10.113.50.207:2181
2011-01-25 23:05:16,410 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for
server null, unexpected error, closing socket connection and attempting
reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
2011-01-25 23:05:16,510 INFO
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: getMaster
attempt 9 of 10 failed; no more retrying.
java.io.IOException:
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for /hbase/master
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:481)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:377)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:381)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:78)
at
org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.<init>(ThriftServer.java:191)
at
org.apache.hadoop.hbase.thrift.ThriftServer.doMain(ThriftServer.java:817)
at
org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:874)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master
at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:921)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:477)
... 6 more
Even more confusing...
--jacob
@thedatachef
On Jan 27, 2011, at 5:19 PM, Dmitriy Ryaboy wrote:
> Looks like it's trying to create the HTable object (to get the splits) and
> failing.
> Check the hbase logs.. it's making the call to hbase ok, and failing there
>
> On Thu, Jan 27, 2011 at 2:58 PM, Jacob Perkins
> <[email protected]>wrote:
>
>> Running this script:
>>
>> data = LOAD '$TABLE' USING HBaseStorage('$CF:field_1');
>> DUMP data;
>>
>> fails with the following error:
>>
>> Failed Jobs:
>> JobId Alias Feature Message Outputs
>> N/A data MAP_ONLY Message:
>> org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Call to /
>> 10.117.86.81:60020 failed on local exception: java.io.EOFException
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:280)
>> at
>> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:899)
>> at
>> org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:916)
>> at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
>> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:834)
>> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:793)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>> at
>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:793)
>> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:767)
>> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
>> at
>> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
>> at
>> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
>> at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.io.IOException: Call to /10.117.86.81:60020 failed on
>> local exception: java.io.EOFException
>> at
>> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:788)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:757)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>> at $Proxy6.getProtocolVersion(Unknown Source)
>> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
>> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
>> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:952)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:969)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:685)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:584)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:559)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:782)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:588)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:553)
>> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)
>> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
>> at
>> com.infochimps.hbase.pig.HBaseStorage.setLocation(HBaseStorage.java:279)
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:261)
>> ... 14 more
>> Caused by: java.io.EOFException
>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:521)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:459)
>>
>> hdfs://ip-10-113-50-207.ec2.internal/tmp/temp-1162684483/tmp278494015
>>
>>
>>
>> I'm not sure what resource, exactly, it's looking for and not finding. Any
>> ideas? Thanks,
>>
>> --jacob
>> @thedatachef
