Hi, We use 0.20.6 to process large amount of data: FILE_BYTES_WRITTEN 132,953,083,977 Map output bytes 300,214,289,928
In two of our mappers which timed out I saw: 2010-11-24 23:16:51,561 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, host=us01-ciqps1-name01.carrieriq.com:2181 sessionTimeout=60000 watcher=org.apache.hadoop.hbase.client.hconnectionmanager$clientzkwatc...@f855562 2010-11-24 23:16:51,563 INFO org.apache.zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is false 2010-11-24 23:16:51,585 INFO org.apache.zookeeper.ClientCnxn: Attempting connection to server us01-ciqps1-name01.carrieriq.com/10.202.50.100:2181 2010-11-24 23:16:51,593 INFO org.apache.zookeeper.ClientCnxn: Priming connection to java.nio.channels.SocketChannel[connected local=/10.202.50.101:63047 remote=us01-ciqps1-name01.carrieriq.com/10.202.50.100:2181] 2010-11-24 23:16:51,596 INFO org.apache.zookeeper.ClientCnxn: Server connection successful 2010-11-24 23:16:55,127 INFO com.carrieriq.m2m.platform.mmp2.input.StripedHBaseTableInputFormat: Starting scan of table 'GRID-GRIDSQL-STAGING-THREEGPPSPEECHCALLS-1290634808555' As of this moment, GRID-GRIDSQL-STAGING-THREEGPPSPEECHCALLS-1290634808555 has been deleted because of failure handling in our flow. Our monitoring script started noticing the following at 2010-11-24 23-39-50 (GMT): HBase Shell; enter 'help<RETURN>' for list of supported commands. Version: 0.20.6, r965666, Mon Jul 19 15:48:07 PDT 2010 get 'GRID-GRIDSQL-STAGING-THREEGPPSPEECHCALLS-1290634808555','7B7C0D0BC834B8BD53422AFA94023223' NativeException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server us01-ciqps1-grid12.carrieriq.com:60020 for region GRID-GRIDSQL-STAGING-THREEGPPSPEECHCALLS-1290634808555,7B7C0D0BC834B8BD53422AFA94023223,1290638846310, row '7B7C0D0BC834B8BD53422AFA94023223', but failed after 7 attempts. Exceptions: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to us01-ciqps1-grid12.carrieriq.com/10.202.50.112:60020 after attempts=1 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to us01-ciqps1-grid12.carrieriq.com/10.202.50.112:60020 after attempts=1 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to us01-ciqps1-grid12.carrieriq.com/10.202.50.112:60020 after attempts=1 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to us01-ciqps1-grid12.carrieriq.com/10.202.50.112:60020 after attempts=1 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to us01-ciqps1-grid12.carrieriq.com/10.202.50.112:60020 after attempts=1 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to us01-ciqps1-grid12.carrieriq.com/10.202.50.112:60020 after attempts=1 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to us01-ciqps1-grid12.carrieriq.com/10.202.50.112:60020 after attempts=1 I have collected region server log (where I found occurrences of GRID-GRIDSQL-STAGING-THREEGPPSPEECHCALLS-1290634808555) and master log I can send the zipped tar ball to you upon request. Have a nice holiday.
