Hi,
I’ve seen this when the HMaster has died on a workstation using the standalone Hadoop. Perhaps you’d see the same behavior if a RegionServer died on a cluster, I don’t know. You may want to check your HBase logs to see if there are any clues there. Given that you’re developing a Trafodion feature, it is possible that something Trafodion did or said to HBase made it blush. J Dave *From:* Eric Owhadi [mailto:[email protected]] *Sent:* Thursday, March 24, 2016 2:32 PM *To:* [email protected] *Subject:* problem with update statistics Hi Trafodioneers, I am trying to create a 10M row table, 1KB each row 10 partitions to test a new feature (parallelScan). But on my dev (so single node), I stumble on update statistics error: update statistics for table t132 on every column sample; *** ERROR[9200] UPDATE STATISTICS for table TRAFODION.SCH.T132 encountered an error (8448) from statement HSCursor::fetchRowset(). *** ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::nextRow returned error HBASE_ACCESS_ERROR(-706). Cause: java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Thu Mar 24 21:01:52 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Call to edevxx.xxxxx.local/10.1.xx.xx:38868 failed on local exception: java.io.EOFException Thu Mar 24 21:02:11 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:02:30 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:02:51 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:03:12 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:03:33 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:04:03 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:04:32 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:05:00 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:05:27 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:06:04 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:06:42 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:07:20 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:08:00 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:08:38 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:09:18 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:09:56 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:10:34 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:11:13 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:11:52 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:12:33 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:13:11 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:13:51 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:14:31 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:15:08 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:15:47 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:16:29 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:17:07 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:17:45 GMT 2016, org.apache.hadoop.hbase.clien'��k *** ERROR[9200] UPDATE STATISTICS for table TRAFODION.SCH.T132 encountered an error (8448) from statement READ_COLS_INTO_MEM. *** ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::nextRow returned error HBASE_ACCESS_ERROR(-706). Cause: java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Thu Mar 24 21:01:52 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Call to xxxxx.xxxxx.local/10.1.xx.xx:38868 failed on local exception: java.io.EOFException Thu Mar 24 21:02:11 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:02:30 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:02:51 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:03:12 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:03:33 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:04:03 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:04:32 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:05:00 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:05:27 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:06:04 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:06:42 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:07:20 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:08:00 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:08:38 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:09:18 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:09:56 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:10:34 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:11:13 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:11:52 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:12:33 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:13:11 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:13:51 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:14:31 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:15:08 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:15:47 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:16:29 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:17:07 GMT 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@7c24d3f1, java.io.IOException: Enable/Disable failed Thu Mar 24 21:17:45 GMT 2016, org.apache.hadoop.hbase.clien'��k The table: create table t132helper (a int not null, primary key(a)); insert into t132helper values(1); create table t132 (k1 int not null, k2 int not null, a int not null, b int not null, c char(1000), primary key (k1,k2)) SALT USING 10 PARTITIONS ON (k1, k2) ATTRIBUTES ALIGNED FORMAT ; upsert using load into t132 select x1000*1000+ x10000*10000 + x100000*100000 + x1000000*1000000, x1+x10*10+x100*100, x1+x10*10+x100*100+ x1000*1000+ x10000*10000 + x100000*100000 + x1000000*1000000, x1+x10*10+x100*100+ x1000*1000+ x10000*10000 + x100000*100000 + x1000000*1000000, 'yo bro' from t132helper transpose 0,1,2,3,4,5,6,7,8,9 as x1 transpose 0,1,2,3,4,5,6,7,8,9 as x10 transpose 0,1,2,3,4,5,6,7,8,9 as x100 transpose 0,1,2,3,4,5,6,7,8,9 as x1000 transpose 0,1,2,3,4,5,6,7,8,9 as x10000 transpose 0,1,2,3,4,5,6,7,8,9 as x100000 transpose 0,1,2,3,4,5,6,7,8,9 as x1000000; update statistics for table t132 on every column sample; is that a known problem? A workaround available? Eric
