[
https://issues.apache.org/jira/browse/TRAFODION-2272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15563519#comment-15563519
]
David Wayne Birdsall commented on TRAFODION-2272:
-------------------------------------------------
The UPDATE STATISTICS command is failing during the sample scan. The
SocketTimeoutException is happening because on this particular cluster the
hbase.client.scanner.timeout.period setting is short, just 60000 (or 1 minute).
The distribution in question is CDH 5.4.5, running HBase 1.0, which lacks the
heartbeat protocol.
When we changed the hbase.client.scanner.timeout.period value in the
hbase-site.xml file to 3600000 (1 hour), the problem went away.
Unfortunately, Cloudera Manager does not allow one to set this particular
property, so we had to do it manually. And that is doubly unfortunate because
when someone changes another setting and does a deploy, it overwrites any
manual change we make.
Therefore a short-term solution that sets this property from the Trafodion
client code is in order.
> UPDATE STATS on 5.5 billion row UNALIGNED table fails with HBase error 706,
> Retries Exhausted
> ---------------------------------------------------------------------------------------------
>
> Key: TRAFODION-2272
> URL: https://issues.apache.org/jira/browse/TRAFODION-2272
> Project: Apache Trafodion
> Issue Type: Bug
> Components: sql-cmp
> Affects Versions: 2.1-incubating
> Environment: Cluster, CDH 5.4 distribution
> Reporter: David Wayne Birdsall
> Assignee: David Wayne Birdsall
>
> Symptom:
> update statistics for table trafodion.javabench.oe_orderline_18432 on every
> column, (OL_W_ID, OL_I_ID), (OL_D_ID, OL_W_ID), (OL_D_ID, OL_I_ID) sample;
> command fails with the following:
> ** ERROR[9200] UPDATE STATISTICS for table
> TRAFODION.JAVABENCH.OE_ORDERLINE_18432 encountered an error (8448) from
> statement HSCursor::fetchRowset(). [2016-09-20 01:15:13]
> *** ERROR[8448] Unable to access Hbase interface. Call to
> ExpHbaseInterface::nextRow returned error HBASE_ACCESS_ERROR(-706). Cause:
> java.util.concurrent.ExecutionException:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=4, exceptions:
> Tue Sep 20 01:13:06 UTC 2016, null, java.net.SocketTimeoutException:
> callTimeout=60000, callDuration=60305: row '?. [2016-09-20 01:15:13]
> *** ERROR[9200] UPDATE STATISTICS for table
> TRAFODION.JAVABENCH.OE_ORDERLINE_18432 encountered an error (8448) from
> statement READ_COLS_INTO_MEM. [2016-09-20 01:15:13]
> *** ERROR[8448] Unable to access Hbase interface. Call to
> ExpHbaseInterface::nextRow returned error HBASE_ACCESS_ERROR(-706). Cause:
> java.util.concurrent.ExecutionException:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=4, exceptions:
> Tue Sep 20 01:13:06 UTC 2016, null, java.net.SocketTimeoutException:
> callTimeout=60000, callDuration=60305: row '?. [2016-09-20 01:15:13]
> The table in question was unaligned format, having 5.5 billion rows. The DDL
> for the table is below:
> CREATE TABLE TRAFODION.JAVABENCH.OE_ORDERLINE_18432
> (
> OL_O_ID INT NO DEFAULT NOT NULL NOT DROPPABLE
> SERIALIZED
> , OL_D_ID INT NO DEFAULT NOT NULL NOT DROPPABLE
> SERIALIZED
> , OL_W_ID INT NO DEFAULT NOT NULL NOT DROPPABLE
> SERIALIZED
> , OL_NUMBER INT NO DEFAULT NOT NULL NOT DROPPABLE
> SERIALIZED
> , OL_I_ID INT NO DEFAULT NOT NULL NOT DROPPABLE
> SERIALIZED
> , OL_SUPPLY_W_ID INT NO DEFAULT NOT NULL NOT DROPPABLE
> SERIALIZED
> , OL_DELIVERY_D TIMESTAMP(6) DEFAULT NULL NOT SERIALIZED
> , OL_QUANTITY NUMERIC(2, 0) NO DEFAULT NOT NULL NOT
> DROPPABLE SERIALIZED
> , OL_AMOUNT NUMERIC(6, 2) NO DEFAULT NOT NULL NOT
> DROPPABLE SERIALIZED
> , OL_DIST_INFO CHAR(24) CHARACTER SET ISO88591 COLLATE
> DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
> , PRIMARY KEY (OL_W_ID ASC, OL_D_ID ASC, OL_O_ID ASC, OL_NUMBER ASC)
> )
> SALT USING 64 PARTITIONS
> ON (OL_W_ID)
> ;
> The cluster in question had 8 nodes. Memory was not an issue during the test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
