[
https://issues.apache.org/jira/browse/HBASE-14180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell resolved HBASE-14180.
------------------------------------
Resolution: Invalid
Please write in to [email protected] for help troubleshooting issues. This
is the project dev tracker. Thanks!
> Change timeout - SocketTimeoutException because of callTimeout
> --------------------------------------------------------------
>
> Key: HBASE-14180
> URL: https://issues.apache.org/jira/browse/HBASE-14180
> Project: HBase
> Issue Type: Bug
> Components: hbase, regionserver, rpc, Zookeeper
> Affects Versions: 1.1.1
> Environment: Hadoop with Ambari 2.1.0
> HBase 1.1.1.2.3
> HDFS 2.7.1.2.3
> Zookeeper 3.4.6.2.3
> Phoenix 4.4.0.2.3
> Reporter: Adrià V.
>
> HBase keeps throwing a timeout exception I have tryed every configuration I
> could think about to increase it.
> Partial stacktrace:
> {quote}
> Caused by: java.io.IOException: Call to
> hdp-w-1.c.dks-hadoop.internal/10.240.2.235:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=43, waitTime=60001,
> operationTimeout=60000 expired.
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1242)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1210)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:213)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:369)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:343)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
> ... 4 more
> Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=43,
> waitTime=60001, operationTimeout=60000 expired.
> at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1184)
> ... 13 more
> {quote}
> I've tryed editing config files and also setting config in Ambari with the
> next keys to increase the timeout with no success:
> - hbase.rpc.timeout
> - dfs.socket.timeout
> - dfs.client.socket-timeout
> - zookeeper.session.timeout
> Also the Phoenix properties, but I think it's mostly an HBase issue:
> - phoenix.query.timeoutMs
> - phoenix.query.keepAliveMs
> Full stack trace:
> {quote}
> Error: Encountered exception in sub plan [0] execution. (state=,code=0)
> java.sql.SQLException: Encountered exception in sub plan [0] execution.
> at
> org.apache.phoenix.execute.HashJoinPlan.iterator(HashJoinPlan.java:157)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:251)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:241)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:240)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1250)
> at sqlline.Commands.execute(Commands.java:822)
> at sqlline.Commands.sql(Commands.java:732)
> at sqlline.SqlLine.dispatch(SqlLine.java:808)
> at sqlline.SqlLine.begin(SqlLine.java:681)
> at sqlline.SqlLine.start(SqlLine.java:398)
> at sqlline.SqlLine.main(SqlLine.java:292)
> Caused by: org.apache.phoenix.exception.PhoenixIOException:
> org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36,
> exceptions:
> Mon Aug 03 16:47:06 UTC 2015, null, java.net.SocketTimeoutException:
> callTimeout=60000, callDuration=60303: row '' on table 'hive_post_topics' at
> region=hive_post_topics,,1438084107396.cdbdc246ff0b7dfed31d481e0bccd2b5.,
> hostname=hdp-w-1.c.dks-hadoop.internal,16020,1438619912282, seqNum=45322
> at
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:542)
> at
> org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176)
> at
> org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91)
> at
> org.apache.phoenix.join.HashCacheClient.serialize(HashCacheClient.java:106)
> at
> org.apache.phoenix.join.HashCacheClient.addHashCache(HashCacheClient.java:82)
> at
> org.apache.phoenix.execute.HashJoinPlan$HashSubPlan.execute(HashJoinPlan.java:339)
> at
> org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:136)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:172)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36,
> exceptions:
> Mon Aug 03 16:47:06 UTC 2015, null, java.net.SocketTimeoutException:
> callTimeout=60000, callDuration=60303: row '' on table 'hive_post_topics' at
> region=hive_post_topics,,1438084107396.cdbdc246ff0b7dfed31d481e0bccd2b5.,
> hostname=hdp-w-1.c.dks-hadoop.internal,16020,1438619912282, seqNum=45322
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:202)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:538)
> ... 11 more
> Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after
> attempts=36, exceptions:
> Mon Aug 03 16:47:06 UTC 2015, null, java.net.SocketTimeoutException:
> callTimeout=60000, callDuration=60303: row '' on table 'hive_post_topics' at
> region=hive_post_topics,,1438084107396.cdbdc246ff0b7dfed31d481e0bccd2b5.,
> hostname=hdp-w-1.c.dks-hadoop.internal,16020,1438619912282, seqNum=45322
> at
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108)
> at
> org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:56)
> at
> org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:104)
> at
> org.apache.phoenix.iterate.LookAheadResultIterator$1.advance(LookAheadResultIterator.java:47)
> at
> org.apache.phoenix.iterate.LookAheadResultIterator.init(LookAheadResultIterator.java:59)
> at
> org.apache.phoenix.iterate.LookAheadResultIterator.peek(LookAheadResultIterator.java:73)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:97)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:85)
> ... 5 more
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
> after attempts=36, exceptions:
> Mon Aug 03 16:47:06 UTC 2015, null, java.net.SocketTimeoutException:
> callTimeout=60000, callDuration=60303: row '' on table 'hive_post_topics' at
> region=hive_post_topics,,1438084107396.cdbdc246ff0b7dfed31d481e0bccd2b5.,
> hostname=hdp-w-1.c.dks-hadoop.internal,16020,1438619912282, seqNum=45322
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:271)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:223)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at
> org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:403)
> at
> org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:364)
> at
> org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:50)
> ... 11 more
> Caused by: java.net.SocketTimeoutException: callTimeout=60000,
> callDuration=60303: row '' on table 'hive_post_topics' at
> region=hive_post_topics,,1438084107396.cdbdc246ff0b7dfed31d481e0bccd2b5.,
> hostname=hdp-w-1.c.dks-hadoop.internal,16020,1438619912282, seqNum=45322
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64)
> ... 3 more
> Caused by: java.io.IOException: Call to
> hdp-w-1.c.dks-hadoop.internal/10.240.2.235:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=43, waitTime=60001,
> operationTimeout=60000 expired.
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1242)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1210)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:213)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:369)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:343)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
> ... 4 more
> Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=43,
> waitTime=60001, operationTimeout=60000 expired.
> at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1184)
> ... 13 more
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
