[
https://issues.apache.org/jira/browse/PHOENIX-5007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16695523#comment-16695523
]
Lars Hofhansl commented on PHOENIX-5007:
----------------------------------------
[~gjacoby] For client side actions we might want to consider increasing the
chunk size (right now its 100 rows or 2MB, whatever is less, right?) Perhaps
1000 rows or 2MB might be better.
For server side mutations (the ones I'm concerned about here) would probably
not benefit. It's already a tight loop on the server.
[~vincentpoon] So it seems HBase misses a coprocessor feature to pass that
allows passing on heart-beating...?
> DELETE FROM times out - even with large HBase timeout setting
> -------------------------------------------------------------
>
> Key: PHOENIX-5007
> URL: https://issues.apache.org/jira/browse/PHOENIX-5007
> Project: Phoenix
> Issue Type: Bug
> Reporter: Lars Hofhansl
> Priority: Major
>
> It seems there is currently no way to make a somewhat large DELETE FROM work
> without timing out. "Large" here includes size equivalent to a single HBase
> region, which means we cannot write a M/R (or Spark) task to delete a lot of
> data.
> This might well be an HBase problem. I'm looking there as well, just keeping
> a record here.
> This is the exception:
> {code:java}
> 0: jdbc:phoenix:localhost> delete from test where v1 < 0.999;
> Error: org.apache.phoenix.exception.PhoenixIOException: Failed after
> attempts=36, exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> (state=08000,code=101)
> org.apache.phoenix.exception.PhoenixIOException:
> org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36,
> exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:144)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1296)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1237)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:50)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:97)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117)
> at
> org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:64)
> at
> org.apache.phoenix.iterate.UngroupedAggregatingResultIterator.next(UngroupedAggregatingResultIterator.java:39)
> at
> org.apache.phoenix.compile.DeleteCompiler$ServerSelectDeleteMutationPlan.execute(DeleteCompiler.java:791)
> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:408)
> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:391)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:390)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:378)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1825)
> at sqlline.Commands.execute(Commands.java:822)
> at sqlline.Commands.sql(Commands.java:732)
> at sqlline.SqlLine.dispatch(SqlLine.java:813)
> at sqlline.SqlLine.begin(SqlLine.java:686)
> at sqlline.SqlLine.start(SqlLine.java:398)
> at sqlline.SqlLine.main(SqlLine.java:291)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36,
> exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:206)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1291)
> ... 19 more
> Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after
> attempts=36, exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:144)
> at
> org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:171)
> at
> org.apache.phoenix.iterate.LookAheadResultIterator$1.advance(LookAheadResultIterator.java:47)
> at
> org.apache.phoenix.iterate.LookAheadResultIterator.init(LookAheadResultIterator.java:59)
> at
> org.apache.phoenix.iterate.LookAheadResultIterator.peek(LookAheadResultIterator.java:73)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:135)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:120)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after
> attempts=36, exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:144)
> at
> org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:156)
> at
> org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:164)
> ... 10 more
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
> after attempts=36, exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:329)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:242)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
> at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:277)
> at
> org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:438)
> at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:312)
> at
> org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:147)
> ... 11 more
> Caused by: java.net.SocketTimeoutException: callTimeout=180000,
> callDuration=180311: Call to lhofhansl-wsl2/127.0.1.1:16201 failed on local
> exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:178)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> ... 3 more
> Caused by: java.io.IOException: Call to lhofhansl-wsl2/127.0.1.1:16201 failed
> on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call
> id=804, waitTime=59689, rpcTimetout=59678
> at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:389)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:94)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:409)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:405)
> at org.apache.hadoop.hbase.ipc.Call.setTimeout(Call.java:96)
> at org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:188)
> at
> org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:625)
> at
> org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:700)
> at
> org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:428)
> ... 1 more
> Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678
> ... 5 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
