[
https://issues.apache.org/jira/browse/PHOENIX-5007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683054#comment-16683054
]
Lars Hofhansl commented on PHOENIX-5007:
----------------------------------------
In fact I think we should (somehow) chunk this better from the client side.
Right now we schedule region (or stats) size delete ranges (assuming no global
indexes), and then on the server we issue 100-size (by default) batches.
> DELETE FROM times out - even with large HBase timeout setting
> -------------------------------------------------------------
>
> Key: PHOENIX-5007
> URL: https://issues.apache.org/jira/browse/PHOENIX-5007
> Project: Phoenix
> Issue Type: Bug
> Reporter: Lars Hofhansl
> Priority: Major
>
> It seems there is currently no way to make a somewhat large DELETE FROM work
> without timing out. "Large" here includes size equivalent to a single HBase
> region, which means we cannot write a M/R (or Spark) task to delete a lot of
> data.
> This might well be an HBase problem. I'm looking there as well, just keeping
> a record here.
> This is the exception:
> {code:java}
> 0: jdbc:phoenix:localhost> delete from test where v1 < 0.999;
> Error: org.apache.phoenix.exception.PhoenixIOException: Failed after
> attempts=36, exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> (state=08000,code=101)
> org.apache.phoenix.exception.PhoenixIOException:
> org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36,
> exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:144)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1296)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1237)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:50)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:97)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117)
> at
> org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:64)
> at
> org.apache.phoenix.iterate.UngroupedAggregatingResultIterator.next(UngroupedAggregatingResultIterator.java:39)
> at
> org.apache.phoenix.compile.DeleteCompiler$ServerSelectDeleteMutationPlan.execute(DeleteCompiler.java:791)
> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:408)
> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:391)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:390)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:378)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1825)
> at sqlline.Commands.execute(Commands.java:822)
> at sqlline.Commands.sql(Commands.java:732)
> at sqlline.SqlLine.dispatch(SqlLine.java:813)
> at sqlline.SqlLine.begin(SqlLine.java:686)
> at sqlline.SqlLine.start(SqlLine.java:398)
> at sqlline.SqlLine.main(SqlLine.java:291)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36,
> exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:206)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1291)
> ... 19 more
> Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after
> attempts=36, exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:144)
> at
> org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:171)
> at
> org.apache.phoenix.iterate.LookAheadResultIterator$1.advance(LookAheadResultIterator.java:47)
> at
> org.apache.phoenix.iterate.LookAheadResultIterator.init(LookAheadResultIterator.java:59)
> at
> org.apache.phoenix.iterate.LookAheadResultIterator.peek(LookAheadResultIterator.java:73)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:135)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:120)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after
> attempts=36, exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:144)
> at
> org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:156)
> at
> org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:164)
> ... 10 more
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
> after attempts=36, exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:329)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:242)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
> at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:277)
> at
> org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:438)
> at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:312)
> at
> org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:147)
> ... 11 more
> Caused by: java.net.SocketTimeoutException: callTimeout=180000,
> callDuration=180311: Call to lhofhansl-wsl2/127.0.1.1:16201 failed on local
> exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:178)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> ... 3 more
> Caused by: java.io.IOException: Call to lhofhansl-wsl2/127.0.1.1:16201 failed
> on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call
> id=804, waitTime=59689, rpcTimetout=59678
> at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:389)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:94)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:409)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:405)
> at org.apache.hadoop.hbase.ipc.Call.setTimeout(Call.java:96)
> at org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:188)
> at
> org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:625)
> at
> org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:700)
> at
> org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:428)
> ... 1 more
> Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678
> ... 5 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
