[
https://issues.apache.org/jira/browse/PHOENIX-5007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16682499#comment-16682499
]
Lars Hofhansl commented on PHOENIX-5007:
----------------------------------------
Thanks [[email protected]].
It's also doing the right thing: callTimeout=180000, callDuration=180311 (i.e.
it waited for the 3 minutes)
I also ran this through the profiler and nothing undue showed up, so it's
probably just slow, because it needs to place all those delete markers.
Not sure how much the batch-size settings would help in this case (which is a
tight server-side loop anyway), but I'll try that.
> DELETE FROM times out - even with large HBase timeout setting
> -------------------------------------------------------------
>
> Key: PHOENIX-5007
> URL: https://issues.apache.org/jira/browse/PHOENIX-5007
> Project: Phoenix
> Issue Type: Bug
> Reporter: Lars Hofhansl
> Priority: Major
>
> It seems there is currently no way to make a somewhat large DELETE FROM work
> without timing out. "Large" here includes size equivalent to a single HBase
> region, which means we cannot write a M/R (or Spark) task to delete a lot of
> data.
> This might well be an HBase problem. I'm looking there as well, just keeping
> a record here.
> This is the exception:
> {code:java}
> 0: jdbc:phoenix:localhost> delete from test where v1 < 0.999;
> Error: org.apache.phoenix.exception.PhoenixIOException: Failed after
> attempts=36, exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> (state=08000,code=101)
> org.apache.phoenix.exception.PhoenixIOException:
> org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36,
> exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:144)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1296)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1237)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:50)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:97)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117)
> at
> org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:64)
> at
> org.apache.phoenix.iterate.UngroupedAggregatingResultIterator.next(UngroupedAggregatingResultIterator.java:39)
> at
> org.apache.phoenix.compile.DeleteCompiler$ServerSelectDeleteMutationPlan.execute(DeleteCompiler.java:791)
> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:408)
> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:391)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:390)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:378)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1825)
> at sqlline.Commands.execute(Commands.java:822)
> at sqlline.Commands.sql(Commands.java:732)
> at sqlline.SqlLine.dispatch(SqlLine.java:813)
> at sqlline.SqlLine.begin(SqlLine.java:686)
> at sqlline.SqlLine.start(SqlLine.java:398)
> at sqlline.SqlLine.main(SqlLine.java:291)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36,
> exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:206)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1291)
> ... 19 more
> Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after
> attempts=36, exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:144)
> at
> org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:171)
> at
> org.apache.phoenix.iterate.LookAheadResultIterator$1.advance(LookAheadResultIterator.java:47)
> at
> org.apache.phoenix.iterate.LookAheadResultIterator.init(LookAheadResultIterator.java:59)
> at
> org.apache.phoenix.iterate.LookAheadResultIterator.peek(LookAheadResultIterator.java:73)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:135)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:120)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after
> attempts=36, exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:144)
> at
> org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:156)
> at
> org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:164)
> ... 10 more
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
> after attempts=36, exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:329)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:242)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
> at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:277)
> at
> org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:438)
> at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:312)
> at
> org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:147)
> ... 11 more
> Caused by: java.net.SocketTimeoutException: callTimeout=180000,
> callDuration=180311: Call to lhofhansl-wsl2/127.0.1.1:16201 failed on local
> exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:178)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> ... 3 more
> Caused by: java.io.IOException: Call to lhofhansl-wsl2/127.0.1.1:16201 failed
> on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call
> id=804, waitTime=59689, rpcTimetout=59678
> at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:389)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:94)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:409)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:405)
> at org.apache.hadoop.hbase.ipc.Call.setTimeout(Call.java:96)
> at org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:188)
> at
> org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:625)
> at
> org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:700)
> at
> org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:428)
> ... 1 more
> Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678
> ... 5 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
