[
https://issues.apache.org/jira/browse/PHOENIX-5007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701093#comment-16701093
]
Lars Hofhansl commented on PHOENIX-5007:
----------------------------------------
Wait... Sorry for the confusing. I had server side mutations turned off in my
previous test.
So it looks like with forcing the deletes to be handled client side and batch
size of 1000 this delete (16m rows on the single server) finishes in about 30s,
while with server side deletes is times out after 180s (I had one run where it
finished in 174s).
Perhaps client side deletes should be the default? We might want to remove the
code for server side mutations all-together!
> DELETE FROM times out - even with large HBase timeout setting
> -------------------------------------------------------------
>
> Key: PHOENIX-5007
> URL: https://issues.apache.org/jira/browse/PHOENIX-5007
> Project: Phoenix
> Issue Type: Bug
> Reporter: Lars Hofhansl
> Priority: Major
>
> It seems there is currently no way to make a somewhat large DELETE FROM work
> without timing out. "Large" here includes size equivalent to a single HBase
> region, which means we cannot write a M/R (or Spark) task to delete a lot of
> data.
> This might well be an HBase problem. I'm looking there as well, just keeping
> a record here.
> This is the exception:
> {code:java}
> 0: jdbc:phoenix:localhost> delete from test where v1 < 0.999;
> Error: org.apache.phoenix.exception.PhoenixIOException: Failed after
> attempts=36, exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> (state=08000,code=101)
> org.apache.phoenix.exception.PhoenixIOException:
> org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36,
> exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:144)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1296)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1237)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:50)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:97)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117)
> at
> org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:64)
> at
> org.apache.phoenix.iterate.UngroupedAggregatingResultIterator.next(UngroupedAggregatingResultIterator.java:39)
> at
> org.apache.phoenix.compile.DeleteCompiler$ServerSelectDeleteMutationPlan.execute(DeleteCompiler.java:791)
> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:408)
> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:391)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:390)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:378)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1825)
> at sqlline.Commands.execute(Commands.java:822)
> at sqlline.Commands.sql(Commands.java:732)
> at sqlline.SqlLine.dispatch(SqlLine.java:813)
> at sqlline.SqlLine.begin(SqlLine.java:686)
> at sqlline.SqlLine.start(SqlLine.java:398)
> at sqlline.SqlLine.main(SqlLine.java:291)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36,
> exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:206)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1291)
> ... 19 more
> Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after
> attempts=36, exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:144)
> at
> org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:171)
> at
> org.apache.phoenix.iterate.LookAheadResultIterator$1.advance(LookAheadResultIterator.java:47)
> at
> org.apache.phoenix.iterate.LookAheadResultIterator.init(LookAheadResultIterator.java:59)
> at
> org.apache.phoenix.iterate.LookAheadResultIterator.peek(LookAheadResultIterator.java:73)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:135)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:120)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after
> attempts=36, exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:144)
> at
> org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:156)
> at
> org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:164)
> ... 10 more
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
> after attempts=36, exceptions:
> Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException:
> callTimeout=180000, callDuration=180311: Call to
> lhofhansl-wsl2/127.0.1.1:16201 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:329)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:242)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
> at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:277)
> at
> org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:438)
> at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:312)
> at
> org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:147)
> ... 11 more
> Caused by: java.net.SocketTimeoutException: callTimeout=180000,
> callDuration=180311: Call to lhofhansl-wsl2/127.0.1.1:16201 failed on local
> exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at
> region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5.,
> hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:178)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> ... 3 more
> Caused by: java.io.IOException: Call to lhofhansl-wsl2/127.0.1.1:16201 failed
> on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call
> id=804, waitTime=59689, rpcTimetout=59678
> at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:389)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:94)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:409)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:405)
> at org.apache.hadoop.hbase.ipc.Call.setTimeout(Call.java:96)
> at org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:188)
> at
> org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:625)
> at
> org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:700)
> at
> org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:428)
> ... 1 more
> Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804,
> waitTime=59689, rpcTimetout=59678
> ... 5 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
