[
https://issues.apache.org/jira/browse/HBASE-15757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274083#comment-15274083
]
Anoop Sam John commented on HBASE-15757:
----------------------------------------
bq.After reducing the start and stop row getting results is pretty fast.
Means when u have less rows to scan, there is no issue.. So it is because of
time outs only. Another comment u said abt increasing the scan caching. That
is not correct in this respect. U should reduce it.. Every RPC will try to
fetch more rows.. And reverse scan is not same performing as the normal forward
scan. It will take more time and when u have more rows to be scanned in one
RPC, it is sure that the time out happening at client side and it gives a
retry. When retry comes, at server, already the old scan request moved the row
position and so we throw seqNo out of sync exception. Either you have to
increase the timeout or decrease the #rows to scan in one RPC.. You dont have
to reduce the range. You have to reduce the scan caching value,
> Reverse scan fails with no obvious cause
> ----------------------------------------
>
> Key: HBASE-15757
> URL: https://issues.apache.org/jira/browse/HBASE-15757
> Project: HBase
> Issue Type: Bug
> Components: Client, Scanners
> Affects Versions: 0.98.12
> Environment: ubuntu 14.04, amazon cloud; 10 datanodes d2.4xlarge -
> 16cores, 12x200GB HDD, 122GB RAM
> Reporter: Robert Fiser
>
> related issue on stackoverflow:
> http://stackoverflow.com/questions/37001169/hbase-reverse-scan-error?noredirect=1#comment61558097_37001169
> this works well:
> scan = new Scan(startRow, stopRow);
> this throws exception sometimes:
> scan = new Scan(stopRow, startRow);
> scan.setReversed(true);
> throwing exception while traffic is at least 100 req/s. there are actually no
> timeouts, exception is fired immediately for 1-10% requests
> hbase: 0.98.12-hadoop2;
> hadoop: 2.7.0;
> cluster in AWS, 10 datanodes: d2.4xlarge
> I think it's maybe related with this issue but I'm not using any filters
> http://apache-hbase.679495.n3.nabble.com/Exception-during-a-reverse-scan-with-filter-td4069721.html
> java.lang.RuntimeException:
> org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of
> OutOfOrderScannerNextException: was there a rpc timeout?
> at
> org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:94)
> at
> com.socialbakers.broker.client.hbase.htable.AbstractHtableListScanner.scanToList(AbstractHtableListScanner.java:30)
> at
> com.socialbakers.broker.client.hbase.htable.AbstractHtableListSingleScanner.invokeOperation(AbstractHtableListSingleScanner.java:23)
> at
> com.socialbakers.broker.client.hbase.htable.AbstractHtableListSingleScanner.invokeOperation(AbstractHtableListSingleScanner.java:11)
> at
> com.socialbakers.broker.client.hbase.AbstractHbaseApi.endPointMethod(AbstractHbaseApi.java:40)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
> Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> com.socialbakers.broker.client.Route.invoke(Route.java:241)
> at
> com.socialbakers.broker.client.handler.EndpointHandler.invoke(EndpointHandler.java:173)
> at
> com.socialbakers.broker.client.handler.EndpointHandler.process(EndpointHandler.java:69)
> at
> com.thetransactioncompany.jsonrpc2.server.Dispatcher.process(Dispatcher.java:196)
> at
> com.socialbakers.broker.client.RejectableRunnable.run(RejectableRunnable.java:38)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by:
> org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of
> OutOfOrderScannerNextException: was there a rpc timeout?
> at
> org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:430)
> at
> org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:333)
> at
> org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:91)
> ... 15 more
> Caused by:
> org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
> org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected
> nextCallSeq: 2 But the nextCallSeq got from client: 1; request=scanner_id:
> 27700695 number_of_rows: 100 close_scanner: false next_call_seq: 1
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3231)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:30946)
> at
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093)
> at
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> at
> sun.reflect.GeneratedConstructorAccessor16.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at
> java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:287)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:214)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:58)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:115)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:91)
> at
> org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:375)
> ... 17 more
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException):
> org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected
> nextCallSeq: 2 But the nextCallSeq got from client: 1; request=scanner_id:
> 27700695 number_of_rows: 100 close_scanner: false next_call_seq: 1
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3231)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:30946)
> at
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093)
> at
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:31392)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:173)
> ... 21 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
