Robert Fiser created HBASE-15757:
------------------------------------
Summary: Reverse scan fails with no obvious cause
Key: HBASE-15757
URL: https://issues.apache.org/jira/browse/HBASE-15757
Project: HBase
Issue Type: Bug
Components: Client, Scanners
Affects Versions: 0.98.12
Environment: ubuntu 14.04, amazon cloud; 10 datanodes d2.4xlarge -
16cores, 12x200GB HDD, 122GB RAM
Reporter: Robert Fiser
related issue on stackoverflow:
http://stackoverflow.com/questions/37001169/hbase-reverse-scan-error?noredirect=1#comment61558097_37001169
this works well:
scan = new Scan(startRow, stopRow);
this throws exception sometimes:
scan = new Scan(stopRow, startRow);
scan.setReversed(true);
throwing exception while traffic is at least 100 req/s. there are actually no
timeouts, exception is fired immediately for 1-10% requests
hbase: 0.98.12-hadoop2;
hadoop: 2.7.0;
cluster in AWS, 10 datanodes: d2.4xlarge
I think it's maybe related with this issue but I'm not using any filters
http://apache-hbase.679495.n3.nabble.com/Exception-during-a-reverse-scan-with-filter-td4069721.html
java.lang.RuntimeException:
org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of
OutOfOrderScannerNextException: was there a rpc timeout?
at
org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:94)
at
com.socialbakers.broker.client.hbase.htable.AbstractHtableListScanner.scanToList(AbstractHtableListScanner.java:30)
at
com.socialbakers.broker.client.hbase.htable.AbstractHtableListSingleScanner.invokeOperation(AbstractHtableListSingleScanner.java:23)
at
com.socialbakers.broker.client.hbase.htable.AbstractHtableListSingleScanner.invokeOperation(AbstractHtableListSingleScanner.java:11)
at
com.socialbakers.broker.client.hbase.AbstractHbaseApi.endPointMethod(AbstractHbaseApi.java:40)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
com.socialbakers.broker.client.Route.invoke(Route.java:241)
at
com.socialbakers.broker.client.handler.EndpointHandler.invoke(EndpointHandler.java:173)
at
com.socialbakers.broker.client.handler.EndpointHandler.process(EndpointHandler.java:69)
at
com.thetransactioncompany.jsonrpc2.server.Dispatcher.process(Dispatcher.java:196)
at
com.socialbakers.broker.client.RejectableRunnable.run(RejectableRunnable.java:38)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by:
org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of
OutOfOrderScannerNextException: was there a rpc timeout?
at
org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:430)
at
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:333)
at
org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:91)
... 15 more
Caused by:
org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected
nextCallSeq: 2 But the nextCallSeq got from client: 1; request=scanner_id:
27700695 number_of_rows: 100 close_scanner: false next_call_seq: 1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3231)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:30946)
at
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093)
at
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
at
sun.reflect.GeneratedConstructorAccessor16.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:287)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:214)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:58)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:115)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:91)
at
org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:375)
... 17 more
Caused by:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException):
org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected
nextCallSeq: 2 But the nextCallSeq got from client: 1; request=scanner_id:
27700695 number_of_rows: 100 close_scanner: false next_call_seq: 1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3231)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:30946)
at
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093)
at
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
at
org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:31392)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:173)
... 21 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
