[
https://issues.apache.org/jira/browse/HBASE-14822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samarth Jain updated HBASE-14822:
---------------------------------
Attachment: HBASE-14822_98_nextseq.diff
[~lhofhansl] - I tried out the latest on 0.98, and looks like there are some
more issues lurking with lease renewal. I noticed that on the region server
side I was still getting the following message even though I made sure Phoenix
was calling renewLease() for the scanners.
INFO [RS:0;localhost:55383.leaseChecker]
org.apache.hadoop.hbase.regionserver.HRegionServer$ScannerListener(2633):
Scanner 59 lease expired on region
After a bit of digging around, it turns out that the lease renewal is actually
causing the regular scan() to fail and vice-versa. This is because renewLease
ends up also increasing the nextCallSeq member variable in the ScannerCallable
object. There are checks in place in the HRegionServer class that causes an
OutOfOrderScannerNextException to be thrown because the nextSeq didn't match.
See this stacktrace:
org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected
nextCallSeq: 2 But the nextCallSeq got from client: 10; request=scanner_id: 56
number_of_rows: 2 close_scanner: false next_call_seq: 10 renew: false
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3277)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31190)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2149)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:298)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:216)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:58)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:115)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:91)
at
org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:387)
at
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:340)
at
org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:57)
at
org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:112)
at
In this case the number of times renewLease() was called was 8 which also
happens to be the difference between the expected nextCallSeq (2) and the
actual nextCallSeq(10). This error isn't surfaced to the clients though because
the HBase client ends up creating a new scanner altogether behind the scenes.
One possible simple fix (in the attached patch) would be to not increment the
nextCallSeq when renewing lease. FWIW, after this change, I no longer see the
OutOfOrderScannerNextException and INFO message about scanner lease expiration
is also gone.
> Renewing leases of scanners doesn't work
> ----------------------------------------
>
> Key: HBASE-14822
> URL: https://issues.apache.org/jira/browse/HBASE-14822
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.98.14
> Reporter: Samarth Jain
> Assignee: Lars Hofhansl
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.3, 0.98.17, 1.0.4
>
> Attachments: 14822-0.98-v2.txt, 14822-0.98-v3.txt, 14822-0.98.txt,
> 14822-v3-0.98.txt, 14822-v4-0.98.txt, 14822-v4.txt, 14822-v5-0.98.txt,
> 14822-v5-1.3.txt, 14822-v5.txt, 14822.txt, HBASE-14822_98_nextseq.diff
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
