It looks like a "race condition" between the check that the lease is expired in HRegionServer.next, and the LeaseListener.leaseExpired methods. If the next call from the client is done before the lease listener do its job, the scanner still exists in the map but is expired, which causes LeaseException instead of UnknownScannerException. Indeed I found this in the logs:
2012-11-15 08:45:34,003 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: org.apache.hadoop.hbase.regionserver.LeaseException: lease '4939065839430667309' does not exist at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2117) at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) Then 2012-11-15 08:48:37,734 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 4939065839430667309 lease expired The scanner is properly closed way too late! I just realized there is already a related issue about this: https://issues.apache.org/jira/browse/HBASE-6856 2012/11/15 Guillaume Perrot <[email protected]> > Hi, > Similar questions have been mailed in the past, but they are not clear > enough to me, so here is a new one: > In HBase 0.90.3, when a client scanner was too slow to advance (no actual > next call within 60s), we had a nice ScannerTimeoutException in client > (UnknownScannerException in region server). > With HBase 0.92.1, when I simulate such a slow scanner I still have this > exception. > > But in our production environment, scanners that used to > produce ScannerTimeoutException in HBase 0.90.3 no longer produce them > since we upgraded to HBase 0.92.1. > Instead we now have: > org.apache.hadoop.hbase.regionserver.LeaseException: > org.apache.hadoop.hbase.regionserver.LeaseException: lease > '2770824890045864909' does not exist > at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2117) > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) > > I don't uderstand the workflow leading to this exception instead of a > ScannerTimeoutException, any local attempt to reproduce this problem leads > to a ScannerTimeoutException and not a LeaseException. > Any idea ? > Thanks in advance. > -- *Guillaume Perrot **Software Engineer *[image: logo] *Contact info * [email protected] www.capptain.com *Rennes Office* Office: +33 2 99 65 69 13 10 rue Jean-Marie Duhamel 35000 Rennes France *Paris Office* Office: +33 1 84 06 13 85 Fax: +33 9 57 72 20 18 18 rue Tronchet 75008 Paris France IMPORTANT NOTICE – UBIKOD and CAPPTAIN are registered trademarks of UBIKOD S.A.R.L., all copyrights are reserved. The contents of this email and attachments are confidential and may be subject to legal privilege and/or protected by copyright. Copying or communicating any part of it to others is prohibited and may be unlawful. If you are not the intended recipient you must not use, copy, distribute or rely on this email and should please return it immediately or notify us by telephone. At present the integrity of email across the Internet cannot be guaranteed. Therefore UBIKOD S.A.R.L. will not accept liability for any claims arising as a result of the use of this medium for transmissions by or to UBIKOD S.A.R.L.. UBIKOD S.A.R.L. may exercise any of its rights under relevant law, to monitor the content of all electronic communications. You should therefore be aware that this communication and any responses might have been monitored, and may be accessed by UBIKOD S.A.R.L. The views expressed in this document are that of the individual and may not necessarily constitute or imply its endorsement or recommendation by UBIKOD S.A.R.L. The content of this electronic mail may be subject to the confidentiality terms of a "Non-Disclosure Agreement" (NDA).
