Thanks for sharing your experience. Have you set hbase.rpc.timeout > hbase.regionserver.lease.period in your environment ?
Cheers On Thu, Nov 15, 2012 at 7:45 AM, Guillaume Perrot <[email protected]>wrote: > It looks like a "race condition" between the check that the lease is > expired in HRegionServer.next, and the LeaseListener.leaseExpired methods. > If the next call from the client is done before the lease listener do its > job, the scanner still exists in the map but is expired, which causes > LeaseException instead of UnknownScannerException. > Indeed I found this in the logs: > > 2012-11-15 08:45:34,003 ERROR > org.apache.hadoop.hbase.regionserver.HRegionServer: > org.apache.hadoop.hbase.regionserver.LeaseException: lease > '4939065839430667309' does not exist > at > org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231) > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2117) > at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) > > Then > > 2012-11-15 08:48:37,734 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner > 4939065839430667309 lease expired > > The scanner is properly closed way too late! > I just realized there is already a related issue about this: > https://issues.apache.org/jira/browse/HBASE-6856 > > 2012/11/15 Guillaume Perrot <[email protected]> > > > Hi, > > Similar questions have been mailed in the past, but they are not clear > > enough to me, so here is a new one: > > In HBase 0.90.3, when a client scanner was too slow to advance (no actual > > next call within 60s), we had a nice ScannerTimeoutException in client > > (UnknownScannerException in region server). > > With HBase 0.92.1, when I simulate such a slow scanner I still have this > > exception. > > > > But in our production environment, scanners that used to > > produce ScannerTimeoutException in HBase 0.90.3 no longer produce them > > since we upgraded to HBase 0.92.1. > > Instead we now have: > > org.apache.hadoop.hbase.regionserver.LeaseException: > > org.apache.hadoop.hbase.regionserver.LeaseException: lease > > '2770824890045864909' does not exist > > at > org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231) > > at > > > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2117) > > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at > > > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) > > at > > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) > > > > I don't uderstand the workflow leading to this exception instead of a > > ScannerTimeoutException, any local attempt to reproduce this problem > leads > > to a ScannerTimeoutException and not a LeaseException. > > Any idea ? > > Thanks in advance. > > > > > > -- > *Guillaume Perrot > **Software Engineer > *[image: logo] > *Contact info * > [email protected] > www.capptain.com > > *Rennes Office* > Office: +33 2 99 65 69 13 > 10 rue Jean-Marie Duhamel > 35000 Rennes > France *Paris Office* > Office: +33 1 84 06 13 85 > Fax: +33 9 57 72 20 18 > 18 rue Tronchet > 75008 Paris > France > > IMPORTANT NOTICE – UBIKOD and CAPPTAIN are registered trademarks of UBIKOD > S.A.R.L., all copyrights are reserved. The contents of this email and > attachments are confidential and may be subject to legal privilege and/or > protected by copyright. Copying or communicating any part of it to others > is prohibited and may be unlawful. If you are not the intended recipient > you must not use, copy, distribute or rely on this email and should please > return it immediately or notify us by telephone. At present the integrity > of email across the Internet cannot be guaranteed. Therefore UBIKOD > S.A.R.L. will not accept liability for any claims arising as a result of > the use of this medium for transmissions by or to UBIKOD S.A.R.L.. UBIKOD > S.A.R.L. may exercise any of its rights under relevant law, to monitor the > content of all electronic communications. You should therefore be aware > that this communication and any responses might have been monitored, and > may be accessed by UBIKOD S.A.R.L. The views expressed in this document are > that of the individual and may not necessarily constitute or imply its > endorsement or recommendation by UBIKOD S.A.R.L. The content of this > electronic mail may be subject to the confidentiality terms of a > "Non-Disclosure Agreement" (NDA). >
