sorry for the delay. I didn't get the lease concept here whether it is specific to hbase or like lease in hadoop?
On Thu, Dec 22, 2016 at 3:13 PM, Phil Yang <ud1...@gmail.com> wrote: > https://github.com/apache/hbase/blob/rel/1.1.1/hbase- > server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices. > java#L2491 > > There is a TTL for scanners at server, to prevent client don't close the > scanners and they leak. The TTL is configured by > hbase.client.scanner.timeout.period at server and refreshed when a scan > RPC > request comes . The TTLs of all scanners are managed by Lease. Your error > happens when server closes a scanner but in Lease it is already expired. So > I think you can try to increase hbase.client.scanner.timeout.period at > server or decrease your hbase.client.scanner.timeout.period at client to > prevent the scanner expired before its scanning done. > hbase.client.scanner.timeout.period is used both at client and server, may > be different if you change one of sides. > > BTW, I still suggest that you can upgrade your cluster and client. 1.1.1 > has some data-loss bugs on scanning. > > Thanks, > Phil > > > 2016-12-22 17:26 GMT+08:00 Rajeshkumar J <rajeshkumarit8...@gmail.com>: > > > can you please explain what is the cause of this lease exception and is > > there any solve this in current version > > > > Thanks > > > > On Thu, Dec 22, 2016 at 2:54 PM, Phil Yang <ud1...@gmail.com> wrote: > > > > > In fact at client the rpc timeout of scan request is also > > > hbase.client.scanner.timeout.period which replaces the > > > deprecated hbase.regionserver.lease.period. > > > > > > Your code that throws LeaseException has been removed by HBASE-16604, > > maybe > > > you can try to upgrade your cluster to 1.1.7? Your client can also > > upgrade > > > to 1.1.7 which will ignore UnknowScannerException and retry when the > > lease > > > is expired at server. > > > > > > Thanks, > > > Phil > > > > > > > > > 2016-12-22 16:51 GMT+08:00 Rajeshkumar J <rajeshkumarit8...@gmail.com > >: > > > > > > > Also there is a solution what i have found from hbase user guide that > > > > hbase.rpc.timeout must be greater than hbase.client.scanner.timeout. > > > > period. > > > > How these two properties plays a part in the above exception. Please > > can > > > > anyone explain? > > > > > > > > On Wed, Dec 21, 2016 at 9:39 PM, Rajeshkumar J < > > > > rajeshkumarit8...@gmail.com> > > > > wrote: > > > > > > > > > I am using hbase version 1.1.1 > > > > > Also I didn't understand something here. Whenever a scanner.next() > is > > > > > called it needs to return rows(based on caching value) within > leasing > > > > > period or else scanner client will be closed eventually throwing > this > > > > > exception. Correct me as I didn't get the clear understanding of > this > > > > issue > > > > > > > > > > On Wed, Dec 21, 2016 at 7:31 PM, Ted Yu <yuzhih...@gmail.com> > wrote: > > > > > > > > > >> Which hbase release are you using ? > > > > >> > > > > >> There is heartbeat support when scanning. > > > > >> Looks like the version you use doesn't have this support. > > > > >> > > > > >> Cheers > > > > >> > > > > >> > On Dec 21, 2016, at 4:02 AM, Rajeshkumar J < > > > > rajeshkumarit8...@gmail.com> > > > > >> wrote: > > > > >> > > > > > >> > Hi, > > > > >> > > > > > >> > Thanks for the reply. I have properties as below > > > > >> > > > > > >> > <property> > > > > >> > <name>hbase.regionserver.lease.period</name> > > > > >> > <value>900000</value> > > > > >> > </property> > > > > >> > <property> > > > > >> > <name>hbase.rpc.timeout</name> > > > > >> > <value>900000>/value> > > > > >> > </property> > > > > >> > > > > > >> > > > > > >> > Correct me If I am wrong. > > > > >> > > > > > >> > I know hbase.regionserver.lease.period, which says how long a > > > scanner > > > > >> > lives between calls to scanner.next(). > > > > >> > > > > > >> > As far as I understand when scanner.next() is called it will > fetch > > > no > > > > >> > of rows as in *hbase.client.scanner.caching. *When this fetching > > > > >> > process takes more than lease period it will close the scanner > > > object. > > > > >> > so this exception occuring? > > > > >> > > > > > >> > > > > > >> > Thanks, > > > > >> > > > > > >> > Rajeshkumar J > > > > >> > > > > > >> > > > > > >> > > > > > >> > On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin < > > > > >> richardstar...@outlook.com > > > > >> >> wrote: > > > > >> > > > > > >> >> It means your lease on a region server has expired during a > call > > to > > > > >> >> resultscanner.next(). This happens on a slow call to next(). > You > > > can > > > > >> either > > > > >> >> embrace it or "fix" it by making sure hbase.rpc.timeout exceeds > > > > >> >> hbase.regionserver.lease.period. > > > > >> >> > > > > >> >> https://richardstartin.com > > > > >> >> > > > > >> >> On 21 Dec 2016, at 11:30, Rajeshkumar J < > > > rajeshkumarit8...@gmail.com > > > > < > > > > >> >> mailto:rajeshkumarit8...@gmail.com>> wrote: > > > > >> >> > > > > >> >> Hi, > > > > >> >> > > > > >> >> I have faced below issue in our production cluster > > > > >> >> > > > > >> >> org.apache.hadoop.hbase.regionserver.LeaseException: > > > > >> >> org.apache.hadoop.hbase.regionserver.LeaseException: lease > > > '166881' > > > > >> does > > > > >> >> not exist > > > > >> >> at org.apache.hadoop.hbase.regionserver.Leases. > > > > >> >> removeLease(Leases.java:221) > > > > >> >> at org.apache.hadoop.hbase.regionserver.Leases. > > > > >> >> cancelLease(Leases.java:206) > > > > >> >> at > > > > >> >> org.apache.hadoop.hbase.regionserver.RSRpcServices. > > > > >> >> scan(RSRpcServices.java:2491) > > > > >> >> at > > > > >> >> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ > > > > >> ClientService$2. > > > > >> >> callBlockingMethod(ClientProtos.java:32205) > > > > >> >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java: > > 2114) > > > > >> >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner. > > java:101) > > > > >> >> at > > > > >> >> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExec > > > > >> utor.java:130) > > > > >> >> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor. > > > > java:107) > > > > >> >> at java.lang.Thread.run(Thread.java:744) > > > > >> >> > > > > >> >> > > > > >> >> Can any one explain what is lease exception > > > > >> >> > > > > >> >> Thanks, > > > > >> >> Rajeshkumar J > > > > >> >> > > > > >> > > > > > > > > > > > > > > > > > > > >
