in hdfs, part of the last contact value increases
Recently, I have increased two disks on a single datanode . I encountered a problem, in hdfs, part of the last contact value increases, resulting in hbase the handler is occupied. The datanode prints the log as follows: 2016-12-21 10:13:50,816 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: RemoteException in offerService org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException at org.apache.hadoop.ipc.Client.call(Client.java:1411) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy14.blockReport(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReport(DatanodeProtocolClientSideTranslatorPB.java:175) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:488) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:711) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:744) 2016-12-21 10:13:51,256 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.90.18.60:50010, dest: /10.90.18.60:39095, bytes: 24894792, op: HDFS_READ, cliID: DFSClient_hb_rs_l-hbase40.dba.cn2.qunar.com,60020,1473253709668_1716681090_37, offset: 0, srvID: b7cb495e-de56-4f4e-b089-abe20a3e0e53, blockid: BP-329468547-10.90.18.19-180193809:blk_1238808018_165078752, duration: 1214526037 2016-12-21 10:13:51,987 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: RemoteException in offerService org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException at org.apache.hadoop.ipc.Client.call(Client.java:1411) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy14.blockReport(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReport(DatanodeProtocolClientSideTranslatorPB.java:175) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:488) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:711) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:744) Use jstack print out the contents of regionserver as follows: "B.defaultRpcServer.handler=62,queue=2,port=60020" daemon prio=10 tid=0x00e91800 nid=0x63b8 in Object.wait() [0x7f9b83f43000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1371) - locked <0x7f9cf1334468> (a java.util.concurrent.atomic.AtomicLong) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1485) at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5968) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2580) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2261) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2213) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2217) at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4257) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3459) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3348) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29593) at qunar.tc.hbase.interceptor.support.CustomCompositeInterceptor.callMethod(CustomCompositeInterceptor.java:76) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2008) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:744) "B.defaultRpcServer.handler=61,queue=1,port=60020" daemon prio=10 tid=0x00e8f800 nid=0x63b7 in Object.wait() [0x7f9b84044000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1371)
Re: 答复: [ANNOUNCE] New HBase committer Guanghao Zhang
Congratulations and welcome Guanghao! -- Cloudera, Inc. On Wed, Dec 21, 2016 at 5:29 PM, Honghua Feng 冯宏华 wrote: > Congratulations and welcome Guanghao！ > > 发件人: saint@gmail.com 代表 Stack > 发送时间: 2016年12月21日 2:01 > 收件人: HBase Dev List > 抄送: hbase-user > 主题: Re: [ANNOUNCE] New HBase committer Guanghao Zhang > > Welcome Guanghao! > St.Ack > > On Mon, Dec 19, 2016 at 5:37 PM, Duo Zhang wrote: > > > On behalf of the Apache HBase PMC, I am pleased to announce that Guanghao > > Zhang has accepted the PMC's invitation to become a committer on the > > project. We appreciate all of Guanghao's generous contributions thus far > > and look forward to his continued involvement. > > > > Congratulations and welcome, Guanghao! > > >
答复: [ANNOUNCE] New HBase committer Guanghao Zhang
Congratulations and welcome Guanghao！ 发件人: saint@gmail.com 代表 Stack 发送时间: 2016年12月21日 2:01 收件人: HBase Dev List 抄送: hbase-user 主题: Re: [ANNOUNCE] New HBase committer Guanghao Zhang Welcome Guanghao! St.Ack On Mon, Dec 19, 2016 at 5:37 PM, Duo Zhang wrote: > On behalf of the Apache HBase PMC, I am pleased to announce that Guanghao > Zhang has accepted the PMC's invitation to become a committer on the > project. We appreciate all of Guanghao's generous contributions thus far > and look forward to his continued involvement. > > Congratulations and welcome, Guanghao! >
Re: Lease exception
I am using hbase version 1.1.1 Also I didn't understand something here. Whenever a scanner.next() is called it needs to return rows(based on caching value) within leasing period or else scanner client will be closed eventually throwing this exception. Correct me as I didn't get the clear understanding of this issue On Wed, Dec 21, 2016 at 7:31 PM, Ted Yu wrote: > Which hbase release are you using ? > > There is heartbeat support when scanning. > Looks like the version you use doesn't have this support. > > Cheers > > > On Dec 21, 2016, at 4:02 AM, Rajeshkumar J > wrote: > > > > Hi, > > > > Thanks for the reply. I have properties as below > > > > > >hbase.regionserver.lease.period > >90 > > > > > >hbase.rpc.timeout > >90>/value> > > > > > > > > Correct me If I am wrong. > > > > I know hbase.regionserver.lease.period, which says how long a scanner > > lives between calls to scanner.next(). > > > > As far as I understand when scanner.next() is called it will fetch no > > of rows as in *hbase.client.scanner.caching. *When this fetching > > process takes more than lease period it will close the scanner object. > > so this exception occuring? > > > > > > Thanks, > > > > Rajeshkumar J > > > > > > > > On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin < > richardstar...@outlook.com > >> wrote: > > > >> It means your lease on a region server has expired during a call to > >> resultscanner.next(). This happens on a slow call to next(). You can > either > >> embrace it or "fix" it by making sure hbase.rpc.timeout exceeds > >> hbase.regionserver.lease.period. > >> > >> https://richardstartin.com > >> > >> On 21 Dec 2016, at 11:30, Rajeshkumar J >> mailto:rajeshkumarit8...@gmail.com>> wrote: > >> > >> Hi, > >> > >> I have faced below issue in our production cluster > >> > >> org.apache.hadoop.hbase.regionserver.LeaseException: > >> org.apache.hadoop.hbase.regionserver.LeaseException: lease '166881' > does > >> not exist > >> at org.apache.hadoop.hbase.regionserver.Leases. > >> removeLease(Leases.java:221) > >> at org.apache.hadoop.hbase.regionserver.Leases. > >> cancelLease(Leases.java:206) > >> at > >> org.apache.hadoop.hbase.regionserver.RSRpcServices. > >> scan(RSRpcServices.java:2491) > >> at > >> org.apache.hadoop.hbase.protobuf.generated. > ClientProtos$ClientService$2. > >> callBlockingMethod(ClientProtos.java:32205) > >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) > >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) > >> at > >> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop( > RpcExecutor.java:130) > >> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) > >> at java.lang.Thread.run(Thread.java:744) > >> > >> > >> Can any one explain what is lease exception > >> > >> Thanks, > >> Rajeshkumar J > >> >
Re: Lease exception
Which hbase release are you using ? There is heartbeat support when scanning. Looks like the version you use doesn't have this support. Cheers > On Dec 21, 2016, at 4:02 AM, Rajeshkumar J > wrote: > > Hi, > > Thanks for the reply. I have properties as below > > >hbase.regionserver.lease.period >90 > > >hbase.rpc.timeout >90>/value> > > > > Correct me If I am wrong. > > I know hbase.regionserver.lease.period, which says how long a scanner > lives between calls to scanner.next(). > > As far as I understand when scanner.next() is called it will fetch no > of rows as in *hbase.client.scanner.caching. *When this fetching > process takes more than lease period it will close the scanner object. > so this exception occuring? > > > Thanks, > > Rajeshkumar J > > > > On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin > wrote: > >> It means your lease on a region server has expired during a call to >> resultscanner.next(). This happens on a slow call to next(). You can either >> embrace it or "fix" it by making sure hbase.rpc.timeout exceeds >> hbase.regionserver.lease.period. >> >> https://richardstartin.com >> >> On 21 Dec 2016, at 11:30, Rajeshkumar J > mailto:rajeshkumarit8...@gmail.com>> wrote: >> >> Hi, >> >> I have faced below issue in our production cluster >> >> org.apache.hadoop.hbase.regionserver.LeaseException: >> org.apache.hadoop.hbase.regionserver.LeaseException: lease '166881' does >> not exist >> at org.apache.hadoop.hbase.regionserver.Leases. >> removeLease(Leases.java:221) >> at org.apache.hadoop.hbase.regionserver.Leases. >> cancelLease(Leases.java:206) >> at >> org.apache.hadoop.hbase.regionserver.RSRpcServices. >> scan(RSRpcServices.java:2491) >> at >> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2. >> callBlockingMethod(ClientProtos.java:32205) >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) >> at >> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) >> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) >> at java.lang.Thread.run(Thread.java:744) >> >> >> Can any one explain what is lease exception >> >> Thanks, >> Rajeshkumar J >>
Re: Lease exception
If your client caching is set to a large value, you will need to do a long scan occasionally, and the rpc itself will be expensive in terms of IO. So it's worth looking at hbase.client.scanner.caching to see if it is too large. If you're scanning the whole table check you aren't churning the block cache. The XML below looks wrong, was that copied verbatim from your site file? https://richardstartin.com > On 21 Dec 2016, at 12:02, Rajeshkumar J wrote: > > Hi, > > Thanks for the reply. I have properties as below > > >hbase.regionserver.lease.period >90 > > >hbase.rpc.timeout >90>/value> > > > > Correct me If I am wrong. > > I know hbase.regionserver.lease.period, which says how long a scanner > lives between calls to scanner.next(). > > As far as I understand when scanner.next() is called it will fetch no > of rows as in *hbase.client.scanner.caching. *When this fetching > process takes more than lease period it will close the scanner object. > so this exception occuring? > > > Thanks, > > Rajeshkumar J > > > > On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin > wrote: > >> It means your lease on a region server has expired during a call to >> resultscanner.next(). This happens on a slow call to next(). You can either >> embrace it or "fix" it by making sure hbase.rpc.timeout exceeds >> hbase.regionserver.lease.period. >> >> https://richardstartin.com >> >> On 21 Dec 2016, at 11:30, Rajeshkumar J > mailto:rajeshkumarit8...@gmail.com>> wrote: >> >> Hi, >> >> I have faced below issue in our production cluster >> >> org.apache.hadoop.hbase.regionserver.LeaseException: >> org.apache.hadoop.hbase.regionserver.LeaseException: lease '166881' does >> not exist >> at org.apache.hadoop.hbase.regionserver.Leases. >> removeLease(Leases.java:221) >> at org.apache.hadoop.hbase.regionserver.Leases. >> cancelLease(Leases.java:206) >> at >> org.apache.hadoop.hbase.regionserver.RSRpcServices. >> scan(RSRpcServices.java:2491) >> at >> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2. >> callBlockingMethod(ClientProtos.java:32205) >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) >> at >> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) >> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) >> at java.lang.Thread.run(Thread.java:744) >> >> >> Can any one explain what is lease exception >> >> Thanks, >> Rajeshkumar J >>
Re: Lease exception
Hi, Thanks for the reply. I have properties as below hbase.regionserver.lease.period 90 hbase.rpc.timeout 90>/value> Correct me If I am wrong. I know hbase.regionserver.lease.period, which says how long a scanner lives between calls to scanner.next(). As far as I understand when scanner.next() is called it will fetch no of rows as in *hbase.client.scanner.caching. *When this fetching process takes more than lease period it will close the scanner object. so this exception occuring? Thanks, Rajeshkumar J On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin wrote: > It means your lease on a region server has expired during a call to > resultscanner.next(). This happens on a slow call to next(). You can either > embrace it or "fix" it by making sure hbase.rpc.timeout exceeds > hbase.regionserver.lease.period. > > https://richardstartin.com > > On 21 Dec 2016, at 11:30, Rajeshkumar J mailto:rajeshkumarit8...@gmail.com>> wrote: > > Hi, > > I have faced below issue in our production cluster > > org.apache.hadoop.hbase.regionserver.LeaseException: > org.apache.hadoop.hbase.regionserver.LeaseException: lease '166881' does > not exist > at org.apache.hadoop.hbase.regionserver.Leases. > removeLease(Leases.java:221) > at org.apache.hadoop.hbase.regionserver.Leases. > cancelLease(Leases.java:206) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices. > scan(RSRpcServices.java:2491) > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2. > callBlockingMethod(ClientProtos.java:32205) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) > at > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) > at java.lang.Thread.run(Thread.java:744) > > > Can any one explain what is lease exception > > Thanks, > Rajeshkumar J >
Re: Lease exception
It means your lease on a region server has expired during a call to resultscanner.next(). This happens on a slow call to next(). You can either embrace it or "fix" it by making sure hbase.rpc.timeout exceeds hbase.regionserver.lease.period. https://richardstartin.com On 21 Dec 2016, at 11:30, Rajeshkumar J mailto:rajeshkumarit8...@gmail.com>> wrote: Hi, I have faced below issue in our production cluster org.apache.hadoop.hbase.regionserver.LeaseException: org.apache.hadoop.hbase.regionserver.LeaseException: lease '166881' does not exist at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:221) at org.apache.hadoop.hbase.regionserver.Leases.cancelLease(Leases.java:206) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2491) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:744) Can any one explain what is lease exception Thanks, Rajeshkumar J
Lease exception
Hi, I have faced below issue in our production cluster org.apache.hadoop.hbase.regionserver.LeaseException: org.apache.hadoop.hbase.regionserver.LeaseException: lease '166881' does not exist at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:221) at org.apache.hadoop.hbase.regionserver.Leases.cancelLease(Leases.java:206) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2491) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:744) Can any one explain what is lease exception Thanks, Rajeshkumar J
Approach: Incremental data load from HBASE
Hello Guys, I would like to understand different approach for Distributed Incremental load from HBase, Is there any *tool / incubactor tool* which satisfy requirement ? *Approach 1:* Write Kafka Producer and maintain manually column flag for events and ingest it with Linkedin Gobblin to HDFS / S3. *Approach 2:* Run Scheduled Spark Job - Read from HBase and do transformations and maintain flag column at HBase Level. In above both approach, I need to maintain column level flags. such as 0 - by default, 1-sent,2-sent and acknowledged. So next time Producer will take another 1000 rows of batch where flag is 0 or 1. I am looking for best practice approach with any distributed tool. Thanks. - Chetan Khatri
