Re: [ANNOUNCE] New HBase committer Mike Drob

[Phil Yang]

Congratulations!

Thanks,
Phil


2017-08-04 9:36 GMT+08:00 Guanghao Zhang :

> Congratulations！
>
> 2017-08-04 5:34 GMT+08:00 Ted Yu :
>
> > Congratulations, Mike.
> >
> > (Just got access to gmail).
> >
> > On Tue, Aug 1, 2017 at 8:38 AM, Josh Elser  wrote:
> >
> > > On behalf of the Apache HBase PMC, I'm pleased to announce that Mike
> Drob
> > > has accepted the PMC's invitation to become a committer.
> > >
> > > Mike has been doing some great things lately in the project and this
> is a
> > > simple way that we can express our thanks. As my boss likes to tell me:
> > the
> > > reward for a job well-done is more work to do! We're all looking
> forward
> > to
> > > your continued involvement :)
> > >
> > > Please join me in congratulating Mike!
> > >
> > > - Josh
> > >
> >
>

Re: [ANNOUNCE] New HBase committer Abhishek Singh Chouhan

[Phil Yang]

Congratulations!

Thanks,
Phil


2017-08-01 3:33 GMT+08:00 Jerry He :

> Congrats Abhishek!
>
>
> Jerry
>
> On Mon, Jul 31, 2017 at 1:41 AM, Anoop John  wrote:
> > Congrats Abhishek
> >
> > On Mon, Jul 31, 2017 at 1:48 PM, Yu Li  wrote:
> >> Congratulations, Abhishek!
> >>
> >> Best Regards,
> >> Yu
> >>
> >> On 31 July 2017 at 15:40, Jingcheng Du  wrote:
> >>
> >>> Congratulations!
> >>>
> >>> Regards,
> >>> Jingcheng
> >>>
> >>> 2017-07-31 12:49 GMT+08:00 ramkrishna vasudevan <
> >>> ramkrishna.s.vasude...@gmail.com>:
> >>>
> >>> > Congratulations Abhishek !!!
> >>> >
> >>> > Regards
> >>> > Ram
> >>> >
> >>> > On Mon, Jul 31, 2017 at 10:16 AM, Pankaj kr 
> >>> wrote:
> >>> >
> >>> > > Congratulations Abhishek..!!
> >>> > >
> >>> > > Thanks & Regards,
> >>> > > Pankaj
> >>> > >
> >>> > > HUAWEI TECHNOLOGIES CO.LTD.
> >>> > > Huawei Tecnologies India Pvt. Ltd.
> >>> > > Near EPIP Industrial Area, Kundalahalli Village
> >>> > > Whitefield, Bangalore-560066
> >>> > > www.huawei.com
> >>> > > 
> >>> > > -
> >>> > > This e-mail and its attachments contain confidential information
> from
> >>> > > HUAWEI, which
> >>> > > is intended only for the person or entity whose address is listed
> >>> above.
> >>> > > Any use of the
> >>> > > information contained herein in any way (including, but not
> limited to,
> >>> > > total or partial
> >>> > > disclosure, reproduction, or dissemination) by persons other than
> the
> >>> > > intended
> >>> > > recipient(s) is prohibited. If you receive this e-mail in error,
> please
> >>> > > notify the sender by
> >>> > > phone or email immediately and delete it!
> >>> > >
> >>> > > -Original Message-
> >>> > > From: Andrew Purtell [mailto:apurt...@apache.org]
> >>> > > Sent: Saturday, July 29, 2017 6:32 AM
> >>> > > To: d...@hbase.apache.org; user@hbase.apache.org
> >>> > > Subject: [ANNOUNCE] New HBase committer Abhishek Singh Chouhan
> >>> > >
> >>> > > On behalf of the Apache HBase PMC, I am pleased to announce that
> >>> Abhishek
> >>> > > Singh Chouhan has accepted the PMC's invitation to become a
> committer
> >>> on
> >>> > > the project.
> >>> > >
> >>> > > We appreciate all of Abhishek's great work thus far and look
> forward to
> >>> > > continued involvement.
> >>> > >
> >>> > > Please join me in congratulating Abhishek!
> >>> > >
> >>> > > --
> >>> > > Best regards,
> >>> > > Andrew
> >>> > >
> >>> >
> >>>
>

Re: [ANNOUNCE] Asynchronous client is available in HBase 2.0

[Phil Yang]

I filed an issue HBASE-18334
 to remove the old sync
implementation. I think we can do it in 3.0 and keep sync logic in
branch-2? Of course we can also remove it in 2.1 or higher 2.x branch but
if we remove it since 2.0 it may be a little radical  :)

Thanks,
Phil


2017-07-21 6:15 GMT+08:00 Stack :

> Very sweet Guanghao. Thanks for all the hard work (you, Duo and all who
> contributed to the async work).
>
> When you think we should move the sync client to be on top of the async
> chassis?
>
> Thanks,
> St.Ack
>
> P.S. It looks like you updated its status in this doc,
> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> ktczrlKHK8N4SZzs/edit#,
> thanks.
>
> On Thu, Jul 20, 2017 at 4:51 AM, Guanghao Zhang 
> wrote:
>
> > Dear all,
> >
> > Asynchronous client aims to provide the ability to access HBase
> > asynchronously. You can obtain an AsyncConnection from ConnectionFactory,
> > and then get an asynchronous table instance (for DML operations) or an
> > asynchronous admin instance (for DDL operations) from it to access HBase.
> > For the asynchronous table or admin, most methods have the same meaning
> > with the old Table or Admin interface, expect that the return value is
> > wrapped with a CompletableFuture (java 8 or higher is required) usually.
> > You can get more details from the documentation[1][2].
> >
> > There are two examples about the asynchronous client.
> AsyncClientExample[3]
> > is a simple example to show you how to use AsyncTable.
> HttpProxyExample[4]
> > is an example for advance user to show you how to use RawAsyncTable to
> > write a fully asynchronous HTTP proxy server. Welcome to try the
> > asynchronous client in your project.
> >
> > We tested the performance of asynchronous client by
> > the PerformanceEvaluation tool. And it showed that async client has
> almost
> > same latency with sync client[5]. Notice: when we did performance test
> for
> > asynchronous client. We found some performance issue when use jdk
> 1.8_111.
> > And it was fixed when use jdk 1.8_131 [6]. So suggest use a bigger
> version
> > than jdk8_131 when you use the asynchronous client.
> >
> > The major umbrella issues include:
> > HBASE-16432 Revisit the asynchronous ipc implementation[7]
> > HBASE-16833 Implement asynchronous hbase client based on HBASE-15921[8]
> > HBASE-17359 Implement async admin[9]
> >
> > There is also an issue HBASE-17856 to track the further improvements for
> > asynchronous hbase client[10].
> >
> > Finally, thanks Duo Zhang and Zheng Hu who worked with me together to
> > implement the asynchronous client.
> >
> > Best Regards,
> > Guanghao
> >
> > [1] https://hbase.apache.org/book.html#async.client
> > [2] https://hbase.apache.org/book.html#async.admin
> > [3]
> > https://github.com/apache/hbase/blob/master/hbase-
> > examples/src/main/java/org/apache/hadoop/hbase/client/
> > example/AsyncClientExample.java
> > [4]
> > https://github.com/apache/hbase/blob/master/hbase-
> > examples/src/main/java/org/apache/hadoop/hbase/client/
> > example/HttpProxyExample.java
> > [5] https://www.slideshare.net/HBaseCon/hbase-at-xiaomi
> > [6]
> > http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/
> > classes/java/util/concurrent/CompletableFuture.java
> > [7] https://issues.apache.org/jira/browse/HBASE-16432
> > [8] https://issues.apache.org/jira/browse/HBASE-16833
> > [9] https://issues.apache.org/jira/browse/HBASE-17359
> > [10] https://issues.apache.org/jira/browse/HBASE-17856
> >
>

Re: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC

[Phil Yang]

Congratulations!

Thanks,
Phil


2017-07-05 13:05 GMT+08:00 ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com>:

> Congratulations !!!
>
> On Wed, Jul 5, 2017 at 10:33 AM, QI Congyun  cn
> > wrote:
>
> > Congratulations, Chunhui.
> >
> > -Original Message-
> > From: Jerry He [mailto:jerry...@gmail.com]
> > Sent: Wednesday, July 05, 2017 1:02 PM
> > To: d...@hbase.apache.org; user@hbase.apache.org
> > Subject: Re: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC
> >
> > Congrats,  Chunhui!
> >
> > Thanks
> >
> > Jerry
> >
> > On Tue, Jul 4, 2017 at 8:37 PM Anoop John  wrote:
> >
> > > Congrats Chunhui..
> > >
> > > On Wed, Jul 5, 2017 at 6:55 AM, Pankaj kr 
> wrote:
> > > > Congratulations Chunhui..!!
> > > >
> > > > Regards,
> > > > Pankaj
> > > >
> > > >
> > > > -Original Message-
> > > > From: Yu Li [mailto:car...@gmail.com]
> > > > Sent: Tuesday, July 04, 2017 1:24 PM
> > > > To: d...@hbase.apache.org; Hbase-User
> > > > Subject: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC
> > > >
> > > > On behalf of the Apache HBase PMC I am pleased to announce that
> > > > Chunhui
> > > Shen has accepted our invitation to become a PMC member on the Apache
> > > HBase project. He has been an active contributor to HBase for past many
> > years.
> > > Looking forward for many more contributions from him.
> > > >
> > > > Please join me in welcoming Chunhui to the HBase PMC!
> > > >
> > > > Best Regards,
> > > > Yu
> > >
> >
>

Re: HBaseCon Update (CfP closes Monday)

[Phil Yang]

Hi Stack
It seems there are only 21 slides here but we have 24 talks totally?

Thanks,
Phil


2017-06-21 3:45 GMT+08:00 Stack :

> Slides from last weeks hbasecon2017 are up and available here:
> https://www.slideshare.net/search/slideshow?searchfrom=
> header=hbasecon2017
>
> Enjoy,
> HBaseCon Program Committee
>
> On Thu, Apr 20, 2017 at 10:57 AM, Stack  wrote:
>
> > A few things.
> >
> > 1. CfP closes Monday. Get those talks in [1]!
> > 2. Registration is full; we are working on opening the 'waiting list' (if
> > you are registered and do not think you can come, please give up your
> spot
> > for another). Please check back on the hbasecon website in next day or so
> > [2].
> > 3. In case you did not realize, sponsorship (courtesy of our gracious
> > hosts) is kinda cool in that you get your logo as sponsor of hbasecon by
> > donating to the Apache Software Foundation (one lump of cash, two good
> > deeds).
> >
> > Write me if I can help w/ questions on any of the above or anything else
> > related to hbasecon.
> >
> > Thanks,
> > St.Ack
> > HBaseCon Secretary
> >
> > P.S. CfP for HBaseCon Asia is also up [3]. If you can't come out west,
> you
> > can head east in August!
> >
> > 1. https://easychair.org/cfp/hbasecon2017
> > 2. hbasecon.com
> > 3. https://easychair.org/cfp/HBaseConAsia2017
> >
>

Re: [ANNOUNCE] New HBase committer Huaxiang Sun

[Phil Yang]

Congratulations!

Thanks,
Phil


2017-06-20 3:30 GMT+08:00 Sean Busbey :

> On behalf of the Apache HBase PMC, I am pleased to announce that
> Huaxiang Sun has accepted the PMC's invitation to become a committer
> on the project. We appreciate all of Huaxiang's great work thus far
> and look forward to continued involvement.
>
> Please join me in congratulating Huaxiang!
>

Re: [ANNOUNCE] New Apache HBase committer Ashu Pachauri

[Phil Yang]

Congratulations!

Thanks,
Phil


2017-06-19 10:47 GMT+08:00 Stephen Jiang :

> Congratulations and Welcome to the team!
>
> Best Regards
> Stephen
>
> On Sun, Jun 18, 2017 at 6:36 PM, Pankaj kr  wrote:
>
> > Congratulations Ashu..!!
> >
> > Thanks & Regards,
> > Pankaj
> >
> > HUAWEI TECHNOLOGIES CO.LTD.
> > Huawei Tecnologies India Pvt. Ltd.
> > Near EPIP Industrial Area, Kundalahalli Village
> > Whitefield, Bangalore-560066
> > www.huawei.com
> > 
> > -
> > This e-mail and its attachments contain confidential information from
> > HUAWEI, which
> > is intended only for the person or entity whose address is listed above.
> > Any use of the
> > information contained herein in any way (including, but not limited to,
> > total or partial
> > disclosure, reproduction, or dissemination) by persons other than the
> > intended
> > recipient(s) is prohibited. If you receive this e-mail in error, please
> > notify the sender by
> > phone or email immediately and delete it!
> >
> > -Original Message-
> > From: Gary Helmling [mailto:ghelml...@gmail.com]
> > Sent: Saturday, June 17, 2017 7:27 AM
> > To: d...@hbase.apache.org; user@hbase.apache.org
> > Subject: [ANNOUNCE] New Apache HBase committer Ashu Pachauri
> >
> > On behalf of the Apache HBase PMC, I am pleased to announce that Ashu
> > Pachauri has accepted the PMC's invitation to become a committer on the
> > project.  We appreciate all of Ashu's generous contributions thus far and
> > look forward to his continued involvement.
> >
> > Congratulations and welcome, Ashu!
> >
>

Re: [ANNOUNCE] New HBase committer Allan Yang

[Phil Yang]

Congratulations!

Thanks,
Phil


2017-06-09 12:13 GMT+08:00 Anoop John :

> Congrats Allan..   Welcome !!
>
> -Anoop-
>
> On Fri, Jun 9, 2017 at 9:27 AM, 张铎(Duo Zhang) 
> wrote:
> > Congratulations!
> >
> > 2017-06-09 11:55 GMT+08:00 Ted Yu :
> >
> >> Congratulations, Allan !
> >>
> >> On Thu, Jun 8, 2017 at 8:49 PM, Yu Li  wrote:
> >>
> >> > On behalf of the Apache HBase PMC, I am pleased to announce that Allan
> >> Yang
> >> > has accepted the PMC's invitation to become a committer on the
> >> > project. We appreciate all of Allan's generous contributions thus far
> and
> >> > look forward to his continued involvement.
> >> >
> >> > Congratulations and welcome, Allan!
> >> >
> >>
>

Re: Lease exception

[Phil Yang]

https://github.com/apache/hbase/blob/rel/1.1.1/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L2491

There is a TTL for scanners at server, to prevent client don't close the
scanners and they leak. The TTL is configured by
hbase.client.scanner.timeout.period at server and refreshed when a scan RPC
request comes . The TTLs of all scanners are managed by Lease. Your error
happens when server closes a scanner but in Lease it is already expired. So
I think you can try to increase  hbase.client.scanner.timeout.period at
server or decrease your hbase.client.scanner.timeout.period at client to
prevent the scanner expired before its scanning done.
hbase.client.scanner.timeout.period is used both at client and server, may
be different if you change one of sides.

BTW, I still suggest that you can upgrade your cluster and client. 1.1.1
has some data-loss bugs on scanning.

Thanks,
Phil


2016-12-22 17:26 GMT+08:00 Rajeshkumar J <rajeshkumarit8...@gmail.com>:

> can you please explain what is the cause of this lease exception and is
> there any solve this in current version
>
> Thanks
>
> On Thu, Dec 22, 2016 at 2:54 PM, Phil Yang <ud1...@gmail.com> wrote:
>
> > In fact at client the rpc timeout of scan request is also
> > hbase.client.scanner.timeout.period which replaces the
> > deprecated hbase.regionserver.lease.period.
> >
> > Your code that throws LeaseException has been removed by HBASE-16604,
> maybe
> > you can try to upgrade your cluster to 1.1.7? Your client can also
> upgrade
> > to 1.1.7 which will ignore UnknowScannerException and retry when the
> lease
> > is expired at server.
> >
> > Thanks,
> > Phil
> >
> >
> > 2016-12-22 16:51 GMT+08:00 Rajeshkumar J <rajeshkumarit8...@gmail.com>:
> >
> > > Also there is a solution what i have found from hbase user guide that
> > > hbase.rpc.timeout must be greater than hbase.client.scanner.timeout.
> > > period.
> > > How these two properties plays a part in the above exception. Please
> can
> > > anyone explain?
> > >
> > > On Wed, Dec 21, 2016 at 9:39 PM, Rajeshkumar J <
> > > rajeshkumarit8...@gmail.com>
> > > wrote:
> > >
> > > > I am using hbase version 1.1.1
> > > > Also I didn't understand something here. Whenever a scanner.next() is
> > > > called it needs to return rows(based on caching value) within leasing
> > > > period or else scanner client will be closed eventually throwing this
> > > > exception. Correct me as I didn't get the clear understanding of this
> > > issue
> > > >
> > > > On Wed, Dec 21, 2016 at 7:31 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> > > >
> > > >> Which hbase release are you using ?
> > > >>
> > > >> There is heartbeat support when scanning.
> > > >> Looks like the version you use doesn't have this support.
> > > >>
> > > >> Cheers
> > > >>
> > > >> > On Dec 21, 2016, at 4:02 AM, Rajeshkumar J <
> > > rajeshkumarit8...@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> > Hi,
> > > >> >
> > > >> >   Thanks for the reply. I have properties as below
> > > >> >
> > > >> > 
> > > >> >hbase.regionserver.lease.period
> > > >> >90
> > > >> >  
> > > >> >  
> > > >> >hbase.rpc.timeout
> > > >> >90>/value>
> > > >> >  
> > > >> >
> > > >> >
> > > >> > Correct me If I am wrong.
> > > >> >
> > > >> > I know hbase.regionserver.lease.period, which says how long a
> > scanner
> > > >> > lives between calls to scanner.next().
> > > >> >
> > > >> > As far as I understand when scanner.next() is called it will fetch
> > no
> > > >> > of rows as in *hbase.client.scanner.caching. *When this fetching
> > > >> > process takes more than lease period it will close the scanner
> > object.
> > > >> > so this exception occuring?
> > > >> >
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > Rajeshkumar J
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Wed, Dec 21, 

Re: Lease exception

[Phil Yang]

In fact at client the rpc timeout of scan request is also
hbase.client.scanner.timeout.period which replaces the
deprecated hbase.regionserver.lease.period.

Your code that throws LeaseException has been removed by HBASE-16604, maybe
you can try to upgrade your cluster to 1.1.7? Your client can also upgrade
to 1.1.7 which will ignore UnknowScannerException and retry when the lease
is expired at server.

Thanks,
Phil


2016-12-22 16:51 GMT+08:00 Rajeshkumar J :

> Also there is a solution what i have found from hbase user guide that
> hbase.rpc.timeout must be greater than hbase.client.scanner.timeout.
> period.
> How these two properties plays a part in the above exception. Please can
> anyone explain?
>
> On Wed, Dec 21, 2016 at 9:39 PM, Rajeshkumar J <
> rajeshkumarit8...@gmail.com>
> wrote:
>
> > I am using hbase version 1.1.1
> > Also I didn't understand something here. Whenever a scanner.next() is
> > called it needs to return rows(based on caching value) within leasing
> > period or else scanner client will be closed eventually throwing this
> > exception. Correct me as I didn't get the clear understanding of this
> issue
> >
> > On Wed, Dec 21, 2016 at 7:31 PM, Ted Yu  wrote:
> >
> >> Which hbase release are you using ?
> >>
> >> There is heartbeat support when scanning.
> >> Looks like the version you use doesn't have this support.
> >>
> >> Cheers
> >>
> >> > On Dec 21, 2016, at 4:02 AM, Rajeshkumar J <
> rajeshkumarit8...@gmail.com>
> >> wrote:
> >> >
> >> > Hi,
> >> >
> >> >   Thanks for the reply. I have properties as below
> >> >
> >> > 
> >> >hbase.regionserver.lease.period
> >> >90
> >> >  
> >> >  
> >> >hbase.rpc.timeout
> >> >90>/value>
> >> >  
> >> >
> >> >
> >> > Correct me If I am wrong.
> >> >
> >> > I know hbase.regionserver.lease.period, which says how long a scanner
> >> > lives between calls to scanner.next().
> >> >
> >> > As far as I understand when scanner.next() is called it will fetch no
> >> > of rows as in *hbase.client.scanner.caching. *When this fetching
> >> > process takes more than lease period it will close the scanner object.
> >> > so this exception occuring?
> >> >
> >> >
> >> > Thanks,
> >> >
> >> > Rajeshkumar J
> >> >
> >> >
> >> >
> >> > On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin <
> >> richardstar...@outlook.com
> >> >> wrote:
> >> >
> >> >> It means your lease on a region server has expired during a call to
> >> >> resultscanner.next(). This happens on a slow call to next(). You can
> >> either
> >> >> embrace it or "fix" it by making sure hbase.rpc.timeout exceeds
> >> >> hbase.regionserver.lease.period.
> >> >>
> >> >> https://richardstartin.com
> >> >>
> >> >> On 21 Dec 2016, at 11:30, Rajeshkumar J  <
> >> >> mailto:rajeshkumarit8...@gmail.com>> wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >>  I have faced below issue in our production cluster
> >> >>
> >> >> org.apache.hadoop.hbase.regionserver.LeaseException:
> >> >> org.apache.hadoop.hbase.regionserver.LeaseException: lease '166881'
> >> does
> >> >> not exist
> >> >> at org.apache.hadoop.hbase.regionserver.Leases.
> >> >> removeLease(Leases.java:221)
> >> >> at org.apache.hadoop.hbase.regionserver.Leases.
> >> >> cancelLease(Leases.java:206)
> >> >> at
> >> >> org.apache.hadoop.hbase.regionserver.RSRpcServices.
> >> >> scan(RSRpcServices.java:2491)
> >> >> at
> >> >> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$
> >> ClientService$2.
> >> >> callBlockingMethod(ClientProtos.java:32205)
> >> >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> >> >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> >> >> at
> >> >> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExec
> >> utor.java:130)
> >> >> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.
> java:107)
> >> >> at java.lang.Thread.run(Thread.java:744)
> >> >>
> >> >>
> >> >> Can any one explain what is lease exception
> >> >>
> >> >> Thanks,
> >> >> Rajeshkumar J
> >> >>
> >>
> >
> >
>

Re: [ANNOUNCE] New HBase Committer Josh Elser

[Phil Yang]

Congratulations!

Thanks,
Phil


2016-12-13 12:56 GMT+08:00 ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com>:

> Congratulations Josh !!!
>
> Regards
> Ram
>
> On Tue, Dec 13, 2016 at 4:18 AM, Enis Söztutar  wrote:
>
> > Congrats Josh!
> >
> > Enis
> >
> > On Mon, Dec 12, 2016 at 11:39 AM, Esteban Gutierrez <
> este...@cloudera.com>
> > wrote:
> >
> > > Congrats and welcome, Josh!
> > >
> > > esteban.
> > >
> > >
> > > --
> > > Cloudera, Inc.
> > >
> > >
> > > On Sun, Dec 11, 2016 at 10:17 PM, Yu Li  wrote:
> > >
> > > > Congratulations and welcome!
> > > >
> > > > Best Regards,
> > > > Yu
> > > >
> > > > On 12 December 2016 at 12:47, Mikhail Antonov 
> > > > wrote:
> > > >
> > > > > Congratulations Josh!
> > > > >
> > > > > -Mikhail
> > > > >
> > > > > On Sun, Dec 11, 2016 at 5:20 PM, 张铎  wrote:
> > > > >
> > > > > > Congratulations!
> > > > > >
> > > > > > 2016-12-12 9:03 GMT+08:00 Jerry He :
> > > > > >
> > > > > > > Congratulations , Josh!
> > > > > > >
> > > > > > > Good work on the PQS too.
> > > > > > >
> > > > > > > Jerry
> > > > > > >
> > > > > > > On Sun, Dec 11, 2016 at 12:14 PM, Josh Elser <
> els...@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > > > Thanks, all. I'm looking forward to continuing to work with
> you
> > > > all!
> > > > > > > >
> > > > > > > >
> > > > > > > > Nick Dimiduk wrote:
> > > > > > > >
> > > > > > > >> On behalf of the Apache HBase PMC, I am pleased to announce
> > that
> > > > > Josh
> > > > > > > >> Elser
> > > > > > > >> has accepted the PMC's invitation to become a committer on
> the
> > > > > > project.
> > > > > > > We
> > > > > > > >> appreciate all of Josh's generous contributions thus far and
> > > look
> > > > > > > forward
> > > > > > > >> to his continued involvement.
> > > > > > > >>
> > > > > > > >> Allow me to be the first to congratulate and welcome Josh
> into
> > > his
> > > > > new
> > > > > > > >> role!
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks,
> > > > > Michael Antonov
> > > > >
> > > >
> > >
> >
>

Re: How to limit a single row being filled with millions of columus?

[Phil Yang]

10 is column-level, if you have many cells in one row but each column is
not too large, I think it will not increase the pressure of GC. You may
need check if your single column's value is too large.

Thanks,
Phil


2016-12-06 17:24 GMT+08:00 聪聪 <175998...@qq.com>:

> 1、The version of my cluster is hbase-0.98.6-cdh5.2.0. Because the table
> have a large of date and many rows, so do you have any other ways to find
> big rows better?
> 2、The parameter default value of hbase.hstore.compaction.kv.max is 10.If
> there has big row, how much should we set the value ?
>
>
>
>
> ------ 原始邮件 --
> 发件人: "Phil Yang";<yangzhe1...@apache.org>;
> 发送时间: 2016年12月6日(星期二) 下午3:46
> 收件人: "hbase-user"<user@hbase.apache.org>;
>
> 主题: Re: How to limit a single row being filled with millions of columus?
>
>
>
> We have hbase.hstore.compaction.kv.max to setBatch on compaction and
> default value is 10, which means we will write each 10 cells to writer. I
> think it can prevent using too much heap while compacting?
>
> Thanks,
> Phil
>
>
> 2016-12-06 15:29 GMT+08:00 Guanghao Zhang <zghao...@gmail.com>:
>
> > Now the scan context has size limit. And scan can break between cells.
> This
> > should help for this. What is the version of your cluster?
> >
> > 2016-12-06 13:35 GMT+08:00 聪聪 <175998...@qq.com>:
> >
> > > I am glad to receive your reply！How can I find a big row quickly? If
> not
> > > ,when we proceed major compact,the regionserver dose not work all the
> > time .
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > -- 原始邮件 --
> > > 发件人: "Guanghao Zhang";<zghao...@gmail.com>;
> > > 发送时间: 2016年12月6日(星期二) 中午12:13
> > > 收件人: "user"<user@hbase.apache.org>;
> > >
> > > 主题: Re: How to limit a single row being filled with millions of
> columus?
> > >
> > >
> > >
> > > There are a config hbase.table.max.rowsize but only used for user
> > get/scan.
> > > It will throw RowTooBigException when you scan a big row with millions
> of
> > > columns. But it doesn't work for compact. If i am not wrong, there are
> > not
> > > a way to prevent a single row being filled with millions of columns.
> > >
> > > 2016-12-06 11:52 GMT+08:00 聪聪 <175998...@qq.com>:
> > >
> > > > Recently, I have a problem that confused me a long time. The problem
> is
> > > > that as we all know in HBase,there are millions of columns in a
> single
> > > row.
> > > > Full gc will happen when region proceeds major compact and it results
> > in
> > > > regionserver and hbase not working. Is there any good way to prevent
> a
> > > > single row being put(wrote) columns overmuch?
> > > > Hope to you soon!
> > >
> >
>

Re: How to limit a single row being filled with millions of columus?

[Phil Yang]

We have hbase.hstore.compaction.kv.max to setBatch on compaction and
default value is 10, which means we will write each 10 cells to writer. I
think it can prevent using too much heap while compacting?

Thanks,
Phil


2016-12-06 15:29 GMT+08:00 Guanghao Zhang :

> Now the scan context has size limit. And scan can break between cells. This
> should help for this. What is the version of your cluster?
>
> 2016-12-06 13:35 GMT+08:00 聪聪 <175998...@qq.com>:
>
> > I am glad to receive your reply！How can I find a big row quickly? If not
> > ,when we proceed major compact,the regionserver dose not work all the
> time .
> >
> >
> >
> >
> >
> >
> >
> > -- 原始邮件 --
> > 发件人: "Guanghao Zhang";;
> > 发送时间: 2016年12月6日(星期二) 中午12:13
> > 收件人: "user";
> >
> > 主题: Re: How to limit a single row being filled with millions of columus?
> >
> >
> >
> > There are a config hbase.table.max.rowsize but only used for user
> get/scan.
> > It will throw RowTooBigException when you scan a big row with millions of
> > columns. But it doesn't work for compact. If i am not wrong, there are
> not
> > a way to prevent a single row being filled with millions of columns.
> >
> > 2016-12-06 11:52 GMT+08:00 聪聪 <175998...@qq.com>:
> >
> > > Recently, I have a problem that confused me a long time. The problem is
> > > that as we all know in HBase,there are millions of columns in a single
> > row.
> > > Full gc will happen when region proceeds major compact and it results
> in
> > > regionserver and hbase not working. Is there any good way to prevent a
> > > single row being put(wrote) columns overmuch?
> > > Hope to you soon!
> >
>

Re: [ANNOUNCE] New HBase committer Phil Yang

[Phil Yang]

Thanks all.
It is a great honor to be a committer and I will try my best to contribute
more to HBase.

Thanks,
Phil


2016-11-30 1:07 GMT+08:00 Stack <st...@duboce.net>:

> Welcome Phil!
> St.Ack
>
> On Tue, Nov 29, 2016 at 1:49 AM, Duo Zhang <zhang...@apache.org> wrote:
>
> > On behalf of the Apache HBase PMC, I am pleased to announce that Phil
> Yang
> > has accepted the PMC's invitation to become a committer on the project.
> We
> > appreciate all of Phil's generous contributions thus far and look forward
> > to his continued involvement.
> >
> > Congratulations and welcome, Phil!
> >
>

Re: Hbase 1.1.1 API documentation link

[Phil Yang]

For javadoc I think users need accurate api docs for the version they use,
so we may also need api docs for 0.98/1.0/1.2 and coming 1.3?

For "doc", which called HBase Book, I think only one book is good but we
may need to tell users which version does HBase starts to support for each
feature/option it mentions. For example, we may add some important options
in hbase-default.xml when we add new features, and we will update HBase
Book according to the latest hbase-default.xml. But some of them may only
in 2.0 and user may use 1.1/1.2, they may be misled by the book. Add a
"since 2.0.0" tag can help users know if a feature in the book is supported
by the released version they use, and also can encourage users to upgrade
to the new version if the feature is what they want but not included by the
version they use.

Thanks,
Phil


2016-11-08 12:13 GMT+08:00 Yu Li :

> Oops, sorry, actually it was me derailing the topic from javadoc to doc...
> And thanks stack for the proposal of setting up 1.1 doc, I think it'll be
> useful (Smile).
>
> However, similar to doc, there's javadoc published for 2.0 and 0.94, but
> none for other version:
> API doc for 0.94: http://hbase.apache.org/0.94/apidocs/
> API doc for 2.0: http://hbase.apache.org/apidocs/
>
> Best Regards,
> Yu
>
> On 8 November 2016 at 08:30, Stack  wrote:
>
> > On Mon, Nov 7, 2016 at 2:43 PM, Sean Busbey  wrote:
> >
> > > I think most folks are looking for javadocs, which vary considerably
> > > across minor release lines.
> > >
> > >
> > Sorry. Yes. Indeed that is what Rajeshkumar asked originally.
> >
> > As penance for my derail of the thread, let me work on putting up a link
> to
> > 1.1. doc. this evening.
> >
> > Thanks,
> > St.Ack
> >
> >
> > > On Mon, Nov 7, 2016 at 3:10 PM, Stack  wrote:
> > > > On Sun, Nov 6, 2016 at 9:23 PM, Yu Li  wrote:
> > > >
> > > >> We do have an hbase book for 0.94 specially:
> > > >> http://hbase.apache.org/0.94/book.html, and it might be a good
> > question
> > > >> that shall we supply similar website of different branches for user
> or
> > > >> always require them to generate the site by themselves? Current
> hbase
> > > book
> > > >> (
> > > >> http://hbase.apache.org/book.html) contains some new features only
> > for
> > > 2.0
> > > >> (plz correct me if I'm wrong) so might not be proper for all users
> to
> > > refer
> > > >> to?
> > > >>
> > > >> btw, if plan to use branch-1.1 release, I'd suggest to use versions
> > > later
> > > >> than (including) 1.1.4 because of HBASE-14460 (
> > > >> https://issues.apache.org/jira/browse/HBASE-14460, which fixes
> > > performance
> > > >> regression), JFYI @Rajeshkumar
> > > >>
> > > >> Best Regards,
> > > >> Yu
> > > >>
> > > >>
> > > > This is a perennial (... that we should publish per-version doc.)
> > > >
> > > > Truth be told, we don't do the work to maintain version specific
> > > > documentation. RMs usually copy the current state of the master doc
> > into
> > > > their branch before cutting an RC so they get the 'latest' changes.
> Doc
> > > is
> > > > done up on the master branch. Doc will have callouts if a
> > > version-specific
> > > > config or feature but we are probably erratic in our practice here
> (and
> > > in
> > > > the converse, deprecating the old and no longer supported).
> > > >
> > > > Doc seems to change at a slow rate so maybe its not so bad that this
> is
> > > the
> > > > case.
> > > >
> > > > That said, if the 1.1 doc has big differences from the master doc.,
> we
> > > > could do the same was we did for 0.94 -- making a subdir for 1.1.
> > > >
> > > > St.Ack
> > > >
> > > >
> > > >
> > > >
> > > >> On 7 November 2016 at 13:06, Ted Yu  wrote:
> > > >>
> > > >> > I actually run the (modified) command in your first email quite
> > often
> > > -
> > > >> > when building tar balls for cluster testing.
> > > >> >
> > > >> > In that case, site goal is not used.
> > > >> >
> > > >> > Cheers
> > > >> >
> > > >> > On Sun, Nov 6, 2016 at 8:44 PM, Josh Elser 
> > > wrote:
> > > >> >
> > > >> > > In hindsight, you're probably right, Ted. I was just copying
> from
> > > the
> > > >> > book
> > > >> > > and forgot to remove that execution :)
> > > >> > >
> > > >> > >
> > > >> > > Ted Yu wrote:
> > > >> > >
> > > >> > >> If I am not mistaken, the 'assembly:single' goal is not needed
> > for
> > > >> > >> building
> > > >> > >> the site.
> > > >> > >>
> > > >> > >> Cheers
> > > >> > >>
> > > >> > >> On Sun, Nov 6, 2016 at 12:57 PM, Josh Elser<
> josh.el...@gmail.com
> > >
> > > >> > wrote:
> > > >> > >>
> > > >> > >> Hi,
> > > >> > >>>
> > > >> > >>> I don't believe all versions of Javadoc are published on the
> > > website.
> > > >> > >>> Something similar to the following should build it for you
> > > locally.
> > > >> > >>>
> > > >> > >>> 1. git checkout rel/1.1.1
> > > >> > >>> 2. `mvn clean install -DskipTests 

Re: Re: Re: Re: What way to improve MTTR other than DLR(distributed log replay)

[Phil Yang]

Hi all

We are also considering how to improve MTTR these days and have a plan. I
just notice this thread, hoping I am not late :)

My original thought is simple: Why we must put entries to MemStore of new
RS? We can read/write WAL entries only once in failover, from log entries
to HFiles, which is similar with Enis's thought. My idea is to flush them
to HFiles in their region's directory at the end of splitting, each HFile
for each region. And then assign the region to a RS. For the new RS it
doesn't replay any logs, it can open the region directly. The difference
from the new RS and the crashed RS is the new RS has one or some small
HFiles, and its MemStore is empty. If we have too many small HFiles we can
submit a minor compaction task but when we compact the region is already
online

Although this thought need writing many small HFiles rather than one big
HFile for each log file, now for ReferenceFile we also write a small file
containing region and split key to region's directory. Duo Zhang told me
this is because HDFS doesn't support hard link. So if we write them to many
HFiles the number of files in HDFS will not be increased. And if we write
entries to many HFiles, we can use current logic for all steps which is
simple to implement. For example, we can use a MemStore to save entries for
one region, and use flush logic to write it to the directory. We don't need
new HFile format and we can still check if we have split logs so we have
compatibility with old version, even if the RSs in cluster have different
split logic, we can still recover all regions.

Thanks,
Phil


2016-10-22 6:24 GMT+08:00 Enis Söztutar :

> A bit late, but let me give my perspective. This can also be moved to jira
> or dev@ I think.
>
> DLR was a nice and had pretty good gains for MTTR. However, dealing with
> the sequence ids, onlining regions etc and the replay paths proved to be
> too difficult in practice. I think the way forward would be to not bring
> DLR back, but actually fix long standing log split problems.
>
> The main gains in DLR is that we do not create lots and lots of tiny files,
> but instead rely on the regular region flushes, to flush bigger files. This
> also helps with handling requests coming from different log files etc. The
> only gain that I can think of that you get with DLR, but not with log split
> is the online enabling of writes while the recovery is going on. However, I
> think it is not worth having DLR just for this feature.
>
> Now, what are the problems with Log Split you ask. The problems are
>   - we create a lot of tiny files
>   - these tiny files are replayed sequentially when the region is assigned
>   - The region has to replay and flush all data sequentially coming from
> all these tiny files.
>
> In terms of IO, we pay the cost of reading original WAL files, and writing
> this same amount of data into many small files where the NN overhead is
> huge. Then for every region, we do serially sort the data by re-reading the
> tiny WAL files (recovered edits) and sorting them in memory and flushing
> the data. Which means we do 2 times the reads and writes that we should do
> otherwise.
>
> The way to solve our log split bottlenecks is re-reading the big table
> paper and implement the WAL recovery as described there.
>  - Implement an HFile format that can contain data from multiple regions.
> Something like a concatinated HFile format where each region has its own
> section, with its own sequence id, etc.
>  - Implement links to these files where a link can refer to this data. This
> is very similar to our ReferenceFile concept.
>  - In each log splitter task, instead of generating tiny WAL files that are
> recovered edits, we instead buffer up in memory, and do a sort (this is the
> same sort of inserting into the memstore) per region. A WAL is ~100 MB on
> average, so should not be a big problem to buffer up this. At the end of
> the WAL split task, write an hfile containing data from all the regions as
> described above. Also do a multi NN request to create links in regions to
> refer to these files (Not sure whether NN has a batch RPC call or not).
>
> The reason this will be on-par or better than DLR is that, we are only
> doing 1 read and 1 write, and the sort is parallelized. The region opening
> does not have to block on replaying anything or waiting for flush, because
> the data is already sorted and in HFile format. These hfiles will be used
> the normal way by adding them to the KVHeaps, etc. When compactions run, we
> will be removing the links to these files using the regular mechanisms.
>
> Enis
>
> On Tue, Oct 18, 2016 at 6:58 PM, Ted Yu  wrote:
>
> > Allan:
> > One factor to consider is that the assignment manager in hbase 2.0 would
> be
> > quite different from those in 0.98 and 1.x branches.
> >
> > Meaning, you may need to come up with two solutions for a single problem.
> >
> > FYI
> >
> > On Tue, Oct 18, 2016 at 6:11 PM, 

Re: get first row of every region

[Phil Yang]

If you pre-split this table with some split points when you create the
table, the start key may not be the prefix of the first row key. So you
should use setStartRow(regionInfo.getStartKey())
and setStopRow(nextRegionInfo.getStartKey()) and setBatch(1), if the result
is still null, this region is indeed empty.

Thanks,
Phil


2016-08-02 10:21 GMT+08:00 jinhong lu :

>  Thanks. Here is my code, but in most case, r is null? why this happened?
>
> byte[] startRowkey =
> regionInfo.getStartKey();
> Scan sc = new Scan();
> sc.setBatch(1);
> sc.setRowPrefixFilter(startRowkey);
> try {
> scanner = table.getScanner(sc);
> r = scanner.next();
> scanner.close();
> }
> Thanks,
> lujinhong
>
> > 在 2016年8月1日，18:49，Ted Yu > 写道：
> >
> > .
>
>