[jira] [Resolved] (HBASE-19145) Look into hbase-2 client going to hbase-1 server

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-19145.
--
Resolution: Done

> Look into hbase-2 client going to hbase-1 server
> 
>
> Key: HBASE-19145
> URL: https://issues.apache.org/jira/browse/HBASE-19145
> Project: HBase
>  Issue Type: Task
>Affects Versions: 2.0.0-beta-1
>    Reporter: Jerry He
>Assignee: Jerry He
>Priority: Major
>
> From the "[DISCUSS] hbase-2.0.0 compatibility expectations" thread.
> Do we support hbase-2 client going against hbase-1 server?
> We seem to be fine mix-and-match the clients and servers within the
> hbase-1 releases.   And IIRC hbase-1 client is ok against 0.98 server.
> Suppose I have a product  that depends and bundles HBase client. I
> want to upgrade the dependency to hbase-2 so that it can take
> advantages of and claim support of hbase-2.
> But does it mean that I will need drop the claims that the new version
> of the product support any hbase-1 backend?
> It has not been an objective. It might work doing basic Client API on a
> later branch-1 but will fail doing Admin functions (and figuring if a Table
> is online).  If it was a small thing to make it
> work, lets get it in.
> Let's look into it to see what works and what not.  Have a statement at least.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-21008) HBase 1.x can not read HBase2 hfiles due to TimeRangeTracker

[Jerry He (JIRA)]

Jerry He created HBASE-21008:


 Summary: HBase 1.x can not read HBase2 hfiles due to 
TimeRangeTracker
 Key: HBASE-21008
 URL: https://issues.apache.org/jira/browse/HBASE-21008
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.4.6, 2.1.0
Reporter: Jerry He


It looks like HBase 1.x can not open hfiiles written by HBase2 still.

I tested the latest HBase 1.4.6 and 2.1.0.  1.4.6 tried to read and open 
regions written by 2.1.0.

{code}
2018-07-30 16:01:31,274 ERROR [StoreFileOpenerThread-info-1] 
regionserver.StoreFile: Error reading timestamp range data from meta -- 
proceeding without
java.lang.IllegalArgumentException: Timestamp cannot be negative. 
minStamp:5783278630776778969, maxStamp:-4698050386518222402
at org.apache.hadoop.hbase.io.TimeRange.check(TimeRange.java:112)
at org.apache.hadoop.hbase.io.TimeRange.(TimeRange.java:100)
at 
org.apache.hadoop.hbase.regionserver.TimeRangeTracker.toTimeRange(TimeRangeTracker.java:214)
at 
org.apache.hadoop.hbase.regionserver.TimeRangeTracker.getTimeRange(TimeRangeTracker.java:198)
at 
org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:507)
at 
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:531)
at 
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:521)
at 
org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:679)
at 
org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:122)
at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:538)
at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:535)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
{code}
Or:
{code}
2018-07-30 16:01:31,305 ERROR [RS_OPEN_REGION-throb1:34004-0] 
handler.OpenRegionHandler: Failed open of 
region=janusgraph,,1532630557542.b0fa15cb0bf1b0bf740997b7056c., starting to 
roll back the global memstore size.
java.io.IOException: java.io.IOException: java.io.EOFException
at 
org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1033)
at 
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:908)
at 
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:876)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6995)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6956)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6927)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6883)
at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6834)
at 
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:364)
at 
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:131)
at 
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: java.io.EOFException
at 
org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:564)
at 
org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:518)
at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:281)
at 
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5378)
at 
org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1007)
at 
org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1004)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at java.io.DataInputStream.readLong(DataInputStream.java:416)
at 
org.apache.hadoop.hbase.regionserver.TimeRangeTracker.readFields(TimeRangeTracker.java:170)
at 
org.apache.hadoop.hbase.util.Writables.copyWritable(Writables.java:161)
at 
org.apache.hadoop.hbase.regionserver.TimeRangeTracker.getTimeRangeTracker(TimeRangeTracker.java:187)
at 
org.apache.hadoop.hbase.regionserver.TimeRangeTracker.getTimeRange(TimeRangeTracker.java:197)
at 
org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:507

[jira] [Created] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4

[Jerry He (JIRA)]

Jerry He created HBASE-20565:


 Summary: ColumnRangeFilter combined with ColumnPaginationFilter 
can produce incorrect result since 1.4
 Key: HBASE-20565
 URL: https://issues.apache.org/jira/browse/HBASE-20565
 Project: HBase
  Issue Type: Bug
  Components: Filters
Affects Versions: 1.4.4
Reporter: Jerry He


When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see 
incorrect result.

Here is a simple example.

One row with 10 columns c0, c1, c2, .., c9.  I have a ColumnRangeFilter for 
range c2 to c9.  Then I have a ColumnPaginationFilter with limit 5 and offset 
0.  FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, 
ColumnPaginationFilter).
We expect 5 columns being returned.  But in HBase 1.4 and after, 4 columns are 
returned.
In 1.2.x, the correct 5 columns are returned.





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Re: [ANNOUNCE] Please welcome Francis Liu to the HBase PMC

[Jerry He]

Congratulations, Francis!

Jerry

On Wed, Apr 11, 2018 at 10:03 PM, Andrew Purtell 
> wrote:
> > On behalf of the Apache HBase PMC I am pleased to announce that Francis
> > Liu has accepted our invitation to become a PMC member on the Apache
> > HBase project. We appreciate Francis stepping up to take more
> > responsibility in the HBase project. He has been an active contributor to
> > HBase for many years and recently took over responsibilities as branch RM
> > for branch-1.3.
> >
> > Please join me in welcoming Francis to the HBase PMC!
> >
> > --
> > Best regards,
> > Andrew
> >
>

Re: Re: [ANNOUNCE] Please welcome Guanghao Zhang to the HBase PMC

[Jerry He]

  Congrats,  Guanghao!

On Wed, Feb 28, 2018 at 6:17 PM Allan Yang  wrote:

>  Congratulations Guanghao!
>
> Best Regards
> Allan Yang
>
> 2018-03-01 10:11 GMT+08:00 OpenInx :
>
> > Congratulations !
> >
> > On Thu, Mar 1, 2018 at 9:54 AM, Chunhui Shen  wrote:
> >
> > > Congrats！
> > >
> > >
> > > At 2018-03-01 08:29:22, "Esteban Gutierrez" 
> > wrote:
> > > >Congrats, Guanghao!
> > > >
> > > >--
> > > >Cloudera, Inc.
> > > >
> > > >
> > > >On Wed, Feb 28, 2018 at 3:56 PM, Andrew Purtell 
> > > wrote:
> > > >
> > > >> Congratulations!
> > > >>
> > > >>
> > > >> On Wed, Feb 28, 2018 at 7:55 AM, Sean Busbey 
> > wrote:
> > > >>
> > > >> > On behalf of the Apache HBase PMC I am pleased to announce that
> > > >> > Guanghao Zhang has accepted our invitation to become a PMC member
> on
> > > >> > the Apache HBase project. We appreciate Guanghao stepping up to
> take
> > > >> > more responsibility in the HBase project.
> > > >> >
> > > >> > As a reminder, if anyone would like to nominate another person as
> a
> > > >> > committer or PMC member, even if you are not currently a committer
> > or
> > > >> > PMC member, you can always drop a note to
> priv...@hbase.apache.org
> > to
> > > >> > let us know!
> > > >> >
> > > >> > - busbey
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Best regards,
> > > >> Andrew
> > > >>
> > > >> Words like orphans lost among the crosstalk, meaning torn from
> truth's
> > > >> decrepit hands
> > > >>- A23, Crosstalk
> > > >>
> > >
> >
> >
> >
> > --
> > ==
> > Openinx  blog : http://openinx.github.io
> >
> > TO BE A GREAT HACKER !
> > ==
> >
>

Re: 回复：Re: [ANNOUNCE] Please welcome Ashish Singhi to the HBase PMC

[Jerry He]

Congrats, Ashish!

On Wed, Feb 28, 2018 at 6:18 PM Allan Yang  wrote:

>  Congratulations  Ashish!
>
> Best Regards
> Allan Yang
>
> 2018-03-01 10:11 GMT+08:00 OpenInx :
>
> > Congratulations !
> >
> > On Thu, Mar 1, 2018 at 9:55 AM, Chunhui Shen  wrote:
> >
> > > Congrats！
> > >
> > >
> > >
> > >
> > > At 2018-03-01 09:24:47, "岭秀"  wrote:
> > > >Congratulations, Ashish!!  well-deserved
> > > >> > On behalf of the Apache HBase PMC I am pleased to announce that
> > Ashish
> > > >> > Singhi has accepted our invitation to become a PMC member on the
> > > >> > Apache HBase project. We appreciate Ashish stepping up to take
> more
> > > >> > responsibility in the HBase project.
> > > >> >
> > > >> > As a reminder, if anyone would like to nominate another person as
> a
> > > >> > committer or PMC member, even if you are not currently a committer
> > or
> > > >> > PMC member, you can always drop a note to
> priv...@hbase.apache.org
> > to
> > > >> > let us know!
> > > >> >
> > > >> > - busbey
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Best regards,
> > > >> Andrew
> > > >>
> > > >> Words like orphans lost among the crosstalk, meaning torn from
> truth's
> > > >> decrepit hands
> > > >>- A23, Crosstalk
> > > >>
> > >
> >
> >
> >
> > --
> > ==
> > Openinx  blog : http://openinx.github.io
> >
> > TO BE A GREAT HACKER !
> > ==
> >
>

Re: Re: [ANNOUNCE] Please welcome Mike Drob to the HBase PMC

[Jerry He]

Congrats,  Mike!

On Wed, Feb 28, 2018 at 6:19 PM Allan Yang  wrote:

>  Congratulations Mike!
>
> Best Regards
> Allan Yang
>
> 2018-03-01 10:12 GMT+08:00 OpenInx :
>
> > Congratulations !
> >
> > On Thu, Mar 1, 2018 at 9:54 AM, Chunhui Shen  wrote:
> >
> > > Congrats！
> > > At 2018-03-01 07:56:11, "Andrew Purtell"  wrote:
> > > >Congrats Mike!
> > > >
> > > >On Wed, Feb 28, 2018 at 7:54 AM, Sean Busbey 
> wrote:
> > > >
> > > >> On behalf of the Apache HBase PMC I am pleased to announce that Mike
> > > >> Drob has accepted our invitation to become a PMC member on the
> Apache
> > > >> HBase project. We appreciate Mike stepping up to take more
> > > >> responsibility in the HBase project.
> > > >>
> > > >>
> > > >> As a reminder, if anyone would like to nominate another person as a
> > > >> committer or PMC member, even if you are not currently a committer
> or
> > > >> PMC member, you can always drop a note to priv...@hbase.apache.org
> to
> > > >> let us know!
> > > >>
> > > >> - busbey
> > > >>
> > > >
> > > >
> > > >
> > > >--
> > > >Best regards,
> > > >Andrew
> > > >
> > > >Words like orphans lost among the crosstalk, meaning torn from truth's
> > > >decrepit hands
> > > >   - A23, Crosstalk
> > >
> >
> >
> >
> > --
> > ==
> > Openinx  blog : http://openinx.github.io
> >
> > TO BE A GREAT HACKER !
> > ==
> >
>

Re: [ANNOUNCE] New HBase committer Peter Somogyi

[Jerry He]

 Congrats, Peter!

On Thu, Feb 22, 2018 at 2:53 PM, Andrew Purtell  wrote:

> Congratulations and welcome, Peter!
>
>
> On Thu, Feb 22, 2018 at 11:08 AM, Sean Busbey  wrote:
>
> > On behalf of the Apache HBase PMC, I am pleased to announce that Peter
> > Somogyi has accepted the PMC's invitation to become a committer on the
> > project.
> >
> > We appreciate all of Peter's great work thus far and look forward to
> > continued involvement.
> >
> > Please join me in congratulating Peter!
> >
>
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

Re: Cleanup and remove the code path where is no hbase.client.rpc.codec

[Jerry He]

On Thu, Dec 21, 2017 at 12:03 AM, Stack <st...@duboce.net> wrote:

> On Tue, Dec 19, 2017 at 7:02 PM, Jerry He <jerry...@gmail.com> wrote:
>
> > RPC_CODEC_CONF_KEY 'hbase.client.rpc.codec' is a property we use on the
> > client side to determine the RPC codec.
> >
> > It currently has a strange logic. Whereas the default is KeyValueCodec,
> we
> > allow  a user to specify an empty string "" as the a way to indicate
> there
> > is no codec class and we should not use any.
> >
> >   Codec getCodec() {
> > // For NO CODEC, "hbase.client.rpc.codec" must be configured with
> empty
> > string AND
> > // "hbase.client.default.rpc.codec" also -- because default is to do
> > cell block encoding.
> > String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> > getDefaultCodec(this.conf));
> > if (className == null || className.length() == 0) {
> >   return null;
> > }
> > try {
> >   return (Codec) Class.forName(className).newInstance();
> > } catch (Exception e) {
> >   throw new RuntimeException("Failed getting codec " + className, e);
> > }
> >   }
> >
> > I don't know the original reason for having this.
> >
>
> IIRC, when we moved to pb's first, KeyValues were all pb'd. This was our
> default serialization.
>
> It was too slow -- duh -- so we had to figure something else. We came up w/
> notion of pb being used to describe the content of the RPC but that the
> actual cells would follow-behind the pb in a 'cellblock' (We used to show a
> picture of a motorcycle with a sidecar as an illustration trying to convey
> that the follow-behind appendage was like a 'sidecar' that went with the
> RPC message). We went out of our way to ensure we allowed shipping both
> forms of message -- with sidecar and without with KVs PB encoded. The
> latter would be useful for not native clients, at least while trying to get
> off the ground.


 Thanks for the background information. The intention for non native client
is a good intention.


> The above code came in with:
>
> tree 6c91d2f4ee7faadea35b238418fcd6b5051e37f5
> parent 823656bf8372e55b5b4a81e72921cb78b0be96d7
> author stack <st...@apache.org> Mon Dec 8 15:23:38 2014 -0800
> committer stack <st...@apache.org> Mon Dec 8 15:23:38 2014 -0800
>
> HBASE-12597 Add RpcClient interface and enable changing of RpcClient
> implementation (Jurriaan Mous)
>
> ... where Jurriaan started in on a client-side refactor whose intent was
> being able to slot in an async client.
>
> Before that was HBASE-10322 which added RPC_CODEC_CONF_KEY. Previous to
> this again, IIRC, Anoop I believe noticed that we werent' cellblocking by
> default and fixed it.
>
> We've been cellblocking by default ever since.
>
>
> > The consequence of this 'no codec' is that we will pb all RPC payload and
> > not using cell blocks.
> >
> >
> Exactly.
>
> I used to think it critical we support this mode for the python messers or
> whoever who wanted to put together a quick client; they wouldn't have to
> rig a non-native cellblock decoder. Rare if ever has anyone made use of
> this facility it seems.


Make sense.


>
> > In the test cases, after these many releases, there is no test that
> > excercises this special case.
> > The code path we test are mostly with a valid or default
> > 'hbase.client.rpc.codec'.
> > The other code path is probably sitting there rotten.
> >
> > For example,
> >
> > In MultiServerCallable:
> >
> >   if (this.cellBlock) {
> > // Build a multi request absent its Cell payload. Send data
> in
> > cellblocks.
> > regionActionBuilder =
> > RequestConverter.buildNoDataRegionAction(regionName,
> > rms, cells,
> >   regionActionBuilder, actionBuilder, mutationBuilder);
> >   } else {
> > regionActionBuilder =
> > RequestConverter.buildRegionAction(regionName,
> > rms);   ==> Will not be exercised in test..
> >   }
> >
> > Proposal:
> >
> > We remove this 'no hbase.rpc.codec' case and all dependent logic. There
> is
> > a default and user can overwrite the default, but have to provide a valid
> > non-empty value.
> >
>
> Only objection is that it makes it harder writing non-native client. With
> all pb encoded, you could do a first-cut easy enough w/o having to do
> non-java decoder for our cryptic cellblock packing mechanism.
>
>
Make sense.  The intention is good.  It may be hard for us to maintain
th

Re: [VOTE] hbase-thirdparty 2.0.0 RC

[Jerry He]

+1

Did a quick check.

Downloaded the src.tar.gz.   Unzipped it.   As Stack mentioned, there is no
hbase-thirdparty-2.0.0 top level directory. README needs minor update to
the package names.

mvn install.  Success.  Checked binary jars, which got installed correctly.

Switched into hbase master branch.  Applied HBASE-19552.v2.patch.  (The
patch needs some re-base.)
Built successfully.

mvn clean assembly:single
No binaries built.   Only produced
the hbase-thirdparty-2.0.0-src.tar.gz.   Unzipping the
src.tar.gz interestingly shows the correct hbase-thirdparty-2.0.0 top level
layout.


On Fri, Dec 22, 2017 at 3:16 PM, Stack  wrote:

> On Fri, Dec 22, 2017 at 11:26 AM, Misty Stanley-Jones 
> wrote:
>
> > +1
> >
> > Tbh I am unclear how to properly exercise this artifact. Can we get some
> > guidance for the future?
> >
> >
> For sure. Its a bit tough exercising this in isolation, this release in
> particular. Needs patch applied to hbase core that picks up new locations
> to try it. But yeah, going forward, for sure. Thanks for the vote Misty.
>
> S
>
>
>
> > On Dec 19, 2017 1:04 PM, "Mike Drob"  wrote:
> >
> > > HBase Devs,
> > >
> > > In preparation for our hbase-2.0.0-beta releases, it would be
> beneficial
> > to
> > > have updated third-party artifacts.
> > >
> > > These artifacts update the version of netty to 4.1.17 (from 4.1.12) and
> > > change the relocation offset to o.a.h.thirdparty to prevent conflicts
> > with
> > > our other shaded artifacts (relocated during the build).
> > >
> > > This artifact was tested locally against current hbase master branch
> > > with HBASE-19552 applied.
> > >
> > > Source artifact, signatures, and checksums are available at
> > > https://dist.apache.org/repos/dist/dev/hbase/hbase-
> thirdparty/2.0.0RC0/
> > >
> > > Signed git commit for the release candidate available at
> > > https://git-wip-us.apache.org/repos/asf?p=hbase-thirdparty.
> > git;a=commit;h=
> > > 2b55dc792196a12d9a365a758a518f26c459391e
> > >
> > > Maven repository available at
> > > https://repository.apache.org/content/repositories/orgapachehbase-1187
> > >
> > > This vote will remain open for at least 72 hours. Please review the
> > > artifacts and cast your votes!
> > >
> > > Here's my +1 (non-binding)
> > >
> > > Thanks,
> > > Mike
> > >
> >
>

[ANNOUNCE] Please welcome new HBase committer YI Liang

[Jerry He]

On behalf of the Apache HBase PMC, I am pleased to announce that
Yi Liang has accepted the PMC's invitation to become a committer
on the project.

We appreciate all of Yi's great work thus far and look forward to
his continued involvement.

Please join me in congratulating Yi!

--
Thanks,
Jerry

Re: Cleanup and remove the code path where is no hbase.client.rpc.codec

[Jerry He]

Ok.  I will open a JIRA.

Thanks,

Jerry

On Wed, Dec 20, 2017 at 7:36 AM, Ted Yu <yuzhih...@gmail.com> wrote:

> Jerry:
> Your proposal sounds good.
>
> On Tue, Dec 19, 2017 at 9:55 PM, Jerry He <jerry...@gmail.com> wrote:
>
> > AbstractTestIPC is a good test. And it will continue to work after this
> > proposed change, because we still want the server to be able to handle no
> > codec (and pb) case, for backward compatibility.
> > The proposal is to remove the support of no codec from the RpcClient at
> the
> > moment.
> > There will always be a default codec, and whoever is using the existence
> of
> > codec to decide pb or no pb will always go pb now.
> >
> > Thanks.
> >
> > Jerry
> >
> >
> > On Tue, Dec 19, 2017 at 9:18 PM, ramkrishna vasudevan <
> > ramkrishna.s.vasude...@gmail.com> wrote:
> >
> > > So the proposal is to avoid the empty config and have a better defined
> > > config if we need No pb way? Ya I agree to it if this empty way seems
> odd
> > > to config.
> > > Any non - java client what will be the value for this config?
> > >
> > > Regards
> > > Ram
> > >
> > > On Wed, Dec 20, 2017 at 8:40 AM, 张铎(Duo Zhang) <palomino...@gmail.com>
> > > wrote:
> > >
> > > > See AbstractTestIPC, there is a testNoCodec. But I agree that we
> should
> > > > have a default fallback codec always.
> > > >
> > > > 2017-12-20 11:02 GMT+08:00 Jerry He <jerry...@gmail.com>:
> > > >
> > > > > RPC_CODEC_CONF_KEY 'hbase.client.rpc.codec' is a property we use on
> > the
> > > > > client side to determine the RPC codec.
> > > > >
> > > > > It currently has a strange logic. Whereas the default is
> > KeyValueCodec,
> > > > we
> > > > > allow  a user to specify an empty string "" as the a way to
> indicate
> > > > there
> > > > > is no codec class and we should not use any.
> > > > >
> > > > >   Codec getCodec() {
> > > > > // For NO CODEC, "hbase.client.rpc.codec" must be configured
> with
> > > > empty
> > > > > string AND
> > > > > // "hbase.client.default.rpc.codec" also -- because default is
> > to
> > > do
> > > > > cell block encoding.
> > > > > String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> > > > > getDefaultCodec(this.conf));
> > > > > if (className == null || className.length() == 0) {
> > > > >   return null;
> > > > > }
> > > > > try {
> > > > >   return (Codec) Class.forName(className).newInstance();
> > > > > } catch (Exception e) {
> > > > >   throw new RuntimeException("Failed getting codec " +
> className,
> > > e);
> > > > > }
> > > > >   }
> > > > >
> > > > > I don't know the original reason for having this.
> > > > > The consequence of this 'no codec' is that we will pb all RPC
> payload
> > > and
> > > > > not using cell blocks.
> > > > >
> > > > > In the test cases, after these many releases, there is no test that
> > > > > excercises this special case.
> > > > > The code path we test are mostly with a valid or default
> > > > > 'hbase.client.rpc.codec'.
> > > > > The other code path is probably sitting there rotten.
> > > > >
> > > > > For example,
> > > > >
> > > > > In MultiServerCallable:
> > > > >
> > > > >   if (this.cellBlock) {
> > > > > // Build a multi request absent its Cell payload. Send
> > data
> > > > in
> > > > > cellblocks.
> > > > > regionActionBuilder =
> > > > > RequestConverter.buildNoDataRegionAction(regionName,
> > > > > rms, cells,
> > > > >   regionActionBuilder, actionBuilder, mutationBuilder);
> > > > >   } else {
> > > > > regionActionBuilder =
> > > > > RequestConverter.buildRegionAction(regionName,
> > > > > rms);   ==> Will not be exercised in test..
> > > > >   }
> > > > >
> > > > > Proposal:
> > > > >
> > > > > We remove this 'no hbase.rpc.codec' case and all dependent logic.
> > There
> > > > is
> > > > > a default and user can overwrite the default, but have to provide a
> > > valid
> > > > > non-empty value.
> > > > > Then we can clean up the code where we choose between pb or no pb.
> > We
> > > > will
> > > > > always do cell block in these cases.
> > > > >
> > > > > There are cases where we currently only do pb, like some of the
> > > > individual
> > > > > ops (append, increment, mutateRow, etc). We can revisit to see if
> > they
> > > > can
> > > > > be non-pb'ed.
> > > > >
> > > > > The proposed change only cleans up the client side (PRC client).
> > > > > I want to keep the server side handling of pb and no-pb both for
> now,
> > > so
> > > > > that the server can accommodate a 'no hbase.rpc.codec' connection
> > > request
> > > > > for now for backward compatibility.
> > > > >
> > > > > Any concerns?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > Jerry
> > > > >
> > > >
> > >
> >
>

Re: Cleanup and remove the code path where is no hbase.client.rpc.codec

[Jerry He]

AbstractTestIPC is a good test. And it will continue to work after this
proposed change, because we still want the server to be able to handle no
codec (and pb) case, for backward compatibility.
The proposal is to remove the support of no codec from the RpcClient at the
moment.
There will always be a default codec, and whoever is using the existence of
codec to decide pb or no pb will always go pb now.

Thanks.

Jerry


On Tue, Dec 19, 2017 at 9:18 PM, ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com> wrote:

> So the proposal is to avoid the empty config and have a better defined
> config if we need No pb way? Ya I agree to it if this empty way seems odd
> to config.
> Any non - java client what will be the value for this config?
>
> Regards
> Ram
>
> On Wed, Dec 20, 2017 at 8:40 AM, 张铎(Duo Zhang) <palomino...@gmail.com>
> wrote:
>
> > See AbstractTestIPC, there is a testNoCodec. But I agree that we should
> > have a default fallback codec always.
> >
> > 2017-12-20 11:02 GMT+08:00 Jerry He <jerry...@gmail.com>:
> >
> > > RPC_CODEC_CONF_KEY 'hbase.client.rpc.codec' is a property we use on the
> > > client side to determine the RPC codec.
> > >
> > > It currently has a strange logic. Whereas the default is KeyValueCodec,
> > we
> > > allow  a user to specify an empty string "" as the a way to indicate
> > there
> > > is no codec class and we should not use any.
> > >
> > >   Codec getCodec() {
> > > // For NO CODEC, "hbase.client.rpc.codec" must be configured with
> > empty
> > > string AND
> > > // "hbase.client.default.rpc.codec" also -- because default is to
> do
> > > cell block encoding.
> > > String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> > > getDefaultCodec(this.conf));
> > > if (className == null || className.length() == 0) {
> > >   return null;
> > > }
> > > try {
> > >   return (Codec) Class.forName(className).newInstance();
> > > } catch (Exception e) {
> > >   throw new RuntimeException("Failed getting codec " + className,
> e);
> > > }
> > >   }
> > >
> > > I don't know the original reason for having this.
> > > The consequence of this 'no codec' is that we will pb all RPC payload
> and
> > > not using cell blocks.
> > >
> > > In the test cases, after these many releases, there is no test that
> > > excercises this special case.
> > > The code path we test are mostly with a valid or default
> > > 'hbase.client.rpc.codec'.
> > > The other code path is probably sitting there rotten.
> > >
> > > For example,
> > >
> > > In MultiServerCallable:
> > >
> > >   if (this.cellBlock) {
> > > // Build a multi request absent its Cell payload. Send data
> > in
> > > cellblocks.
> > > regionActionBuilder =
> > > RequestConverter.buildNoDataRegionAction(regionName,
> > > rms, cells,
> > >   regionActionBuilder, actionBuilder, mutationBuilder);
> > >   } else {
> > > regionActionBuilder =
> > > RequestConverter.buildRegionAction(regionName,
> > > rms);   ==> Will not be exercised in test..
> > >   }
> > >
> > > Proposal:
> > >
> > > We remove this 'no hbase.rpc.codec' case and all dependent logic. There
> > is
> > > a default and user can overwrite the default, but have to provide a
> valid
> > > non-empty value.
> > > Then we can clean up the code where we choose between pb or no pb.  We
> > will
> > > always do cell block in these cases.
> > >
> > > There are cases where we currently only do pb, like some of the
> > individual
> > > ops (append, increment, mutateRow, etc). We can revisit to see if they
> > can
> > > be non-pb'ed.
> > >
> > > The proposed change only cleans up the client side (PRC client).
> > > I want to keep the server side handling of pb and no-pb both for now,
> so
> > > that the server can accommodate a 'no hbase.rpc.codec' connection
> request
> > > for now for backward compatibility.
> > >
> > > Any concerns?
> > >
> > > Thanks.
> > >
> > > Jerry
> > >
> >
>

Cleanup and remove the code path where is no hbase.client.rpc.codec

[Jerry He]

RPC_CODEC_CONF_KEY 'hbase.client.rpc.codec' is a property we use on the
client side to determine the RPC codec.

It currently has a strange logic. Whereas the default is KeyValueCodec, we
allow  a user to specify an empty string "" as the a way to indicate there
is no codec class and we should not use any.

  Codec getCodec() {
// For NO CODEC, "hbase.client.rpc.codec" must be configured with empty
string AND
// "hbase.client.default.rpc.codec" also -- because default is to do
cell block encoding.
String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
getDefaultCodec(this.conf));
if (className == null || className.length() == 0) {
  return null;
}
try {
  return (Codec) Class.forName(className).newInstance();
} catch (Exception e) {
  throw new RuntimeException("Failed getting codec " + className, e);
}
  }

I don't know the original reason for having this.
The consequence of this 'no codec' is that we will pb all RPC payload and
not using cell blocks.

In the test cases, after these many releases, there is no test that
excercises this special case.
The code path we test are mostly with a valid or default
'hbase.client.rpc.codec'.
The other code path is probably sitting there rotten.

For example,

In MultiServerCallable:

  if (this.cellBlock) {
// Build a multi request absent its Cell payload. Send data in
cellblocks.
regionActionBuilder =
RequestConverter.buildNoDataRegionAction(regionName,
rms, cells,
  regionActionBuilder, actionBuilder, mutationBuilder);
  } else {
regionActionBuilder =
RequestConverter.buildRegionAction(regionName,
rms);   ==> Will not be exercised in test..
  }

Proposal:

We remove this 'no hbase.rpc.codec' case and all dependent logic. There is
a default and user can overwrite the default, but have to provide a valid
non-empty value.
Then we can clean up the code where we choose between pb or no pb.  We will
always do cell block in these cases.

There are cases where we currently only do pb, like some of the individual
ops (append, increment, mutateRow, etc). We can revisit to see if they can
be non-pb'ed.

The proposed change only cleans up the client side (PRC client).
I want to keep the server side handling of pb and no-pb both for now, so
that the server can accommodate a 'no hbase.rpc.codec' connection request
for now for backward compatibility.

Any concerns?

Thanks.

Jerry

[jira] [Created] (HBASE-19557) Build and release source jars for hbase-shaded-client and others

[Jerry He (JIRA)]

Jerry He created HBASE-19557:


 Summary: Build and release source jars for hbase-shaded-client and 
others
 Key: HBASE-19557
 URL: https://issues.apache.org/jira/browse/HBASE-19557
 Project: HBase
  Issue Type: Improvement
Affects Versions: 1.2.6, 1.3.1
Reporter: Jerry He


It seems that currently we don't build and release source jars for 
hbase-shaded-client (and server or mapreduce).  IDEs will complain from the 
dependent users. We should provide them.

http://central.maven.org/maven2/org/apache/hbase/hbase-shaded-client/1.3.1/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HBASE-19171) Update package references to match new shaded offset in hbase-thirdparty

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-19171.
--
   Resolution: Duplicate
Fix Version/s: (was: 2.0.0-beta-1)

Resolve as dup of the other task [~mdrob] is working on.

> Update package references to match new shaded offset in hbase-thirdparty
> 
>
> Key: HBASE-19171
> URL: https://issues.apache.org/jira/browse/HBASE-19171
> Project: HBase
>  Issue Type: Sub-task
>        Reporter: Jerry He
>Priority: Critical
>
> This has dependency on the parent task, and can only be done after a new 
> hbase-thirdparty release.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] hbase-thirdparty versioning

[Jerry He]

Is it an intention to use the same version as hbase 2. Or is it just
coincidental, and we can not really peg them?

On Tue, Nov 28, 2017 at 4:08 PM Josh Elser  wrote:

>
>
> On 11/28/17 4:54 PM, Stack wrote:
> > On Tue, Nov 28, 2017 at 1:14 PM, Mike Drob  wrote:
> >
> >> Wanted to get some input on the versioning scheme for our
> hbase-thirdparty
> >> artifacts.
> >>
> >> We are moving all of the relocation from o.a.h.shaded to
> o.a.h.thirdparty
> >> due to conflicts with our non-thirdparty shaded libraries. Currently,
> the
> >> next release is slated to be of the thirdparty libs is slated to be
> 1.0.2,
> >> however the package change seems like a big deal.
> >>
> >> I propose that we go to 1.1.0 or 2.0.0 even with this. Version numbers
> are
> >> cheap, we won't run out, so we can afford to be aggressive with
> >> incrementing them. And we don't have to worry about other users of this,
> >> since it's all designed to be internal.
> >>
> >> Thoughts?
> >>
> >>
> > 2.0.0
>
> +1
>

Re: Fix version maintenance for 1.4.0

[Jerry He]

It is a good discussion.  I had confusions in rare cases where I couldn't
rely on the JIRA clearly if a fix was in a release.  It will be really nice
to explicitly document the implicit assumptions.

Is there anything or change that committers need to do?  For example, don't
mark 'fixed in 1.5' on the JIRA until 1.4.0 is out, and don't mark 3.0
until 2.0 is out?  Or it will be up to RMs' batch job?

Thanks.

Jerry

On Thu, Nov 9, 2017 at 12:37 PM Stack  wrote:

> On Thu, Nov 9, 2017 at 12:22 PM, Mike Drob  wrote:
>
> > By this same token, there are a lot of issues with fix version
> 2.0.0-alphaX
> > or -betaY and also 3.0.0
> >
> > Should we drop the 3.0.0 from these?
> >
> >
> Yes (says the 2.0.0 RM). I've been doing this as I come across them.
>
> I filed HBASE-19230 to write-up what we've said out loud about our practice
> here.
>
> St.Ack
>
>
>
>
>
> > On Thu, Nov 9, 2017 at 1:42 PM, Andrew Purtell 
> > wrote:
> >
> > > > once 1.4.0 RC0 comes out, we need to include 1.5.0 because if RC0
> > passes,
> > > such an issue will actually be in 1.4.1 and not 1.4.0
> > >
> > > Ok, I'll remember that.
> > >
> > >
> > > On Thu, Nov 9, 2017 at 11:40 AM, Sean Busbey 
> wrote:
> > >
> > > > That all sounds correct. The one edge case is that when the .0
> release
> > > > hasn't been cut yet but RCs exist, it's important to include in fix
> > > > versions any releases that would need to be included if the current
> RC
> > > > passed.
> > > >
> > > > e.g. once 1.4.0 RC0 comes out, we need to include 1.5.0 because if
> RC0
> > > > passes, such an issue will actually be in 1.4.1 and not 1.4.0. I'm
> not
> > > > sure if we should set 1.4.0 or 1.4.1 in such a case. When prepping
> for
> > > > 1.2.0 I tried to account for folks who had picked either 1.2.0 or
> > > > 1.2.1 when generating the next RC, by correcting fix versions as
> > > > needed.
> > > >
> > > > On Thu, Nov 9, 2017 at 1:28 PM, Dave Latham 
> > wrote:
> > > > > Cool.  Don't mean to suggest this is a change or new, just thinking
> > > > through
> > > > > and writing down what I think I've observed and folks seem to be
> > doing,
> > > > > which makes sense.
> > > > > I don't have the bandwidth at the moment to figure out the doc
> format
> > > and
> > > > > go through the process to create a patch, but if any of the above
> is
> > > > > helpful, would be happy for anyone inclined to get it into the doc.
> > > > >
> > > > > On Thu, Nov 9, 2017 at 11:24 AM, Andrew Purtell <
> apurt...@apache.org
> > >
> > > > wrote:
> > > > >
> > > > >> > Yes, those are the rules I applied.
> > > > >>
> > > > >> To clarify: Those are the rules I applied after going back and
> > adding
> > > > back
> > > > >> fixversions such that the set {1.3.1, ...} becomes {1.4.0, 1.3.1,
> > ...}
> > > > as
> > > > >> Sean suggested.
> > > > >>
> > > > >> Now, 1.4.0 was dropped only if there is a fixversion 1.3.0 in the
> > set.
> > > > And,
> > > > >> 1.5.0 was dropped wherever we have 1.4.0 in the set,  and that is
> > > > currently
> > > > >> everywhere because at this moment branch-1.4 == branch-1, but will
> > > > begin to
> > > > >> diverge as soon as 1.4.0 is released.
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Thu, Nov 9, 2017 at 11:21 AM, Andrew Purtell <
> > apurt...@apache.org>
> > > > >> wrote:
> > > > >>
> > > > >> > Yes, those are the rules I applied.
> > > > >> >
> > > > >> > While I was in there adjusting fixversions I noticed that
> > generally
> > > we
> > > > >> > follow exactly that approach for numbering. I think we inherit
> it
> > > from
> > > > >> > Hadoop, because our old timers came from that community.
> > > > >> >
> > > > >> > Dave - Please consider submitting a patch for our online book
> for
> > > > this,
> > > > >> if
> > > > >> > the text doesn't already exist in there somewhere.
> > > > >> >
> > > > >> > On Thu, Nov 9, 2017 at 11:17 AM, Dave Latham <
> lat...@davelink.net
> > >
> > > > >> wrote:
> > > > >> >
> > > > >> >> +1 for making it as simple as possible to determine if a given
> > fix
> > > is
> > > > >> in a
> > > > >> >> given release purely from the release numbers, without having
> to
> > > > consult
> > > > >> >> the dates of when branches were made or release candidates were
> > > > built.
> > > > >> >>
> > > > >> >> I think the rules should be
> > > > >> >>
> > > > >> >> Fix version of X.Y.Z => fixed in all releases X.Y.Z' where Z'
> >=
> > Z
> > > > >> >> Fix version of X.Y.0 => fixed in all releases X.Y'.* where Y'
> >=
> > Y
> > > > >> >> Fix version of X.0.0 => fixed in all releases X'.*.* where X'
> >=
> > X
> > > > >> >>
> > > > >> >> By this policy, fix version of 1.3.0 implies 1.4.0, but 1.3.2
> > does
> > > > not
> > > > >> >> imply 1.4.0, as we could not tell purely from the numbers which
> > > > release
> > > > >> >> came first.
> > > > >> >>
> > > > >> >> To track a fix then, I think that means that there should
> usually
> > > be
> > 

[jira] [Created] (HBASE-19171) Update package references to match new shaded offset in hbase-thirdparty

[Jerry He (JIRA)]

Jerry He created HBASE-19171:


 Summary: Update package references to match new shaded offset in 
hbase-thirdparty
 Key: HBASE-19171
 URL: https://issues.apache.org/jira/browse/HBASE-19171
 Project: HBase
  Issue Type: Sub-task
Reporter: Jerry He
Priority: Critical
 Fix For: 2.0.0


This has dependency on the parent task, and can only be done after a new 
hbase-thirdparty release.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-19170) [hbase-thirdparty] Change the relocation offset of shaded artifacts

[Jerry He (JIRA)]

Jerry He created HBASE-19170:


 Summary: [hbase-thirdparty] Change the relocation offset of shaded 
artifacts
 Key: HBASE-19170
 URL: https://issues.apache.org/jira/browse/HBASE-19170
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.0.1
Reporter: Jerry He
Priority: Critical
 Fix For: 1.0.2


On the dev@hbase list, we conclude that we need to change the relocation offset 
in hbase-thirdparty to avoid shading conflicts with the other hbase shaded 
components (hbase-shaded-client and hbase-shaded-mapreduce components).
https://lists.apache.org/thread.html/1aa5d1d7f6d176df49e72096926b011cafe1315932515346d06e8342@%3Cdev.hbase.apache.org%3E
The suggestion is to use "o.a.h.hbase.thirdparty" in hbase-thirdparty to 
differentiate between "shaded" for downstream of us vs "thirdparty" for our 
internal use.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Problem with double shading in hbase-thirdparty and hbase-shaded-client?

[Jerry He]

o.a.h.hbase.thirdparty does look like a better choice.
Let me open two JIRAs to change the hbase-thirdparty and then all the
references in the main code (more tedious part).
Thanks, guys!



On Thu, Nov 2, 2017 at 1:54 PM Sean Busbey <bus...@apache.org> wrote:

> I think something like o.a.h.hbase.thirdparty might be better. then we
> can differentiate between "shaded" for downstream of us vs
> "thirdparty" for our internal use.
>
> On Thu, Nov 2, 2017 at 3:52 PM, Stack <st...@duboce.net> wrote:
> > Good one Jerry He.
> >
> > I can try changing the offset in hbase-thirdparty in a new release
> > (o.a.hbase.shaded?). There used to be a reason why it had to be
> > o.a.h.hbase.shaded but IIRC it went away after all shaded dependencies
> made
> > it out to hbase-thirdparty (including the pb3 that we used to have
> checked
> > into core).
> >
> > S
> >
> > On Thu, Nov 2, 2017 at 11:45 AM, Jerry He <jerry...@gmail.com> wrote:
> >
> >> Mike from another thread asked about shading jackson into
> >> hbase-thirdparty. That reminded me of a question I was thinking
> >> recently,
> >> in conjunction with HBASE-18800 "[Propaganda] Push shaded hbase-client
> >> as gateway to an hbase cluster going forward".
> >>
> >> We have shaded artifacts in hbase-thirdparty
> >> (hbase-shaded-miscellaneous, hbase-shaded-protobuf and
> >> hbase-shaded-netty). The purpose is to shade and bundle the jars with
> >> versions hbase uses internally.
> >>
> >> Then in hbase-shaded-client we do shading again for some of the same
> >> jars, but they are more non-hbase, transitive dependencies.
> >> For example, in hbase-shaded-miscellaneous, we shade guava 22. But
> >> hbase-client will bring in guava 11 from hadoop-common. In
> >> hbase-shaded-protobuf, we shade protobuf 3. But hadoop brings in
> >> protobuf 2.5.
> >> We will be doing shading of all these in hbase-shaded-client (and the
> >> hbase-shaded-mapreduce), with the same relocation
> >> (org.apache.hadoop.hbase.shaded).
> >>
> >> The end result is shaded guava 22 or guava 11?  Protobuf 3 or protobuf
> 2.5?
> >> I checked hbase-shaded-client and found it had the shaded guava 22
> classes.
> >> Looks like the shading process will only add one version of the
> >> classes with the same name.
> >>
> >> This will potentially have problem, and potentially make it unusable
> >> in unlucky situations.
> >>
> >> I bring it up here for awareness and discussion. First, is this really
> >> a problem?
> >> Second, how to fix it?  Maybe a different relocation offset in
> >> hbase-thirdparty?
> >>
> >> Thanks,
> >>
> >> Jerry
> >>
>

Problem with double shading in hbase-thirdparty and hbase-shaded-client?

[Jerry He]

Mike from another thread asked about shading jackson into
hbase-thirdparty. That reminded me of a question I was thinking
recently,
in conjunction with HBASE-18800 "[Propaganda] Push shaded hbase-client
as gateway to an hbase cluster going forward".

We have shaded artifacts in hbase-thirdparty
(hbase-shaded-miscellaneous, hbase-shaded-protobuf and
hbase-shaded-netty). The purpose is to shade and bundle the jars with
versions hbase uses internally.

Then in hbase-shaded-client we do shading again for some of the same
jars, but they are more non-hbase, transitive dependencies.
For example, in hbase-shaded-miscellaneous, we shade guava 22. But
hbase-client will bring in guava 11 from hadoop-common. In
hbase-shaded-protobuf, we shade protobuf 3. But hadoop brings in
protobuf 2.5.
We will be doing shading of all these in hbase-shaded-client (and the
hbase-shaded-mapreduce), with the same relocation
(org.apache.hadoop.hbase.shaded).

The end result is shaded guava 22 or guava 11?  Protobuf 3 or protobuf 2.5?
I checked hbase-shaded-client and found it had the shaded guava 22 classes.
Looks like the shading process will only add one version of the
classes with the same name.

This will potentially have problem, and potentially make it unusable
in unlucky situations.

I bring it up here for awareness and discussion. First, is this really
a problem?
Second, how to fix it?  Maybe a different relocation offset in hbase-thirdparty?

Thanks,

Jerry

Re: [DISCUSS] hbase-2.0.0 compatibility expectations

[Jerry He]

Thanks.

I filed a JIRA to track it.
https://issues.apache.org/jira/browse/HBASE-19145

On Mon, Oct 30, 2017 at 9:36 PM, Stack <st...@duboce.net> wrote:
> On Mon, Oct 30, 2017 at 11:46 AM, Jerry He <jerry...@gmail.com> wrote:
>
>> Hi, Stack
>>
>> Coming back to this thread, i have meant to ask this question long ago.
>> Do we support hbase-2 client going against hbase-1 server?
>>
>
> It has not been an objective. It might work doing basic Client API on a
> later branch-1 but will fail doing Admin functions (and figuring if a Table
> is online). I've not tried it Jerry. If it was a small thing to make it
> work, lets get it in.
>
> St.Ack
>
>
>
>> We seem to be fine mix-and-match the clients and servers within the
>> hbase-1 releases.   And IIRC hbase-1 client is ok against 0.98 server.
>> Suppose I have a product  that depends and bundles HBase client. I
>> want to upgrade the dependency to hbase-2 so that it can take
>> advantages of and claim support of hbase-2.
>> But does it mean that I will need drop the claims that the new version
>> of the product support any hbase-1 backend?
>>
>> Thanks.
>>
>>
>>
>> On Tue, Sep 12, 2017 at 10:21 AM, Stack <st...@duboce.net> wrote:
>> > Let me put this one on this thread, G1GC on by default in hbase2?
>> > St.Ack
>> >
>> > On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
>> >
>> >> What are our expectations regards compatibility between hbase1 and
>> hbase2?
>> >>
>> >> Lets have a chat about it. Here are some goal posts.
>> >>
>> >> + You have to upgrade to hbase-1.x before you can migrate to hbase-2. No
>> >> migration from < hbase-1 (Is this too onerous? Should we support 0.98 =>
>> >> 2.0?).
>> >> + You do NOT have to upgrade to the latest release of hbase1 to migrate
>> to
>> >> hbase2; being up on hbase-1.0.0+ will be sufficient.
>> >> + You'll have to update your hbase1 coprocessors to deploy them on
>> hbase2.
>> >> A bunch of CP API has/will change by the time hbase2 comes out; e.g.
>> >> watching for region split on RegionServer no longer makes sense given
>> >> Master runs all splits now.
>> >> + An hbase1 client can run against an hbase2 cluster but it will only be
>> >> able to do DML (Get/Put/Scan, etc.). We do not allow being able to do
>> admin
>> >> ops using an hbase1 Admin client against an hbase2 cluster. We have some
>> >> egregious API violations in branch-1; e.g. we have protobuf in our API
>> (See
>> >> HBASE-15607). The notion is that we can't afford a deprecation cycle
>> >> purging this stuff from our Admin API.
>> >>
>> >> What you all think?
>> >>
>> >> St.Ack
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>>

[jira] [Created] (HBASE-19145) Look into hbase-2 client going to hbase-1 server

[Jerry He (JIRA)]

Jerry He created HBASE-19145:


 Summary: Look into hbase-2 client going to hbase-1 server
 Key: HBASE-19145
 URL: https://issues.apache.org/jira/browse/HBASE-19145
 Project: HBase
  Issue Type: Task
Affects Versions: 2.0.0-beta-1
Reporter: Jerry He
Priority: Major


>From the "[DISCUSS] hbase-2.0.0 compatibility expectations" thread.

Do we support hbase-2 client going against hbase-1 server?
We seem to be fine mix-and-match the clients and servers within the
hbase-1 releases.   And IIRC hbase-1 client is ok against 0.98 server.
Suppose I have a product  that depends and bundles HBase client. I
want to upgrade the dependency to hbase-2 so that it can take
advantages of and claim support of hbase-2.
But does it mean that I will need drop the claims that the new version
of the product support any hbase-1 backend?

It has not been an objective. It might work doing basic Client API on a
later branch-1 but will fail doing Admin functions (and figuring if a Table
is online).  If it was a small thing to make it
work, lets get it in.

Let's look into it to see what works and what not.  Have a statement at least.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [ANNOUNCE] Please welcome new HBase committer Rahul Gidwani (churro)

[Jerry He]

Congrats, and welcome Rahul!

On Sat, Oct 28, 2017 at 10:34 AM, Abhishek Singh Chouhan
 wrote:
> Congrats and welcome Rahul!!! :)
>
> On Sat, Oct 28, 2017 at 9:10 AM Mike Drob  wrote:
>
>> Welcome churro!
>>
>> On Fri, Oct 27, 2017 at 4:49 PM, Sukumar Maddineni <
>> smaddin...@salesforce.com> wrote:
>>
>> > Congrats Rahul...
>> >
>> > On Fri, Oct 27, 2017 at 2:48 PM, Ted Yu  wrote:
>> >
>> > > Congratulations, Rahul !
>> > >
>> > > On Fri, Oct 27, 2017 at 2:44 PM, Andrew Purtell 
>> > > wrote:
>> > >
>> > > > On behalf of the Apache HBase PMC, I am pleased to announce that
>> Rahul
>> > > > Gidwani has accepted the PMC's invitation to become a committer on
>> the
>> > > > project.
>> > > >
>> > > > We appreciate all of Rahul's great work thus far and look forward to
>> > > > continued involvement.
>> > > >
>> > > > Please join me in congratulating Rahul!
>> > > >
>> > > > --
>> > > > Best regards,
>> > > > Andrew
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> >
>> > 
>> >
>>

Re: [DISCUSS] hbase-2.0.0 compatibility expectations

[Jerry He]

Hi, Stack

Coming back to this thread, i have meant to ask this question long ago.
Do we support hbase-2 client going against hbase-1 server?
We seem to be fine mix-and-match the clients and servers within the
hbase-1 releases.   And IIRC hbase-1 client is ok against 0.98 server.
Suppose I have a product  that depends and bundles HBase client. I
want to upgrade the dependency to hbase-2 so that it can take
advantages of and claim support of hbase-2.
But does it mean that I will need drop the claims that the new version
of the product support any hbase-1 backend?

Thanks.



On Tue, Sep 12, 2017 at 10:21 AM, Stack  wrote:
> Let me put this one on this thread, G1GC on by default in hbase2?
> St.Ack
>
> On Wed, Aug 2, 2017 at 4:38 PM, Stack  wrote:
>
>> What are our expectations regards compatibility between hbase1 and hbase2?
>>
>> Lets have a chat about it. Here are some goal posts.
>>
>> + You have to upgrade to hbase-1.x before you can migrate to hbase-2. No
>> migration from < hbase-1 (Is this too onerous? Should we support 0.98 =>
>> 2.0?).
>> + You do NOT have to upgrade to the latest release of hbase1 to migrate to
>> hbase2; being up on hbase-1.0.0+ will be sufficient.
>> + You'll have to update your hbase1 coprocessors to deploy them on hbase2.
>> A bunch of CP API has/will change by the time hbase2 comes out; e.g.
>> watching for region split on RegionServer no longer makes sense given
>> Master runs all splits now.
>> + An hbase1 client can run against an hbase2 cluster but it will only be
>> able to do DML (Get/Put/Scan, etc.). We do not allow being able to do admin
>> ops using an hbase1 Admin client against an hbase2 cluster. We have some
>> egregious API violations in branch-1; e.g. we have protobuf in our API (See
>> HBASE-15607). The notion is that we can't afford a deprecation cycle
>> purging this stuff from our Admin API.
>>
>> What you all think?
>>
>> St.Ack
>>
>>
>>
>>
>>
>>
>>
>>
>>

[jira] [Resolved] (HBASE-19003) Make sure all balancer actions respect decommissioned server

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-19003.
--
Resolution: Done

> Make sure all balancer actions respect decommissioned server
> 
>
> Key: HBASE-19003
> URL: https://issues.apache.org/jira/browse/HBASE-19003
> Project: HBase
>  Issue Type: Sub-task
>        Reporter: Jerry He
>    Assignee: Jerry He
> Fix For: 2.0.0-beta-1
>
>
> There have been questions raised in HBASE-10367 and other related JIRAs. We 
> want to make sure all aspects of the balancer respect the draining flag. We 
> will have a good look, and fix if any violation is found.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-19096) Add RowMutions batch support in AsyncTable

[Jerry He (JIRA)]

Jerry He created HBASE-19096:


 Summary: Add RowMutions batch support in AsyncTable
 Key: HBASE-19096
 URL: https://issues.apache.org/jira/browse/HBASE-19096
 Project: HBase
  Issue Type: Sub-task
Reporter: Jerry He
Assignee: Jerry He


Batch support for RowMutations has been added in the Table interface, but is 
not in AsyncTable. This JIRA will add it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [ANNOUNCE] New HBase committer Zheng Hu

[Jerry He]

Congrats and welcome!


On Mon, Oct 23, 2017 at 10:29 AM, Huaxiang Sun  wrote:
> Congratulations, Zheng!
>

[jira] [Resolved] (HBASE-17369) Add ACL to the new region server drain related API

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-17369.
--
Resolution: Implemented

Implemented as part of HBASE-10367.

> Add ACL to the new region server drain related API
> --
>
> Key: HBASE-17369
> URL: https://issues.apache.org/jira/browse/HBASE-17369
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>    Reporter: Jerry He
>Priority: Critical
>
> Add ACL to the new region server drain related API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-19021) Restore a few important missing logics for balancer in 2.0

[Jerry He (JIRA)]

Jerry He created HBASE-19021:


 Summary: Restore a few important missing logics for balancer in 2.0
 Key: HBASE-19021
 URL: https://issues.apache.org/jira/browse/HBASE-19021
 Project: HBase
  Issue Type: Bug
Reporter: Jerry He
Assignee: Jerry He
Priority: Critical


After looking at the code, and some testing, I see the following things are 
missing for balancer to work properly after AMv2.

# hbase.master.loadbalance.bytable is not respected. It is always 'bytable'. 
Previous default is cluster wide, not by table.
# Servers with no assignments is not added for balance consideration.
# Crashed server is not removed from the in-memory server map in RegionStates, 
which affects balance.
# Draining marker is not respected when balance.

Also try to re-enable {{TestRegionRebalancing}}, which has a 
{{testRebalanceOnRegionServerNumberChange}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-19003) Make sure all balancer actions respect decommissioned server

[Jerry He (JIRA)]

Jerry He created HBASE-19003:


 Summary: Make sure all balancer actions respect decommissioned 
server
 Key: HBASE-19003
 URL: https://issues.apache.org/jira/browse/HBASE-19003
 Project: HBase
  Issue Type: Sub-task
Reporter: Jerry He


There have been questions raised in HBASE-10367 and other related JIRAs. We 
want to make sure all aspects of the balancer respect the draining flag. We 
will have a good look, and fix if any violation is found.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HBASE-16010) Put draining function through Admin API

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-16010.
--
Resolution: Fixed

> Put draining function through Admin API
> ---
>
> Key: HBASE-16010
> URL: https://issues.apache.org/jira/browse/HBASE-16010
> Project: HBase
>  Issue Type: Improvement
>        Reporter: Jerry He
>Assignee: Matt Warhaftig
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16010-v3.patch, hbase-16010-v1.patch, 
> hbase-16010-v2.patch
>
>
> Currently, there is no Amdin API for draining function. Client has to 
> interact directly with Zookeeper draining node to add and remove draining 
> servers.
> For example, in draining_servers.rb:
> {code}
>   zkw = org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.new(config, 
> "draining_servers", nil)
>   parentZnode = zkw.drainingZNode
>   begin
> for server in servers
>   node = ZKUtil.joinZNode(parentZnode, server)
>   ZKUtil.createAndFailSilent(zkw, node)
> end
>   ensure
> zkw.close()
>   end
> {code}
> This is not good in cases like secure clusters with protected Zookeeper nodes.
> Let's put draining function through Admin API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HBASE-11608) Add synchronous split

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-11608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-11608.
--
Resolution: Duplicate

Finally, marked as dup.  It has been added in HBASE-18229.

> Add synchronous split
> -
>
> Key: HBASE-11608
> URL: https://issues.apache.org/jira/browse/HBASE-11608
> Project: HBase
>  Issue Type: New Feature
>  Components: Admin
>Affects Versions: 0.99.0, 0.98.5
>Reporter: Jerry He
>
> Users have been asking for this. We have an internal requirement for this as 
> well.
> The goal is a provide a Admin API (and shell command) so that users can 
> request to split a region or table and get the split completion result 
> synchronously.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: HBase Backup Mode

[Jerry He]

I stumbled across it a long time ago as well.

It was brought in by the very original hfile archiving work (put obsolete
hfiles temperately in archive dir).

Probably it was intended to show, as an example, how to make use of the
archiving work to backup hbase table (with a tracker on zk node of table
names and a new hfile cleaner, etc).

I don't think we should keep it there anymore, particularly after the new
backup work.  We can move it to the hbase-examples if we want to keep it at
all.

Thanks,

Jerry




On Fri, Sep 1, 2017 at 4:13 AM, Lars George  wrote:
> Hi,
>
> I stumbled across this:
>
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveManager.java
>
> I am wondering what the state of this is. Also, why is this in an
> _example_ package, if it is usable code. Should be in `util` or so
> instead? If it isn't really used at all or just really an example as
> it states, should this be in GitHub instead?
>
> Let me know what you think.
>
> Best,
> Lars

[jira] [Created] (HBASE-18740) Upgrade Zookeeper version in branch 1.4 and branch-1

[Jerry He (JIRA)]

Jerry He created HBASE-18740:


 Summary: Upgrade Zookeeper version in branch 1.4 and branch-1
 Key: HBASE-18740
 URL: https://issues.apache.org/jira/browse/HBASE-18740
 Project: HBase
  Issue Type: Improvement
Reporter: Jerry He
Assignee: Jerry He


Branch 1.4 and branch 1 are still on Zookeeper 3.4.6.
Branch 2 and master branch have upgraded to 3.4.9.

There are some important fixes we'd like to have. See the linked JIRAs.
Another critical fix is ZOOKEEPER-2146, which can be explored maliciously.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HBASE-9417) SecureBulkLoadEndpoint should be folded in core

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-9417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-9417.
-
   Resolution: Duplicate
Fix Version/s: (was: 2.0.0-alpha-3)

> SecureBulkLoadEndpoint should be folded in core
> ---
>
> Key: HBASE-9417
> URL: https://issues.apache.org/jira/browse/HBASE-9417
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver, security
>Reporter: Enis Soztutar
>Priority: Critical
>
> In unsecure bulk loading, the client creates the files to be bulk loaded, and 
> asks the regionservers to do the operation. Bulk loading is performed by a 
> move, which would mean that the hbase user has to have WRITE permissions for 
> the bulk loaded files. If the client who has generated the files is different 
> than the hbase user, this creates an access denied exception if complete bulk 
> load is not run as the hbase user.
> I think even for unsecure mode, we should mimic what SecureBulkLoadEndpoint 
> does, where hbase creates a staging directory and the client hands off the 
> files to that directory with global perms. 
> Update: Now that HBASE-12052 enables running SecureBulkLoadEndpoint even in 
> unsecure deployments, we should consider bringing SecureBulkLoad into core 
> HBase (meaning implement the functionality in RegionServer instead of in the 
> coprocessor). 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] First release candidate for HBase 1.1.12 (RC0) is available

[Jerry He]

+1

- Downloaded the src tarball.
  - Checked the md5 and signature.  Looks good.
  - Built from source.  openjdk version "1.8.0_131"
  - Ran unit test suite.  All passed.
  - mvn apache-rat:check   Passed

- Download the bin tarball.
 -  Checked the md5 and signature.  Looks good.
 -  Extract it.   Layout looks good.
 -  Started a local instance.  Run the 'hbase shell' with some basic table
commands.  Looks good.
 -  Nothing unusual in the logs.
 -  Check master and region server UI.  Looks good.

-  Bulit YCSB 0.12.0 from git source with HBase binding version pointing to
1.1.12.

-  Run YCSB workloads against the local RC instance.  Worked fine.

Thanks,

Jerry


On Tue, Aug 15, 2017 at 7:33 PM, Josh Elser  wrote:
> +1 (binding)
>
> * xsums/sigs OK
> * apache-rat:check OK
> * Poked at L in both src/bin (bin's NOTICE seems bloated with ASLv2
> entries, but this isn't a blocker)
> * Compat report is great (thanks for publishing)
> * Tag is published
>
> Thanks for putting together, Nick!
>
>
> On 8/12/17 6:09 PM, Nick Dimiduk wrote:
>>
>> I'm happy to announce the first release candidate of HBase 1.1.12
>> (HBase-1.1.12RC0) is available for download at https://dist.apache.org/
>> repos/dist/dev/hbase/hbase-1.1.12RC0/
>>
>> Maven artifacts are also available in the staging repository
>> https://repository.apache.org/content/repositories/orgapachehbase-1172
>>
>> Artifacts are signed with my code signing subkey 0xAD9039071C3489BD,
>> available in the Apache keys directory https://people.apache.org/
>> keys/committer/ndimiduk.asc and in our KEYS file
>> http://www-us.apache.org/dist/hbase/KEYS.
>>
>> There's also a signed tag for this release at
>> https://git-wip-us.apache.org/
>> repos/asf?p=hbase.git;a=tag;h=29a930cdcbef2a64b601a110bad72955b3b04afc
>>
>> The detailed source and binary compatibility report vs 1.1.9 has been
>> published for your review, at https://home.apache.org/~
>> ndimiduk/1.1.11_1.1.12RC0_compat_report.html
>>
>> HBase 1.1.12 is the twelfth patch release in the HBase 1.1 line,
>> continuing
>> on the theme of bringing a stable, reliable database to the Hadoop and
>> NoSQL communities. This release includes nearly 10 bug fixes since the
>> 1.1.11 release. Notable correctness fixes include HBASE-9393,
>> HBASE-15691, HBASE-18390,
>> HBASE-18330.
>>
>> The full list of fixes included in this release is available at
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>> projectId=12310753=12340864 and and in the CHANGES.txt file
>> included in the distribution.
>>
>> Please try out this candidate and vote +/-1 by 23:59 Pacific time on
>> Saturday, 2017-08-19 as to whether we should release these artifacts as
>> HBase 1.1.12.
>>
>> Thanks,
>> Nick
>>
>

[jira] [Resolved] (HBASE-18590) branch-1.4 needs a Jenkins commit build job

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-18590.
--
Resolution: Fixed

> branch-1.4 needs a Jenkins commit build job
> ---
>
> Key: HBASE-18590
> URL: https://issues.apache.org/jira/browse/HBASE-18590
> Project: HBase
>  Issue Type: Bug
>        Reporter: Jerry He
>Assignee: Ted Yu
>Priority: Critical
>
> The current HBase-1.4 job is actually branch-1.
> https://builds.apache.org/job/HBase-1.4/
> Need a separate job for branch-1.4.  And rename the current job to HBase-1.5.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Jenkins build for 1.4

[Jerry He]

Then we need another Jenkins job for branch-1 (HBase 1.5)?  Or there
is already one?

Thanks.

Jerry

On Sun, Aug 13, 2017 at 2:38 PM, Ted Yu  wrote:
> Hi,
> https://builds.apache.org/job/HBase-1.4 was pointing to branch-1.
>
> I have made the config change to point to branch-1.4
>
> Thanks to Jerry who reported build break in HBASE-18589

[jira] [Created] (HBASE-18590) branch-1.4 needs a Jenkins commit build job

[Jerry He (JIRA)]

Jerry He created HBASE-18590:


 Summary: branch-1.4 needs a Jenkins commit build job
 Key: HBASE-18590
 URL: https://issues.apache.org/jira/browse/HBASE-18590
 Project: HBase
  Issue Type: Bug
Reporter: Jerry He
Priority: Critical


The current HBase-1.4 job is actually branch-1.
https://builds.apache.org/job/HBase-1.4/

Need a separate job for branch-1.4.  And rename the current job to HBase-1.5.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18589) branch-1.4 build compile is broken

[Jerry He (JIRA)]

Jerry He created HBASE-18589:


 Summary: branch-1.4 build compile is broken
 Key: HBASE-18589
 URL: https://issues.apache.org/jira/browse/HBASE-18589
 Project: HBase
  Issue Type: Bug
Reporter: Jerry He
Assignee: Jerry He
Priority: Critical


{noformat}
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile 
(default-testCompile) on project hbase-server: Compilation failure: Compilation 
failure:
[ERROR] 
/hbase-apache/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRegionSnapshotTask.java:[164,46]
 error: cannot find symbol
[ERROR] symbol:   class SnapshotDescription
[ERROR] location: class HBaseProtos
{noformat}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Notes from dev meetup in Shenzhen, August 5th, 2017

[Jerry He]

Looks like you folks had a good time there.  I wish I could've made it.
Good write-up too.

Thanks.

Jerry

On Mon, Aug 7, 2017 at 10:38 AM, ramkrishna vasudevan
 wrote:
> Thanks for the write up Stack. I could not make it to Shenzhen. Nice to
> know the conference and meet up went great.
>
> Regards
> Ram
>
> On Mon, Aug 7, 2017 at 9:36 PM, Stack  wrote:
>
>> At fancy Huawei headquarters, 10:00-12:00AM or so (with nice coffee and
>> fancy little cake squares provided about half way through the session).
>>
>> For list of attendees, see picture at end of this email.
>>
>> Discussion was mostly in Chinese with about 25% in English plus some
>> gracious sideline translation so the below is patchy. Hopefully you get the
>> gist.
>>
>> For client-side scanner going against hfiles directly; is there a means of
>> being able to pass the permissions from hbase to hdfs?
>>
>> Issues w/ the hbase 99th percentile were brought up. "DynamoDB can do
>> 10ms". How to do better?
>>
>> SSD is not enough.
>>
>> GC messes us up.
>>
>> Will the Distributed Log Replay come back to help improve MTTR? We could
>> redo on new ProcedureV2 basis. ZK timeout is the biggest issue. Do as we
>> used to and just rely on the regionserver heartbeating...
>>
>> Read replica helps w/ MTTR.
>>
>> Ratis incubator project to do a quorum based hbase?
>>
>> Digression on licensing issues around fb wangle and folly.
>>
>> Redo of hbase but quorum based would be another project altogether.
>>
>> Decided to go around the table to talk about concerns and what people are
>> working on.
>>
>> Jieshan wondered what could be done to improve OLAP over hbase.
>>
>> Client side scanner was brought up again as means of skipping RS overhead
>> and doing better OLAP.
>>
>> Have HBase compact to parquet files. Query parquet and hbase.
>>
>> At Huawei, they are using 1.0 hbase. Most problems are assignment. They
>> have .5M regions. RIT is a killer. Double assignment issues. And RIT. They
>> run their own services. Suggested they upgrade to get fixes at least. Then
>> 2.0.
>>
>> Will HBase federate like HDFS? Can Master handle load at large scale? It
>> needs to do federation too?
>>
>> Anyone using Bulk loaded replication? (Yes, it just works so no one talks
>> about it...)
>>
>> Request that fixes be backported to all active branches, not just most
>> current.
>>
>> Andrew was good at backporting... not all RMs are.
>>
>> Too many branches. What should we do?
>>
>> Proliferation of branches makes for too much work.
>>
>> Need to cleanup bugs in 1.3. Make it stable release now.
>>
>> Lets do more active EOL'ing of branches. 1.1?.
>>
>> Hubert asked if we can have clusters where RS are differently capable?
>> i.e. several generations of HW all running in the same cluster.
>>
>> What if fat server goes down.
>>
>> Balancer could take of it all. RS Capacity. Balancer can take it into
>> account.
>> Regionserver labels like YARN labels. Characteristics.
>>
>> Or run it all in docker when heterogeneous cluster. The K8 talk from day
>> before was mentioned; we should all look at being able to deploy in k8 and
>> docker.
>>
>> Lets put out kubernetes blog...(Doing).
>>
>> Alibaba looking at HBase as native YARN app.
>>
>> i/o is hard even when containers.
>>
>> Use autoscaler of K8 when heavy user.
>>
>> Limit i/o use w/ CP. Throttle.
>>
>> Spark and client-side scanner came up again.
>>
>> Snapshot input format in spark.
>>
>> HBase federation came up again. jd.com talking of 3k to 4k nodes in a
>> cluster. Millions of regions. Region assignment is messing them up.
>>
>> Maybe federation is good idea? Argument that it is too much operational
>> conplexity. Can we fix master load w/ splittable meta, etc?
>>
>> Was brought up that even w/ 100s of RS there is scale issue, nvm thousands.
>>
>> Alibaba talked about disaster recovery. Described issue where HDFS has
>> fencing problem during an upgrade. There was no active NN. All RS went down.
>> ZK is another POF. If ZK is not available. Operators were being asked how
>> much longer the cluster was going to be down but they could not answer the
>> question. No indicators from HBase on how much longer it will be down or
>> how many WALs its processed and how many more to go. Operator unable to
>> tell his org how long it would be before it all came back on line. Should
>> say how many regions are online and how many more to do.
>>
>> Alibaba use SQL to lower cost. HBase API is low-level. Row-key
>> construction is tricky. New users make common mistakes. If you don't do
>> schema right, high-performance is difficult.
>>
>> Alibaba are using a subset of Phoenix... simple sql only; throws
>> exceptions if user tries to do joins, etc.., anything but basic ops.
>>
>> HareQL is using hive for meta store.  Don't have data typing in hbase.
>>
>> HareQL could perhaps contribute some piece... or a module in hbase to
>> sql... From phoenix?
>>
>> Secondary index.
>>
>> Client is 

[jira] [Created] (HBASE-18522) Add RowMutations support to Batch

[Jerry He (JIRA)]

Jerry He created HBASE-18522:


 Summary: Add RowMutations support to Batch
 Key: HBASE-18522
 URL: https://issues.apache.org/jira/browse/HBASE-18522
 Project: HBase
  Issue Type: Improvement
Affects Versions: 1.2.6
Reporter: Jerry He
Assignee: Jerry He
 Fix For: 2.0.0, 3.0.0, 1.4.0, 1.5.0


RowMutations is multiple Puts and/or Deletes atomically on a single row. 
Current Batch call does not support RowMutations as part of the batch.
We should add this missing part. We should be able to batch RowMutations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] Planning changes on RegionServer totalRequestCount metrics

[Jerry He]

I like the ideal to clean it up and make it clear.  I had the same
confusion as well.
Will look at the JIRA and comment there as well.

Thanks.

Jerry

On Wed, Aug 2, 2017 at 11:48 PM, Yu Li  wrote:
> Dear all,
>
> Recently in HBASE-18469 
> we found some inconsistency on regionserver request related metrics,
> including:
> 1. totalRequestCount could be less than readRequestCount+writeRequestCount
> 2. For multi request, we count action count into totalRequestCount, while
> for scan with caching we count only one.
>
> To fix the inconsistency, we plan to make below changes:
> 1. Make totalRequestCount only counts rpc request, thus multi request will
> only count as one for totalRequestCount
> 2. Introduce a new metrics in name of "totalRowsRequestCount", which will
> count the DML workloads on RS by row-level action, and for this metrics we
> will count how many rows included for multi and scan-with-caching request.
>
> After the change, there won't be any compatibility issue -- existing
> monitoring system could still work -- only that totalRequestCount will be
> less than previous. And it's recommended to use totalRowsRequestCount to
> check the RS DML workload.
>
> Please kindly let us know if you have any different idea or suggestion
> (operators' opinion is especially welcomed).
>
> Let's make this discussion open for 72 hours and will make the change if no
> objections.
>
> Thanks!
>
> Best Regards,
> Yu

Re: Re: [ANNOUNCE] New HBase committer Vikas Vishwakarma

[Jerry He]

Congrats Vikas!


Jerry

On Mon, Jul 31, 2017 at 3:25 AM, Chunhui Shen  wrote:
> Congrats, Vikas!
>
>
> At 2017-07-31 16:42:01, "Anoop John"  wrote:
>>Congrats Vikas...
>>
>>On Mon, Jul 31, 2017 at 1:48 PM, Yu Li  wrote:
>>> Congratulations, Vikas!
>>>
>>> Best Regards,
>>> Yu
>>>
>>> On 31 July 2017 at 15:41, Jingcheng Du  wrote:
>>>
 Congratulations!

 2017-07-31 14:57 GMT+08:00 Vikas Vishwakarma :

 > Thanks everyone !
 >
 > On Mon, Jul 31, 2017 at 10:16 AM, Pankaj kr 
 wrote:
 >
 > > Congratulations Vikas..!!
 > >
 > > Thanks & Regards,
 > > Pankaj
 > >
 > > HUAWEI TECHNOLOGIES CO.LTD.
 > > Huawei Tecnologies India Pvt. Ltd.
 > > Near EPIP Industrial Area, Kundalahalli Village
 > > Whitefield, Bangalore-560066
 > > www.huawei.com
 > > 
 > > -
 > > This e-mail and its attachments contain confidential information from
 > > HUAWEI, which
 > > is intended only for the person or entity whose address is listed
 above.
 > > Any use of the
 > > information contained herein in any way (including, but not limited to,
 > > total or partial
 > > disclosure, reproduction, or dissemination) by persons other than the
 > > intended
 > > recipient(s) is prohibited. If you receive this e-mail in error, please
 > > notify the sender by
 > > phone or email immediately and delete it!
 > >
 > >
 > > -Original Message-
 > > From: Andrew Purtell [mailto:apurt...@apache.org]
 > > Sent: Saturday, July 29, 2017 6:33 AM
 > > To: dev@hbase.apache.org; u...@hbase.apache.org
 > > Subject: [ANNOUNCE] New HBase committer Vikas Vishwakarma
 > >
 > > On behalf of the Apache HBase PMC, I am pleased to announce that Vikas
 > > Vishwakarma has accepted the PMC's invitation to become a committer on
 > the
 > > project.
 > >
 > > We appreciate all of Vikas's great work thus far and look forward to
 > > continued involvement.
 > >
 > > Please join me in congratulating Vikas!
 > >
 > > --
 > > Best regards,
 > > Andrew
 > >
 >

Re: [ANNOUNCE] New HBase committer Abhishek Singh Chouhan

[Jerry He]

Congrats Abhishek!


Jerry

On Mon, Jul 31, 2017 at 1:41 AM, Anoop John  wrote:
> Congrats Abhishek
>
> On Mon, Jul 31, 2017 at 1:48 PM, Yu Li  wrote:
>> Congratulations, Abhishek!
>>
>> Best Regards,
>> Yu
>>
>> On 31 July 2017 at 15:40, Jingcheng Du  wrote:
>>
>>> Congratulations!
>>>
>>> Regards,
>>> Jingcheng
>>>
>>> 2017-07-31 12:49 GMT+08:00 ramkrishna vasudevan <
>>> ramkrishna.s.vasude...@gmail.com>:
>>>
>>> > Congratulations Abhishek !!!
>>> >
>>> > Regards
>>> > Ram
>>> >
>>> > On Mon, Jul 31, 2017 at 10:16 AM, Pankaj kr 
>>> wrote:
>>> >
>>> > > Congratulations Abhishek..!!
>>> > >
>>> > > Thanks & Regards,
>>> > > Pankaj
>>> > >
>>> > > HUAWEI TECHNOLOGIES CO.LTD.
>>> > > Huawei Tecnologies India Pvt. Ltd.
>>> > > Near EPIP Industrial Area, Kundalahalli Village
>>> > > Whitefield, Bangalore-560066
>>> > > www.huawei.com
>>> > > 
>>> > > -
>>> > > This e-mail and its attachments contain confidential information from
>>> > > HUAWEI, which
>>> > > is intended only for the person or entity whose address is listed
>>> above.
>>> > > Any use of the
>>> > > information contained herein in any way (including, but not limited to,
>>> > > total or partial
>>> > > disclosure, reproduction, or dissemination) by persons other than the
>>> > > intended
>>> > > recipient(s) is prohibited. If you receive this e-mail in error, please
>>> > > notify the sender by
>>> > > phone or email immediately and delete it!
>>> > >
>>> > > -Original Message-
>>> > > From: Andrew Purtell [mailto:apurt...@apache.org]
>>> > > Sent: Saturday, July 29, 2017 6:32 AM
>>> > > To: dev@hbase.apache.org; u...@hbase.apache.org
>>> > > Subject: [ANNOUNCE] New HBase committer Abhishek Singh Chouhan
>>> > >
>>> > > On behalf of the Apache HBase PMC, I am pleased to announce that
>>> Abhishek
>>> > > Singh Chouhan has accepted the PMC's invitation to become a committer
>>> on
>>> > > the project.
>>> > >
>>> > > We appreciate all of Abhishek's great work thus far and look forward to
>>> > > continued involvement.
>>> > >
>>> > > Please join me in congratulating Abhishek!
>>> > >
>>> > > --
>>> > > Best regards,
>>> > > Andrew
>>> > >
>>> >
>>>

Re: [ANNOUNCE] Devaraj Das joins the Apache HBase PMC

[Jerry He]

Congrats, Devaraj !


Thanks,

Jerry

On Wed, Jul 5, 2017 at 1:36 PM, Nick Dimiduk  wrote:
> Congratulations Devaraj!
>
> On Wed, Jul 5, 2017 at 9:27 AM, Josh Elser  wrote:
>
>> I'm pleased to announce yet another PMC addition in the form of Devaraj
>> Das. One of the "old guard" in the broader Hadoop umbrella, he's also a
>> long-standing member in our community. We all look forward to the continued
>> contributions and project leadership.
>>
>> Please join me in welcoming Devaraj!
>>
>> - Josh (on behalf of the PMC)
>>

Re: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC

[Jerry He]

Congrats,  Chunhui!

Thanks

Jerry

On Tue, Jul 4, 2017 at 8:37 PM Anoop John  wrote:

> Congrats Chunhui..
>
> On Wed, Jul 5, 2017 at 6:55 AM, Pankaj kr  wrote:
> > Congratulations Chunhui..!!
> >
> > Regards,
> > Pankaj
> >
> >
> > -Original Message-
> > From: Yu Li [mailto:car...@gmail.com]
> > Sent: Tuesday, July 04, 2017 1:24 PM
> > To: dev@hbase.apache.org; Hbase-User
> > Subject: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC
> >
> > On behalf of the Apache HBase PMC I am pleased to announce that Chunhui
> Shen has accepted our invitation to become a PMC member on the Apache HBase
> project. He has been an active contributor to HBase for past many years.
> Looking forward for many more contributions from him.
> >
> > Please join me in welcoming Chunhui to the HBase PMC!
> >
> > Best Regards,
> > Yu
>

Re: [DISCUSS] status of and plans for our hbase-spark integration

[Jerry He]

>> We currently have code in the o.a.spark namespace. I don't think there is a
>> JIRA for it yet, but this seems like cross-project trouble waiting to
>> happen. https://github.com/apache/hbase/tree/master/
>> hbase-spark/src/main/scala/org/apache/spark
>>
>
> IIRC, this was something we had to do because of how Spark architected
> their stuff. So long as we're marking all of that stuff IA.Private I
> think we're good, since we can fix it later if/when Spark changes.
>

Yes.  IIRC The trick is needed because we use a construct from spark
sql package private for Spark 1.6.
This trick is no longer needed if we only support Spark 2.x.


> >> The way I see it, the options are a) ship both 1.6 and 2.y support, b)
> >> ship just 2.y support, c) ship 1.6 in branch-1 and ship 2.y in
> >> branch-2. Does anyone have preferences here?
> >
> > I think I prefer option B here as well. It sounds like Spark 2.2 will be
> > out Very Soon, so we should almost certainly have a story for that. If
> > there are no compatibility issues, then we can support >= 2.0 or 2.1,
> > otherwise there's no reason to try and hit the moving target and we can
> > focus on supporting the newest release. Like you said earlier, there's been
> > no official release of this module yet, so I have to imagine that the
> > current consumers are knowingly bleeding edge and can handle an upgrade or
> > recompile on their own.
> >
>
> Yeah, the bleeding-edge bit sounds fair. (someone please shout if it ain't)


I am for Option b) as well!
Even better, I am for  we only ship support for Scala 2.11.   Start clean?


>>> 4) Packaging all this probably will be a pain no matter what we do
>>
>> Do we have to package this in our assembly at all? Currently, we include
>> the hbase-spark module in the branch-2 and master assembly, but I'm not
>> convinced this needs to be the case. Is it too much to ask users to build a
>> jar with dependencies (which I think we already do) and include the
>> appropriate spark/scala/hbase jars in it (pulled from maven)? I think this
>> problem can be better solved through docs and client tooling rather than
>> going through awkward gymnastics to package m*n versions in our tarball
>> _and_ making sure that we get all the classpaths right.
>>
>
>
> Even if we don't put it in the assembly, we still have to package m*n
> versions to put up in Maven, right?
>
> I'm not sure on the jar-with-deps bit. It's super nice to just include
> one known-deployed jar in your spark classpath instead of putting that
> size into each application jar your run. Of course, to your point on
> classpaths, right now they'd need to grab things besides that jar.
> Maybe these should be shaded jars sooner rather than later?

There is a Filter class from the hbase-spark module that needs to be
on the server classpath.
If we don't have the whole jar there, we have to do some trick to
separate it out.

Great write-up from Sean.

Thanks,

Jerry

[jira] [Resolved] (HBASE-18123) Hbase indexer

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-18123.
--
Resolution: Invalid

> Hbase indexer
> -
>
> Key: HBASE-18123
> URL: https://issues.apache.org/jira/browse/HBASE-18123
> Project: HBase
>  Issue Type: Task
>Affects Versions: 1.2.3
> Environment: Debian / Hadoop 2.6 / Solr 6.4.2
>Reporter: Fred
>
> Hi all,
> I want to extract fields from PDF files and store it into Hbase. Then I want 
> to link the database with a Solr collection. To do this, I installed 
> Hbase-indexer.
> What is the best way to do it ?
> Actually, I can write data into Hbase but not into my Solr collection. When I 
> launch Hbase-indexer server, I get some errors :
> Cannot connect to cluster at myIPaddress:2181/solr: cluster not found/not 
> ready
> Somebody ti o help me ? Thanks in advance.
> Fred



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] First release candidate for HBase 1.1.10 (RC0) is available

[Jerry He]

+1

- Downloaded the src tarball.
  - Checked the md5 and signature.  Looks good.
  - Built from source.  openjdk version "1.8.0_111"
  - Ran unit test suite.  All passed.
  - mvn apache-rat:check   Passed

- Download the bin tarball.
 -  Checked the md5 and signature.  Looks good.
 -  Extract it.   Layout looks good.
 -  Started a local instance.  Run the 'hbase shell' with some basic table
commands.  Looks good.
 -  Nothing unusual in the logs.
 -  Check master and region server UI.  Clicked on the tabs.  Looks good.

- Built Phoenix 4.9-HBase-1.1 with hbase version 1.1.10 pointing to the RC
maven repo URL.
  Ran unit tests.  All passed.

-  Bulit YCSB 0.12.0 from git source with HBase binding version 1.1.10
pointing to RC.
-  Run YCSB workloads against the local RC instance.  Worked fine.

On Wed, Apr 19, 2017 at 4:57 PM, Andrew Purtell  wrote:

> ​+1
>
> Checked sums and signatures: ok
> Build from source: ok (7u80)
> RAT check: ok (7u80)
> Unit tests pass: ok (7u80)
> 1M rows LTT: ok (8u102)
> ​
>
> On Tue, Apr 18, 2017 at 10:00 PM, Nick Dimiduk 
> wrote:
>
> > I'm happy to announce the first release candidate of HBase 1.1.10
> > (HBase-1.1.10RC0) is available for download at
> > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.1.10RC0/
> >
> > Maven artifacts are also available in the staging repository
> > https://repository.apache.org/content/repositories/orgapachehbase-1167
> >
> > Artifacts are signed with my code signing subkey 0xAD9039071C3489BD,
> > available in the Apache keys directory
> > https://people.apache.org/keys/committer/ndimiduk.asc and in our KEYS
> file
> > http://www-us.apache.org/dist/hbase/KEYS.
> >
> > Please note that this signing key is set to expire at the end of this
> > month. I will be taking steps to extend its expiration date. This should
> be
> > transparent, but may cause some hiccups.
> >
> > There's also a signed tag for this release at
> > https://git-wip-us.apache.org/repos/asf?p=hbase.git;a=tag;h=
> > 4a7f2062f5776d6423d19f6d88bc4161c33ccd0c
> >
> > The detailed source and binary compatibility report vs 1.1.9 has been
> > published for your review, at
> > https://home.apache.org/~ndimiduk/1.1.9_1.1.10RC0_compat_report.html
> >
> > HBase 1.1.10 is the tenth patch release in the HBase 1.1 line, continuing
> > on the theme of bringing a stable, reliable database to the Hadoop and
> > NoSQL communities. This release includes over 10 bug fixes since the
> 1.1.9
> > release. Notable correctness fixes include HBASE-17675, HBASE-17682,
> > HBASE-17287, and HBASE-17717.
> >
> > The full list of fixes included in this release is available at
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
> > ctId=12310753=12339651
> > and and in the CHANGES.txt file included in the distribution.
> >
> > Please try out this candidate and vote +/-1 by 23:59 Pacific time on
> > Tuesday, 2017-04-24 as to whether we should release these artifacts as
> > HBase 1.1.10.
> >
> > Thanks,
> > Nick
> >
>
>
>
> --
> Best regards,
>
>- Andy
>
> If you are given a choice, you believe you have acted freely. - Raymond
> Teller (via Peter Watts)
>

Re: ANNOUNCE: Yu Li joins the Apache HBase PMC

[Jerry He]

Congratulations and welcome, Yu!

On Fri, Apr 14, 2017 at 6:47 PM, Andrew Purtell  wrote:

> Congratulations and welcome!
>
>
> On Fri, Apr 14, 2017 at 7:22 AM, Anoop John  wrote:
>
> > On behalf of the Apache HBase PMC I"m pleased to announce that Yu Li
> > has accepted our invitation to become a PMC member on the Apache HBase
> > project. He has been an active contributor to HBase for past many
> > years. Looking forward for
> > many more contributions from him.
> >
> > Welcome to the PMC, Yu Li...
> >
> >
> > -Anoop-
> >
>
>
>
> --
> Best regards,
>
>- Andy
>
> If you are given a choice, you believe you have acted freely. - Raymond
> Teller (via Peter Watts)
>

Re: Is HBase RPC-Handling idempotent for reads?

[Jerry He]

Yes.  In the context to the underlying physical region or database,. read
is idempotent.


Thanks

Jerry

On Apr 9, 2017 9:15 PM, "Yu Li" <car...@gmail.com> wrote:

> Correct me if I'm wrong, but I think we should assume no other but the
> single operation when checking whether it's idempotent. Similar to the
> wikipedia
> example <https://en.wikipedia.org/wiki/Idempotence#Examples>: "A function
> looking up a customer's name and address in a database
> <https://en.wikipedia.org/wiki/Database> is typically idempotent, since
> this will not cause the database to change", I think all Get/MultiGet/Scan
> operations in hbase are idempotent.
>
> About "speculative rpc handling", I doubt whether it benefits in hbase.
> Normally if a request already arrives at server side but with slow
> execution, the problem might be:
> 1. The server is too busy and request get queued
> 2. The processing itself is slow due to the request pattern or some
> hardware failure
> I don't think a speculative execution of the request could help in any of
> the above cases. It's different from the speculative task execution in MR,
> there we could choose another node to execute the task while here we have
> no choice.
>
> OTOH, we already have timeout mechanism to make sure server resource won't
> be wasted:
> 1. For scan
> - When a request handling timeouts, server will stop further
> processing, refer to RSRpcServices#getTimeLimit and
> ScannerContext#checkTimeLimit
> - If the client went away during processing, server will also stop
> processing, check the SimpleRpcServer#disconnectSince and
> RegionScannerImpl#nextInternal methods for more details.
>
> 2. For single Get
> - Controlled by rpc and operation timeout
>
> 3. For MultiGet
> - I think this is something we could improve. On client side we have
> timeout mechanism but on server side there seems to be no relative
> interrupt logic.
>
>
> Best Regards,
> Yu
>
> On 10 April 2017 at 11:12, Jerry He <jerry...@gmail.com> wrote:
>
> > Again, it depends on how you abort and 'idempotent' can have different
> > definitions.
> >
> > For example, even if you are only concerned about read,
> > there are resources on the HRegion that the read touches or acquires
> > (scanner, lock, mvcc etc) that hopefully will be cleaned/releases with
> the
> > abort.
> > Or you may have it in a bad/inconsistent state.
> >
> > Thanks.
> >
> > Jerry
> >
> >
> > On Sun, Apr 9, 2017 at 7:14 PM, 张铎(Duo Zhang) <palomino...@gmail.com>
> > wrote:
> >
> > > I think this depends on how you model the problem. At server side, if
> you
> > > re-execute a read operation with a new mvcc, then you may read a value
> > that
> > > should not be visible if you use the old mvcc. If you define this as an
> > > error then I think there will be conflicts.
> > >
> > > But at client side, there is guarantee that the request you send first
> > will
> > > be executed first. So as long as the read request does not return, I
> > think
> > > it is OK to read a value which is written by a write request which is
> > sent
> > > after the read request?
> > >
> > > Thanks.
> > >
> > > 2017-04-10 9:52 GMT+08:00 杨苏立 Yang Su Li <yangs...@gmail.com>:
> > >
> > > > We are only concerned about read operations here. Are you suggesting
> > they
> > > > are completely idempotent?
> > > > Are there any read-after-write conflicts?
> > > >
> > > > Thanks
> > > >
> > > > Sui
> > > >
> > > > On Sun, Apr 9, 2017 at 8:48 PM, 张铎(Duo Zhang) <palomino...@gmail.com
> >
> > > > wrote:
> > > >
> > > > > It depends on how you about the rpc request. For hbase, there will
> be
> > > no
> > > > > write conflict, but a write operation can only be finished iff all
> > the
> > > > > write operations with a lower mvcc number have been finished. So if
> > you
> > > > > just stop a write operation without recovering the mvcc(I do not
> know
> > > how
> > > > > to recover but I think you need to something...) then the writes
> will
> > > be
> > > > > stuck.
> > > > >
> > > > > And one more thing, for read operation you may interrupt it at any
> > > time,
> > > > > but for write operation, I do not think you can re-execute it with
> a
> > > new
> &

Re: Is HBase RPC-Handling idempotent for reads?

[Jerry He]

Again, it depends on how you abort and 'idempotent' can have different
definitions.

For example, even if you are only concerned about read,
there are resources on the HRegion that the read touches or acquires
(scanner, lock, mvcc etc) that hopefully will be cleaned/releases with the
abort.
Or you may have it in a bad/inconsistent state.

Thanks.

Jerry


On Sun, Apr 9, 2017 at 7:14 PM, 张铎(Duo Zhang) <palomino...@gmail.com> wrote:

> I think this depends on how you model the problem. At server side, if you
> re-execute a read operation with a new mvcc, then you may read a value that
> should not be visible if you use the old mvcc. If you define this as an
> error then I think there will be conflicts.
>
> But at client side, there is guarantee that the request you send first will
> be executed first. So as long as the read request does not return, I think
> it is OK to read a value which is written by a write request which is sent
> after the read request?
>
> Thanks.
>
> 2017-04-10 9:52 GMT+08:00 杨苏立 Yang Su Li <yangs...@gmail.com>:
>
> > We are only concerned about read operations here. Are you suggesting they
> > are completely idempotent?
> > Are there any read-after-write conflicts?
> >
> > Thanks
> >
> > Sui
> >
> > On Sun, Apr 9, 2017 at 8:48 PM, 张铎(Duo Zhang) <palomino...@gmail.com>
> > wrote:
> >
> > > It depends on how you about the rpc request. For hbase, there will be
> no
> > > write conflict, but a write operation can only be finished iff all the
> > > write operations with a lower mvcc number have been finished. So if you
> > > just stop a write operation without recovering the mvcc(I do not know
> how
> > > to recover but I think you need to something...) then the writes will
> be
> > > stuck.
> > >
> > > And one more thing, for read operation you may interrupt it at any
> time,
> > > but for write operation, I do not think you can re-execute it with a
> new
> > > mvcc number if the WAL entry has already been flushed out. That means,
> > the
> > > re-execution process will be different if you about the write operation
> > at
> > > different stages.
> > >
> > > Thanks.
> > >
> > > 2017-04-10 6:47 GMT+08:00 杨苏立 Yang Su Li <yangs...@gmail.com>:
> > >
> > > > We are trying to implement speculative rpc handling for our
> workloads.
> > So
> > > > we want allow RPC Handler to stop executing an RPC call, put it back
> to
> > > the
> > > > queue, and later re-execute it.
> > > >
> > > > If at time t1, we execute and RPC call half way, aborts, and put the
> > call
> > > > back to the queue.
> > > > Then at time t2 another RPC handler picks the call and re-execute it.
> > > > I understand that we might get a different mvcc number and different
> > > > results at t2 compared to we execute it at t1.
> > > > My question is that: would this situation any different compared to
> the
> > > > situation where the call was never executed at t1, and is executed at
> > t2
> > > > for the first time.
> > > >
> > > >
> > > > My guess is that since at t1 we may already gotten an mvcc number, so
> > it
> > > > might potentially cause some write conflicts and certain write
> > operations
> > > > to retry. But correctness wise, is there any difference?
> > > >
> > > > Thanks a lot!
> > > >
> > > > Suli
> > > >
> > > >
> > > > On Sun, Apr 9, 2017 at 5:14 PM, Jerry He <jerry...@gmail.com> wrote:
> > > >
> > > > > I don't know what your intention and your context are.
> > > > >
> > > > > You may get a different mvcc number and get different results next
> > time
> > > > > around if there are concurrent writes.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jerry
> > > > >
> > > > > On Sun, Apr 9, 2017 at 12:48 PM 杨苏立 Yang Su Li <yangs...@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I am wondering, for read requests like Get/MultiGet/Scan, is the
> > RPC
> > > > > > handling idempotent in HBase?
> > > > > >
> > > > > > More specifically, if in the middle of RPC handling we stop the
> > > > handling
> > > > > > threads, puts the RPC call back to the queue, and later another
> RPC
> > > > > Handler
> > > > > > picks up this call and starts all over again, will the result be
> > the
> > > > same
> > > > > > as if this call is being handled for the first time now? Or are
> > their
> > > > any
> > > > > > unexpected side effects?
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Suli
> > > > > >
> > > > > > --
> > > > > > Suli Yang
> > > > > >
> > > > > > Department of Physics
> > > > > > University of Wisconsin Madison
> > > > > >
> > > > > > 4257 Chamberlin Hall
> > > > > > Madison WI 53703
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Suli Yang
> > > >
> > > > Department of Physics
> > > > University of Wisconsin Madison
> > > >
> > > > 4257 Chamberlin Hall
> > > > Madison WI 53703
> > > >
> > >
> >
> >
> >
> > --
> > Suli Yang
> >
> > Department of Physics
> > University of Wisconsin Madison
> >
> > 4257 Chamberlin Hall
> > Madison WI 53703
> >
>

Re: Is HBase RPC-Handling idempotent for reads?

[Jerry He]

I don't know what your intention and your context are.

You may get a different mvcc number and get different results next time
around if there are concurrent writes.

Thanks,

Jerry

On Sun, Apr 9, 2017 at 12:48 PM 杨苏立 Yang Su Li  wrote:

> Hi,
>
> I am wondering, for read requests like Get/MultiGet/Scan, is the RPC
> handling idempotent in HBase?
>
> More specifically, if in the middle of RPC handling we stop the handling
> threads, puts the RPC call back to the queue, and later another RPC Handler
> picks up this call and starts all over again, will the result be the same
> as if this call is being handled for the first time now? Or are their any
> unexpected side effects?
>
> Thanks!
>
> Suli
>
> --
> Suli Yang
>
> Department of Physics
> University of Wisconsin Madison
>
> 4257 Chamberlin Hall
> Madison WI 53703
>

[jira] [Created] (HBASE-17890) FuzzyRow tests fail if unaligned support is false

[Jerry He (JIRA)]

Jerry He created HBASE-17890:


 Summary: FuzzyRow tests fail if unaligned support is false
 Key: HBASE-17890
 URL: https://issues.apache.org/jira/browse/HBASE-17890
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.2.5, 2.0.0
Reporter: Jerry He


When unaligned support is false, FuzzyRow tests fail:
{noformat}
Failed tests:
  TestFuzzyRowAndColumnRangeFilter.Test:134->runTest:157->runScanner:186 
expected:<10> but was:<0>
  TestFuzzyRowFilter.testSatisfiesForward:81 expected: but was:
  TestFuzzyRowFilter.testSatisfiesReverse:121 expected: but 
was:
  TestFuzzyRowFilterEndToEnd.testEndToEnd:247->runTest1:278->runScanner:343 
expected:<6250> but was:<0>
  TestFuzzyRowFilterEndToEnd.testFilterList:385->runTest:417->runScanner:445 
expected:<5> but was:<0>
  TestFuzzyRowFilterEndToEnd.testHBASE14782:204 expected:<6> but was:<0>
{noformat}
This can be reproduced in the case described in HBASE-17869. Or on a platform 
really without unaligned support.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HBASE-17869) UnsafeAvailChecker wrongly returns false on ppc

[Jerry He (JIRA)]

Jerry He created HBASE-17869:


 Summary: UnsafeAvailChecker wrongly returns false on ppc
 Key: HBASE-17869
 URL: https://issues.apache.org/jira/browse/HBASE-17869
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.2.4
Reporter: Jerry He
Assignee: Jerry He
Priority: Minor


On ppc64 arch,  java.nio.Bits.unaligned() wrongly returns false due to a JDK 
bug.
https://bugs.openjdk.java.net/browse/JDK-8165231
This causes some problem for HBase. i.e. FuzzyRowFilter test fails.
Fix it by providing a hard-code workaround for the JDK bug.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: About the InterfaceStability annotation for public API

[Jerry He]

Looks fine.

Incorporating another public dimension probably makes the version scheme
too complex.   Keep it simple.

Jerry

On Fri, Mar 31, 2017 at 5:30 PM 张铎(Duo Zhang) <palomino...@gmail.com> wrote:

> Some progress here.
>
> In HBASE-17857, a sub task of HBASE-17828, I've created a script to remove
> all the IS annotations for the IA.Public API. And I've also changed the
> IA.Public annotations for several classes which are marked as IS.Unstable
> to IA.Private. We can change them back when we think they are stable
> enough.
>
> The patch is ready to land. Will commit it today if no objections.
>
> Thanks.
>
> 2017-03-24 9:48 GMT+08:00 张铎(Duo Zhang) <palomino...@gmail.com>:
>
> > Filed https://issues.apache.org/jira/browse/HBASE-17828
> >
> > 2017-03-21 8:36 GMT+08:00 张铎(Duo Zhang) <palomino...@gmail.com>:
> >
> >> bq.  If someone is
> >> comfortable with the risk of an API that can change in minor or
> >> maintenance releases, what's gained by calling it IA.Public +
> >> IS.Evolving or Unstable rather than just labeling it IA.Private or
> >> IA.LimitedPrivate?
> >>
> >> Indeed. We can not stop users use IA.Private classes if they are
> declared
> >> as public class. The users take the risk by themselves.
> >>
> >> Anyway, seems we all agree that at least we need to update our
> >> documentation. So let me open a issue first. We can continue the
> discussion
> >> there.
> >>
> >> Thanks.
> >>
> >> 2017-03-21 4:27 GMT+08:00 Jerry He <jerry...@gmail.com>:
> >>
> >>> Just to bring up an alternative idea.
> >>> The Spark InterfaceStability explicitly spells out what can or can not
> >>> change from major or minor releases. (I was onto it recently).
> >>> See [1]
> >>>
> >>> HBase is probably a more stable project that does not have to do the
> >>> same.
> >>> But it is also possible that we may have new features (i.e.
> AsyncClient,
> >>> Backup, etc) that we want to evolve within a major release.
> >>> We should see if we want to provide such flexibility via the
> >>> InterfaceStability annotations and document it explicitly if yes.
> >>>
> >>> Thanks,
> >>>
> >>> Jerry
> >>>
> >>>
> >>> 1. https://github.com/apache/spark/blob/master/common/tags/
> >>> src/main/java/org/apache/spark/annotation/InterfaceStability.java
> >>>
> >>>
> >>>
> >>>
> >>> On Mon, Mar 20, 2017 at 10:46 AM, Yu Li <car...@gmail.com> wrote:
> >>>
> >>> > +1 on removing InterfaceStability annotation for IA.Public. Even
> more,
> >>> is
> >>> > it possible to forbid using these two annotations together in Yetus
> at
> >>> > code-level if we are migrating to it (as mentioned in another
> thread)?
> >>> >
> >>> > For IA.Private or IA.LimitedPrivate, personally I think
> >>> InterfaceStability
> >>> > is still a useful annotation.
> >>> >
> >>> > Best Regards,
> >>> > Yu
> >>> >
> >>> > On 20 March 2017 at 22:07, Sean Busbey <bus...@apache.org> wrote:
> >>> >
> >>> > > I really dislike having InterfaceStability markings on IA.Public
> >>> > > interfaces, because to me it reads like us essentially saying we
> >>> > > didn't invest enough time in deciding what something should look
> like
> >>> > > before declaring it safe for downstream folks. If someone is
> >>> > > comfortable with the risk of an API that can change in minor or
> >>> > > maintenance releases, what's gained by calling it IA.Public +
> >>> > > IS.Evolving or Unstable rather than just labeling it IA.Private or
> >>> > > IA.LimitedPrivate?
> >>> > >
> >>> > > So I'd be +1 on updating our docs to state that InterfaceStability
> is
> >>> > > just for IA.LimitedPrivate or even discontinuing our use of it
> >>> > > entirely.
> >>> > >
> >>> > > On Sun, Mar 19, 2017 at 11:28 PM, Duo Zhang <zhang...@apache.org>
> >>> wrote:
> >>> > > > In the compatibility section of our refguide, the compatibility
> for
> >>> > patch
> >>> > > > version, minor version and 

Re: [DISCUSS] What are we going to call the releases leading up to HBase 2.0?

[Jerry He]

+1 on 2.0.0-alpha/beta. I think each of the alpha or beta will go through
RCs as well.

Thanks.

Jerry

On Tue, Mar 28, 2017 at 11:18 PM, Phil Yang  wrote:

> +1 on alpha/beta, too. If they are not enough we can also have 2.0.0-rc :)
>
> Thanks,
> Phil
>
>
> 2017-03-29 12:18 GMT+08:00 Yu Li :
>
> > +1 on -alpha/-beta, and cannot wait to see an alpha1 out (smile)
> >
> > Best Regards,
> > Yu
> >
> > On 29 March 2017 at 10:28, 张铎(Duo Zhang)  wrote:
> >
> > > +1 on 2.0.0-alpha[x]/2.0.0-beta[x].
> > >
> > > 2017-03-29 10:07 GMT+08:00 Andrew Purtell :
> > >
> > > > That settles it. :-)
> > > >
> > > > I'd also be cool with -alpha, -beta, etc.
> > > >
> > > > > On Mar 28, 2017, at 1:25 PM, Enis Söztutar 
> wrote:
> > > > >
> > > > > I would automatically -1 any release with a number like 1.99
> > regardless
> > > > of
> > > > > content.
> > > > >
> > > > > Semantic versioning which we are following already provides an
> answer
> > > for
> > > > > this:
> > > > > http://semver.org/#spec-item-9
> > > > >
> > > > > From my experience as RM for 0.99.x series and 1.0.x series, I
> would
> > > > > suggest we do 2.0.0-alpha1 and alpha2, and one or two betas. I
> think
> > we
> > > > > should start the alpha1 release now which does not have to wait for
> > > > > anything but packaging work.
> > > > >
> > > > > Enis
> > > > >
> > > > >> On Tue, Mar 28, 2017 at 1:19 PM, Sean Busbey 
> > > wrote:
> > > > >>
> > > > >> Hi folks!
> > > > >>
> > > > >> What are folks opinions on how we name releases leading up to
> HBase
> > > > >> 2.0 that aren't quite done yet?
> > > > >>
> > > > >> For 1.0, we used 0.99 as a placeholder for "what we expect will be
> > in
> > > > >> 1.0 but is not yet ready for production use." That got us 0.99.0,
> > > > >> 0.99.1, and 0.99.2 before we declared 1.0.0 ready for use. For
> 2.0,
> > > > >> continuing this pattern would be done with 1.99, I suppose.
> > > > >>
> > > > >> This issue I take with this approach is that back before 1.0, we
> > could
> > > > >> count on users thinking of 0.99 as a different major release train
> > > > >> than 0.98. Now, I'm concerned that some might lump 1.99 in with
> the
> > > > >> 1.y major release series.
> > > > >>
> > > > >> Alternatively we could expressly label the releases as alpha/beta
> > > > >> based on our confidence. This would give us 2.0.0-alpha1,
> > > > >> 2.0.0-alpha2, etc, 2.0.0-beta1, etc. This has the disadvantage of
> > > > >> futzing with sort order, but clearly conveys that these releases
> are
> > > > >> both part of what will be the 2.y major release series and not for
> > the
> > > > >> faint of heart.
> > > > >>
> > > > >>
> > > > >> Thoughts?
> > > > >>
> > > >
> > >
> >
>

Re: [ANNOUNCE] - Welcome our new HBase committer Anastasia Braginsky

[Jerry He]

Congrats and welcome!

Jerry

Re: About the InterfaceStability annotation for public API

[Jerry He]

Just to bring up an alternative idea.
The Spark InterfaceStability explicitly spells out what can or can not
change from major or minor releases. (I was onto it recently).
See [1]

HBase is probably a more stable project that does not have to do the same.
But it is also possible that we may have new features (i.e. AsyncClient,
Backup, etc) that we want to evolve within a major release.
We should see if we want to provide such flexibility via the
InterfaceStability annotations and document it explicitly if yes.

Thanks,

Jerry


1. https://github.com/apache/spark/blob/master/common/tags/
src/main/java/org/apache/spark/annotation/InterfaceStability.java




On Mon, Mar 20, 2017 at 10:46 AM, Yu Li  wrote:

> +1 on removing InterfaceStability annotation for IA.Public. Even more, is
> it possible to forbid using these two annotations together in Yetus at
> code-level if we are migrating to it (as mentioned in another thread)?
>
> For IA.Private or IA.LimitedPrivate, personally I think InterfaceStability
> is still a useful annotation.
>
> Best Regards,
> Yu
>
> On 20 March 2017 at 22:07, Sean Busbey  wrote:
>
> > I really dislike having InterfaceStability markings on IA.Public
> > interfaces, because to me it reads like us essentially saying we
> > didn't invest enough time in deciding what something should look like
> > before declaring it safe for downstream folks. If someone is
> > comfortable with the risk of an API that can change in minor or
> > maintenance releases, what's gained by calling it IA.Public +
> > IS.Evolving or Unstable rather than just labeling it IA.Private or
> > IA.LimitedPrivate?
> >
> > So I'd be +1 on updating our docs to state that InterfaceStability is
> > just for IA.LimitedPrivate or even discontinuing our use of it
> > entirely.
> >
> > On Sun, Mar 19, 2017 at 11:28 PM, Duo Zhang  wrote:
> > > In the compatibility section of our refguide, the compatibility for
> patch
> > > version, minor version and major version is not related
> > > to InterfaceStability annotation. The only place we mention it is for
> > > Server-Side Limited API compatibility.
> > >
> > > And  in the Developer Guidelines section, we say this
> > > @InterfaceStability.Evolving
> > >
> > > Public packages marked as evolving may be changed, but it is
> discouraged.
> > > I think this is a little confusing, esepecially that the comment
> > > of InterfaceStability also mentions the compatibility for patch, minor
> > and
> > > major release.
> > >
> > > For me, I think only InterfaceStability.Unstable is useful for public
> > API.
> > > It means the API is still experimental and will not respect the
> > > compatibility rule.
> > >
> > > So here I suggest we just remove the InterfaceStability annoation for
> the
> > > classes which are marked as InterfaceAudience.Public, and change the
> > > comment of InterfaceStability and also the refguide to be more
> specific.
> > >
> > > Suggestions are welcomed.
> > >
> > > Thanks.
> >
>

Re: [VOTE] First release candidate for HBase 1.2.5 is available

[Jerry He]

Sure, what the RM feels it is ok and reasonable.

Jerry

On Mon, Mar 20, 2017 at 8:01 AM, Sean Busbey <sean.bus...@gmail.com> wrote:

> Hi Jerry!
>
> Are you comfortable with me considering your vote as binding now that
> you're a PMC member?
>
> On Wed, Mar 15, 2017 at 7:12 PM, Jerry He <jerry...@gmail.com> wrote:
> > +1
> >
> > (non-binding)
> >
> >
> > - Downloaded the src tarball.
> >   - Checked the md5 and signature.  Looks good.
> >   - Built from source.  openjdk version "1.8.0_121"
> >   - Ran unit test suite twice.  All passed.
> >   - mvn apache-rat:check   Passed
> >
> > - Download the bin tarball.
> >  -  Checked the md5 and signature.  Looks good.
> >  -  Extract it.   Layout looks good.
> >  -  Started a local instance.  Run the 'hbase shell' with some basic
> table
> > commands, split, flush, etc.  Looks good.
> >  -  Nothing unusual in the logs.
> >  -  Check master and region server UI.  Clicked on the tabs, looked at
> the
> > tasks running, Debug dump, Configuration dump, zk dump. Looks good.
> >
> > - Built Phoenix 4.9-HBase-1.2 with hbase version 1.2.5 pointing to the RC
> > maven repo URL.
> >   Ran unit tests.  All passed.
> >
> > -  Bulit YCSB 0.12.0 from git source with HBase binding version 1.2.5
> > pointing to RC.
> > -  Run YCSB workloads again the local RC instance.  Worked fine.
> >
> > Jerry
> >
> >
> > On Mon, Mar 13, 2017 at 9:50 PM, Andrew Purtell <apurt...@apache.org>
> wrote:
> >
> >> +1
> >>
> >> Checked sums and signatures: ok
> >> Built from source: ok (7u80)
> >> RAT check passed: ok (7u80)
> >> Unit tests pass: ok (8u102)
> >> Loaded 1M rows with LTT: ok (8u102), all keys verified, latencies in the
> >> ballpark, nothing unusual in the logs
> >>
> >>
> >> On Fri, Mar 10, 2017 at 2:31 PM, Sean Busbey <bus...@apache.org> wrote:
> >>
> >> > The first release candidate for HBase 1.2.5 is available for download
> at:
> >> >
> >> > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.2.5RC0/
> >> >
> >> > Maven artifacts are also available in a staging repository at:
> >> >
> >> > https://repository.apache.org/content/repositories/
> orgapachehbase-1164/
> >> >
> >> > Artifacts are signed with my key (0D80DB7C) published in our KEYS
> >> > file at http://www.apache.org/dist/hbase/KEYS
> >> >
> >> > The RC corresponds to the signed tag 1.2.5RC0, which currently points
> >> > to commit ref
> >> >
> >> > d7b05f79dee10e0ada614765bb354b93d615a157
> >> >
> >> > The detailed source and binary compatibility report vs 1.2.4 has been
> >> > published for your review, at:
> >> >
> >> > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.2.5RC0/
> >> > 1.2.4_1.2.5RC0_compat_report.html
> >> >
> >> > Note that this report calls out the issue discussed in HBASE-17725.
> >> >
> >> > HBase 1.2.5 is the fifth maintenance release in the HBase 1.2 line,
> >> > continuing on
> >> > the theme of bringing a stable, reliable database to the Hadoop and
> NoSQL
> >> > communities. This release includes over 50 bug fixes since 1.2.4.
> >> > Critical fixes include:
> >> >
> >> > * HBASE-17069 RegionServer writes invalid META entries for split
> >> > daughters in some circumstances
> >> > * HBASE-17044 Fix merge failed before creating merged region leaves
> >> > meta inconsistent
> >> > * HBASE-17206 FSHLog may roll a new writer successfully with unflushed
> >> > entries
> >> > * HBASE-16765 New SteppingRegionSplitPolicy, avoid too aggressive
> >> > spread of regions for small tables.
> >> >
> >> >
> >> > The full list of fixes included in this release is available at:
> >> >
> >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> >> > version=12338339=12310753
> >> >
> >> > and in the CHANGES.txt file included in the distribution.
> >> >
> >> > Please try out this candidate and vote +1/-1 on whether we should
> >> > release these artifacts as HBase 1.2.5.
> >> >
> >> > The VOTE will remain open for atleast 72 hours. Given sufficient votes
> >> > I would like to close it on March 17th, 2017.
> >> >
> >> > thanks!
> >> >
> >>
> >>
> >>
> >> --
> >> Best regards,
> >>
> >>- Andy
> >>
> >> If you are given a choice, you believe you have acted freely. - Raymond
> >> Teller (via Peter Watts)
> >>
>
>
>
> --
> Sean
>

Re: [ANNOUNCE] New HBase committer Chia-Ping Tsai

[Jerry He]

Congrats and welcome, Chia-Ping!


Jerry

On Fri, Mar 17, 2017 at 9:31 PM, Chia-Ping Tsai  wrote:

> thank you for inviting me here!
>
> On 2017-03-18 06:30 (+0800), Anoop John  wrote:
> > On behalf of the Apache HBase PMC, I am pleased to announce that
> Chia-Ping Tsai
> > has accepted the PMC's invitation to become a committer on the project.
> We
> > appreciate all of his contributions thus far and look forward
> > to his continued involvement.
> >
> > Congratulation and welcome Chia-Ping Tsai!
> >
> > -Anoop-
> >
>

Re: [VOTE] First release candidate for HBase 1.2.5 is available

[Jerry He]

+1

(non-binding)


- Downloaded the src tarball.
  - Checked the md5 and signature.  Looks good.
  - Built from source.  openjdk version "1.8.0_121"
  - Ran unit test suite twice.  All passed.
  - mvn apache-rat:check   Passed

- Download the bin tarball.
 -  Checked the md5 and signature.  Looks good.
 -  Extract it.   Layout looks good.
 -  Started a local instance.  Run the 'hbase shell' with some basic table
commands, split, flush, etc.  Looks good.
 -  Nothing unusual in the logs.
 -  Check master and region server UI.  Clicked on the tabs, looked at the
tasks running, Debug dump, Configuration dump, zk dump. Looks good.

- Built Phoenix 4.9-HBase-1.2 with hbase version 1.2.5 pointing to the RC
maven repo URL.
  Ran unit tests.  All passed.

-  Bulit YCSB 0.12.0 from git source with HBase binding version 1.2.5
pointing to RC.
-  Run YCSB workloads again the local RC instance.  Worked fine.

Jerry


On Mon, Mar 13, 2017 at 9:50 PM, Andrew Purtell  wrote:

> +1
>
> Checked sums and signatures: ok
> Built from source: ok (7u80)
> RAT check passed: ok (7u80)
> Unit tests pass: ok (8u102)
> Loaded 1M rows with LTT: ok (8u102), all keys verified, latencies in the
> ballpark, nothing unusual in the logs
>
>
> On Fri, Mar 10, 2017 at 2:31 PM, Sean Busbey  wrote:
>
> > The first release candidate for HBase 1.2.5 is available for download at:
> >
> > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.2.5RC0/
> >
> > Maven artifacts are also available in a staging repository at:
> >
> > https://repository.apache.org/content/repositories/orgapachehbase-1164/
> >
> > Artifacts are signed with my key (0D80DB7C) published in our KEYS
> > file at http://www.apache.org/dist/hbase/KEYS
> >
> > The RC corresponds to the signed tag 1.2.5RC0, which currently points
> > to commit ref
> >
> > d7b05f79dee10e0ada614765bb354b93d615a157
> >
> > The detailed source and binary compatibility report vs 1.2.4 has been
> > published for your review, at:
> >
> > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.2.5RC0/
> > 1.2.4_1.2.5RC0_compat_report.html
> >
> > Note that this report calls out the issue discussed in HBASE-17725.
> >
> > HBase 1.2.5 is the fifth maintenance release in the HBase 1.2 line,
> > continuing on
> > the theme of bringing a stable, reliable database to the Hadoop and NoSQL
> > communities. This release includes over 50 bug fixes since 1.2.4.
> > Critical fixes include:
> >
> > * HBASE-17069 RegionServer writes invalid META entries for split
> > daughters in some circumstances
> > * HBASE-17044 Fix merge failed before creating merged region leaves
> > meta inconsistent
> > * HBASE-17206 FSHLog may roll a new writer successfully with unflushed
> > entries
> > * HBASE-16765 New SteppingRegionSplitPolicy, avoid too aggressive
> > spread of regions for small tables.
> >
> >
> > The full list of fixes included in this release is available at:
> >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > version=12338339=12310753
> >
> > and in the CHANGES.txt file included in the distribution.
> >
> > Please try out this candidate and vote +1/-1 on whether we should
> > release these artifacts as HBase 1.2.5.
> >
> > The VOTE will remain open for atleast 72 hours. Given sufficient votes
> > I would like to close it on March 17th, 2017.
> >
> > thanks!
> >
>
>
>
> --
> Best regards,
>
>- Andy
>
> If you are given a choice, you believe you have acted freely. - Raymond
> Teller (via Peter Watts)
>

[jira] [Reopened] (HBASE-14161) Add hbase-spark integration tests to IT jenkins job

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He reopened HBASE-14161:
--

> Add hbase-spark integration tests to IT jenkins job
> ---
>
> Key: HBASE-14161
> URL: https://issues.apache.org/jira/browse/HBASE-14161
> Project: HBase
>  Issue Type: Task
>  Components: build
>Reporter: Sean Busbey
> Fix For: 2.0.0
>
>
> expand the set of ITs we run to include the new hbase-spark tests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (HBASE-14161) Add hbase-spark integration tests to IT jenkins job

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-14161.
--
Resolution: Duplicate

> Add hbase-spark integration tests to IT jenkins job
> ---
>
> Key: HBASE-14161
> URL: https://issues.apache.org/jira/browse/HBASE-14161
> Project: HBase
>  Issue Type: Task
>  Components: build
>Reporter: Sean Busbey
> Fix For: 2.0.0
>
>
> expand the set of ITs we run to include the new hbase-spark tests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (HBASE-14161) Add hbase-spark integration tests to IT jenkins job

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-14161.
--
Resolution: Fixed

> Add hbase-spark integration tests to IT jenkins job
> ---
>
> Key: HBASE-14161
> URL: https://issues.apache.org/jira/browse/HBASE-14161
> Project: HBase
>  Issue Type: Task
>  Components: build
>Reporter: Sean Busbey
> Fix For: 2.0.0
>
>
> expand the set of ITs we run to include the new hbase-spark tests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (HBASE-14167) hbase-spark integration tests do not respect -DskipIntegrationTests

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-14167.
--
   Resolution: Duplicate
Fix Version/s: 2.0.0

> hbase-spark integration tests do not respect -DskipIntegrationTests
> ---
>
> Key: HBASE-14167
> URL: https://issues.apache.org/jira/browse/HBASE-14167
> Project: HBase
>  Issue Type: Improvement
>  Components: pom, spark
>Affects Versions: 2.0.0
>Reporter: Andrew Purtell
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-14167.11.patch, HBASE-14167-master-v2.patch
>
>
> When running a build with {{mvn ... -DskipIntegrationTests}}, the hbase-spark 
> module's integration tests do not respect the flag and run anyway. Fix. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Moving 2.0 forward

[Jerry He]

Thanks for the write-up, Stack.

I take the liberty to make the Spark item as:

4.3 hbase-spark


4.3.1 Status=

IN_PROGRESS

4.3.2 Owner=
Jerry
and team

We will see how much we can dot to fix-up and improve.

Thanks.

Jerry

Re: [VOTE] First release candidate for HBase 1.1.9 (RC0) is available

[Jerry He]

+1
(non-binding)

- Downloaded the src tarball.
  - Checked the md5 and signature.  Looks good.
  - Built from source.  openjdk version "1.8.0_121"
  - Ran unit test suite.  All passed.
  - mvn apache-rat:check   Passed

- Download the bin tarball.
 -  Checked the md5 and signature.  Looks good.
 -  Extract it.   Looks good.
 -  Started a local instance.  Run the 'hbase shell' with some basic table
commands, split, flush, etc.  Looks good.

- Built Phoenix 4.8-HBase-1.1 with hbase version 1.1.9 pointing to the RC
using https://repository.apache.org/content/repositories/orgapachehbase-1163
  Ran unit tests.  All pass.

-  Bulit YCSB 0.12.0 from git source with HBase binding pointing to RC.
-  Run YCSB workloads again the local RC instance.  Worked fine.
-  Run the same YSCB against a HBase 1.2.0 cluster.  Worked fine.


Jerry



On Sat, Feb 25, 2017 at 7:20 PM, Nick Dimiduk  wrote:

> Thank you Andrew and Josh. For the rest of you lot, there's 30 hours
> remaining.
>
> On Sat, Feb 25, 2017 at 3:22 PM Josh Elser  wrote:
>
> > +1 (non-binding)
> >
> > * sigs/xsums are good
> > * Can build from source
> > * No unexpected binaries in source tarball
> > * apache-rat:check passes on source tarball
> > * Can run locally from bin tarball
> >
> > Nick Dimiduk wrote:
> > > I'm happy to announce the first release candidate of HBase 1.1.9
> > (HBase-1.1.
> > > 9RC0) is available for download at
> > > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.1.9RC0/
> > >
> > > Maven artifacts are also available in the staging repository
> > > https://repository.apache.org/content/repositories/orgapachehbase-1163
> > >
> > > Artifacts are signed with my code signing subkey 0xAD9039071C3489BD,
> > > available in the Apache keys directory https://people.
> > > apache.org/keys/committer/ndimiduk.asc
> > >
> > > There's also a signed tag for this release at
> > >
> > https://git-wip-us.apache.org/repos/asf?p=hbase.git;a=commit
> ;h=0d1feabed5295495ed2257d31fab9e6553e8a9d7
> > >
> > > The detailed source and binary compatibility report vs 1.1.8 has been
> > > published for your review, at
> > > http://home.apache.org/~ndimiduk/1.1.8_1.1.9RC0_compat_report.html
> > >
> > > HBase 1.1.9 is the ninth patch release in the HBase 1.1 line,
> continuing
> > on
> > > the theme of bringing a stable, reliable database to the Hadoop and
> NoSQL
> > > communities. This release includes nearly 20 bug fixes since the 1.1.8
> > > release. Notable correctness fixes include HBASE-17238,
> > > HBASE-17587, HBASE-17275, and HBASE-17265.
> > >
> > > The full list of fixes included in this release is available at
> > >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
> ctId=12310753=12338734
> > >   and in the CHANGES.txt file included in the distribution.
> > >
> > > Please try out this candidate and vote +/-1 by 23:59 Pacific time on
> > > Sunday, 2017-02-26 as to whether we should release these artifacts as
> > HBase
> > > 1.1.9.
> > >
> > > Thanks,
> > > Nick
> > >
> >
>

Re: [DISCUSSION] Upgrading core dependencies

[Jerry He]

I was more on the thinking to avoid/reduce pulling in hbase dependencies
into hbase-spark, and that maybe even hbase-spark can depend on shaded
client and server -- it will be easier and more feasible if the shaded
client becomes the default as you mentioned.

Your idea that hbase-spark itself becomes a shaded artifact sounds better
if I understand you correctly. The spark/scala dependencies are 'provided'
already.

Jerry



On Wed, Feb 8, 2017 at 6:14 PM, Nick Dimiduk <ndimi...@gmail.com> wrote:

> On Wed, Feb 8, 2017 at 10:24 AM Jerry He <jerry...@gmail.com> wrote:
>
> > Yeah.  Talking about the dependency, the hbase-spark module already has
> > dependency on hbase-server (coming from the spark bulk load producing
> > hfiles).
> > This is not very good. We have to be careful not entangling it more.
> > Also, there is already problem running the hbase-spark due to dependency
> > conflict, and one has to be careful about the order of the classpath to
> > make it work.
>
> We own the hbase-spark module, do we not? In that case, we control our own
> destiny. An explicit goal of this effort would be to make use of that
> module agnostic to classpath load order. As per my earlier reply,
> hbase-spark could itself be an artifact shaded over hbase-server and all of
> its dependencies. That way the user doesn't need to think about it at all.
>
> It further seems to me that the maven-shade-plugin could gain a new analyze
> goal, similar to the that of the dependency-plugin, which would audit the
> classes packaged in a jar vs the contract defined in configuration. This
> could further be used to fail the build of there's any warnings reported.
> I've found myself wanting this very functionality as I consume ES and
> Phoenix, shaded in downstream projects.
>

Re: [DISCUSSION] Upgrading core dependencies

[Jerry He]

Yeah.  Talking about the dependency, the hbase-spark module already has
dependency on hbase-server (coming from the spark bulk load producing
hfiles).
This is not very good. We have to be careful not entangling it more.
Also, there is already problem running the hbase-spark due to dependency
conflict, and one has to be careful about the order of the classpath to
make it work.

Jerry

[jira] [Resolved] (HBASE-17502) Document hadoop pre-2.6.1 and Java 1.8 Kerberos problem in our hadoop support matrix

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-17502.
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.0.0

Pushed.
Thanks for the review, Ted.

> Document hadoop pre-2.6.1 and Java 1.8 Kerberos problem in our hadoop support 
> matrix
> 
>
> Key: HBASE-17502
> URL: https://issues.apache.org/jira/browse/HBASE-17502
> Project: HBase
>  Issue Type: Task
>    Reporter: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-17502.patch, HBASE-17502-v2.patch
>
>
> Hadoop pre-2.6.1 on JDK 1.8 has problem with Kerberos keytabe relogin.
> HADOOP-10786 fixed the problem in Hadoop 2.6.1.
> Let's document it in the Hadoop support matrix.
> This was brought up in HBase 1.3.0 RC0 voting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17502) Document hadoop pre-2.6.1 and Java 1.8 Kerberos problem in our hadoop support matrix

[Jerry He (JIRA)]

Jerry He created HBASE-17502:


 Summary: Document hadoop pre-2.6.1 and Java 1.8 Kerberos problem 
in our hadoop support matrix
 Key: HBASE-17502
 URL: https://issues.apache.org/jira/browse/HBASE-17502
 Project: HBase
  Issue Type: Task
Reporter: Jerry He


Hadoop pre-2.6.1 on JDK 1.8 has problem with Kerberos keytabe relogin.
HADOOP-10786 fixed the problem in Hadoop 2.6.1.
Let's document it in the Hadoop support matrix.

This was brought up in HBase 1.3.0 RC0 voting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] The 1st HBase 1.3.0 release candidate (RC0) is available

[Jerry He]

Sounds good.
I was thinking we could put a note in the Reference Guide for this problem.
There are sections on JDK support metrics and Hadoop support metrics (
https://hbase.apache.org/book.html#basic.prerequisites). They are good
places to document this and suggest to use a later version of Hadoop if
needed.  I can open a JIRA to do it.

Thanks.

Jerry



On Mon, Jan 16, 2017 at 10:01 PM, Mikhail Antonov 
wrote:

> Thanks Stack. I expect starting merges for 1.3.1 in next coming weeks, I
> think there's a dozen or so patches waiting for review
> and backport, and the more large-scale testing we can get the better.
>
> Thanks for testing that kerberized setup Jerry. Looking at the patch
> though, I imagine older HBase releases should
> also suffer from it, if they run on 2.5.1 Hadoop? Also as was noted in the
> jira, people were able to reproduce this
> on recent builds of JDK 1.7, so it's not 1.8 specific.
>
> We do strive to support older Hadoop releases and should be able to run on
> 2.5.1 as such, but this bug in Hadoop Common
> have been resolved year and a half ago and people running production
> kerberized clusters should have upgraded Hadoop
> to deal with such issues.
>
> I think it might worth separate thread to discuss how to deal with it,
> should we just say that we support 2.5.1 but recommend something later,
> or should we say that we recommend newer versions for security setups, or
> should we just make broad claim that we support 2.5.1 but
> in certain deployments users will be subject to bugs in underlying
> dependencies. What do you think?
>
> -Mikhail
>
>
>
>
> On Mon, Jan 16, 2017 at 9:11 PM, Stack  wrote:
>
> > Late +1
> >
> > Checked signature and hash on src tgz.
> > Built tgz from src w/ jdk7.
> > Loaded some data. Verified it made it.
> > Browsed UI, logs, and docs.
> >
> > All looks good.
> >
> > St.Ack
> >
> > P.S. Sorry, didn't get a chance to deploy and run RC test at larger
> scale.
> > Will do for 1.3.1. Have done a bunch of larger scale ITBLLs w/ Chaos on
> > recent tips of 1.3 and it seemed sound.
> >
> >
> > On Fri, Jan 6, 2017 at 12:42 PM, Mikhail Antonov 
> > wrote:
> >
> > > Hello everyone,
> > >
> > > I'm pleased to announce the first release candidate for HBase 1.3.0 is
> > > available to download and testing.
> > >
> > > Artifacts are available here:
> > >
> > > https://dist.apache.org/repos/dist/dev/hbase/1.3.0RC0/
> > >
> > > Maven artifacts are available in the staging repository:
> > >
> > > https://repository.apache.org/content/repositories/
> orgapachehbase-1162/
> > >
> > > All artifacts are signed with my code signing key 35A4ABE2, which is
> also
> > > in the project KEYS file at
> > >
> > > http://www.apache.org/dist/hbase/KEYS
> > >
> > > these artifacts correspond to commit hash
> > >
> > > e359c76e8d9fd0d67396456f92bcbad9ecd7a710 tagged as 1.3.0RC0.
> > >
> > > HBase 1.3.0 is the third minor release in the HBase 1.x line and
> includes
> > > approximately 1700 resolved issues.
> > >
> > > Notable new features include:
> > >
> > >  - Date-based tiered compactions (HBASE-15181, HBASE-15339)
> > >  - Maven archetypes for HBase client applications (HBASE-14877)
> > >  - Throughput controller for flushes (HBASE-14969)
> > >  - Controlled delay (CoDel) based RPC scheduler (HBASE-15136)
> > >  - Bulk loaded HFile replication (HBASE-13153)
> > >  - More improvements to Procedure V2
> > >  - Improvements to Multi WAL (HBASE-14457)
> > >  - Many improvements and optimizations in metrics subsystem
> > >  - Reduced memory allocation in RPC layer (HBASE-15177)
> > >  - Region location lookups optimizations in HBase client
> > >
> > > and numerous bugfixes and performance improvements.
> > >
> > > The full list of issues addressed is available at
> > >
> > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
> > > ctId=12310753=12332794
> > >
> > > and also in the CHANGES.txt file in the root directory of the source
> > > tarball.
> > >
> > > Please take a few minutes to verify the release and vote on
> > > releasing it:
> > >
> > > [ ] +1 Release this package as Apache HBase 1.3.0
> > > [ ] +0 no opinion
> > > [ ] -1 Do not release this package because...
> > >
> > > This Vote will run for one week and close Fri Jan 13, 2017 22:00 PST.
> > >
> > > Thank you!
> > > Mikhail
> > >
> >
>
>
>
> --
> Thanks,
> Michael Antonov
>

Re: [VOTE] The 1st HBase 1.3.0 release candidate (RC0) is available

[Jerry He]

Hi, Mikhail

It looks like the hadoop 2.5.1 bundled with the HBase 1.3.0 would make the
HBase servers (master and region server) fail/abort in a Kerberos
environment after the initial server login expires. It happens on Java 1.8.
This seems to be the related JIRA:
https://issues.apache.org/jira/browse/HADOOP-10786
The root problem is UGI#reloginFromKeytab.  It happened on my HBase 1.3.0
RC0 testing cluster with OpenJDk 1.8.0_101-b13 after 24 hours.

Since Java 1.8 is supported by HBase 1.3.0.  I think we probably should
call it out in a documentation.

Thanks.

Jerry


On Mon, Jan 16, 2017 at 2:30 AM, Mikhail Antonov <olorinb...@gmail.com>
wrote:

> With 4 binding +1 (including mine), 2 non-binding +1 and no other votes
> this vote passes and RC0 becomes 1.3.0 release.
>
> Thanks for everyone who tested the RC and voted!
>
> I'll publish artifacts and make an announcements soon,
>
> Thanks!
> Mikhail
>
>
> On Sat, Jan 14, 2017 at 12:12 PM, Jerry He <jerry...@gmail.com> wrote:
>
> > +1
> >
> > - Downloaded the src tarball.
> >   - Checked the md5 and signature.  All good.
> >   - Built from source.   OpenJDk 1.8.0_101-b13
> >   - Unit test suite.  Passed.
> >   - mvn apache-rat:check   Passed
> >
> > - Download the bin tarball.
> >  -  Checked the md5 and signature.  All good.
> >  -  Untar it.  Layout and files look good.
> >
> > - Put the binaries on a distributed Kerberos cluster
> >  - With all the security co-processors enabled.
> >  - Ran hbase shell. Table put, scan, split, major-compact, merge-region.
> > Permission checkes good.
> >  - SecureBulkLoad good.
> >  - Security audit trace looks good. Master and region server logs look
> > good.
> >
> >  - Check master and region server UI.  Clicked on the tabs, looked at the
> > tasks running, Debug dump, Configuration dump, zk dump. Looks good.
> >
> >  - Got this exception when running PerformanceEvaluation or other MR job:
> >   org.codehaus.jackson.map.exc.UnrecognizedPropertyException:
> Unrecognized
> > field "Token" (Class
> > org.apache.hadoop.yarn.api.records.timeline.
> TimelineDelegationTokenRespons
> > e),
> > not marked as ignorable
> >  at [Source: N/A; line: -1, column: -1] (through reference chain:
> > org.apache.hadoop.yarn.api.records.timeline.
> TimelineDelegationTokenRespons
> > e["Token"])
> >
> >My Hadoop/Yarn cluster is 2.7.2.  HBase 1.3.0 bundles 2.5.1. That may
> be
> > the problem.
> >After copying hadoop 2.7.2 jars into hbase.  The jobs run fine.
> >Loaded data.
> >
> > On Sat, Jan 14, 2017 at 12:51 AM, Elliott Clark <ecl...@apache.org>
> wrote:
> >
> > > +1 Checked sigs.
> > > Downloaded everything looks good.
> > > License and Notice files look good.
> > >
> > > On Mon, Jan 9, 2017 at 7:11 PM, Andrew Purtell <apurt...@apache.org>
> > > wrote:
> > >
> > > > +1
> > > >
> > > > Checked sums and signatures: ok
> > > > Spot check LICENSE and NOTICE files: ok
> > > > Built from source (7u80): ok
> > > > RAT audit (7u80): ok
> > > > Unit test suite passes (8u102):
> > > > Loaded 1 million rows with LTT (8u102, 10 readers, 10 writers, 10
> > > updaters
> > > > @ 20%): all keys verified, no unexpected or unusual messages in the
> > logs,
> > > > latencies in the ballpark
> > > > 1 billion row ITBLL (8u102): failed, but known issue HBASE-17069
> > > >
> > > >
> > > >
> > > > On Fri, Jan 6, 2017 at 12:42 PM, Mikhail Antonov <
> olorinb...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hello everyone,
> > > > >
> > > > > I'm pleased to announce the first release candidate for HBase 1.3.0
> > is
> > > > > available to download and testing.
> > > > >
> > > > > Artifacts are available here:
> > > > >
> > > > > https://dist.apache.org/repos/dist/dev/hbase/1.3.0RC0/
> > > > >
> > > > > Maven artifacts are available in the staging repository:
> > > > >
> > > > > https://repository.apache.org/content/repositories/
> > > orgapachehbase-1162/
> > > > >
> > > > > All artifacts are signed with my code signing key 35A4ABE2, which
> is
> > > also
> > > > > in the project KEYS file at
> > > > >
> > > > > http://www.apache.org/dist/hbase

Re: [DISCUSS] hbase-spark module in branch-1 and branch-2

[Jerry He]

Hi, Andrew

Stack was talking to me about this area when I met him in the HBase Meetup
last December.
Let me take a shot at HBASE-14375.

Thanks,

Jerry

On Sat, Jan 14, 2017 at 9:22 PM, Andrew Purtell <andrew.purt...@gmail.com>
wrote:

>
>
> > On Jan 14, 2017, at 9:07 PM, Jerry He <jerry...@gmail.com> wrote:
> >
> > I think it will be a big disappointment for the community if the
> > hbase-spark module is not going into 2.0.
> > I understand there are still a few blockers, including HBASE-16179.
>
> Patches welcome. :-)
>
>
> > We have it in our distribution, probably in other vendors' as well.  It
> is
> > little easier for us because we can be flexible on the supported
> > Spark/Scala version combinations and the APIs.
> > But a major release still without a good Spark story for the HBase open
> > source community does not look good.
> >
> > Jerry
> >
> >> On Sat, Jan 14, 2017 at 4:52 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> >>
> >> I agree with Devaraj's assessment w.r.t. hbase-spark module in master
> >> (which is becoming branch-2).
> >>
> >> Cheers
> >>
> >>
> >>
> >> On Mon, Nov 21, 2016 at 11:46 AM, Devaraj Das <d...@hortonworks.com>
> >> wrote:
> >>
> >>> Hi Sean, I did a quick check with someone from the Spark team here and
> >> his
> >>> opinion was that the hbase-spark module as it currently stands can be
> >> used
> >>> by downstream users to do basic stuff and to try some simple things
> out,
> >>> etc. The integration is improving.
> >>> I think we should get what we have in 2.0 (which is the default action
> >>> anyways).
> >>> Thanks
> >>> Devaraj
> >>> 
> >>> From: Sean Busbey <bus...@apache.org>
> >>> Sent: Wednesday, November 16, 2016 9:49 AM
> >>> To: dev
> >>> Subject: [DISCUSS] hbase-spark module in branch-1 and branch-2
> >>>
> >>> Hi folks!
> >>>
> >>> With 2.0 releases coming up, I'd like to revive our prior discussion
> >>> on the readiness of the hbase-spark module for downstream users.
> >>>
> >>> We've had a ticket for tracking the milestones set up for inclusion in
> >>> branch-1 releases for about 1.5 years:
> >>>
> >>> https://issues.apache.org/jira/browse/HBASE-14160
> >>>
> >>> We still haven't gotten all of the blocker issues completed, AFAIK.
> >>>
> >>> Is anyone interested in volunteering to knock the rest of these out?
> >>>
> >>> If they aren't, shall we plan to leave hbase-spark in master and
> >>> revert it from branch-2 once it forks for the HBase 2.0 release line?
> >>>
> >>> This feature isn't a blocker for 2.0; just as we've been planning to
> >>> add the hbase-spark module to some 1.y release we can also include it
> >>> in a 2.1+ release.
> >>>
> >>> This does appear to be a feature our downstream users could benefit
> >>> from, so I'd hate to continue the current situation where no official
> >>> releases include it. This is especially true now that we're looking at
> >>> ways to handle changes between Spark 1.6 and Spark 2.0 in HBASE-16179.
> >>>
> >>> -
> >>> busbey
> >>>
> >>>
> >>
>

Re: [DISCUSS] hbase-spark module in branch-1 and branch-2

[Jerry He]

I think it will be a big disappointment for the community if the
hbase-spark module is not going into 2.0.
I understand there are still a few blockers, including HBASE-16179.
We have it in our distribution, probably in other vendors' as well.  It is
little easier for us because we can be flexible on the supported
Spark/Scala version combinations and the APIs.
But a major release still without a good Spark story for the HBase open
source community does not look good.

Jerry

On Sat, Jan 14, 2017 at 4:52 PM, Ted Yu  wrote:

> I agree with Devaraj's assessment w.r.t. hbase-spark module in master
> (which is becoming branch-2).
>
> Cheers
>
>
>
> On Mon, Nov 21, 2016 at 11:46 AM, Devaraj Das 
> wrote:
>
> > Hi Sean, I did a quick check with someone from the Spark team here and
> his
> > opinion was that the hbase-spark module as it currently stands can be
> used
> > by downstream users to do basic stuff and to try some simple things out,
> > etc. The integration is improving.
> > I think we should get what we have in 2.0 (which is the default action
> > anyways).
> > Thanks
> > Devaraj
> > 
> > From: Sean Busbey 
> > Sent: Wednesday, November 16, 2016 9:49 AM
> > To: dev
> > Subject: [DISCUSS] hbase-spark module in branch-1 and branch-2
> >
> > Hi folks!
> >
> > With 2.0 releases coming up, I'd like to revive our prior discussion
> > on the readiness of the hbase-spark module for downstream users.
> >
> > We've had a ticket for tracking the milestones set up for inclusion in
> > branch-1 releases for about 1.5 years:
> >
> > https://issues.apache.org/jira/browse/HBASE-14160
> >
> > We still haven't gotten all of the blocker issues completed, AFAIK.
> >
> > Is anyone interested in volunteering to knock the rest of these out?
> >
> > If they aren't, shall we plan to leave hbase-spark in master and
> > revert it from branch-2 once it forks for the HBase 2.0 release line?
> >
> > This feature isn't a blocker for 2.0; just as we've been planning to
> > add the hbase-spark module to some 1.y release we can also include it
> > in a 2.1+ release.
> >
> > This does appear to be a feature our downstream users could benefit
> > from, so I'd hate to continue the current situation where no official
> > releases include it. This is especially true now that we're looking at
> > ways to handle changes between Spark 1.6 and Spark 2.0 in HBASE-16179.
> >
> > -
> > busbey
> >
> >
>

Re: [VOTE] The 1st HBase 1.3.0 release candidate (RC0) is available

[Jerry He]

+1

- Downloaded the src tarball.
  - Checked the md5 and signature.  All good.
  - Built from source.   OpenJDk 1.8.0_101-b13
  - Unit test suite.  Passed.
  - mvn apache-rat:check   Passed

- Download the bin tarball.
 -  Checked the md5 and signature.  All good.
 -  Untar it.  Layout and files look good.

- Put the binaries on a distributed Kerberos cluster
 - With all the security co-processors enabled.
 - Ran hbase shell. Table put, scan, split, major-compact, merge-region.
Permission checkes good.
 - SecureBulkLoad good.
 - Security audit trace looks good. Master and region server logs look good.

 - Check master and region server UI.  Clicked on the tabs, looked at the
tasks running, Debug dump, Configuration dump, zk dump. Looks good.

 - Got this exception when running PerformanceEvaluation or other MR job:
  org.codehaus.jackson.map.exc.UnrecognizedPropertyException: Unrecognized
field "Token" (Class
org.apache.hadoop.yarn.api.records.timeline.TimelineDelegationTokenResponse),
not marked as ignorable
 at [Source: N/A; line: -1, column: -1] (through reference chain:
org.apache.hadoop.yarn.api.records.timeline.TimelineDelegationTokenResponse["Token"])

   My Hadoop/Yarn cluster is 2.7.2.  HBase 1.3.0 bundles 2.5.1. That may be
the problem.
   After copying hadoop 2.7.2 jars into hbase.  The jobs run fine.
   Loaded data.

On Sat, Jan 14, 2017 at 12:51 AM, Elliott Clark  wrote:

> +1 Checked sigs.
> Downloaded everything looks good.
> License and Notice files look good.
>
> On Mon, Jan 9, 2017 at 7:11 PM, Andrew Purtell 
> wrote:
>
> > +1
> >
> > Checked sums and signatures: ok
> > Spot check LICENSE and NOTICE files: ok
> > Built from source (7u80): ok
> > RAT audit (7u80): ok
> > Unit test suite passes (8u102):
> > Loaded 1 million rows with LTT (8u102, 10 readers, 10 writers, 10
> updaters
> > @ 20%): all keys verified, no unexpected or unusual messages in the logs,
> > latencies in the ballpark
> > 1 billion row ITBLL (8u102): failed, but known issue HBASE-17069
> >
> >
> >
> > On Fri, Jan 6, 2017 at 12:42 PM, Mikhail Antonov 
> > wrote:
> >
> > > Hello everyone,
> > >
> > > I'm pleased to announce the first release candidate for HBase 1.3.0 is
> > > available to download and testing.
> > >
> > > Artifacts are available here:
> > >
> > > https://dist.apache.org/repos/dist/dev/hbase/1.3.0RC0/
> > >
> > > Maven artifacts are available in the staging repository:
> > >
> > > https://repository.apache.org/content/repositories/
> orgapachehbase-1162/
> > >
> > > All artifacts are signed with my code signing key 35A4ABE2, which is
> also
> > > in the project KEYS file at
> > >
> > > http://www.apache.org/dist/hbase/KEYS
> > >
> > > these artifacts correspond to commit hash
> > >
> > > e359c76e8d9fd0d67396456f92bcbad9ecd7a710 tagged as 1.3.0RC0.
> > >
> > > HBase 1.3.0 is the third minor release in the HBase 1.x line and
> includes
> > > approximately 1700 resolved issues.
> > >
> > > Notable new features include:
> > >
> > >  - Date-based tiered compactions (HBASE-15181, HBASE-15339)
> > >  - Maven archetypes for HBase client applications (HBASE-14877)
> > >  - Throughput controller for flushes (HBASE-14969)
> > >  - Controlled delay (CoDel) based RPC scheduler (HBASE-15136)
> > >  - Bulk loaded HFile replication (HBASE-13153)
> > >  - More improvements to Procedure V2
> > >  - Improvements to Multi WAL (HBASE-14457)
> > >  - Many improvements and optimizations in metrics subsystem
> > >  - Reduced memory allocation in RPC layer (HBASE-15177)
> > >  - Region location lookups optimizations in HBase client
> > >
> > > and numerous bugfixes and performance improvements.
> > >
> > > The full list of issues addressed is available at
> > >
> > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > projectId=12310753=12332794
> > >
> > > and also in the CHANGES.txt file in the root directory of the source
> > > tarball.
> > >
> > > Please take a few minutes to verify the release and vote on
> > > releasing it:
> > >
> > > [ ] +1 Release this package as Apache HBase 1.3.0
> > > [ ] +0 no opinion
> > > [ ] -1 Do not release this package because...
> > >
> > > This Vote will run for one week and close Fri Jan 13, 2017 22:00 PST.
> > >
> > > Thank you!
> > > Mikhail
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> >- Andy
> >
> > If you are given a choice, you believe you have acted freely. - Raymond
> > Teller (via Peter Watts)
> >
>

Re: Moving 2.0 forward

[Jerry He]

I agree we need a long and stable 1.x release. Branch-1 is a good fit for
that role.
It has the stability and compatibility of 1.x, and it has still been quite
open for flow of improvements and commits.

+1


Jerry

On Fri, Jan 6, 2017 at 1:01 PM, Mikhail Antonov 
wrote:

> I support that idea of cutting branch-2 early. Yes it will create some
> burden for the RM and committers to port things between the
> branches, but until the branch is cut we won't have that sense of imminense
> of approaching release, and more importantly, until
> branch is cut _all_ commits will continue to go there, making it hard to
> stabilize.
>
> Regarding branch-1 and branch-2 release lines, agree those are unrelated
> questions. I'm all for frequent and fast updates to new versions, but
> obviously we can't drop support and development on branch-1 until 2.0 is
> released and probed by early adopters, and then not until 2.0 is as stable
> as what people running late 1.* branches currently have.
>
> Thanks,
> Mikhail
>
> On Fri, Jan 6, 2017 at 10:40 AM, Andrew Purtell 
> wrote:
>
> > Considerations for a new branch-2 and branch-1 are orthogonal in my
> > opinion.
> >
> > I intend to volunteer to be the RM for branch-1 itself (we've not had one
> > before) as necessary for it to become a stable source of incremental
> > releases for a long time, similar to how we had 0.98 active for almost
> > three years while 1.x development took place. Where I work we plan to
> have
> > branch-1 based code in production for at least one year, probably longer.
> >
> > Given the above arrangement, releases from branch-1 and branch-2 would
> > have independent roadmaps and release timelines.
> >
> > Does this sound reasonable?
> >
> >
> > > On Jan 5, 2017, at 11:51 PM, Phil Yang  wrote:
> > >
> > > Hi all
> > > After cutting branch-2, what will we do for branch-1? If I am not
> wrong,
> > > 1.4 may be the last 1.x release branch? Should 1.4.0 release before
> > 2.0.0?
> > > If not, will it confuse users?
> > >
> > > Thanks,
> > > Phil
> > >
> > >
> > > 2017-01-01 5:20 GMT+08:00 Andrew Purtell :
> > >
> > >> On the other hand branching will force the issue. There will always be
> > >> lists of issues to get in. How long have we been talking about 2.0? At
> > >> least a year and a half. At some point it's time to stop talking and
> > take
> > >> action. Let me revisit progress at the end of January and bring this
> up
> > >> again. As a member of the PMC I'm advising all concerned that 2.0 is
> > >> talking too long and I am considering steps to move it forward.
> > >>
> > >>
> > >>> On Dec 31, 2016, at 12:54 PM, Ted Yu  wrote:
> > >>>
> > >>> I agree with Stephen on not branching too early.
> > >>>
> > >>> When people come back from vacation, we can poll relevant parties on
> > >>> estimate of respective project to get a sense of when would be proper
> > >> time
> > >>> for branching.
> > >>>
> > >>> On Sat, Dec 31, 2016 at 12:16 PM, Stephen Jiang <
> > syuanjiang...@gmail.com
> > >>>
> > >>> wrote:
> > >>>
> >  Hello, Andrew, I was a helper on Matteo so that we can help each
> other
> >  while we are focusing on the new Assignment Manager work.  Now he is
> > not
> >  available (at least in the next few months).  I have to be more
> > focused
> > >> on
> >  the new AM work; plus other work in my company; it would be too much
> > >> for me
> >  to 2.0 RM alone.  I am happy someone would help to take primary 2.0
> RM
> > >> role
> >  while I am still help to make this 2.0 release smooth.
> > 
> >  For branch-2, I think it is too early to cut it, as we still have a
> > lot
> > >> of
> >  moving parts and on-going project that needs to be part of 2.0.  For
> >  example, the mentioned new AM (and other projects, such as
> > HBASE-14414,
> >  HBASE-15179, HBASE-14070, HBASE-14850, HBASE-16833, HBASE-15531,
> just
> > >> name
> >  a few).  Cutting branch now would add burden to complete those
> > projects.
> > 
> >  thanks
> >  Stephen
> > 
> >  On Sat, Dec 31, 2016 at 10:54 AM, Andrew Purtell <
> > >> andrew.purt...@gmail.com
> > >
> >  wrote:
> > 
> > > Hi all,
> > >
> > > I've heard a rumor the co-RM situation with 2.0 may have changed.
> Can
> > >> we
> > > get an update from co-RMs Matteo and Steven on their availability
> and
> > > interest in continuing in this role?
> > >
> > > To assist in moving 2.0 forward I intend to branch branch-2 from
> > master
> > > next week. Unless there is an objection I will take this action
> under
> > > assumption of lazy consensus. Master branch will be renumbered to
> > > 3.0.0-SNAPSHOT. Once we have a branch-2 I will immediately begin
> > scale
> > > tests and stabilization (via bug fixes or reverts of unfinished
> work)
> > >> and
> > > invite interested collaborators to 

Re: [VOTE] The 1st HBase 0.98.24 release candidate (RC0) is available

[Jerry He]

Happy new year!

+1

- Downloaded the src tarball.
  - Checked the md5 and signature.  Looks good.
  - Built from source.
  - Unit test suite, 2 times.  All passed.
  - mvn apache-rat:check   Passed

- Download the hadoop-2 bin tarball.
 -  Checked the md5 and signature.  Looks good.
 -  Untar it.  Layout looks good.
 -  Started a local instance.  Run the 'hbase shell' with some basic table
commands.  Looks good.

-  Bulit YCSB 0.12.0 from git source with HBase binding pointing to 0.98.24
RC0 using https://repository.apache.org/content/repositories/
orgapachehbase-1161/ .
-  Run YCSB workloads again the local 0.98.24 RC0 instance.  Works fine.
-  Run the same YSCB against a HBase 1.2.0 cluster.  Works fine.

Thanks.

Jerry


On Sun, Jan 1, 2017 at 5:42 PM, Stack  wrote:

> On Sat, Dec 31, 2016 at 10:44 AM, Andrew Purtell  >
> wrote:
>
> > Happy new year!
> >
> >
> HNY yourself Mr. Purtell.
>
>
>
> > Here's to 2017 being better than 2016.
> >
> >
> Amen.
>
>
>
> > As you return from vacation et cetera in the coming week, please consider
> > taking the time to vote on one last 0.98 release candidate. We only need
> > two +1s besides my own to get it out.
> >
> >
> Here is my valedictory +1 on the 0.98 branch.
>
> Downloaded src bundle. Checked hash and signature.
> Built it. Started it. Checked logs.
> Loaded some data. Verified it made it in.
> Built site. Poked around. Looks good.
>
> Thanks,
> St.Ack
>
>
>
> >
> > > On Dec 21, 2016, at 6:55 PM, Andrew Purtell 
> wrote:
> > >
> > > The 1st HBase 0.98.2​4 release candidate (RC0) is available for
> download
> > at https://dist.apache.org/repos/dist/dev/hbase/hbase-0.98.24RC0 and
> > Maven artifacts are also available in the temporary repository
> > https://repository.apache.org/content/repositories/orgapachehbase-1161/
> .
> > The git tag corresponding to the candidate is '0.98.24RC0' (9c13a1c).
> > >
> > > The detailed source and binary compatibility report for this release
> > with respect to the previous is available for your review at
> > https://dist.apache.org/repos/dist/dev/hbase/hbase-0.98.
> > 24RC0/0.98.23_0.98.24RC0_compat_report.html ​. There are no reported
> > compatibility issues.
> > >
> > > The ​33​ issues resolved in this release can be found at
> > https://s.apache.org/qDNI .
> > >
> > > I have made the following assessments of this candidate:
> > >
> > > - Release audit check​: pass​ (7u80)
> > >
> > > -​ Unit test suite: pass 20/20 iterations (7u80)​
> > >
> > > - Loaded 1M keys with LTT (10 readers, 10 writers, 10 updaters
> > (20%): all keys verified, no unusual messages or errors, latencies in the
> > ballpark (8u102)
> > >
> > > - IntegrationTestBigLinkedList ​1B rows: 100% referenced, no errors
> > (8u102, Hadoop 2.7.3)
> > >
> > > - Built head of Apache Phoenix 4.x-HBase-0.98 branch​:​ ​no errors
> > (7u80)
> > >
> > > Signed with my code signing key D5365CCD.
> > >
> > > Please try out the candidate and vote +1/0/-1. This vote will be open
> > for at least 72 hours. Unless objection I will try to close it ​Tuesday
> > January 3, 2017 if we have sufficient votes.
> > >
> > >
> > > --
> > > Best regards,
> > >
> > >- Andy
> > >
> > > If you are given a choice, you believe you have acted freely. - Raymond
> > Teller (via Peter Watts)
> >
>

Re: Draining mode and graceful decommissioning

[Jerry He]

Please see https://issues.apache.org/jira/browse/HBASE-16010 as well.
Nice testing you did.  zk_dump not dumping draining nodes is probably a
small bug.
Yes, we need to update/improve the related scripts.

Thanks.

Jerry

On Sun, Jan 1, 2017 at 6:56 AM, Lars George  wrote:

> Hi,
>
> This is along the lines of https://issues.apache.org/
> jira/browse/HBASE-10367
>
> Since we have draining mode, should we not in the meantime at least
> replace the balancer stop/start step in graceful_stop.sh with simply
> setting the draining mode for the server? This would less impact any
> other balancer work.
>
> BTW, I tried on 1.3 to figure the state of the balancer handling the
> draining mode flag for a node, and it seems to do the right thing (see
> https://www.evernote.com/l/ACHWQmzwekZN9LIVcInLjviefhwngwuhqJE) - are
> there any concerns that it does not? Enis (and Stack in the original
> JIRA) mentioned something about it only being implemented in one part,
> but not the other?
>
> Cheers,
> Lars
>

Re: Approach: Incremental data load from HBASE

[Jerry He]

There is no magic in the sqoop incremental import. You need a key column or
timestamp column to let sqoop know where to start each incremental.

HBase has built in timestamp.  Please look at the hbase bundled MR tool.
Export. https://hbase.apache.org/book.html#tools
There are options that let you specify starttime and endtime.
You can also write your own MR or Spark job to do incremental export of
HBase data by providing timestamps to the Scan or providing filter.

Jerry

On Sat, Dec 24, 2016 at 7:24 PM, Chetan Khatri 
wrote:

> Hello HBase Community,
>
> What is suggested approach for Incremental import from HBase to HDFS, like
> RDBMS to HDFS Sqoop provides support with below script
>
> sqoop job --create myssb1 -- import --connect
> jdbc:mysql://:/sakila --username admin --password admin
> --driver=com.mysql.jdbc.Driver --query "SELECT address_id, address,
> district, city_id, postal_code, alast_update, cityid, city, country_id,
> clast_update FROM(SELECT a.address_id as address_id, a.address as address,
> a.district as district, a.city_id as city_id, a.postal_code as postal_code,
> a.last_update as alast_update, c.city_id as cityid, c.city as city,
> c.country_id as country_id, c.last_update as clast_update FROM
> sakila.address a INNER JOIN sakila.city c ON a.city_id=c.city_id) as sub
> WHERE $CONDITIONS" --incremental lastmodified --check-column alast_update
> --last-value 1900-01-01 --target-dir /user/cloudera/ssb7 --hive-import
> --hive-table test.sakila -m 1 --hive-drop-import-delims --map-column-java
> address=String
>
>
> Thanks.
>
> On Wed, Dec 21, 2016 at 3:58 PM, Chetan Khatri <
> chetan.opensou...@gmail.com>
> wrote:
>
> > Hello Guys,
> >
> > I would like to understand different approach for Distributed Incremental
> > load from HBase, Is there any *tool / incubactor tool* which satisfy
> > requirement ?
> >
> > *Approach 1:*
> >
> > Write Kafka Producer and maintain manually column flag for events and
> > ingest it with Linkedin Gobblin to HDFS / S3.
> >
> > *Approach 2:*
> >
> > Run Scheduled Spark Job - Read from HBase and do transformations and
> > maintain flag column at HBase Level.
> >
> > In above both approach, I need to maintain column level flags. such as 0
> -
> > by default, 1-sent,2-sent and acknowledged. So next time Producer will
> take
> > another 1000 rows of batch where flag is 0 or 1.
> >
> > I am looking for best practice approach with any distributed tool.
> >
> > Thanks.
> >
> > - Chetan Khatri
> >
>

[jira] [Created] (HBASE-17370) Fix or provide shell scripts to drain and decommission region server

[Jerry He (JIRA)]

Jerry He created HBASE-17370:


 Summary: Fix or provide shell scripts to drain and decommission 
region server
 Key: HBASE-17370
 URL: https://issues.apache.org/jira/browse/HBASE-17370
 Project: HBase
  Issue Type: Sub-task
Reporter: Jerry He


1. Update the existing shell scripts to use the new drain related API.
2  Or provide new shell scripts.
3. Provide a 'decommission' shell tool that puts the server in drain mode and 
offload the server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17369) Add ACL to the new region server drain related API

[Jerry He (JIRA)]

Jerry He created HBASE-17369:


 Summary: Add ACL to the new region server drain related API
 Key: HBASE-17369
 URL: https://issues.apache.org/jira/browse/HBASE-17369
 Project: HBase
  Issue Type: Task
Reporter: Jerry He
Priority: Critical


Add ACL to the new region server drain related API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-17322) New API to get the list of draining region servers

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-17322.
--
Resolution: Duplicate

> New API to get the list of draining region servers
> --
>
> Key: HBASE-17322
> URL: https://issues.apache.org/jira/browse/HBASE-17322
> Project: HBase
>  Issue Type: Improvement
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
>
> In various scenarios it would be useful to have a list of draining region 
> servers so as to avoid them while doing certain operations such as region 
> moving during batch rolling upgrades.
> Jira to add a method getDrainingServers() in ClusterStatus so that this info 
> can be retrieved through HBaseAdmin.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [ANNOUNCE] New HBase Committer Josh Elser

[Jerry He]

Congratulations , Josh!

Good work on the PQS too.

Jerry

On Sun, Dec 11, 2016 at 12:14 PM, Josh Elser  wrote:

> Thanks, all. I'm looking forward to continuing to work with you all!
>
>
> Nick Dimiduk wrote:
>
>> On behalf of the Apache HBase PMC, I am pleased to announce that Josh
>> Elser
>> has accepted the PMC's invitation to become a committer on the project. We
>> appreciate all of Josh's generous contributions thus far and look forward
>> to his continued involvement.
>>
>> Allow me to be the first to congratulate and welcome Josh into his new
>> role!
>>
>>

[jira] [Resolved] (HBASE-16796) Correct RpcServer responseSize and requestSize to account for cellScanner payload

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-16796.
--
Resolution: Duplicate

> Correct RpcServer responseSize and requestSize to account for cellScanner 
> payload
> -
>
> Key: HBASE-16796
> URL: https://issues.apache.org/jira/browse/HBASE-16796
> Project: HBase
>  Issue Type: Bug
>    Reporter: Jerry He
>Assignee: Jerry He
>
> The reponseSize and requestSize on the RpcServer don't seem to account for 
> the cellScanner payload.  We should correct them.
> The metrics numbers and the logging for reponseTooLarge  and TooSlow won't  
> show up correctly otherwise.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17221) Abstract out an interface for RpcServer.Call

[Jerry He (JIRA)]

Jerry He created HBASE-17221:


 Summary: Abstract out an interface for RpcServer.Call
 Key: HBASE-17221
 URL: https://issues.apache.org/jira/browse/HBASE-17221
 Project: HBase
  Issue Type: Improvement
Reporter: Jerry He
Assignee: Jerry He
 Fix For: 2.0.0


RpcServer.Call is a concrete class, but it is marked as:
{noformat}
@InterfaceAudience.LimitedPrivate({HBaseInterfaceAudience.COPROC, 
HBaseInterfaceAudience.PHOENIX})
{noformat}

Let's abstract out an interface out of it for potential consumers that want to 
pass it around.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [ANNOUNCE] New HBase committer Phil Yang

[Jerry He]

Congratulations and welcome, Phil.

On Tue, Nov 29, 2016 at 9:35 AM, Huaxiang Sun  wrote:

> Congratulations, Phil!
>
> > On Nov 29, 2016, at 5:29 AM, Yu Li  wrote:
> >
> > Congrats and welcome Phil!
> >
> > Best Regards,
> > Yu
> >
> > On 29 November 2016 at 19:01, Anoop John  wrote:
> >
> >> Congrats and welcome Phil...  Looking forward for many more
> >> contributions from u.
> >>
> >> -Anoop-
> >>
> >> On Tue, Nov 29, 2016 at 4:30 PM, Anoop John 
> wrote:
> >>> Congrats and welcome Binlijin.
> >>>
> >>> -Anoop-
> >>>
> >>> On Tue, Nov 29, 2016 at 4:29 PM, Guanghao Zhang 
> >> wrote:
>  Congratulations!
> 
>  2016-11-29 17:49 GMT+08:00 Duo Zhang :
> 
> > On behalf of the Apache HBase PMC, I am pleased to announce that Phil
> >> Yang
> > has accepted the PMC's invitation to become a committer on the
> >> project. We
> > appreciate all of Phil's generous contributions thus far and look
> >> forward
> > to his continued involvement.
> >
> > Congratulations and welcome, Phil!
> >
> >>
>
>

Re: [ANNOUNCE] New HBase committer Lijin Bin

[Jerry He]

*Congratulations, *宾莉金!

On Tue, Nov 29, 2016 at 9:34 AM, Huaxiang Sun  wrote:

>
> Congratulations, Lijin!
>
> > On Nov 29, 2016, at 5:15 AM, 宾莉金 or binlijin  wrote:
> >
> > Thanks everyone. :)
> >
> > Looking forward to keep doing cool stuff with you guys.
> >
> > 2016-11-29 19:01 GMT+08:00 ramkrishna vasudevan <
> > ramkrishna.s.vasude...@gmail.com>:
> >
> >> Congratulations !! All the best !!!
> >>
> >> On Tue, Nov 29, 2016 at 4:29 PM, Guanghao Zhang 
> >> wrote:
> >>
> >>> Congratulations!
> >>>
> >>> 2016-11-29 17:48 GMT+08:00 Duo Zhang :
> >>>
>  On behalf of the Apache HBase PMC, I am pleased to announce that Lijin
>  Bin(binlijin) has accepted the PMC's invitation to become a committer
> >> on
>  the project. We appreciate all of Lijin's generous contributions thus
> >> far
>  and look forward to his continued involvement.
> 
>  Congratulations and welcome, Lijin!
> 
> >>>
> >>
> >
> >
> >
> > --
> > *Best Regards,*
> > lijin bin
>
>

Re: [DISCUSS] Drop legacy hadoop support at the 2.0 release

[Jerry He]

Concur here.  Our distribution has on 2.7.x lines for a while.

Jerry

On Oct 19, 2016 6:26 PM, "张铎"  wrote:

> OK, seems no objection. I will file a issue to modify the hadoop version
> support matrix.
>
> And I think we also need to change the hadoopcheck versions for our
> precommit check?
>
> Thanks all.
>
> 2016-10-20 1:14 GMT+08:00 Andrew Purtell :
>
> > FWIW, we are running 2.7.x in production and it's stable.
> >
> >
> > On Tue, Oct 18, 2016 at 10:18 PM, Sean Busbey  wrote:
> >
> > > we had not decided yet AFAIK. a big concern was the lack of
> > > maintenance releases on more recent Hadoop versions and the perception
> > > that 2.4 and 2.5 were the last big stable release lines.
> > >
> > > 2.6 and 2.7 have both gotten a few maintenance releases now, so maybe
> > > this isn't a concern any more?
> > >
> > > On Tue, Oct 18, 2016 at 7:00 PM, Enis Söztutar 
> > wrote:
> > > > I thought we already decided to do that, no?
> > > >
> > > > Enis
> > > >
> > > > On Tue, Oct 18, 2016 at 6:56 PM, Ted Yu  wrote:
> > > >
> > > >> Looking at http://hadoop.apache.org/releases.html , 2.5.x hasn't
> got
> > > new
> > > >> release for almost two years.
> > > >>
> > > >> Seems fine to drop support for 2.4 and 2.5
> > > >>
> > > >> On Tue, Oct 18, 2016 at 6:42 PM, Duo Zhang 
> > wrote:
> > > >>
> > > >> > This is the current hadoop version support matrix
> > > >> >
> > > >> > https://hbase.apache.org/book.html#hadoop
> > > >> >
> > > >> > 2016-10-19 9:40 GMT+08:00 Duo Zhang :
> > > >> >
> > > >> > > To be specific, hadoop-2.4.x and hadoop-2.5.x.
> > > >> > >
> > > >> > > The latest releases for these two lines are about two years ago,
> > so
> > > I
> > > >> > > think it is the time to drop the support of them when 2.0 out.
> > Then
> > > we
> > > >> > > could drop some code in our hadoop-compat module as we may need
> to
> > > add
> > > >> > some
> > > >> > > code for the incoming hadoop-3.0...
> > > >> > >
> > > >> > > Thanks.
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> >- Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>

[jira] [Created] (HBASE-16882) Expose the number of open scanners on region server via metics

[Jerry He (JIRA)]

Jerry He created HBASE-16882:


 Summary: Expose the number of open scanners on region server via 
metics
 Key: HBASE-16882
 URL: https://issues.apache.org/jira/browse/HBASE-16882
 Project: HBase
  Issue Type: Improvement
Reporter: Jerry He
Assignee: Jerry He
Priority: Minor


Metrics on the number of open scanners and their live span on the region server 
are useful in doing server performance diagnostics, and are useful in doing 
client application diagnostics, profiling as well. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [ANNOUNCE] Stephen Yuan Jiang joins Apache HBase PMC

[Jerry He]

Congratulations, Stephen.

Jerry

On Fri, Oct 14, 2016 at 12:56 PM, Dima Spivak  wrote:

> Congrats, Stephen!
>
> -Dima
>
> On Fri, Oct 14, 2016 at 11:27 AM, Enis Söztutar  wrote:
>
> > On behalf of the Apache HBase PMC, I am happy to announce that Stephen
> has
> > accepted our invitation to become a PMC member of the Apache HBase
> project.
> >
> > Stephen has been working on HBase for a couple of years, and is already a
> > committer for more than a year. Apart from his contributions in proc v2,
> > hbck and other areas, he is also helping for the 2.0 release which is the
> > most important milestone for the project this year.
> >
> > Welcome to the PMC Stephen,
> > Enis
> >
>

[jira] [Created] (HBASE-16843) Manage and secure dynamic jar dir with client API

[Jerry He (JIRA)]

Jerry He created HBASE-16843:


 Summary: Manage and secure dynamic jar dir with client API
 Key: HBASE-16843
 URL: https://issues.apache.org/jira/browse/HBASE-16843
 Project: HBase
  Issue Type: Sub-task
Reporter: Jerry He
Assignee: Jerry He


The dynamic jar dir is specified with the property 'hbase.dynamic.jars.dir'
The permissions on this dir is a cause of confusion and concern.

Let's secure this dir by folding it into root.dir, and provide client API to 
add, remove jars.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16822) Enable restore-snapshot and clone-snapshot to use external specified snapshot locatioin

[Jerry He (JIRA)]

Jerry He created HBASE-16822:


 Summary: Enable restore-snapshot and clone-snapshot to use 
external specified snapshot locatioin 
 Key: HBASE-16822
 URL: https://issues.apache.org/jira/browse/HBASE-16822
 Project: HBase
  Issue Type: Improvement
Reporter: Jerry He


Currently restore-snapshot and clone-snapshot only work with the snapshots that 
are under hbase root.dir.

In combination with export-snapshot, this means the snapshot needs to be 
exported out to another hbase root.dir, or back and forth eventually to a hbase 
root.dir.

There are a few issues with the approach.

We've know that export-snapshot has a limitation dealing with secure cluster, 
where the external user needs to have read access to hbase root.dir data, 
by-passing table ACL check.

The second problem is when we try to use or bring back the exported snapshot 
for restore/clone.  They have to in the target hbase root.dir, and needs write 
permission to get it in there.

Again we will have permission problem.

This ticket tries to deal with the second problem, clone and restore from a 
exported snapshot.  The exported snapshots can be on the same cluster but the 
user may not have write permission to move it to hbase.root.dir.

We should have a solution that allow clone/restore snapshot from a external 
path that keeps snapshot backups. And also do it with security permission in 
mind.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16798) Integrate MultiRowMutation CPEP into HBase core

[Jerry He (JIRA)]

Jerry He created HBASE-16798:


 Summary: Integrate MultiRowMutation CPEP into HBase core
 Key: HBASE-16798
 URL: https://issues.apache.org/jira/browse/HBASE-16798
 Project: HBase
  Issue Type: Sub-task
Reporter: Jerry He


Integrate MultiRowMutation CPEP into HBase core as discussed in the parent task.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16797) Integrate TokenProvider to HBase core

[Jerry He (JIRA)]

Jerry He created HBASE-16797:


 Summary: Integrate TokenProvider to HBase core
 Key: HBASE-16797
 URL: https://issues.apache.org/jira/browse/HBASE-16797
 Project: HBase
  Issue Type: Sub-task
Reporter: Jerry He


Integrate TokenProvider to HBase core as discussed in the parent task.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16796) Correct RpcServer responseSize and requestSize to account for cellScanner payload

[Jerry He (JIRA)]

Jerry He created HBASE-16796:


 Summary: Correct RpcServer responseSize and requestSize to account 
for cellScanner payload
 Key: HBASE-16796
 URL: https://issues.apache.org/jira/browse/HBASE-16796
 Project: HBase
  Issue Type: Bug
Reporter: Jerry He
Assignee: Jerry He


The reponseSize and requestSize on the RpcServer don't seem to account for the 
cellScanner payload.  We should correct them.
The metrics numbers and the logging for reponseTooLarge  and TooSlow won't  
show up correctly otherwise.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)