Re: [VOTE] The second hbase-2.0.0-beta-1 Release Candidate is available for download

[Stack]

On Wed, Jan 10, 2018 at 6:37 AM, Jean-Marc Spaggiari <
jean-m...@spaggiari.org> wrote:

> "Remaining KVs" field in master WebUI shows negative values... Is that
> tracked anywhere?
>
>
No.

Yeah, this one is sunk. Trying to put up a new one. Taking a while!

Thanks JMS (and Balazs, Yu Li).

St.Ack



> 2018-01-10 5:12 GMT-05:00 Jean-Marc Spaggiari :
>
> > Oh, you're right! I missed it! Ok. Thanks. Will wait for the next one.
> >
> > JM
> >
> > 2018-01-10 5:00 GMT-05:00 Yu Li :
> >
> >> I believe this one is already sunk according to stack's reply:
> >>
> >> -- Forwarded message --
> >> From: Stack 
> >> Date: 10 January 2018 at 02:09
> >> Subject: Re: [VOTE] The second hbase-2.0.0-beta-1 Release Candidate is
> >> available for download
> >> To: HBase Dev List 
> >>
> >>
> >> Thanks Andrew.
> >>
> >> Let me work on this.
> >>
> >> RC is sunk. Will put up a new one after I address the above failure.
> >>
> >> Best Regards,
> >> Yu
> >>
> >> On 10 January 2018 at 17:54, Jean-Marc Spaggiari <
> jean-m...@spaggiari.org
> >> >
> >> wrote:
> >>
> >> > Are we going to sink this one because of Andrew's -1?
> >> >
> >> > If so I will wait for the next one to test it...
> >> >
> >> > Le 10 janv. 2018 03 h 24, "Balazs Meszaros" <
> >> balazs.mesza...@cloudera.com>
> >> > a écrit :
> >> >
> >> > > +1
> >> > >
> >> > > - signatures, checksums OK,
> >> > > - unit tests passes (8u112),
> >> > > - shell works,
> >> > > - load test tool also successed.
> >> > >
> >> > > Best regards,
> >> > > Balazs
> >> > >
> >> > > On Tue, Jan 9, 2018 at 10:38 PM, Andrew Purtell <
> apurt...@apache.org>
> >> > > wrote:
> >> > >
> >> > > > Linux here.
> >> > > >
> >> > > >
> >> > > > On Tue, Jan 9, 2018 at 1:13 PM, Stack  wrote:
> >> > > >
> >> > > > > Andrew:
> >> > > > >
> >> > > > > I don't see this:
> >> > > > >
> >> > > > > [INFO] ---
> >> > > > > [INFO]  T E S T S
> >> > > > > [INFO] ---
> >> > > > > [INFO] Running
> >> > > > > org.apache.hadoop.hbase.regionserver.TestMemstoreLABWithoutPool
> >> > > > > [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time
> >> > elapsed:
> >> > > > > 4.409 s - in org.apache.hadoop.hbase.regionserver.
> >> > > > > TestMemstoreLABWithoutPool
> >> > > > > [INFO]
> >> > > > > [INFO] Results:
> >> > > > > [INFO]
> >> > > > > [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
> >> > > > >
> >> > > > > Linux and Mac.
> >> > > > >
> >> > > > > What you reckon sir?
> >> > > > >
> >> > > > > St.Ack
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Tue, Jan 9, 2018 at 10:03 AM, Andrew Purtell <
> >> apurt...@apache.org
> >> > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > -1
> >> > > > > >
> >> > > > > > Checked sums and signatures: ok
> >> > > > > > RAT check passes: ok (8u144)
> >> > > > > > Built from source: ok (8u144)
> >> > > > > > Unit tests pass: failed (8u144), TestMemstoreLABWithoutPool
> >> always
> >> > > > fails
> >> > > > > >
> >> > > > > >
> >> > > > > > TestMemstoreLABWithoutPool.org.apache.hadoop.hbase.regionser
> >> ver.
> >> > > > > > TestMemstoreLABWithoutPool
> >> > > > > > can fail with OOME if not run in isolation.
> >> > > > > >
> >> > > > > > If run in iso

Re: [VOTE] The first hbase-2.0.0-beta-1 Release Candidate is available

[Stack]

egion server exiting
> > > > > java.lang.RuntimeException: HRegionServer Aborted
> > > > > at
> > > > > org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.
> start(
> > > > > HRegionServerCommandLine.java:66)
> > > > > at
> > > > > org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(
> > > > > HRegionServerCommandLine.java:85)
> > > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > > > > at
> > > > > org.apache.hadoop.hbase.util.ServerCommandLine.doMain(
> > > > > ServerCommandLine.java:149)
> > > > > at
> > > > > org.apache.hadoop.hbase.regionserver.HRegionServer.
> > > > > main(HRegionServer.java:3016)
> > > > >
> > > > > Which is very surprising, because I can clearly see the directory
> > being
> > > > > created.
> > > > >
> > > > > Another attempt here, we I even look one step deeper and can see
> the
> > > > > generated file:
> > > > > 2018-01-10 05:27:58,116 WARN  [regionserver/node8.com/192.
> > > 168.23.2:16020
> > > > ]
> > > > > wal.AsyncFSWAL: create wal log writer
> > > > > hdfs://node2.com:8020*/hbase/WALs/node8.com
> > > > > <http://node8.com>,16020,1515580031417/node8.com
> > > > > <http://node8.com>%2C16020%2C1515580031417.1515580037373* failed,
> > > retry
> > > > =
> > > > > 7
> > > > > org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$
> > > > > AnnotatedConnectException:
> > > > > syscall:getsockopt(..) failed: Connexion refusée: /
> > 192.168.23.2:50010
> > > > > at
> > > > > org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.
> > > > > finishConnect(..)(Unknown
> > > > > Source)
> > > > > Caused by:
> > > > > org.apache.hbase.thirdparty.io.netty.channel.unix.Errors$
> > > > > NativeConnectException:
> > > > > syscall:getsockopt(..) failed: Connexion refusée
> > > > > ... 1 more
> > > > > 2018-01-10 05:28:08,210 INFO  [regionserver/node8.com/192.
> > > 168.23.2:16020
> > > > ]
> > > > > util.FSHDFSUtils: Recover lease on dfs file /hbase/WALs/node8.com
> > > > > ,16020,1515580031417/node8.com%2C16020%2C1515580031417.
> 1515580037373
> > > > > 2018-01-10 05:28:08,228 INFO  [regionserver/node8.com/192.
> > > 168.23.2:16020
> > > > ]
> > > > > util.FSHDFSUtils: Failed to recover lease, attempt=0 on
> > > file=/hbase/WALs/
> > > > > node8.com,16020,1515580031417/node8.com%2C16020%
> > > > > 2C1515580031417.1515580037373
> > > > > after 17ms
> > > > >
> > > > > hbase@node8:~/hbase-2.0.0-beta-1/logs$
> > /home/hadoop/hadoop-2.7.5/bin/
> > > > hdfs
> > > > > dfs -ls -R /hbase/WALs/ | grep node8
> > > > > drwxr-xr-x   - hbase supergroup  0 2018-01-10 05:28
> > > /hbase/WALs/
> > > > > node8.com,16020,1515580031417
> > > > > -rw-r--r--   3 hbase supergroup  0 2018-01-10 05:28
> > > > > */hbase/WALs/node8.com
> > > > > <http://node8.com>,16020,1515580031417/node8.com
> > > > > <http://node8.com>%2C16020%2C1515580031417.1515580037373*
> > > > >
> > > > >
> > > > > But still says it fails. Any clue? all other nodes are working
> fine.
> > > > >
> > > > > 2018-01-09 16:25 GMT-05:00 Stack :
> > > > >
> > > > > > On Tue, Jan 9, 2018 at 10:07 AM, Andrew Purtell <
> > apurt...@apache.org
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I just vetoed the RC because TestMemstoreLABWithoutPool always
> > > fails
> > > > > for
> > > > > > > me. It was the same with the last RC too. My Java is Oracle
> Java
> > > > 8u144
> > > > > > > running on x64 Linux (Ubuntu xenial). Let me know if you need
> me
> > to
> > > > > > provide
> > > > > > > the test output.
> > > > > > >
> > > > > > >
> > > > > > Ok. I can't make it fail. I'm going to disable it and file an
> issue
> > 

Re: disabled Jenkins jobs

[Stack]

Please see tail of the "Re: Testing and CI -- Apache Jenkins Builds (WAS ->
Re: Testing)" thread which ends with note on this disabling. These
'old-style' jobs have been disabled in favor the nightly runs.
S

On Wed, Jan 10, 2018 at 8:44 AM, Ted Yu  wrote:

> Hi,
> I was looking at:
> https://builds.apache.org/job/HBase-1.4/
>
> I found that the last build was from Dec 8th.
>
> Should the job be enabled now that there would be more 1.4 releases down
> the road ?
>

Re: [VOTE] The first hbase-2.0.0-beta-1 Release Candidate is available

[Stack]

Thanks JMS.
S

On Thu, Jan 11, 2018 at 9:36 AM, Jean-Marc Spaggiari <
jean-m...@spaggiari.org> wrote:

> Opened HBASE-19767 <https://issues.apache.org/jira/browse/HBASE-19767>
> and HBASE-19768.
> Regarding the issue to create the log writer, it fails even if the DN is
> already declared dead on the NN side...
>
> 2018-01-10 21:37 GMT-05:00 Apekshit Sharma :
>
> > On Wed, Jan 10, 2018 at 11:25 AM, Zach York <
> zyork.contribut...@gmail.com>
> > wrote:
> >
> > > What is the expectation for flaky tests? I was going to post some test
> > > failures, but saw that they were included in the excludes for flaky
> > tests.
> > >
> > > I understand we might be okay with having flaky tests for this beta-1
> > (and
> > > obviously for dev), but I would assume that we want consistent test
> > results
> > > for the official 2.0.0 release.
> > >
> >
> > Yeah, that's the goal, but sadly not many hands on deck are working on
> > that, so doesn't seem in reach.
> >
> >
> > > Do we have JIRAs created for all the flaky tests so that we can start
> > > fixing them before the beta-2/official RCs get put up?
> > >
> >
> > Whenever i start working on one, i search for it in jira first in case
> > someone's already working on it, if not I create a new one. (treating
> jira
> > as a lock to avoid redundant work).
> > Creating just the jiras doesn't really help until someone takes them and
> so
> > most just remain open. But chicken and egg problem maybe? Might be good
> to
> > create jira for a few simple ones to see if peeps starting contributing
> on
> > this front?
> >
> >
> > > I'd be happy to help try to track down the root causes of the flakiness
> > and
> > > try to fix these problematic tests.
> > >
> > Any help here would be great!
> > Here's a personal thank you :
> > http://calmcoolcollective.net/wp-content/uploads/2016/08/
> chocolatechip.jpg
> > :)
> >
> >
> > >
> > > Thanks,
> > > Zach
> > >
> > > On Wed, Jan 10, 2018 at 9:37 AM, Stack  wrote:
> > >
> > > > Put up a JIRA and dump this stuff in JMS. Sounds like we need a bit
> > more
> > > > test coverage at least. Thanks sir.
> > > > St.Ack
> > > >
> > > > On Wed, Jan 10, 2018 at 2:52 AM, Jean-Marc Spaggiari <
> > > > jean-m...@spaggiari.org> wrote:
> > > >
> > > > > The DN was dead since December 31st... I really hope the DN figured
> > > that
> > > > > :-/
> > > > >
> > > > > I will retry with making sure that the NN is aware the local DN is
> > > dead,
> > > > > and see. I let you know.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > JMS
> > > > >
> > > > > 2018-01-10 5:50 GMT-05:00 张铎(Duo Zhang) :
> > > > >
> > > > > > The problem maybe that the DN is dead, but NN does not know and
> > keep
> > > > > > telling RS that you should try to connect to it. And for the new
> > > > > > AsyncFSWAL, we need to connect to all the 3 DNs successfully
> > before
> > > > > > writing actual data to it, so the RS sucks...
> > > > > >
> > > > > > This maybe a problem.
> > > > > >
> > > > > > 2018-01-10 18:40 GMT+08:00 Jean-Marc Spaggiari <
> > > > jean-m...@spaggiari.org
> > > > > >:
> > > > > >
> > > > > > > You're correct. It was dead. I thought HBase will be able to
> > > survive
> > > > > > that.
> > > > > > > Same the DN dies after the RS has started, RS will fail closing
> > > > nicely
> > > > > :(
> > > > > > >
> > > > > > > 2018-01-10 5:38 GMT-05:00 张铎(Duo Zhang)  >:
> > > > > > >
> > > > > > > > Connection refuse? Have you checked the status of the
> datanode
> > on
> > > > > > node8?
> > > > > > > >
> > > > > > > > 2018-01-10 18:31 GMT+08:00 Jean-Marc Spaggiari <
> > > > > > jean-m...@spaggiari.org
> > > > > > > >:
> > > > > > > >
> > > > > > > > > I know, 

Re: Flaky dashboard for current branch-2

[Stack]

Thanks Appy. Looks beautiful. Is Nightly now using a list of flakes? Thanks.
S

On Thu, Jan 11, 2018 at 6:10 PM, Apekshit Sharma  wrote:

> https://builds.apache.org/job/HBase-Find-Flaky-Tests-
> branch2.0/lastSuccessfulBuild/artifact/dashboard.html
>
> @stack: when you branch out branch-2.0, let me know, i'll update the jobs
> to point to that branch so that it's helpful in release. Once release is
> done, i'll move them back to "branch-2".
>
>
> -- Appy
>

[VOTE] The third hbase-2.0.0-beta-1 Release Candidate is available for download

[Stack]

The third release candidate for HBase 2.0.0-beta-1 is up at:

  https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-beta-1-RC2/

Maven artifacts are available from a staging directory here:

  https://repository.apache.org/content/repositories/orgapachehbase-1191

All was signed with my key at 8ACC93D2 [1]

I tagged the RC as 2.0.0-beta-1-RC1.6 at hash
3e6f80dcd5b6630769f310d429f403f1fa8bae23

This third RC has fix nice bug fixes over RC0 and RC1 including fix for
MOST of the failing and flakey tests (We are still working through them).

hbase-2.0.0-beta-1 is our first beta release. It includes all that was in
previous alphas (new assignment manager, offheap read/write path, in-memory
compactions, etc.). The APIs and feature-set are sealed.

hbase-2.0.0-beta-1 is a not-for-production preview of hbase-2.0.0. It is
meant for devs and downstreamers to test drive and flag us if we messed up
on anything ahead of our rolling GAs. We are particular interested in
hearing from Coprocessor developers.

The list of features addressed in 2.0.0 so far can be found here [3]. There
are thousands. The list of ~2k+ fixes in 2.0.0 exclusively can be found
here [4] (My JIRA JQL foo is a bit dodgy -- forgive me if mistakes).

I've updated our overview doc. on the state of 2.0.0 [6]. We'll do one more
beta before we put up our first 2.0.0 Release Candidate by the end of
January, 2.0.0-beta-2. Its focus will be making it so users can do a
rolling upgrade on to hbase-2.x from hbase-1.x (and any bug fixes found
running beta-1). Here is the list of what we have targeted so far for
beta-2 [5]. Check it out.

One known issue is that the User API has not been properly filtered so it
shows more than just InterfaceAudience Public content (HBASE-19663, to be
fixed by beta-2).

Please take this beta for a spin. Please vote on whether it ok to put out
this RC as our first beta (Note CHANGES has not yet been updated). Let the
VOTE be open for 72 hours (Lets say Monday morning!)

Thanks,
Your 2.0.0 Release Manager

1. http://pgp.mit.edu/pks/lookup?op=get&search=0x9816C7FC8ACC93D2
3. https://goo.gl/scYjJr
4. https://goo.gl/dFFT8b
5. https://issues.apache.org/jira/projects/HBASE/versions/12340862
6. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
z9iEu_ktczrlKHK8N4SZzs/

Re: Flaky dashboard for current branch-2

[Stack]

Dashboard doesn't capture timed out tests, right Appy?
Thanks,
S

On Thu, Jan 11, 2018 at 6:10 PM, Apekshit Sharma  wrote:

> https://builds.apache.org/job/HBase-Find-Flaky-Tests-
> branch2.0/lastSuccessfulBuild/artifact/dashboard.html
>
> @stack: when you branch out branch-2.0, let me know, i'll update the jobs
> to point to that branch so that it's helpful in release. Once release is
> done, i'll move them back to "branch-2".
>
>
> -- Appy
>

Re: [VOTE] The third hbase-2.0.0-beta-1 Release Candidate is available for download

[Stack]

There seem to be too many tests failing for this to pass. Just a heads up
that if you are seeing this, its not you. I'm looking at it. Will report
back.

FYI, I need to look into RC-building time too. It takes many hours to
produce one.

Thanks,
S


On Thu, Jan 11, 2018 at 10:30 PM, Stack  wrote:

> The third release candidate for HBase 2.0.0-beta-1 is up at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-beta-1-RC2/
>
> Maven artifacts are available from a staging directory here:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1191
>
> All was signed with my key at 8ACC93D2 [1]
>
> I tagged the RC as 2.0.0-beta-1-RC1.6 at hash 3e6f80dcd5b6630769f310d42
> 9f403f1fa8bae23
>
> This third RC has fix nice bug fixes over RC0 and RC1 including fix for
> MOST of the failing and flakey tests (We are still working through them).
>
> hbase-2.0.0-beta-1 is our first beta release. It includes all that was in
> previous alphas (new assignment manager, offheap read/write path, in-memory
> compactions, etc.). The APIs and feature-set are sealed.
>
> hbase-2.0.0-beta-1 is a not-for-production preview of hbase-2.0.0. It is
> meant for devs and downstreamers to test drive and flag us if we messed up
> on anything ahead of our rolling GAs. We are particular interested in
> hearing from Coprocessor developers.
>
> The list of features addressed in 2.0.0 so far can be found here [3].
> There are thousands. The list of ~2k+ fixes in 2.0.0 exclusively can be
> found here [4] (My JIRA JQL foo is a bit dodgy -- forgive me if mistakes).
>
> I've updated our overview doc. on the state of 2.0.0 [6]. We'll do one
> more beta before we put up our first 2.0.0 Release Candidate by the end of
> January, 2.0.0-beta-2. Its focus will be making it so users can do a
> rolling upgrade on to hbase-2.x from hbase-1.x (and any bug fixes found
> running beta-1). Here is the list of what we have targeted so far for
> beta-2 [5]. Check it out.
>
> One known issue is that the User API has not been properly filtered so it
> shows more than just InterfaceAudience Public content (HBASE-19663, to be
> fixed by beta-2).
>
> Please take this beta for a spin. Please vote on whether it ok to put out
> this RC as our first beta (Note CHANGES has not yet been updated). Let the
> VOTE be open for 72 hours (Lets say Monday morning!)
>
> Thanks,
> Your 2.0.0 Release Manager
>
> 1. http://pgp.mit.edu/pks/lookup?op=get&search=0x9816C7FC8ACC93D2
> 3. https://goo.gl/scYjJr
> 4. https://goo.gl/dFFT8b
> 5. https://issues.apache.org/jira/projects/HBASE/versions/12340862
> 6. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> z9iEu_ktczrlKHK8N4SZzs/
>

Re: [VOTE] The third hbase-2.0.0-beta-1 Release Candidate is available for download

[Stack]

On Fri, Jan 12, 2018 at 9:37 AM, Andrew Purtell 
wrote:

> The bulk of the build time seems due to the retargeting of the Maven
> repository cache to a new empty directory, requiring a jarcrawl of the
> world at every build. If we had a flag to make this optional and nondefault
> I bet that would resolve it.
>
>
Yeah. Let me add that. I find this a goodly portion but enforcer checking
and for some reason, hangs in archetypes eats up loads of time too (I'm
finding that my build machines seem to have rotted which doesn't help the
situation).

S



>
> > On Jan 12, 2018, at 9:31 AM, Stack  wrote:
> >
> > There seem to be too many tests failing for this to pass. Just a heads up
> > that if you are seeing this, its not you. I'm looking at it. Will report
> > back.
> >
> > FYI, I need to look into RC-building time too. It takes many hours to
> > produce one.
> >
> > Thanks,
> > S
> >
> >
> >> On Thu, Jan 11, 2018 at 10:30 PM, Stack  wrote:
> >>
> >> The third release candidate for HBase 2.0.0-beta-1 is up at:
> >>
> >>  https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-beta-1-RC2/
> >>
> >> Maven artifacts are available from a staging directory here:
> >>
> >>  https://repository.apache.org/content/repositories/orgapachehbase-1191
> >>
> >> All was signed with my key at 8ACC93D2 [1]
> >>
> >> I tagged the RC as 2.0.0-beta-1-RC1.6 at hash 3e6f80dcd5b6630769f310d42
> >> 9f403f1fa8bae23
> >>
> >> This third RC has fix nice bug fixes over RC0 and RC1 including fix for
> >> MOST of the failing and flakey tests (We are still working through
> them).
> >>
> >> hbase-2.0.0-beta-1 is our first beta release. It includes all that was
> in
> >> previous alphas (new assignment manager, offheap read/write path,
> in-memory
> >> compactions, etc.). The APIs and feature-set are sealed.
> >>
> >> hbase-2.0.0-beta-1 is a not-for-production preview of hbase-2.0.0. It is
> >> meant for devs and downstreamers to test drive and flag us if we messed
> up
> >> on anything ahead of our rolling GAs. We are particular interested in
> >> hearing from Coprocessor developers.
> >>
> >> The list of features addressed in 2.0.0 so far can be found here [3].
> >> There are thousands. The list of ~2k+ fixes in 2.0.0 exclusively can be
> >> found here [4] (My JIRA JQL foo is a bit dodgy -- forgive me if
> mistakes).
> >>
> >> I've updated our overview doc. on the state of 2.0.0 [6]. We'll do one
> >> more beta before we put up our first 2.0.0 Release Candidate by the end
> of
> >> January, 2.0.0-beta-2. Its focus will be making it so users can do a
> >> rolling upgrade on to hbase-2.x from hbase-1.x (and any bug fixes found
> >> running beta-1). Here is the list of what we have targeted so far for
> >> beta-2 [5]. Check it out.
> >>
> >> One known issue is that the User API has not been properly filtered so
> it
> >> shows more than just InterfaceAudience Public content (HBASE-19663, to
> be
> >> fixed by beta-2).
> >>
> >> Please take this beta for a spin. Please vote on whether it ok to put
> out
> >> this RC as our first beta (Note CHANGES has not yet been updated). Let
> the
> >> VOTE be open for 72 hours (Lets say Monday morning!)
> >>
> >> Thanks,
> >> Your 2.0.0 Release Manager
> >>
> >> 1. http://pgp.mit.edu/pks/lookup?op=get&search=0x9816C7FC8ACC93D2
> >> 3. https://goo.gl/scYjJr
> >> 4. https://goo.gl/dFFT8b
> >> 5. https://issues.apache.org/jira/projects/HBASE/versions/12340862
> >> 6. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> >> z9iEu_ktczrlKHK8N4SZzs/
> >>
>

Re: [VOTE] The third hbase-2.0.0-beta-1 Release Candidate is available for download

[Stack]

I'm sinking this RC. Pardon the inconvenience.

Appy put up this dashboard for branch-2 last night:
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html
There are a bunch of tests that fail always in the RC. I only realized the
extent when I ran a full build on a borrowed machine (My build boxes have
been mis-performing). hadoopqa filters out the permanently broken so I was
missing the true picture.

I will be back w/ a new RC as soon as the build finishes. A good few of the
permanent failures came in w/ MY "HBASE-19694 The initialization order for
a fresh cluster is incorrect". It raises a bunch of interesting issues that
we'll have to deal with in a beta-2 (that were hidden, but present
nonetheless, by the previous start order).

Pardon my trying your patience.
Yours,
St.Ack


On Thu, Jan 11, 2018 at 10:30 PM, Stack  wrote:

> The third release candidate for HBase 2.0.0-beta-1 is up at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-beta-1-RC2/
>
> Maven artifacts are available from a staging directory here:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1191
>
> All was signed with my key at 8ACC93D2 [1]
>
> I tagged the RC as 2.0.0-beta-1-RC1.6 at hash 3e6f80dcd5b6630769f310d42
> 9f403f1fa8bae23
>
> This third RC has fix nice bug fixes over RC0 and RC1 including fix for
> MOST of the failing and flakey tests (We are still working through them).
>
> hbase-2.0.0-beta-1 is our first beta release. It includes all that was in
> previous alphas (new assignment manager, offheap read/write path, in-memory
> compactions, etc.). The APIs and feature-set are sealed.
>
> hbase-2.0.0-beta-1 is a not-for-production preview of hbase-2.0.0. It is
> meant for devs and downstreamers to test drive and flag us if we messed up
> on anything ahead of our rolling GAs. We are particular interested in
> hearing from Coprocessor developers.
>
> The list of features addressed in 2.0.0 so far can be found here [3].
> There are thousands. The list of ~2k+ fixes in 2.0.0 exclusively can be
> found here [4] (My JIRA JQL foo is a bit dodgy -- forgive me if mistakes).
>
> I've updated our overview doc. on the state of 2.0.0 [6]. We'll do one
> more beta before we put up our first 2.0.0 Release Candidate by the end of
> January, 2.0.0-beta-2. Its focus will be making it so users can do a
> rolling upgrade on to hbase-2.x from hbase-1.x (and any bug fixes found
> running beta-1). Here is the list of what we have targeted so far for
> beta-2 [5]. Check it out.
>
> One known issue is that the User API has not been properly filtered so it
> shows more than just InterfaceAudience Public content (HBASE-19663, to be
> fixed by beta-2).
>
> Please take this beta for a spin. Please vote on whether it ok to put out
> this RC as our first beta (Note CHANGES has not yet been updated). Let the
> VOTE be open for 72 hours (Lets say Monday morning!)
>
> Thanks,
> Your 2.0.0 Release Manager
>
> 1. http://pgp.mit.edu/pks/lookup?op=get&search=0x9816C7FC8ACC93D2
> 3. https://goo.gl/scYjJr
> 4. https://goo.gl/dFFT8b
> 5. https://issues.apache.org/jira/projects/HBASE/versions/12340862
> 6. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> z9iEu_ktczrlKHK8N4SZzs/
>

[VOTE] The fourth hbase-2.0.0-beta-1 Release Candidate is available for download

[Stack]

The fourth release candidate for HBase 2.0.0-beta-1 is up at:

  https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-beta-1-RC3/

Maven artifacts are available from a staging directory here:

  https://repository.apache.org/content/repositories/orgapachehbase-1193

All was signed with my key at 8ACC93D2 [1].

I tagged the RC as 2.0.0-beta-1-RC1.7 at hash 026f535a7747b89003252ded9
585e827686aa79f

This RC has some nice bug fixes over RC0-2 including fixing MOST of the
failing and flakey tests (We are still working through them).

hbase-2.0.0-beta-1 is our first beta release. It includes all that was in
previous alphas (new assignment manager, offheap read/write path, in-memory
compactions, etc.). The APIs and feature-set are sealed.

hbase-2.0.0-beta-1 is a not-for-production preview of hbase-2.0.0. It is
meant for devs and downstreamers to test drive and flag us if we messed up
on anything ahead of our rolling GAs. We are particular interested in
hearing from Coprocessor developers.

The list of features addressed in 2.0.0 so far can be found here [3]. There
are thousands. The list of ~2k+ fixes in 2.0.0 exclusively can be found
here [4] (My JIRA JQL foo is a bit dodgy -- forgive me if mistakes).

I've updated our overview doc. on the state of 2.0.0 [6]. We'll do one more
beta before we put up our first 2.0.0 Release Candidate by the end of
January, 2.0.0-beta-2. Its focus will be making it so users can do a
rolling upgrade on to hbase-2.x from hbase-1.x (and any bug fixes found
running beta-1). Here is the list of what we have targeted so far for
beta-2 [5]. Check it out.

One known issue is that the User API has not been properly filtered so it
shows more than just InterfaceAudience Public content (HBASE-19663, to be
fixed by beta-2).

Please take this beta for a spin. Please vote on whether it ok to put out
this RC as our first beta (Note CHANGES has not yet been updated). Let the
VOTE be open for 72 hours (Lets say Tuesday morning!)

Thanks,
Your 2.0.0 Release Manager

1. http://pgp.mit.edu/pks/lookup?op=get&search=0x9816C7FC8ACC93D2
3. https://goo.gl/scYjJr
4. https://goo.gl/dFFT8b
5. https://issues.apache.org/jira/projects/HBASE/versions/12340862
6. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
z9iEu_ktczrlKHK8N4SZzs/

Re: [VOTE] The fourth hbase-2.0.0-beta-1 Release Candidate is available for download

[Stack]

+1

Checked sig and hash.
Downloaded locally. Undid and loaded data. Logs look good.
Put on cluster. Picks up old data fine. Nothing odd in logs (too much
logging but that is another story).

St.Ack

On Fri, Jan 12, 2018 at 8:48 PM, Stack  wrote:

> The fourth release candidate for HBase 2.0.0-beta-1 is up at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-beta-1-RC3/
>
> Maven artifacts are available from a staging directory here:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1193
>
> All was signed with my key at 8ACC93D2 [1].
>
> I tagged the RC as 2.0.0-beta-1-RC1.7 at hash 026f535a7747b89003252ded9
> 585e827686aa79f
>
> This RC has some nice bug fixes over RC0-2 including fixing MOST of the
> failing and flakey tests (We are still working through them).
>
> hbase-2.0.0-beta-1 is our first beta release. It includes all that was in
> previous alphas (new assignment manager, offheap read/write path, in-memory
> compactions, etc.). The APIs and feature-set are sealed.
>
> hbase-2.0.0-beta-1 is a not-for-production preview of hbase-2.0.0. It is
> meant for devs and downstreamers to test drive and flag us if we messed up
> on anything ahead of our rolling GAs. We are particular interested in
> hearing from Coprocessor developers.
>
> The list of features addressed in 2.0.0 so far can be found here [3].
> There are thousands. The list of ~2k+ fixes in 2.0.0 exclusively can be
> found here [4] (My JIRA JQL foo is a bit dodgy -- forgive me if mistakes).
>
> I've updated our overview doc. on the state of 2.0.0 [6]. We'll do one
> more beta before we put up our first 2.0.0 Release Candidate by the end of
> January, 2.0.0-beta-2. Its focus will be making it so users can do a
> rolling upgrade on to hbase-2.x from hbase-1.x (and any bug fixes found
> running beta-1). Here is the list of what we have targeted so far for
> beta-2 [5]. Check it out.
>
> One known issue is that the User API has not been properly filtered so it
> shows more than just InterfaceAudience Public content (HBASE-19663, to be
> fixed by beta-2).
>
> Please take this beta for a spin. Please vote on whether it ok to put out
> this RC as our first beta (Note CHANGES has not yet been updated). Let the
> VOTE be open for 72 hours (Lets say Tuesday morning!)
>
> Thanks,
> Your 2.0.0 Release Manager
>
> 1. http://pgp.mit.edu/pks/lookup?op=get&search=0x9816C7FC8ACC93D2
> 3. https://goo.gl/scYjJr
> 4. https://goo.gl/dFFT8b
> 5. https://issues.apache.org/jira/projects/HBASE/versions/12340862
> 6. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> z9iEu_ktczrlKHK8N4SZzs/
>

[DISCUSS] Purge hbase-archetypes?

[Stack]

Anyone using it? As far as I know it goes unsupported (anyone interested in
this project?).

Building a release takes two hours, one hour of which is archetype
machinations, so its not my favorite module these times.

Purge it from hbase?

St.Ack

Re: [VOTE] The fourth hbase-2.0.0-beta-1 Release Candidate is available for download

[Stack]

Thanks to the votes and testing so far.

Last night, a nightly build on hadoop2 completed but I notice that it has a
bunch of tests filtered out [1] (I'm trying to get this facility turned
off). There is also a report by Duo that came in last night a test that
passes for me fails for him (HBASE-19791). It happens to be one of those
filtered out unfortunately.

So, running the test suite (JMS!), you might want to apply the same filter
for now. See [1] below. I'll work on the failing tests in meantime (Yeah,
they were supposed to be fixed by now but its been taking a while).

Thanks again for your patience/understanding.

Yours,
Your RM

1.
https://builds.apache.org/job/HBase%20Nightly/job/branch-2/183/artifact/output-jdk8-hadoop2/patch-unit-root.txt


On Fri, Jan 12, 2018 at 8:48 PM, Stack  wrote:

> The fourth release candidate for HBase 2.0.0-beta-1 is up at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-beta-1-RC3/
>
> Maven artifacts are available from a staging directory here:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1193
>
> All was signed with my key at 8ACC93D2 [1].
>
> I tagged the RC as 2.0.0-beta-1-RC1.7 at hash 026f535a7747b89003252ded9
> 585e827686aa79f
>
> This RC has some nice bug fixes over RC0-2 including fixing MOST of the
> failing and flakey tests (We are still working through them).
>
> hbase-2.0.0-beta-1 is our first beta release. It includes all that was in
> previous alphas (new assignment manager, offheap read/write path, in-memory
> compactions, etc.). The APIs and feature-set are sealed.
>
> hbase-2.0.0-beta-1 is a not-for-production preview of hbase-2.0.0. It is
> meant for devs and downstreamers to test drive and flag us if we messed up
> on anything ahead of our rolling GAs. We are particular interested in
> hearing from Coprocessor developers.
>
> The list of features addressed in 2.0.0 so far can be found here [3].
> There are thousands. The list of ~2k+ fixes in 2.0.0 exclusively can be
> found here [4] (My JIRA JQL foo is a bit dodgy -- forgive me if mistakes).
>
> I've updated our overview doc. on the state of 2.0.0 [6]. We'll do one
> more beta before we put up our first 2.0.0 Release Candidate by the end of
> January, 2.0.0-beta-2. Its focus will be making it so users can do a
> rolling upgrade on to hbase-2.x from hbase-1.x (and any bug fixes found
> running beta-1). Here is the list of what we have targeted so far for
> beta-2 [5]. Check it out.
>
> One known issue is that the User API has not been properly filtered so it
> shows more than just InterfaceAudience Public content (HBASE-19663, to be
> fixed by beta-2).
>
> Please take this beta for a spin. Please vote on whether it ok to put out
> this RC as our first beta (Note CHANGES has not yet been updated). Let the
> VOTE be open for 72 hours (Lets say Tuesday morning!)
>
> Thanks,
> Your 2.0.0 Release Manager
>
> 1. http://pgp.mit.edu/pks/lookup?op=get&search=0x9816C7FC8ACC93D2
> 3. https://goo.gl/scYjJr
> 4. https://goo.gl/dFFT8b
> 5. https://issues.apache.org/jira/projects/HBASE/versions/12340862
> 6. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> z9iEu_ktczrlKHK8N4SZzs/
>

Re: disabled Jenkins jobs

[Stack]

Yeah. HBASE-18467. Patches welcome.
S

On Sat, Jan 13, 2018 at 10:13 AM, Ted Yu  wrote:

> I noticed that for JIRAs which were applied to branch-1.x , there was no
> post from Hudson about the corresponding build.
>
> It would be desirable to have such post for branch-1.x .
>
> Cheers
>
> On Wed, Jan 10, 2018 at 9:45 AM, Mike Drob  wrote:
>
> > I believe we're migrating everything to the nightly branch matrix job.
> >
> > https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/
> >
> > On Wed, Jan 10, 2018 at 10:44 AM, Ted Yu  wrote:
> >
> > > Hi,
> > > I was looking at:
> > > https://builds.apache.org/job/HBase-1.4/
> > >
> > > I found that the last build was from Dec 8th.
> > >
> > > Should the job be enabled now that there would be more 1.4 releases
> down
> > > the road ?
> > >
> >
>

Re: disabled Jenkins jobs

[Stack]

Its unassigned now.
S

On Sat, Jan 13, 2018 at 12:31 PM, Ted Yu  wrote:

> That was assigned to Sean.
>
> Do you know when Sean would be back ?
>
> On Sat, Jan 13, 2018 at 12:29 PM, Stack  wrote:
>
> > Yeah. HBASE-18467. Patches welcome.
> > S
> >
> > On Sat, Jan 13, 2018 at 10:13 AM, Ted Yu  wrote:
> >
> > > I noticed that for JIRAs which were applied to branch-1.x , there was
> no
> > > post from Hudson about the corresponding build.
> > >
> > > It would be desirable to have such post for branch-1.x .
> > >
> > > Cheers
> > >
> > > On Wed, Jan 10, 2018 at 9:45 AM, Mike Drob  wrote:
> > >
> > > > I believe we're migrating everything to the nightly branch matrix
> job.
> > > >
> > > > https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/
> > > >
> > > > On Wed, Jan 10, 2018 at 10:44 AM, Ted Yu 
> wrote:
> > > >
> > > > > Hi,
> > > > > I was looking at:
> > > > > https://builds.apache.org/job/HBase-1.4/
> > > > >
> > > > > I found that the last build was from Dec 8th.
> > > > >
> > > > > Should the job be enabled now that there would be more 1.4 releases
> > > down
> > > > > the road ?
> > > > >
> > > >
> > >
> >
>

Re: HBase 1.3.2-SNAPSHOT (20180113T013910Z)

[Stack]

+1

Checked hash and signature.
Downloaded bin. Checked layout and doc.
Started it up. Checked UI and logs.
Loaded data. Verified it made it in.

St.Ack

On Fri, Jan 12, 2018 at 6:14 PM, Andrew Purtell  wrote:

> A snapshot of HBase 1.3.2 is available for testing in Apache's Maven or at
> ​​
> https://dist.apache.org/repos/dist/dev/hbase/hbase-1.3.2-SNAPSHOT/ . This
> snapshot was generated from the git SHA b6f4f511a6 .
>
> ​The compatibility report with respect to the previous release can be found
> at ​
> https://dist.apache.org/repos/dist/dev/hbase/hbase-1.3.2-
> SNAPSHOT/compat-check-report.html
> .
>
> This is a snapshot, not an official release. These artifacts are provided
> for development and testing convenience. Please use with caution. Do not
> deploy to production.
>
> We should be releasing 1.3.2 within the next month or so. On behalf of the
> HBase project I apologize for the length of time it has been since the
> 1.3.1 release, and for the number of changes ​included in the upcoming
> patch release as a result.
>
> The changes in this snapshot since the last 1.3 release or snapshot are:
>
>- HBASE-8758 Error in RegionCoprocessorHost class preScanner method
>documentation
>- HBASE-9393 Hbase does not closing a closed socket resulting in many
>CLOSE_WAIT
>- HBASE-9899 for idempotent operation dups, return the result instead of
>throwing conflict exception (addendum for branch-1)
>- HBASE-14220 nightly check that we can build a source tarball.
>- HBASE-15497 Incorrect javadoc for atomicity guarantee of Increment and
>Append
>- HBASE-15548 SyncTable: sourceHashDir is supposed to be optional but
>won't work without (Dave Latham)
>- HBASE-15607 Deprecating SnapShotInfo (Ram)
>- HBASE-15691 ConcurrentModificationException in BucketAllocator
>- HBASE-15947 Classes used only for tests included in main code base
>- HBASE-16011 TableSnapshotScanner and TableSnapshotInputFormat can
>produce duplicate rows if split table.
>- HBASE-16090 ResultScanner is not closed in
>SyncTable#finishRemainingHashRanges()
>- HBASE-16116 Remove redundant pattern *.iml
>- HBASE-16351 Improve error reporting for license failures
>- HBASE-16459 Remove unused hbase shell --format option
>- HBASE-16615 Fix flaky TestScannerHeartbeatMessages (Duo Zhang)
>- HBASE-16939 ExportSnapshot: set owner and permission on right
> directory
>- HBASE-17131 Avoid livelock caused by HRegion#processRowsWithLocks
>- HBASE-17352 Fix hbase-assembly build with bash 4
>- HBASE-17441 Fix invalid quoting around hadoop-3 build in yetus
>personality
>- HBASE-17514 emit a warning if thrift1 proxy user is configured but
>hbase.regionserver.thrift.http is not
>- HBASE-17534 Avoid re-wrapping IOExceptions as IOExceptions
>- HBASE-17617 Backport HBASE-16731 (Inconsistent results from the
>Get/Scan if we use the empty FilterList) to branch-1
>- HBASE-17648 HBase Table-level synchronization fails between two
>secured(kerberized) cluster
>- HBASE-17658 Fix bookkeeping error with max regions for a table
>- HBASE-17803 PE always re-creates table when we specify the split
> policy
>- HBASE-17817 add table name to output (if available) when removing
>coprocessors
>- HBASE-17862 Fix a condition that always returns true
>- HBASE-17877 Improve HBase's byte[] comparator.
>- HBASE-17887 Row-level consistency is broken for read
>- HBASE-17902 Backport HBASE-16367 "Race between master and region
>server initialization may lead to premature server abort" to 1.3
>- HBASE-17916 Error message not clear when the permission of staging dir
>is not as expected
>- HBASE-17925 mvn assembly:single fails against hadoop3-alpha2
>- HBASE-17934 Backport HBASE-17779 "disable_table_replication returns
>misleading message and does not turn off replication"
>- HBASE-17934 Backport HBASE-17779 "disable_table_replication returns
>misleading message and does not turn off replication" (Janos Gub)
>- HBASE-17937 Memstore size becomes negative in case of expensive
>postPut/Delete Coprocessor call
>- HBASE-17944 Removed unused JDK version parsing from ClassSize.
>- HBASE-17954 Switched findbugs implementation to spotbugs
>- HBASE-17968 Fix NOTICE.txt for src-release
>- HBASE-17985 Inline apt-get update calls in Yetus Dockerfile
>- HBASE-18000 Make sure we always return the scanner id with
> ScanResponse
>- HBASE-18014 A case of Region remain unassigned when table enabled
>(Allan Yang)
>- HBASE-18020 Update API Compliance Checker to Incorporate Improvements
>Done in Hadoop
>- HBASE-18023 Log multi-* requests for more than threshold number of
> rows
>- HBASE-18023 Update row threshold warning from 1k to 5k (addendum)
>- HBASE-18024 HRegion#initializeRegionInternals should not re-create
>.hregioninfo file when the region directory no longer exi

Re: HBase 1.3.2-SNAPSHOT (20180113T013910Z)

[Stack]

More Still +1.

Downloaded src. Built a tgz. Ran it. Poked around (logs, UI).

Looks good,
S


On Sat, Jan 13, 2018 at 1:19 PM, Stack  wrote:

> +1
>
> Checked hash and signature.
> Downloaded bin. Checked layout and doc.
> Started it up. Checked UI and logs.
> Loaded data. Verified it made it in.
>
> St.Ack
>
> On Fri, Jan 12, 2018 at 6:14 PM, Andrew Purtell 
> wrote:
>
>> A snapshot of HBase 1.3.2 is available for testing in Apache's Maven or at
>> ​​
>> https://dist.apache.org/repos/dist/dev/hbase/hbase-1.3.2-SNAPSHOT/ . This
>> snapshot was generated from the git SHA b6f4f511a6 .
>>
>> ​The compatibility report with respect to the previous release can be
>> found
>> at ​
>> https://dist.apache.org/repos/dist/dev/hbase/hbase-1.3.2-SNA
>> PSHOT/compat-check-report.html
>> .
>>
>> This is a snapshot, not an official release. These artifacts are provided
>> for development and testing convenience. Please use with caution. Do not
>> deploy to production.
>>
>> We should be releasing 1.3.2 within the next month or so. On behalf of the
>> HBase project I apologize for the length of time it has been since the
>> 1.3.1 release, and for the number of changes ​included in the upcoming
>> patch release as a result.
>>
>> The changes in this snapshot since the last 1.3 release or snapshot are:
>>
>>- HBASE-8758 Error in RegionCoprocessorHost class preScanner method
>>documentation
>>- HBASE-9393 Hbase does not closing a closed socket resulting in many
>>CLOSE_WAIT
>>- HBASE-9899 for idempotent operation dups, return the result instead
>> of
>>throwing conflict exception (addendum for branch-1)
>>- HBASE-14220 nightly check that we can build a source tarball.
>>- HBASE-15497 Incorrect javadoc for atomicity guarantee of Increment
>> and
>>Append
>>- HBASE-15548 SyncTable: sourceHashDir is supposed to be optional but
>>won't work without (Dave Latham)
>>- HBASE-15607 Deprecating SnapShotInfo (Ram)
>>- HBASE-15691 ConcurrentModificationException in BucketAllocator
>>- HBASE-15947 Classes used only for tests included in main code base
>>- HBASE-16011 TableSnapshotScanner and TableSnapshotInputFormat can
>>produce duplicate rows if split table.
>>- HBASE-16090 ResultScanner is not closed in
>>SyncTable#finishRemainingHashRanges()
>>- HBASE-16116 Remove redundant pattern *.iml
>>- HBASE-16351 Improve error reporting for license failures
>>- HBASE-16459 Remove unused hbase shell --format option
>>- HBASE-16615 Fix flaky TestScannerHeartbeatMessages (Duo Zhang)
>>- HBASE-16939 ExportSnapshot: set owner and permission on right
>> directory
>>- HBASE-17131 Avoid livelock caused by HRegion#processRowsWithLocks
>>- HBASE-17352 Fix hbase-assembly build with bash 4
>>- HBASE-17441 Fix invalid quoting around hadoop-3 build in yetus
>>personality
>>- HBASE-17514 emit a warning if thrift1 proxy user is configured but
>>hbase.regionserver.thrift.http is not
>>- HBASE-17534 Avoid re-wrapping IOExceptions as IOExceptions
>>- HBASE-17617 Backport HBASE-16731 (Inconsistent results from the
>>Get/Scan if we use the empty FilterList) to branch-1
>>- HBASE-17648 HBase Table-level synchronization fails between two
>>secured(kerberized) cluster
>>- HBASE-17658 Fix bookkeeping error with max regions for a table
>>- HBASE-17803 PE always re-creates table when we specify the split
>> policy
>>- HBASE-17817 add table name to output (if available) when removing
>>coprocessors
>>- HBASE-17862 Fix a condition that always returns true
>>- HBASE-17877 Improve HBase's byte[] comparator.
>>- HBASE-17887 Row-level consistency is broken for read
>>- HBASE-17902 Backport HBASE-16367 "Race between master and region
>>server initialization may lead to premature server abort" to 1.3
>>- HBASE-17916 Error message not clear when the permission of staging
>> dir
>>is not as expected
>>- HBASE-17925 mvn assembly:single fails against hadoop3-alpha2
>>- HBASE-17934 Backport HBASE-17779 "disable_table_replication returns
>>misleading message and does not turn off replication"
>>- HBASE-17934 Backport HBASE-17779 "disable_table_replication returns
>>misleading message and does not turn off replication" (Janos Gub)
>>- HBASE-17937 Memstore size becomes negative in case of expensive
>>postPut/Delete Coprocess

Re: [VOTE] The fourth hbase-2.0.0-beta-1 Release Candidate is available for download

[Stack]

Thanks to all who have voted so far. It is great to see the votes coming
in. Need one more PMC-er +1 please.
Thanks,
S

On Fri, Jan 12, 2018 at 8:48 PM, Stack  wrote:

> The fourth release candidate for HBase 2.0.0-beta-1 is up at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-beta-1-RC3/
>
> Maven artifacts are available from a staging directory here:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1193
>
> All was signed with my key at 8ACC93D2 [1].
>
> I tagged the RC as 2.0.0-beta-1-RC1.7 at hash 026f535a7747b89003252ded9
> 585e827686aa79f
>
> This RC has some nice bug fixes over RC0-2 including fixing MOST of the
> failing and flakey tests (We are still working through them).
>
> hbase-2.0.0-beta-1 is our first beta release. It includes all that was in
> previous alphas (new assignment manager, offheap read/write path, in-memory
> compactions, etc.). The APIs and feature-set are sealed.
>
> hbase-2.0.0-beta-1 is a not-for-production preview of hbase-2.0.0. It is
> meant for devs and downstreamers to test drive and flag us if we messed up
> on anything ahead of our rolling GAs. We are particular interested in
> hearing from Coprocessor developers.
>
> The list of features addressed in 2.0.0 so far can be found here [3].
> There are thousands. The list of ~2k+ fixes in 2.0.0 exclusively can be
> found here [4] (My JIRA JQL foo is a bit dodgy -- forgive me if mistakes).
>
> I've updated our overview doc. on the state of 2.0.0 [6]. We'll do one
> more beta before we put up our first 2.0.0 Release Candidate by the end of
> January, 2.0.0-beta-2. Its focus will be making it so users can do a
> rolling upgrade on to hbase-2.x from hbase-1.x (and any bug fixes found
> running beta-1). Here is the list of what we have targeted so far for
> beta-2 [5]. Check it out.
>
> One known issue is that the User API has not been properly filtered so it
> shows more than just InterfaceAudience Public content (HBASE-19663, to be
> fixed by beta-2).
>
> Please take this beta for a spin. Please vote on whether it ok to put out
> this RC as our first beta (Note CHANGES has not yet been updated). Let the
> VOTE be open for 72 hours (Lets say Tuesday morning!)
>
> Thanks,
> Your 2.0.0 Release Manager
>
> 1. http://pgp.mit.edu/pks/lookup?op=get&search=0x9816C7FC8ACC93D2
> 3. https://goo.gl/scYjJr
> 4. https://goo.gl/dFFT8b
> 5. https://issues.apache.org/jira/projects/HBASE/versions/12340862
> 6. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> z9iEu_ktczrlKHK8N4SZzs/
>

[RESULT][VOTE] The fourth hbase-2.0.0-beta-1 Release Candidate is available for download

[Stack]

With 6 +1s (three binding) and none against, the vote passes. Thanks to all
who helped with this difficult delivery.

St.Ack



On Fri, Jan 12, 2018 at 8:48 PM, Stack  wrote:

> The fourth release candidate for HBase 2.0.0-beta-1 is up at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-beta-1-RC3/
>
> Maven artifacts are available from a staging directory here:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1193
>
> All was signed with my key at 8ACC93D2 [1].
>
> I tagged the RC as 2.0.0-beta-1-RC1.7 at hash 026f535a7747b89003252ded9
> 585e827686aa79f
>
> This RC has some nice bug fixes over RC0-2 including fixing MOST of the
> failing and flakey tests (We are still working through them).
>
> hbase-2.0.0-beta-1 is our first beta release. It includes all that was in
> previous alphas (new assignment manager, offheap read/write path, in-memory
> compactions, etc.). The APIs and feature-set are sealed.
>
> hbase-2.0.0-beta-1 is a not-for-production preview of hbase-2.0.0. It is
> meant for devs and downstreamers to test drive and flag us if we messed up
> on anything ahead of our rolling GAs. We are particular interested in
> hearing from Coprocessor developers.
>
> The list of features addressed in 2.0.0 so far can be found here [3].
> There are thousands. The list of ~2k+ fixes in 2.0.0 exclusively can be
> found here [4] (My JIRA JQL foo is a bit dodgy -- forgive me if mistakes).
>
> I've updated our overview doc. on the state of 2.0.0 [6]. We'll do one
> more beta before we put up our first 2.0.0 Release Candidate by the end of
> January, 2.0.0-beta-2. Its focus will be making it so users can do a
> rolling upgrade on to hbase-2.x from hbase-1.x (and any bug fixes found
> running beta-1). Here is the list of what we have targeted so far for
> beta-2 [5]. Check it out.
>
> One known issue is that the User API has not been properly filtered so it
> shows more than just InterfaceAudience Public content (HBASE-19663, to be
> fixed by beta-2).
>
> Please take this beta for a spin. Please vote on whether it ok to put out
> this RC as our first beta (Note CHANGES has not yet been updated). Let the
> VOTE be open for 72 hours (Lets say Tuesday morning!)
>
> Thanks,
> Your 2.0.0 Release Manager
>
> 1. http://pgp.mit.edu/pks/lookup?op=get&search=0x9816C7FC8ACC93D2
> 3. https://goo.gl/scYjJr
> 4. https://goo.gl/dFFT8b
> 5. https://issues.apache.org/jira/projects/HBASE/versions/12340862
> 6. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> z9iEu_ktczrlKHK8N4SZzs/
>

[ANNOUNCE] Apache HBase 2.0.0-beta-1 is now available for download

[Stack]

The HBase team is happy to announce the immediate availability of Apache
​HBase​ 2.0.0-beta-1!

Apache HBase is an open-source, distributed, versioned, non-relational
database. Apache HBase gives you low latency random access to billions of
rows with millions of columns atop non-specialized hardware. To learn more
about HBase, see https://hbase.apache.org/.

Download through an ASF mirror:

https://www.apache.org/dyn/closer.lua/hbase/

(It may take a while to show up on a particular mirror site)

hbase-2.0.0-beta-1 is our first beta release. It includes all that was in
previous alphas (new assignment manager, offheap read/write path, in-memory
compactions, etc.) only now the
APIs and feature-set are sealed as we run up to our hbase-2.0.0 release.

hbase-2.0.0-beta-1 is a not-for-production preview of hbase-2.0.0. It is
meant for devs and downstreamers to test drive and flag us if we messed up
on anything ahead of our rolling GAs. We are particular interested in
hearing from Coprocessor developers.

The list of features addressed in 2.0.0 so far can be found here [3]. There
are thousands. The list of ~2k+ fixes in 2.0.0 exclusively can be found
here [4].

I've updated our overview doc. on the state of 2.0.0 [6]. We'll do one more
beta before we put up our first 2.0.0 Release Candidate by the end of
February, 2.0.0-beta-2. Its focus will be making it so users can do a
rolling upgrade onto hbase-2.x from hbase-1.x (and any bug fixes found
running beta-1). Here is the list of what we have targeted so far for
beta-2 [5]. Check it out.

One known issue is that the User API has not been properly filtered so it
shows more than just InterfaceAudience Public content (HBASE-19663, to be
fixed by beta-2).

Please take this beta for a spin. Let us know if you find any issue so we
can fix it before we GA.

For instructions on verifying ASF release downloads, please see

https://www.apache.org/dyn/closer.cgi#verify

Project member signature keys can be found at

https://www.apache.org/dist/hbase/KEYS

This release was tagged with rel/2.0.0-beta-1
(026f535a7747b89003252ded9585e827686aa79f)

Thanks to all the contributors who made this release possible!

Yours,
The HBase Dev Team

3. https://goo.gl/scYjJr
4. https://goo.gl/dFFT8b
5. https://issues.apache.org/jira/projects/HBASE/versions/12340862
6. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
z9iEu_ktczrlKHK8N4SZzs/

Re: Temporary jenkins job to run tests for 2.0 against hadoop3

[Stack]

Thanks Appy. We need this.
S

On Fri, Jan 19, 2018 at 4:25 PM, Apekshit Sharma  wrote:

> Here's the job: https://builds.apache.org/job/HBase-2.0-hadoop3-tests
> (Still trying to make it work).
>
> Why do we need separate temporary job?
> - Jenkins pipeline doesn't yet support running step B if step A fails, and
> our hadoop2 tests (which run before hadoop3 tests) always fail.
> - Parallelizing two test runs will require some major refactor of
> Jenkinsfile. Will need to split out test run into separate job so that they
> can run on different machines. Single Apache machine isn't big enough to
> run our two mvn test executions simultaneously.
>
> Other better suggestions to make the same happen would be great!
>
> -- Appy
>

Re: Two master branches?

[Stack]

Looks like Master was added a few days ago, by mistake is my guess:
https://github.com/apache/hbase/branches

Kill it after ensuring master has all its commits?

Shout if you want me to do it sir.

S

On Fri, Jan 19, 2018 at 2:50 PM, Apekshit Sharma  wrote:

> I see following three branches:
>   remotes/apache/HEAD -> apache/master
>   remotes/apache/Master
>   remotes/apache/master
>
> Was "Master" always there?
>
> --
>
> -- Appy
>

Re: Testing and CI -- Apache Jenkins Builds (WAS -> Re: Testing)

[Stack]

Our brothers and sisters over in yetus-land made a release that deals w/
the changed JIRA behavior regards ordering attached-patches. No need of
deleting all but the intended patch going forward nor gymnastics with
prefixes when naming.  It seems to be working properly. The one-liner
change that moves us from yetus 0.6.0 to 0.7.0 has been pushed to all
active branches and our hadoopqa up on jenkins has been configured to use
it going forward.

FYI,
S

On Fri, Dec 8, 2017 at 12:13 PM, Stack  wrote:

> Thanks Andrew. I disabled the job. Use the nightly going forward. The jdk7
> builds seem to run fine. The jdk8 has some timeout going on. Need to dig
> in. You can see here:
>
> https://builds.apache.org/view/H-L/view/HBase/job/HBase%
> 20Nightly/job/branch-1.4/
>
> Thanks,
> M
>
> On Fri, Dec 8, 2017 at 11:29 AM, Andrew Purtell 
> wrote:
>
>> Ok with me, Stack. Thanks for asking.
>>
>>
>> On Thu, Nov 30, 2017 at 5:33 PM, Stack  wrote:
>>
>> > On the move over to nightly test runs:
>> >
>> > 1.2 nightly had a successful build last night after the branch-1
>> > stabilization effort (HBASE-19204) and fixing a few unit test failures.
>> See
>> > build 150
>> > https://builds.apache.org/view/H-L/view/HBase/job/HBase%
>> > 20Nightly/job/branch-1.2/
>> > It then failed, 151, because of timed out test. Need to dig in. Clean
>> up a
>> > few more unit tests and branch-1.2 is probably ready for a
>> release-cutting.
>> >
>> > 1.3 has a few flakies. The last build failed because of:
>> >
>> >   Test Result (1 failure / ±0)
>> > org.apache.hadoop.hbase.regionserver.TestEncryptionKeyRotation.
>> > testCFKeyRotation
>> >
>> > Just a little effort should turn 1.3 green.
>> >
>> > I was going to disable the 1.4 job,
>> > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.4/,  in
>> favor of
>> > the 1.4 nightly,
>> > https://builds.apache.org/view/H-L/view/HBase/job/HBase%
>> > 20Nightly/job/branch-1.4/,
>> > if ok w/ you Andrew Purtell... And move over the branch-1, branch-2, and
>> > master too.
>> >
>> > Thanks,
>> > S
>> >
>> >
>> >
>> > On Wed, Nov 29, 2017 at 8:06 AM, Stack  wrote:
>> >
>> > > Example of the new nice reporting: vhttps://builds.apache.org/
>> > > view/H-L/view/HBase/job/HBase%20Nightly/job/branch-1.2/
>> > > S
>> > >
>> > > On Wed, Nov 29, 2017 at 8:06 AM, Stack  wrote:
>> > >
>> > >> Note that I have disabled the HBase-1.2-JDK7, HBase-1.2-JDK8,
>> > >> HBase-1.3-JDK7, and HBase-1.3-JDK8 jobs. They have been broken for a
>> > good
>> > >> while now. In their place, refer to an ongoing Sean "Nightly"
>> project,
>> > an
>> > >> effort he has been at for a while. It does more checking with pretty
>> > >> reports that will help figuring general stability over time. See
>> under
>> > >> https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/
>> > >> See the nightly builds for 1.2 and 1.3. They have some teething
>> issues
>> > >> still but are almost there. See the 1.2 build from last night. In
>> recent
>> > >> days, the 1.2 branch went from trash-can fire to stable. See how all
>> > tests
>> > >> passed in the last build but then we failed generating the src
>> bundle on
>> > >> the end (this is what I mean by 'teething' issue). Will work on
>> fixing
>> > this
>> > >> last step and moving over 1.4, etc., in the next few days.
>> > >>
>> > >> FYI,
>> > >> St.Ack
>> > >>
>> > >>
>> > >> On Tue, Nov 7, 2017 at 7:45 AM, Stack  wrote:
>> > >>
>> > >>> On Tue, Nov 7, 2017 at 6:10 AM, Sean Busbey 
>> wrote:
>> > >>>
>> > >>>> > Should I be able to see the machine dir when I look at nightlies
>> > >>>> output?
>> > >>>> > (Was trying to see what else is running).
>> > >>>>
>> > >>>> Ah. we don't have the same machine sampling on nightly as we do in
>> > >>>> precommit. I am 80% on a patch for HBASE-19189 (run test ad-hoc
>> > >>>> repeatedly)  that includes pulling that information gathering into
>> a
>> > >>>> place where we could also u

Re: Precommit OutOfMemoryError

[Stack]

Looking...
S

On Tue, Jan 30, 2018 at 2:37 AM, 张铎(Duo Zhang) 
wrote:

> Seems like an infra issue. In the morning(in China) there is no available
> node under the label ubuntu...
>
> Anyone who can access the infra mailing list please send an email? I'm not
> in that list yet...
>
> Thanks.
>
> 2018-01-30 18:11 GMT+08:00 Peter Somogyi :
>
> > The Nightly build started to fail as well.
> > branch-2: https://builds.apache.org/job/HBase%20Nightly/job/branch-2/
> 223/
> > artifact/output-jdk8-hadoop2/patch-unit-root.txt
> >
> > On Tue, Jan 30, 2018 at 11:01 AM, Peter Somogyi 
> > wrote:
> >
> > > Hi,
> > >
> > > Recent precommit jobs are failing with OutOfMemoryError or "[ERROR]
> > unable
> > > to create new native thread".
> > > https://builds.apache.org/job/PreCommit-HBASE-Build/11250/
> > > artifact/patchprocess/patch-mvninstall-root.txt
> > >  > 11250/artifact/patchprocess/patch-mvninstall-root.txt>
> > > https://builds.apache.org/job/PreCommit-HBASE-Build/11251/
> > > artifact/patchprocess/patch-unit-hbase-server.txt
> > >
> > > Can someone with access rights to Apache Jenkins take a look?
> > >
> > > Thanks,
> > > Peter
> > >
> >
>

Re: Precommit OutOfMemoryError

[Stack]

Oh, concurrent, there seems to be an errant geode build process making its
way around the build boxes according to INFRA.
S

On Tue, Jan 30, 2018 at 5:01 PM, Stack  wrote:

> Thanks Sean. Allen showed up on issues to report same. Trying to backfill
> his recommendation on how to up proclimit count.
> S
>
> On Tue, Jan 30, 2018 at 4:30 PM, Sean Busbey  wrote:
>
>> This sounds like it could be caused by the new feature in yetus 0.7.0 to
>> provide a configurable bound on resource usage, fyi.
>>
>> On Jan 30, 2018 08:53, "Stack"  wrote:
>>
>> > Looking...
>> > S
>> >
>> > On Tue, Jan 30, 2018 at 2:37 AM, 张铎(Duo Zhang) 
>> > wrote:
>> >
>> > > Seems like an infra issue. In the morning(in China) there is no
>> available
>> > > node under the label ubuntu...
>> > >
>> > > Anyone who can access the infra mailing list please send an email? I'm
>> > not
>> > > in that list yet...
>> > >
>> > > Thanks.
>> > >
>> > > 2018-01-30 18:11 GMT+08:00 Peter Somogyi :
>> > >
>> > > > The Nightly build started to fail as well.
>> > > > branch-2: https://builds.apache.org/job/
>> HBase%20Nightly/job/branch-2/
>> > > 223/
>> > > > artifact/output-jdk8-hadoop2/patch-unit-root.txt
>> > > >
>> > > > On Tue, Jan 30, 2018 at 11:01 AM, Peter Somogyi <
>> psomo...@cloudera.com
>> > >
>> > > > wrote:
>> > > >
>> > > > > Hi,
>> > > > >
>> > > > > Recent precommit jobs are failing with OutOfMemoryError or
>> "[ERROR]
>> > > > unable
>> > > > > to create new native thread".
>> > > > > https://builds.apache.org/job/PreCommit-HBASE-Build/11250/
>> > > > > artifact/patchprocess/patch-mvninstall-root.txt
>> > > > > <https://builds.apache.org/job/PreCommit-HBASE-Build/
>> > > > 11250/artifact/patchprocess/patch-mvninstall-root.txt>
>> > > > > https://builds.apache.org/job/PreCommit-HBASE-Build/11251/
>> > > > > artifact/patchprocess/patch-unit-hbase-server.txt
>> > > > >
>> > > > > Can someone with access rights to Apache Jenkins take a look?
>> > > > >
>> > > > > Thanks,
>> > > > > Peter
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Re: Precommit OutOfMemoryError

[Stack]

Thanks Sean. Allen showed up on issues to report same. Trying to backfill
his recommendation on how to up proclimit count.
S

On Tue, Jan 30, 2018 at 4:30 PM, Sean Busbey  wrote:

> This sounds like it could be caused by the new feature in yetus 0.7.0 to
> provide a configurable bound on resource usage, fyi.
>
> On Jan 30, 2018 08:53, "Stack"  wrote:
>
> > Looking...
> > S
> >
> > On Tue, Jan 30, 2018 at 2:37 AM, 张铎(Duo Zhang) 
> > wrote:
> >
> > > Seems like an infra issue. In the morning(in China) there is no
> available
> > > node under the label ubuntu...
> > >
> > > Anyone who can access the infra mailing list please send an email? I'm
> > not
> > > in that list yet...
> > >
> > > Thanks.
> > >
> > > 2018-01-30 18:11 GMT+08:00 Peter Somogyi :
> > >
> > > > The Nightly build started to fail as well.
> > > > branch-2: https://builds.apache.org/job/
> HBase%20Nightly/job/branch-2/
> > > 223/
> > > > artifact/output-jdk8-hadoop2/patch-unit-root.txt
> > > >
> > > > On Tue, Jan 30, 2018 at 11:01 AM, Peter Somogyi <
> psomo...@cloudera.com
> > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Recent precommit jobs are failing with OutOfMemoryError or "[ERROR]
> > > > unable
> > > > > to create new native thread".
> > > > > https://builds.apache.org/job/PreCommit-HBASE-Build/11250/
> > > > > artifact/patchprocess/patch-mvninstall-root.txt
> > > > > <https://builds.apache.org/job/PreCommit-HBASE-Build/
> > > > 11250/artifact/patchprocess/patch-mvninstall-root.txt>
> > > > > https://builds.apache.org/job/PreCommit-HBASE-Build/11251/
> > > > > artifact/patchprocess/patch-unit-hbase-server.txt
> > > > >
> > > > > Can someone with access rights to Apache Jenkins take a look?
> > > > >
> > > > > Thanks,
> > > > > Peter
> > > > >
> > > >
> > >
> >
>

Re: Testing and CI -- Apache Jenkins Builds (WAS -> Re: Testing)

[Stack]

Note that I reverted our yetus version last night. It discombobulated our
builds (OOMEs). Meantime, you'll have to do the patch naming trick for
another day or so. Our test runs seem to use an ungodly number of file
descriptors Stay tuned.
S

On Mon, Jan 29, 2018 at 10:56 PM, Stack  wrote:

> Our brothers and sisters over in yetus-land made a release that deals w/
> the changed JIRA behavior regards ordering attached-patches. No need of
> deleting all but the intended patch going forward nor gymnastics with
> prefixes when naming.  It seems to be working properly. The one-liner
> change that moves us from yetus 0.6.0 to 0.7.0 has been pushed to all
> active branches and our hadoopqa up on jenkins has been configured to use
> it going forward.
>
> FYI,
> S
>
> On Fri, Dec 8, 2017 at 12:13 PM, Stack  wrote:
>
>> Thanks Andrew. I disabled the job. Use the nightly going forward. The
>> jdk7 builds seem to run fine. The jdk8 has some timeout going on. Need to
>> dig in. You can see here:
>>
>> https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Ni
>> ghtly/job/branch-1.4/
>>
>> Thanks,
>> M
>>
>> On Fri, Dec 8, 2017 at 11:29 AM, Andrew Purtell 
>> wrote:
>>
>>> Ok with me, Stack. Thanks for asking.
>>>
>>>
>>> On Thu, Nov 30, 2017 at 5:33 PM, Stack  wrote:
>>>
>>> > On the move over to nightly test runs:
>>> >
>>> > 1.2 nightly had a successful build last night after the branch-1
>>> > stabilization effort (HBASE-19204) and fixing a few unit test
>>> failures. See
>>> > build 150
>>> > https://builds.apache.org/view/H-L/view/HBase/job/HBase%
>>> > 20Nightly/job/branch-1.2/
>>> > It then failed, 151, because of timed out test. Need to dig in. Clean
>>> up a
>>> > few more unit tests and branch-1.2 is probably ready for a
>>> release-cutting.
>>> >
>>> > 1.3 has a few flakies. The last build failed because of:
>>> >
>>> >   Test Result (1 failure / ±0)
>>> > org.apache.hadoop.hbase.regionserver.TestEncryptionKeyRotation.
>>> > testCFKeyRotation
>>> >
>>> > Just a little effort should turn 1.3 green.
>>> >
>>> > I was going to disable the 1.4 job,
>>> > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.4/,  in
>>> favor of
>>> > the 1.4 nightly,
>>> > https://builds.apache.org/view/H-L/view/HBase/job/HBase%
>>> > 20Nightly/job/branch-1.4/,
>>> > if ok w/ you Andrew Purtell... And move over the branch-1, branch-2,
>>> and
>>> > master too.
>>> >
>>> > Thanks,
>>> > S
>>> >
>>> >
>>> >
>>> > On Wed, Nov 29, 2017 at 8:06 AM, Stack  wrote:
>>> >
>>> > > Example of the new nice reporting: vhttps://builds.apache.org/
>>> > > view/H-L/view/HBase/job/HBase%20Nightly/job/branch-1.2/
>>> > > S
>>> > >
>>> > > On Wed, Nov 29, 2017 at 8:06 AM, Stack  wrote:
>>> > >
>>> > >> Note that I have disabled the HBase-1.2-JDK7, HBase-1.2-JDK8,
>>> > >> HBase-1.3-JDK7, and HBase-1.3-JDK8 jobs. They have been broken for a
>>> > good
>>> > >> while now. In their place, refer to an ongoing Sean "Nightly"
>>> project,
>>> > an
>>> > >> effort he has been at for a while. It does more checking with pretty
>>> > >> reports that will help figuring general stability over time. See
>>> under
>>> > >> https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/
>>> > >> See the nightly builds for 1.2 and 1.3. They have some teething
>>> issues
>>> > >> still but are almost there. See the 1.2 build from last night. In
>>> recent
>>> > >> days, the 1.2 branch went from trash-can fire to stable. See how all
>>> > tests
>>> > >> passed in the last build but then we failed generating the src
>>> bundle on
>>> > >> the end (this is what I mean by 'teething' issue). Will work on
>>> fixing
>>> > this
>>> > >> last step and moving over 1.4, etc., in the next few days.
>>> > >>
>>> > >> FYI,
>>> > >> St.Ack
>>> > >>
>>> > >>
>>> > >> On Tue, Nov 7, 2017 at 7:45 AM, Stack  wrote:
>>> >

Re: Testing and CI -- Apache Jenkins Builds (WAS -> Re: Testing)

[Stack]

I just set hadoopqa to be 0.7.0 again with an upped proclimit to see if
this fixes our OOME failures.. HadoopQA builds numbered 11295 and later
will have this change.
Thanks
S

On Wed, Jan 31, 2018 at 6:46 AM, Stack  wrote:

> Note that I reverted our yetus version last night. It discombobulated our
> builds (OOMEs). Meantime, you'll have to do the patch naming trick for
> another day or so. Our test runs seem to use an ungodly number of file
> descriptors Stay tuned.
> S
>
> On Mon, Jan 29, 2018 at 10:56 PM, Stack  wrote:
>
>> Our brothers and sisters over in yetus-land made a release that deals w/
>> the changed JIRA behavior regards ordering attached-patches. No need of
>> deleting all but the intended patch going forward nor gymnastics with
>> prefixes when naming.  It seems to be working properly. The one-liner
>> change that moves us from yetus 0.6.0 to 0.7.0 has been pushed to all
>> active branches and our hadoopqa up on jenkins has been configured to use
>> it going forward.
>>
>> FYI,
>> S
>>
>> On Fri, Dec 8, 2017 at 12:13 PM, Stack  wrote:
>>
>>> Thanks Andrew. I disabled the job. Use the nightly going forward. The
>>> jdk7 builds seem to run fine. The jdk8 has some timeout going on. Need to
>>> dig in. You can see here:
>>>
>>> https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Ni
>>> ghtly/job/branch-1.4/
>>>
>>> Thanks,
>>> M
>>>
>>> On Fri, Dec 8, 2017 at 11:29 AM, Andrew Purtell 
>>> wrote:
>>>
>>>> Ok with me, Stack. Thanks for asking.
>>>>
>>>>
>>>> On Thu, Nov 30, 2017 at 5:33 PM, Stack  wrote:
>>>>
>>>> > On the move over to nightly test runs:
>>>> >
>>>> > 1.2 nightly had a successful build last night after the branch-1
>>>> > stabilization effort (HBASE-19204) and fixing a few unit test
>>>> failures. See
>>>> > build 150
>>>> > https://builds.apache.org/view/H-L/view/HBase/job/HBase%
>>>> > 20Nightly/job/branch-1.2/
>>>> > It then failed, 151, because of timed out test. Need to dig in. Clean
>>>> up a
>>>> > few more unit tests and branch-1.2 is probably ready for a
>>>> release-cutting.
>>>> >
>>>> > 1.3 has a few flakies. The last build failed because of:
>>>> >
>>>> >   Test Result (1 failure / ±0)
>>>> > org.apache.hadoop.hbase.regionserver.TestEncryptionKeyRotation.
>>>> > testCFKeyRotation
>>>> >
>>>> > Just a little effort should turn 1.3 green.
>>>> >
>>>> > I was going to disable the 1.4 job,
>>>> > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.4/,  in
>>>> favor of
>>>> > the 1.4 nightly,
>>>> > https://builds.apache.org/view/H-L/view/HBase/job/HBase%
>>>> > 20Nightly/job/branch-1.4/,
>>>> > if ok w/ you Andrew Purtell... And move over the branch-1, branch-2,
>>>> and
>>>> > master too.
>>>> >
>>>> > Thanks,
>>>> > S
>>>> >
>>>> >
>>>> >
>>>> > On Wed, Nov 29, 2017 at 8:06 AM, Stack  wrote:
>>>> >
>>>> > > Example of the new nice reporting: vhttps://builds.apache.org/
>>>> > > view/H-L/view/HBase/job/HBase%20Nightly/job/branch-1.2/
>>>> > > S
>>>> > >
>>>> > > On Wed, Nov 29, 2017 at 8:06 AM, Stack  wrote:
>>>> > >
>>>> > >> Note that I have disabled the HBase-1.2-JDK7, HBase-1.2-JDK8,
>>>> > >> HBase-1.3-JDK7, and HBase-1.3-JDK8 jobs. They have been broken for
>>>> a
>>>> > good
>>>> > >> while now. In their place, refer to an ongoing Sean "Nightly"
>>>> project,
>>>> > an
>>>> > >> effort he has been at for a while. It does more checking with
>>>> pretty
>>>> > >> reports that will help figuring general stability over time. See
>>>> under
>>>> > >> https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/
>>>> > >> See the nightly builds for 1.2 and 1.3. They have some teething
>>>> issues
>>>> > >> still but are almost there. See the 1.2 build from last night. In
>>>> recent
>>>> > >> days, the 1.2 branch went

Re: Testing and CI -- Apache Jenkins Builds (WAS -> Re: Testing)

[Stack]

All seems to have settled now. Hadoopqa is running 'normally' again with
yetus 0.7.0 and some new configs (Thanks to Allen Wittenhauer for the
help/input...). That said, we need to work on curbing resources used during
test runs
St.Ack

On Wed, Jan 31, 2018 at 9:01 AM, Stack  wrote:

> I just set hadoopqa to be 0.7.0 again with an upped proclimit to see if
> this fixes our OOME failures.. HadoopQA builds numbered 11295 and later
> will have this change.
> Thanks
> S
>
> On Wed, Jan 31, 2018 at 6:46 AM, Stack  wrote:
>
>> Note that I reverted our yetus version last night. It discombobulated our
>> builds (OOMEs). Meantime, you'll have to do the patch naming trick for
>> another day or so. Our test runs seem to use an ungodly number of file
>> descriptors Stay tuned.
>> S
>>
>> On Mon, Jan 29, 2018 at 10:56 PM, Stack  wrote:
>>
>>> Our brothers and sisters over in yetus-land made a release that deals w/
>>> the changed JIRA behavior regards ordering attached-patches. No need of
>>> deleting all but the intended patch going forward nor gymnastics with
>>> prefixes when naming.  It seems to be working properly. The one-liner
>>> change that moves us from yetus 0.6.0 to 0.7.0 has been pushed to all
>>> active branches and our hadoopqa up on jenkins has been configured to use
>>> it going forward.
>>>
>>> FYI,
>>> S
>>>
>>> On Fri, Dec 8, 2017 at 12:13 PM, Stack  wrote:
>>>
>>>> Thanks Andrew. I disabled the job. Use the nightly going forward. The
>>>> jdk7 builds seem to run fine. The jdk8 has some timeout going on. Need to
>>>> dig in. You can see here:
>>>>
>>>> https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Ni
>>>> ghtly/job/branch-1.4/
>>>>
>>>> Thanks,
>>>> M
>>>>
>>>> On Fri, Dec 8, 2017 at 11:29 AM, Andrew Purtell 
>>>> wrote:
>>>>
>>>>> Ok with me, Stack. Thanks for asking.
>>>>>
>>>>>
>>>>> On Thu, Nov 30, 2017 at 5:33 PM, Stack  wrote:
>>>>>
>>>>> > On the move over to nightly test runs:
>>>>> >
>>>>> > 1.2 nightly had a successful build last night after the branch-1
>>>>> > stabilization effort (HBASE-19204) and fixing a few unit test
>>>>> failures. See
>>>>> > build 150
>>>>> > https://builds.apache.org/view/H-L/view/HBase/job/HBase%
>>>>> > 20Nightly/job/branch-1.2/
>>>>> > It then failed, 151, because of timed out test. Need to dig in.
>>>>> Clean up a
>>>>> > few more unit tests and branch-1.2 is probably ready for a
>>>>> release-cutting.
>>>>> >
>>>>> > 1.3 has a few flakies. The last build failed because of:
>>>>> >
>>>>> >   Test Result (1 failure / ±0)
>>>>> > org.apache.hadoop.hbase.regionserver.TestEncryptionKeyRotation.
>>>>> > testCFKeyRotation
>>>>> >
>>>>> > Just a little effort should turn 1.3 green.
>>>>> >
>>>>> > I was going to disable the 1.4 job,
>>>>> > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.4/,  in
>>>>> favor of
>>>>> > the 1.4 nightly,
>>>>> > https://builds.apache.org/view/H-L/view/HBase/job/HBase%
>>>>> > 20Nightly/job/branch-1.4/,
>>>>> > if ok w/ you Andrew Purtell... And move over the branch-1, branch-2,
>>>>> and
>>>>> > master too.
>>>>> >
>>>>> > Thanks,
>>>>> > S
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Wed, Nov 29, 2017 at 8:06 AM, Stack  wrote:
>>>>> >
>>>>> > > Example of the new nice reporting: vhttps://builds.apache.org/
>>>>> > > view/H-L/view/HBase/job/HBase%20Nightly/job/branch-1.2/
>>>>> > > S
>>>>> > >
>>>>> > > On Wed, Nov 29, 2017 at 8:06 AM, Stack  wrote:
>>>>> > >
>>>>> > >> Note that I have disabled the HBase-1.2-JDK7, HBase-1.2-JDK8,
>>>>> > >> HBase-1.3-JDK7, and HBase-1.3-JDK8 jobs. They have been broken
>>>>> for a
>>>>> > good
>>>>> > >> while now. In their place, refer to an ongoing Sean "Nightly&qu

Re: Please invite me slack

[Stack]

On Thu, Feb 1, 2018 at 4:01 AM, Kang Minwoo  wrote:

> Hello, dev users.
>
> I want to join the slack channel.
> Please invite me slack.
>
> Best regards,
> Minwoo Kang
>

Looks like an invite has been sent?
S

Re: QuotaExceededException as a DoNotRetryIOException?

[Stack]

QEE being a DNRIOE seems right on the face of it.

But if throttling, a DNRIOE is inappropriate. Where you seeing a QEE in a
throttling scenario Huaxiang?

Thanks,
S


On Tue, Feb 6, 2018 at 4:56 PM, Huaxiang Sun  wrote:

> Hi HBase devs,
>
> I found that QuotaExceededException  is a DoNotRetryIOException, which
> is a bit strange from user’s point of view.
> For rpc throttling, the exception is retryable and it tells app to
> slow down and retry later.
>
> Any thoughts?
>
> Thanks,
> Huaxiang

Re: HBaseCon Plans?

[Stack]

On Fri, Feb 2, 2018 at 9:13 PM, Mike Drob  wrote:

> Hi folks, has there been any consideration put forth toward the next
> HBaseCon? The last one was very productive for me personally, but I hadn't
> heard anything about the schedule for 2018 so figured I could ask on list.
>
> Mike
>

Is been kinda quiet this year in terms of hbasecon2018. We, the community,
have been running the last bunch hosted by a generous, main sponsor (Huawei
in Shenzhen and Google on east and west coast). If there was the interest,
we could go beat the bushes to turn up a venue and a date. Wouldn't have to
be a grand affair.

Thanks,
St.Ack

Re: HBaseCon Plans?

[Stack]

On Thu, Feb 8, 2018 at 9:37 AM, Andrew Purtell 
wrote:

> This is Lars H. on Andy's phone.
>
> I'm looking into hosting HBaseCon at Salesforce this year. Hopefully in
> the new Salesforce tower. Gimme a few days perhaps a week.
>
Thanks.
>
>
Only if it is the top floor Lars. Otherwise, HBaseConEastBay -- Tacubaya,
Starline Social Club -- here we come (smile).
St.Ack





> -- Lars
>
> > On Feb 8, 2018, at 4:18 AM, Jean-Marc Spaggiari 
> wrote:
> >
> > So who's jumping in for NY or SF? ;)
> >
> > 2018-02-08 4:46 GMT-05:00 Bijieshan :
> >
> >> Huawei can continue to hold HBaseCon Asia 2018 :-)
> >>
> >> Best Regards,
> >> Jieshan.
> >> -Original Message-
> >> From: saint@gmail.com [mailto:saint@gmail.com] On Behalf Of
> Stack
> >> Sent: 2018年2月8日 0:37
> >> To: HBase Dev List 
> >> Subject: Re: HBaseCon Plans?
> >>
> >>> On Fri, Feb 2, 2018 at 9:13 PM, Mike Drob  wrote:
> >>>
> >>> Hi folks, has there been any consideration put forth toward the next
> >>> HBaseCon? The last one was very productive for me personally, but I
> >>> hadn't heard anything about the schedule for 2018 so figured I could
> ask
> >> on list.
> >>>
> >>> Mike
> >>>
> >>
> >> Is been kinda quiet this year in terms of hbasecon2018. We, the
> community,
> >> have been running the last bunch hosted by a generous, main sponsor
> (Huawei
> >> in Shenzhen and Google on east and west coast). If there was the
> interest,
> >> we could go beat the bushes to turn up a venue and a date. Wouldn't
> have to
> >> be a grand affair.
> >>
> >> Thanks,
> >> St.Ack
> >>
>

Re: HBaseCon Plans?

[Stack]

Super sweet Yu Li!

Between your kind offer and Bijieshan's note, we should be good in China
(smile).

St.Ack

On Thu, Feb 8, 2018 at 12:39 AM, Yu Li  wrote:

> Boss, we (Alibaba) are interested to host this year's hbasecon (or hbasecon
> asia) in China (smile)
>
> Best Regards,
> Yu
>
> On 8 February 2018 at 00:36, Stack  wrote:
>
> > On Fri, Feb 2, 2018 at 9:13 PM, Mike Drob  wrote:
> >
> > > Hi folks, has there been any consideration put forth toward the next
> > > HBaseCon? The last one was very productive for me personally, but I
> > hadn't
> > > heard anything about the schedule for 2018 so figured I could ask on
> > list.
> > >
> > > Mike
> > >
> >
> > Is been kinda quiet this year in terms of hbasecon2018. We, the
> community,
> > have been running the last bunch hosted by a generous, main sponsor
> (Huawei
> > in Shenzhen and Google on east and west coast). If there was the
> interest,
> > we could go beat the bushes to turn up a venue and a date. Wouldn't have
> to
> > be a grand affair.
> >
> > Thanks,
> > St.Ack
> >
>

Re: HBaseCon Plans?

[Stack]

On Mon, Feb 12, 2018 at 8:13 AM, Artem Ervits  wrote:

> Who's the contact for coordinating HBaseCon on West Coast? Please provide
> contact details here or privately.
>
>
You could start with priv...@hbase.apache.org. This is the HBase Project
Management Committee.
Thanks,
S




> On Feb 8, 2018 10:57 PM, "Stack"  wrote:
>
> > Super sweet Yu Li!
> >
> > Between your kind offer and Bijieshan's note, we should be good in China
> > (smile).
> >
> > St.Ack
> >
> > On Thu, Feb 8, 2018 at 12:39 AM, Yu Li  wrote:
> >
> > > Boss, we (Alibaba) are interested to host this year's hbasecon (or
> > hbasecon
> > > asia) in China (smile)
> > >
> > > Best Regards,
> > > Yu
> > >
> > > On 8 February 2018 at 00:36, Stack  wrote:
> > >
> > > > On Fri, Feb 2, 2018 at 9:13 PM, Mike Drob 
> wrote:
> > > >
> > > > > Hi folks, has there been any consideration put forth toward the
> next
> > > > > HBaseCon? The last one was very productive for me personally, but I
> > > > hadn't
> > > > > heard anything about the schedule for 2018 so figured I could ask
> on
> > > > list.
> > > > >
> > > > > Mike
> > > > >
> > > >
> > > > Is been kinda quiet this year in terms of hbasecon2018. We, the
> > > community,
> > > > have been running the last bunch hosted by a generous, main sponsor
> > > (Huawei
> > > > in Shenzhen and Google on east and west coast). If there was the
> > > interest,
> > > > we could go beat the bushes to turn up a venue and a date. Wouldn't
> > have
> > > to
> > > > be a grand affair.
> > > >
> > > > Thanks,
> > > > St.Ack
> > > >
> > >
> >
>

Re: use of Apache Arrow?

[Stack]

On Thu, Feb 15, 2018 at 11:39 AM, Sean Busbey  wrote:

> Heya folks,
>
> I've been reading about underlying technologies we might use to
> improve coprocessors, with an eye towards a long term goal of better
> isolating them.
>
> Today I started reading about Apache Arrow, which is a data format and
> support library that has a focus on in-memory perf. (in particular
> they claim their libs are all zero-copy.)
>
> Their main website[1] calls out Apache HBase as a use case and our
> community as backers. I can't find anything in our code base though,
> anyone know of any previous or ongoing efforts?
>
>
>
Some noise on instantiation but nought thereafter that I know of.

What you thinking? We'd pass an external process blocks made by hbase for
it to play with and then we'd persist whatever it gave us back?

Thanks Sean,
S (Apache Arrow PMC -- Smile).



> [1]: http://arrow.apache.org/
>

Re: Region split caching issue

[Stack]

On Wed, Feb 14, 2018 at 3:19 PM, Zach York 
wrote:

> Hello,
>
> I have been trying for some time to figure out an issue where splits fail
> due to trying to cache the same key, but different value.


Whats the exception look like Mighty Zach?

...


> What I know/have observed:
> This occurs when prefetch on open is true
> Both daugher store openers will try to access and cache the first and last
> values in the store (HalfStoreFile). [1] [2] [3]
> Both daugher store openers will access the same key (the splitkey, I
> believe).
>


The lastkey in one daughter will be the firstkey in the other?




> * However they will access two different paths and retrieve two different
> values.
>

Different values? Or is that the same path (the half store files resolve to
same backing storefile) but different values somehow?



> Then when they try to cache these values, it fails because the other thread
> cached it's value and the two values aren't equivalent. [4]
> This is past the point of no return for the region split so it tries to
> roll forward, but fails. These regions are then corrupted and cannot be
> recovered.
>
> One thing I have notices is that these blocks are all leaf index blocks. I
> guess these index blocks would be different if it was in a different file.
> This is on HBase 1.3.1.
>
> It is definitely a concurrency bug as both daughter openers need to try to
> get the block from disk before the other caches the block. Why would it be
> trying to access two different paths for the same key? Is that normal in a
> reference file?
>
>
The two paths are probably the two reference file paths, one for each
daughter. That the paths are different should be fine (even if same value);
the key the block cache uses is the based off filename.

But you seem to be saying the key is the same so ultimately the file and
offset we are getting the value with -- used to make the block cache key --
are the same but somehow the value gotten differs?



> Does anyone have any idea as to what might be occurring?
> If you need more info, I can show you the code path that it is running
> through.
>
>
Say more Zach. Perhaps we can help out.

Thanks,

S



> Thanks,
> Zach
>
> [1]
> https://github.com/apache/hbase/blob/rel/1.3.1/hbase-
> server/src/main/java/org/apache/hadoop/hbase/io/
> HalfStoreFileReader.java#L357
> [2]
> https://github.com/apache/hbase/blob/rel/1.3.1/hbase-
> server/src/main/java/org/apache/hadoop/hbase/io/hfile/
> HFileReaderV2.java#L454
> [3]
> https://github.com/apache/hbase/blob/rel/1.3.1/hbase-
> server/src/main/java/org/apache/hadoop/hbase/io/hfile/
> HFileReaderV2.java#L461
> [4]
> https://github.com/apache/hbase/blob/rel/1.3.1/hbase-
> server/src/main/java/org/apache/hadoop/hbase/io/hfile/
> LruBlockCache.java#L367
>

Re: [VOTE] The first HBase 1.4.2 release candidate (RC0) is available

[Stack]

+1 (but one question on bottom)

Checked hash and signature.
Built tgz from src download and ran it.
Poked around. Looks good.
Loaded data. Verified it present.
Shutdown. Logs looked good.


Question: For some reason my master port was homed on 63502... anyone else
have this issue? I didn't spend much time digging figuring no one else saw
this

Thanks,
S





On Tue, Feb 20, 2018 at 3:24 AM, Peter Somogyi 
wrote:

> +1 (non-binding)
>
> Checked signature, sums
> Rat check
> Web UI: split, merge tables
> Shell basic functionalities
> LTT 1M rows
> Build with full test suite
> PE 1M rows
> Basic operations from java client
>
>
> On Fri, Feb 16, 2018 at 10:46 PM, Andrew Purtell 
> wrote:
>
> > The first HBase 1.4.2 release candidate (RC0) is available for download
> at
> > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.4.2RC0/ and Maven
> > artifacts are available in the temporary repository
> > https://repository.apache.org/content/repositories/orgapachehbase-1195/
> .
> >
> > The git tag corresponding to the candidate is '1.4.2RC0' (9519ec2ead).
> >
> > A detailed source and binary compatibility report for this release is
> > available for your review at https://dist.apache.org/repos/
> > dist/dev/hbase/hbase-1.4.2RC0/compat-check-report.html .
> >
> > A list of the 19 issues resolved in this release can be found at
> > https://s.apache.org/aGcb .
> >
> > Please try out the candidate and vote +1/0/-1.
> >
> > This vote will be open for at least 72 hours. Unless objection I will try
> > to close it Friday February 23, 2018 if we have sufficient votes.
> >
> > Prior to making this announcement I made the following preflight checks:
> >
> >- RAT check passes (7u80)
> >- Unit test suite passes (8u131)
> >- LTT load 1M rows with 100% verification and 20% updates (8u131)
> >- PE sequentialWrite, sequentialRead, randomWrite, randomRead,
> >scanRange100 (8u131)
> >- ITBLL Loop 1 100M rows (8u131)
> >
> >
> > --
> > Best regards,
> > Andrew
> >
>

Re: Best way to help on beta2?

[Stack]

On Tue, Feb 20, 2018 at 11:36 AM, Josh Elser  wrote:

> Hi folks,
>
> Just curious where I can offer a hand on the remaining 2.0.0-beta2 work. I
> know the theme was upgrade, and I also saw the good test-fixing work that's
> also happening now.
>
> Anything still pending on the radar I can help out with? Feel free to
> directly ping for any help on reviews.
>
> - Josh
>

Thanks Josh for asking.

You have one you are trying to land and I just asked you take a look at
another that I think lands in your corner.

In general, test fixing has been taking forever...  (Almost 200 issues
since beta-1). The flakey list was long and if you dug, there was a dirty
root cause that needed addressing. We are almost there now thanks to the
great work of a bunch of you all [1] (Duo has done a mountain of work in
here). The nightlies are starting to pass with hbase2 on hadoop2 [2] and we
are now into test failures running hbase2 over hadoop3 (Drob has some fun
fixes coming here). Correct me if I have it wrong but I'd think it'd be
hard to release an hbase2 if unit tests don't pass (most of the time).

Thereafter, its test and doc, mostly[3]. Help is needed here, in particular
enabling new stuff and trying it. Little has been done on scale and perf
that I know of. I was going to take a look at a rolling upgrade to see what
is involved and a last run through the compatibility report. Making this
work might have to be a post-2.0.0 thing; hbase-2.0.0 has been going on too
long.

Was hoping to beta-2 this weekend or start of next week.

Thanks,
St.Ack

1.
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests-branch2.0/lastSuccessfulBuild/artifact/dashboard.html
2. https://builds.apache.org/job/HBase%20Nightly/job/branch-2/
3. https://issues.apache.org/jira/projects/HBASE/versions/12340862

Re: client.HBaseAdmin No serialized HRegionInfo

[Stack]

On Fri, Feb 23, 2018 at 11:10 AM, Ted Yu  wrote:

> Can you check whether region c15b13946fa4318a0a956e067d59ebd4 is healthy
> (via hbck) ?
>
>

hbck does not work against hbase2 [1].
S

1.
https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#heading=h.3jy69yxm0gxm



> You may find some clue in master log.
>
> Cheers
>
>
>
> On Fri, Feb 23, 2018 at 11:04 AM, Jean-Marc Spaggiari <
> jean-m...@spaggiari.org> wrote:
>
> > Hi guys,
> >
> > In HBase 2.0.0-beta-1, getting this:
> >
> > 2018-02-23 14:01:55,499 WARN  [main] client.HBaseAdmin
> > (HBaseAdmin.java:visit(1948)) - No serialized HRegionInfo in
> > keyvalues={page_proposed,,1384198043055.c15b13946fa4318a0a956e067d59eb
> > d4./info:server/1384198854149/Put/vlen=11/seqid=0,
> > page_proposed,,1384198043055.c15b13946fa4318a0a956e067d59eb
> > d4./info:serverstartcode/1384198854149/Put/vlen=8/seqid=0}
> >
> >
> > I'm not getting that on other versions. And I'm not sure what it means...
> >
> > Any idea?
> >
> > Thanks,
> >
> > JMS
> >
>

Re: client.HBaseAdmin No serialized HRegionInfo

[Stack]

On Fri, Feb 23, 2018 at 11:04 AM, Jean-Marc Spaggiari <
jean-m...@spaggiari.org> wrote:

> Hi guys,
>
> In HBase 2.0.0-beta-1, getting this:
>
> 2018-02-23 14:01:55,499 WARN  [main] client.HBaseAdmin
> (HBaseAdmin.java:visit(1948)) - No serialized HRegionInfo in
> keyvalues={page_proposed,,1384198043055.c15b13946fa4318a0a956e067d59eb
> d4./info:server/1384198854149/Put/vlen=11/seqid=0,
> page_proposed,,1384198043055.c15b13946fa4318a0a956e067d59eb
> d4./info:serverstartcode/1384198854149/Put/vlen=8/seqid=0}
>
>
> I'm not getting that on other versions. And I'm not sure what it means...
>
> Any idea?
>
>
The Admin client is getting detail from hbase:meta. It found a row where
there was no serialized regioninfo and is WARNing you of this. This should
never happen. It is dumping out all it did find in the meta table.

Why no regioninfo in hbase:meta? This a fresh hbase2 or are you starting an
hbase2 over an old hbase1 dataset? If latter, could it have been null
before the update?

Thanks,
S



> Thanks,
>
> JMS
>

Re: Temporarily disabled the timeout setting for flakey tests jenkins job

[Stack]

Duo:

A bad nightly build ruined our dashboard. A bad run produced many tests for
the flakies list. The flakies list then timed out because too many tests to
run.  Appy cleaned it up, mostly. It will take a few hours for the failures
to clear out (it seems to be coming along nicely).

I should have dumped note in here on dev that we were working on it so you
didn't have to waste your time.

Thanks,
S



On Tue, Feb 27, 2018 at 6:27 PM, 张铎(Duo Zhang) 
wrote:

> Not sure why it happens but the build timeout is set to the default value
> which is 50 minutes instead of depending on the previous build time.
> Disable the timeout setting to see if it helps.
>

Please hold on commits to branch-2; cutting an hbase-2.0.0-beta-2 Release Candidate

[Stack]

Short Version: Please hold on making commits to branch-2. I am in the
process of cutting a hbase-2.0.0-beta-2 release candidate. It make take me
a day or so.

Long Version:

A miracle happened last night. Our nightly for branch-2 passed [1]. This
means all unit tests ran against hadoop2 and then subsequently, all unit
tests ran against hadoop3. It had been threatening for a while. A few
recent builds passed all unit tests only to fail on a few findbugs issues.

Given this momentous event and that there are over 200 fixes since beta-1,
it is time to push out a beta-2.

Originally beta-2 was supposed to be about making sure we could do a
rolling upgrade from branch-1 to branch-2. This was its theme. The work has
not been done. Also, a bunch of work is still to be done around testing,
compatibility checking, and performance. Perhaps folks are waiting on the
2.0.0 release candidate before they dig in (smile)?

I'll give the all clear once I've cut my release. Thank you for your
patience.
St.Ack


1. See build #415 here
https://builds.apache.org/job/HBase%20Nightly/job/branch-2/

Re: Moving 2.0 forward

[Stack]

In another thread in this list I flag the coming 2.0.0-beta-2. Look for an
RC in the next day or so.

After 2.0.0-beta-2 makes it out, I'll cut branch-2.0. Soon after will come
our first hbase-2.0.0 release candidate. Expect this within a week or so
after beta-2 after remaining blockers are addressed (At this stage,
blockers are mostly documentation). Scream if there is something that has
to make 2.0.0 that I may have missed, but at this point, only critical
bug-fixes or compatibility breakages will be entertained.

I just did a pass over the issues filed against hbase-2.0.0. I did a pretty
brutal edit, sparse on commentary mostly because the count of issues
against 2.0.0 was high. There were a fair amount filed against '2.0.0' just
because 2.0.0 used to be thought of as the end of the rainbow, an event
that was way off on the horizon that would never arrive[1]. I unscheduled
the bulk of these. Reschedule against 2.0.0 if I got it wrong.  A good
bunch were great ideas almost- or half-done or never started (HBASE-14055
RegionServer Refactor, HBASE-15531 Favored Nodes, Andy's security
improvements, etc.). These were also unscheduled. Others that were really
old without recent updates I resolved. A few had been realized via other
channels.

Here is the current list [2]. Help on blockers would be much appreciated.

Your 2.0.0RM

P.S.No scale testing, perf compare, compatibility, or exercise of new
features (at least to my knowledge) has been done. We could do with some
exercise along any of these dimensions if any of you have a few spare
cycles. Thanks.

1. hbase-2.0.0 still has this smell to it, but we do our best.
2.
https://issues.apache.org/jira/projects/HBASE/versions/12327188#release-report-tab-body


On Wed, Dec 27, 2017 at 9:54 AM, Stack  wrote:

> Heads-up, I'm working on cutting a beta-1. Shout if there is something you
> think needs to make it in (The Chia-Ping Tsai cleanups look good). I'm
> mainly waiting on the hbase-thirdparty changes to go in. The rest can wait
> it.
>
> Thanks,
> S
>
>
> On Wed, Dec 13, 2017 at 11:28 PM, Stack  wrote:
>
>> Update. See below.
>>
>> On Tue, Dec 5, 2017 at 10:43 AM, Stack  wrote:
>>
>>> Update on hbase-2.0.0-beta-1. I'm going to put up a beta-1 RC before
>>> Christmas day.
>>>
>>>
>> Still aiming for an hbase2 beta-1 showing up in your christmas stocking.
>>
>>
>>> Here is the list [1]. We're doing pretty good. We've fixed a load since
>>> the last update including some nice cleanups  two weeks ago (e.g. undoing
>>> client dependency on curator/zk watching making the dependence read-only)
>>> but there are still a few hairy ones hanging out there.
>>>
>>>  * HBASE-18946 Stochastic load balancer assigns replica regions to the
>>> same RS
>>>This one is proving a little tough. Ram has been banging his head on
>>> it. We'll be able to clear a bunch of test failures once this is done.
>>>
>>>
>> This should be going in tomorrow.
>>
>>
>>
>>>  * HBASE-19112 Suspect methods on Cell to be deprecated
>>>Good discussion up on the JIRA. We need to be crystal clear around
>>> Cell and derivatives API and Audience when we ship 2.0.
>>>
>>>
>> Ram, Chia-Ping Tsai, and Anoop are doing a nice job here and it should be
>> done in next day or so.
>>
>>
>>
>>>  * HBASE-17204 Make L2 off heap cache default ON
>>>We need to try this at least so offheaping can be default.
>>>
>>>
>> Chatted with Anoop and Ram. Because it is so late in the day and because
>> it critical that the user have a good experience out of the box -- which
>> requires time and testing -- we are going to punt this from 2.0. Will look
>> at it for 2.1 (or 3.0 if soon). 2.0 will still be carrying all of the boys
>> offheap read/write path goodness.
>>
>>
>>
>>> Making asyncfswal default is also ongoing, HBASE-16890
>>> <https://issues.apache.org/jira/browse/HBASE-16890>, making good
>>> progress.
>>>
>>>
>>
>> Good progress here. In perf testing asyncfswal makes us generally faster.
>> Duo making good progress on last few tests that are in the way of our
>> making this default for hbase2.
>>
>> A load of goodness has been landing these last few weeks. Keep up the
>> great work.
>>
>> Your hbase-2 RM.
>>
>>
>>
>>
>>> I punted HBASE-19147 "All branch-2 unit tests pass" out to beta-2. Our
>>> unit test story has gotten better and a few of us are actively working on
>>> flakies [2] but we may make a beta-1 

[VOTE] The first hbase-2.0.0-beta-2 Release Candidate is available for download

[Stack]

The first release candidate for HBase 2.0.0-beta-2 is up at

 https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-beta-2.RC0/

Maven artifacts are available from a staging directory here:

  https://repository.apache.org/content/repositories/orgapachehbase-1199

All was signed with my key at 8ACC93D2 [1]

I tagged the RC as 2.0.0-beta-2RC0.2 at
9e9b347d667e1fc6165c9f8ae5ae7052147e8895

hbase-2.0.0-beta-2 is a not-for-production preview of hbase-2.0.0. It is
meant for devs and downstreamers to test drive and flag us if we messed up
on anything ahead of our rolling
actual 2.0.0 release candidates ("GAs").

hbase-2.0.0-beta-2 is our second beta release. More than 200 fixes have
gone in since
beta-1. Unit tests generallly pass when run against hadoop2 and hadoop3[5].
It includes
all that was in previous alphas and beta (new assignment manager, offheap
read/write
path, in-memory compactions, etc).The list of features addressed in 2.0.0
so far can be
found here [3]. There are thousands. The list of ~3k+ fixes in 2.0.0
exclusively can be
found here [4]. Our overview doc. on the state of 2.0.0 is at [6].

This beta was supposed to have as its focus rolling upgrade from hbase-1.x
versions but
this is work not complete (At this late stage, it is looking like it will
be a post-2.0.0 project).

This is our last hbase-2.0.0 beta release. Next up, we'll be rolling an
actual 2.0.0 release
candidate. Look for this in a week or two after beta-2 goes out, after
we've done more
testing and documentation (and we fix issues raised by you all against this
beta).

One known issue, still unaddressed, is that the User API has not been
properly filtered
so it shows more than just InterfaceAudience Public content (HBASE-19663,
to be fixed
by release).

Please take this beta for a spin. Please vote on whether it ok to put out
this RC as our second
beta (Note CHANGES has not yet been updated). Let the VOTE be open for at
least 72 hours
(Lets say Wednesday morning, March 7th).

Thanks,
Your 2.0.0 Release Manager

1. http://pgp.mit.edu/pks/lookup?op=get&search=0x9816C7FC8ACC93D2
3. https://goo.gl/scYjJr
4. https://goo.gl/dFFT8b
5. https://builds.apache.org/job/HBase%20Nightly/job/branch-2/

6. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
z9iEu_ktczrlKHK8N4SZzs/

Re: [VOTE] The first hbase-2.0.0-beta-2 Release Candidate is available for download

[Stack]

Thank Umesh for giving it a run.

Did you start the hbase2 over an hbase1 dataset. I've seen the exception
you note when I've done this. The startup keeps going over this exception,
right? (IIRC, its a complaint reading a file written w/ hbase1... We fail
to read in the bloom filter which is not the end-of-the-world).

On the replication failures, you can reproduce reliably?

Thanks,
St.Ack



On Sun, Mar 4, 2018 at 9:04 AM, Umesh Agashe  wrote:

> -1 non-binding (concerns noted below)
>
> download src & bin tar ball   - OK
> signatures & sums- OK
> build from source (openjdk version "1.8.0_151")  - OK
> rat check   -
> OK
> unit tests   -
> NOT OK
> start local instance from bin & CRUD from shell  - OK
> LTT write, read1 million rows, 2 cols/row  - OK
> Upgrade from 1.4.2   - OK
> check logs -
> NOT OK
>
> NOTE:
> * Error message in the log is concerning.
>
> Found following exception logged multiple times (~11) in the log:
> ERROR [StoreFileOpenerThread-test_cf-1] regionserver.StoreFileReader:
> Error
> reading bloom filter meta for GENERAL_BLOOM_META -- proceeding without
> java.io.IOException: Comparator class
> org.apache.hadoop.hbase.KeyValue$RawBytesComparator
> is not instantiable
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComp
> arator(FixedFileTrailer.java:628)
> at org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.(
> CompoundBloomFilter.java:79)
> at org.apache.hadoop.hbase.util.BloomFilterFactory.createFromMe
> ta(BloomFilterFactory.java:104)
> at org.apache.hadoop.hbase.regionserver.StoreFileReader.loadBlo
> omfilter(StoreFileReader.java:479)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.open(HStoreF
> ile.java:425)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.initReader(H
> StoreFile.java:460)
> at org.apache.hadoop.hbase.regionserver.HStore.createStoreFileA
> ndReader(HStore.java:671)
> at org.apache.hadoop.hbase.regionserver.HStore.lambda$openStore
> Files$0(HStore.java:537)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executor
> s.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
>
>
> * Some of the replication UTs failed with message "peer doesn't exist" in
> the log
>
> [ERROR] Errors:
> [ERROR]   TestReplicationAdmin.testAddPeerWithState:171 » RetriesExhausted
> Failed after ...
> [ERROR]   TestReplicationAdmin.testAppendPeerTableCFs:273 »
> RetriesExhausted Failed afte...
> [ERROR]   TestReplicationAdmin.testEnableDisable:247 » RetriesExhausted
> Failed after att...
> [ERROR]   TestReplicationAdmin.testPeerBandwidth:763 » RetriesExhausted
> Failed after att...
> [ERROR]   TestReplicationAdmin.testPeerClusterKey:779 » RetriesExhausted
> Failed after at...
> [ERROR]   TestReplicationAdmin.testPeerConfig:192 » RetriesExhausted
> Failed
> after attemp...
> [ERROR]   TestReplicationAdmin.testPeerConfigConflict:656 »
> RetriesExhausted Failed afte...
> [ERROR]   TestReplicationAdmin.testPeerExcludeNamespaces:515 »
> RetriesExhausted Failed a...
> [ERROR]   TestReplicationAdmin.testPeerReplicationEndpointImpl:795 »
> RetriesExhausted Fa...
> [ERROR]   TestReplicationAdmin.testSetPeerNamespaces:457 »
> RetriesExhausted
> Failed after...
> [ERROR]   TestReplicationAdmin.testSetReplicateAllUserTables:490 »
> RetriesExhausted Fail...
> [INFO]
> [ERROR] Tests run: 1906, Failures: 0, Errors: 11, Skipped: 21
>
>
> On Fri, Mar 2, 2018 at 3:40 PM, Stack  wrote:
>
> > The first release candidate for HBase 2.0.0-beta-2 is up at
> >
> >  https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-beta-2.RC0/
> >
> > Maven artifacts are available from a staging directory here:
> >
> >   https://repository.apache.org/content/repositories/orgapachehbase-1199
> >
> > All was signed with my key at 8ACC93D2 [1]
> >
> > I tagged the RC as 2.0.0-beta-2RC0.2 at
> > 9e9b347d667e1fc6165c9f8ae5ae7052147e8895
> >
> > hbase-2.0

Re: [VOTE] The first hbase-2.0.0-beta-2 Release Candidate is available for download

[Stack]

Thanks  Umesh. File an issue though? Stuff Anoops comment in it. While
'harmless', it will scare users; we should fix it.
Thanks for trying the RC,
S

On Sun, Mar 4, 2018 at 8:18 PM, Umesh Agashe  wrote:

> Yes, I started hbase2 over hbase1 when I got the exception. I also tried
> running TestReplicationAdmin a couple of times and can not reliably
> reproduce the problem.
>
> I am changing my vote to +1.
>
> Thanks,
> Umesh
>
>
> On Sun, Mar 4, 2018 at 2:43 PM, Stack  wrote:
>
> > Thank Umesh for giving it a run.
> >
> > Did you start the hbase2 over an hbase1 dataset. I've seen the exception
> > you note when I've done this. The startup keeps going over this
> exception,
> > right? (IIRC, its a complaint reading a file written w/ hbase1... We fail
> > to read in the bloom filter which is not the end-of-the-world).
> >
> > On the replication failures, you can reproduce reliably?
> >
> > Thanks,
> > St.Ack
> >
> >
> >
> > On Sun, Mar 4, 2018 at 9:04 AM, Umesh Agashe 
> wrote:
> >
> > > -1 non-binding (concerns noted below)
> > >
> > > download src & bin tar ball   - OK
> > > signatures & sums- OK
> > > build from source (openjdk version "1.8.0_151")  - OK
> > > rat check
>  -
> > > OK
> > > unit tests
> >  -
> > > NOT OK
> > > start local instance from bin & CRUD from shell  - OK
> > > LTT write, read1 million rows, 2 cols/row  - OK
> > > Upgrade from 1.4.2   - OK
> > > check logs
>  -
> > > NOT OK
> > >
> > > NOTE:
> > > * Error message in the log is concerning.
> > >
> > > Found following exception logged multiple times (~11) in the log:
> > > ERROR [StoreFileOpenerThread-test_cf-1] regionserver.StoreFileReader:
> > > Error
> > > reading bloom filter meta for GENERAL_BLOOM_META -- proceeding without
> > > java.io.IOException: Comparator class
> > > org.apache.hadoop.hbase.KeyValue$RawBytesComparator
> > > is not instantiable
> > > at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.
> createComp
> > > arator(FixedFileTrailer.java:628)
> > > at org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.<
> init>(
> > > CompoundBloomFilter.java:79)
> > > at org.apache.hadoop.hbase.util.BloomFilterFactory.
> createFromMe
> > > ta(BloomFilterFactory.java:104)
> > > at org.apache.hadoop.hbase.regionserver.StoreFileReader.
> loadBlo
> > > omfilter(StoreFileReader.java:479)
> > > at org.apache.hadoop.hbase.regionserver.HStoreFile.open(
> HStoreF
> > > ile.java:425)
> > > at org.apache.hadoop.hbase.regionserver.HStoreFile.
> initReader(H
> > > StoreFile.java:460)
> > > at org.apache.hadoop.hbase.regionserver.HStore.
> createStoreFileA
> > > ndReader(HStore.java:671)
> > > at org.apache.hadoop.hbase.regionserver.HStore.lambda$
> openStore
> > > Files$0(HStore.java:537)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at java.util.concurrent.Executors$RunnableAdapter.
> call(Executor
> > > s.java:511)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPool
> > > Executor.java:1149)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoo
> > > lExecutor.java:624)
> > > at java.lang.Thread.run(Thread.java:748)
> > > Caused by: java.lang.NullPointerException
> > >
> > >
> > > * Some of the replication UTs failed with message "peer doesn't exist"
> in
> > > the log
> > >
> > > [ERROR] Errors:
> > > [ERROR]   TestReplicationAdmin.testAddPeerWithState:171 »
> > RetriesExhausted
> > > Failed after ...
> > > [ERROR]   TestReplicationAdmin.testAppendPeerTableCFs:273 »
> > > RetriesExhausted Failed afte...
> > > [ERROR]   TestReplicationAdmin.testEnableDisable:247 »
> RetriesExhausted
> > > Failed after att...
> > > [ERROR]   TestReplicationAdmin.testPeerBandwidth:763 »
> RetriesExhausted
> > > Failed after att...
> > > [ERROR]   TestReplicationAdmin.testPeerClusterKey:779 »
> RetriesExhausted
> > > Failed

Re: [VOTE] The first hbase-2.0.0-beta-2 Release Candidate is available for download

[Stack]

On Mon, Mar 5, 2018 at 6:56 AM, Chia-Ping Tsai  wrote:

> +1 (binding) with some questions
> unit test (oracle jdk-8u161) - all pass
> deploy binary (3 nodes) - ok
> browse master/regionserver web - LGTM
> put/delete/get/scan 500W rows - ok
> create/disable/drop/put/scan/count by shell - ok
>
> Q1:
> the 2.0 binary get fatter than 1.4. (384 MB -> 887MB). Seems we package
> the test docs and source code to binary tar. Are all of them necessary?
>
>
No. Let me review.



> Q2:
> Why some test jars of hadoop are included to hbase/lib? such as
> hadoop-common-2.7.4-tests and hadoop-hdfs-2.7.4-tests
>
>
We have it as an explicit dependency in our top-level pom. Doing
dependency:tree, hadoop minicluster needs it? Its need to run tests? You
think we should not include in our binary sir?

[INFO] +- org.apache.hadoop:hadoop-minicluster:jar:2.7.4:test
[INFO] |  +- org.apache.hadoop:hadoop-common:test-jar:tests:2.7.4:test


Thanks for trying the RC Chia-Ping,
S







>
> On 2018/03/02 23:40:53, Stack  wrote:
> > The first release candidate for HBase 2.0.0-beta-2 is up at
> >
> >  https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-beta-2.RC0/
> >
> > Maven artifacts are available from a staging directory here:
> >
> >   https://repository.apache.org/content/repositories/orgapachehbase-1199
> >
> > All was signed with my key at 8ACC93D2 [1]
> >
> > I tagged the RC as 2.0.0-beta-2RC0.2 at
> > 9e9b347d667e1fc6165c9f8ae5ae7052147e8895
> >
> > hbase-2.0.0-beta-2 is a not-for-production preview of hbase-2.0.0. It is
> > meant for devs and downstreamers to test drive and flag us if we messed
> up
> > on anything ahead of our rolling
> > actual 2.0.0 release candidates ("GAs").
> >
> > hbase-2.0.0-beta-2 is our second beta release. More than 200 fixes have
> > gone in since
> > beta-1. Unit tests generallly pass when run against hadoop2 and
> hadoop3[5].
> > It includes
> > all that was in previous alphas and beta (new assignment manager, offheap
> > read/write
> > path, in-memory compactions, etc).The list of features addressed in 2.0.0
> > so far can be
> > found here [3]. There are thousands. The list of ~3k+ fixes in 2.0.0
> > exclusively can be
> > found here [4]. Our overview doc. on the state of 2.0.0 is at [6].
> >
> > This beta was supposed to have as its focus rolling upgrade from
> hbase-1.x
> > versions but
> > this is work not complete (At this late stage, it is looking like it will
> > be a post-2.0.0 project).
> >
> > This is our last hbase-2.0.0 beta release. Next up, we'll be rolling an
> > actual 2.0.0 release
> > candidate. Look for this in a week or two after beta-2 goes out, after
> > we've done more
> > testing and documentation (and we fix issues raised by you all against
> this
> > beta).
> >
> > One known issue, still unaddressed, is that the User API has not been
> > properly filtered
> > so it shows more than just InterfaceAudience Public content (HBASE-19663,
> > to be fixed
> > by release).
> >
> > Please take this beta for a spin. Please vote on whether it ok to put out
> > this RC as our second
> > beta (Note CHANGES has not yet been updated). Let the VOTE be open for at
> > least 72 hours
> > (Lets say Wednesday morning, March 7th).
> >
> > Thanks,
> > Your 2.0.0 Release Manager
> >
> > 1. http://pgp.mit.edu/pks/lookup?op=get&search=0x9816C7FC8ACC93D2
> > 3. https://goo.gl/scYjJr
> > 4. https://goo.gl/dFFT8b
> > 5. https://builds.apache.org/job/HBase%20Nightly/job/branch-2/
> > <https://goo.gl/dFFT8b>
> > 6. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> > z9iEu_ktczrlKHK8N4SZzs/
> >
>

[DISCUSS] What should we bundle in our bin tarball?

[Stack]

On the vote thread for hbase-2.0.0-beta-2, it was noted that the bulk of
our convenience bin tarball size is test javadoc [1] (~ > 50%). Should our
bin tarball include all possible doc both api and dev api for both test and
user?

I could make it so we do not build dev doc by default or that we do not
include it in our bin tarball? What do you all think?

Thanks,
St.Ack


1.
$ cd hbase-2.0.0-beta-2
$ du -sh *
256K CHANGES.txt
4.0K LEGAL
128K LICENSE.txt
420K NOTICE.txt
4.0K README.txt
184K bin
 40K conf
818M docs
620K hbase-webapps
129M lib

$ cd docs/
$ du -sh *
  0B _chapters
 28K acid-semantics.html
 11M apache_hbase_reference_guide.pdf
152M apidocs
 32K asciidoctor.css
4.0K book
1.9M book.html
 16K bulk-loads.html
 20K coc.html
4.0K coderay-asciidoctor.css
116K css
 36K cygwin.html
 20K dependencies.html
272K dependency-convergence.html
 16K dependency-info.html
 48K dependency-management.html
332M devapidocs
4.0K doap_Hbase.rdf
 16K export_control.html
200K fonts
 32K hbase.css
2.5M images
 28K img
 20K index.html
 16K integration.html
 16K issue-tracking.html
148K js
 28K license.html
 16K mail-lists.html
 20K metrics.html
 24K old_news.html
 20K plugin-management.html
 20K plugins.html
 40K poweredbyhbase.html
 16K project-info.html
 16K project-reports.html
 16K project-summary.html
 16K pseudo-distributed.html
 16K replication.html
372K repo
 16K resources.html
 16K source-repository.html
 16K sponsors.html
 24K supportingprojects.html
 32K team-list.html
112M testapidocs
205M testdevapidocs

Re: [DISCUSS] What should we bundle in our bin tarball?

[Stack]

Thanks lads. Let me look at purge of non-user javadocs and at building a
javadoc-only artifiact (then could purge all javadoc as per Andy).

S

On Tue, Mar 6, 2018 at 11:45 AM, Sean Busbey  wrote:

> if we keep any javadocs in the tarball they should just be the user
> facing javadocs (wether that includes testapidocs I could go either
> way). I'd be in favor of dumping the javadocs entirely from the binary
> tarball.
>
> do we know why the apidocs are bigger than our binaries? are we
> mistakingly include a source rendition in html again?
>
> We've been a bit spotty on keeping per-release-line javadocs up to
> date on the website. To give ourselves an out on that, what about
> including a release aritfact that's just a tarball of javadocs?
>
> On Tue, Mar 6, 2018 at 1:33 PM, Andrew Purtell 
> wrote:
> >> I could make it so we do not build dev doc by default or that we do not
> > include it in our bin tarball?
> >
> > The binary tarball is so huge at this point I think getting the size down
> > by pruning javadoc is reasonable, at least in the short term. We have
> > javadocs up on our site. They will also be available in Maven and your
> > favorite IDE should handle the automatic download and use of javadoc
> jars.
> >
> >
> > On Tue, Mar 6, 2018 at 11:29 AM, Stack  wrote:
> >
> >> On the vote thread for hbase-2.0.0-beta-2, it was noted that the bulk of
> >> our convenience bin tarball size is test javadoc [1] (~ > 50%). Should
> our
> >> bin tarball include all possible doc both api and dev api for both test
> and
> >> user?
> >>
> >> I could make it so we do not build dev doc by default or that we do not
> >> include it in our bin tarball? What do you all think?
> >>
> >> Thanks,
> >> St.Ack
> >>
> >>
> >> 1.
> >> $ cd hbase-2.0.0-beta-2
> >> $ du -sh *
> >> 256K CHANGES.txt
> >> 4.0K LEGAL
> >> 128K LICENSE.txt
> >> 420K NOTICE.txt
> >> 4.0K README.txt
> >> 184K bin
> >>  40K conf
> >> 818M docs
> >> 620K hbase-webapps
> >> 129M lib
> >>
> >> $ cd docs/
> >> $ du -sh *
> >>   0B _chapters
> >>  28K acid-semantics.html
> >>  11M apache_hbase_reference_guide.pdf
> >> 152M apidocs
> >>  32K asciidoctor.css
> >> 4.0K book
> >> 1.9M book.html
> >>  16K bulk-loads.html
> >>  20K coc.html
> >> 4.0K coderay-asciidoctor.css
> >> 116K css
> >>  36K cygwin.html
> >>  20K dependencies.html
> >> 272K dependency-convergence.html
> >>  16K dependency-info.html
> >>  48K dependency-management.html
> >> 332M devapidocs
> >> 4.0K doap_Hbase.rdf
> >>  16K export_control.html
> >> 200K fonts
> >>  32K hbase.css
> >> 2.5M images
> >>  28K img
> >>  20K index.html
> >>  16K integration.html
> >>  16K issue-tracking.html
> >> 148K js
> >>  28K license.html
> >>  16K mail-lists.html
> >>  20K metrics.html
> >>  24K old_news.html
> >>  20K plugin-management.html
> >>  20K plugins.html
> >>  40K poweredbyhbase.html
> >>  16K project-info.html
> >>  16K project-reports.html
> >>  16K project-summary.html
> >>  16K pseudo-distributed.html
> >>  16K replication.html
> >> 372K repo
> >>  16K resources.html
> >>  16K source-repository.html
> >>  16K sponsors.html
> >>  24K supportingprojects.html
> >>  32K team-list.html
> >> 112M testapidocs
> >> 205M testdevapidocs
> >>
> >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> >- A23, Crosstalk
>

Re: [VOTE] The first hbase-2.0.0-beta-2 Release Candidate is available for download

[Stack]

On Tue, Mar 6, 2018 at 12:52 PM, Peter Somogyi  wrote:

> +1 (non-binding)
>
> - Signature, checksum OK
> - Test suite using 1.8.0_161 OK
> - Build and run from source OK
> - Run from bin tarball OK
> - PE 1M rows OK
> - LTT 1M rows OK
> - Basic operations from shell and Java client OK
>
> One thing I noticed: CHANGES.txt isn't updated, latest information is about
> 0.93.0 - Unreleased. The 1.4.2 release contains up-to-date CHANGES.txt.
> Will the CHANGES.txt be updated only for the final 2.0.0 release?
>
> Thanks Peter for trying the RC. Yeah, CHANGES.txt needs to be up-to-date
by RC (I tried to note that this had not been done in the head of this
thread 'Note CHANGES has not yet been updated').

S



>
> On Tue, Mar 6, 2018 at 10:58 AM, Artem Ervits 
> wrote:
>
> > +1 (non-binding)
> >
> > Hadoop Pseudo-distrbibuted: 2.7.5
> > $M2_HOME from scratch
> > Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d;
> > 2017-10-18T07:58:13Z)
> > Maven home: /opt/maven/apache-maven-3.5.2
> > Java version: 1.8.0_161, vendor: Oracle Corporation
> > Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.
> > x86_64/jre
> > Default locale: en_US, platform encoding: UTF-8
> > OS name: "linux", version: "3.10.0-693.11.6.el7.x86_64", arch: "amd64",
> > family: "unix"
> >
> > Binary Release:
> > Java 1M rows: OK
> > LTT 1M rows: OK
> > PE 1M rows: OK
> > MD5: OK
> >
> > hbase shell: OK
> > create, list, scan, count, truncate, disable, drop
> > snapshot, restore_snapshot
> > UI:
> >   split: OK
> >   merge: OK
> >
> > Source Release:
> > Build with: mvn clean -DskipTests -Dhadoop-two.version=2.7.5
> > install && mvn clean -DskipTests -Dhadoop-two.version=2.7.5 package
> > assembly:single   OK
> > MD5: OK
> > installed and ran from src: OK
> >
> > mvn test -P runSmallTests: NOK (this can be my own environment and I've
> > surfaced this in votes for 1.4.1 and 1.4.2 but failing class is the same
> > but failure is different.
> >
> >   [ESC[1;34mINFOESC[m] Results:
> > [ESC[1;34mINFOESC[m]
> > [ESC[1;31mERRORESC[m] ESC[1;31mFailures: ESC[m
> > [ESC[1;31mERRORESC[m] ESC[1;31m
> > TestSpnegoHttpServer.testAllowedClient:243->Assert.
> > assertEquals:631->Assert.assertEquals:645->Assert.
> > failNotEquals:834->Assert.fail:88
> > expected:<200> but was:<401>ESC[m
> > [ESC[1;34mINFOESC[m]
> >
> > On Tue, Mar 6, 2018 at 10:57 AM, Josh Elser  wrote:
> >
> > > +1 (binding)
> > >
> > > * src release OK
> > > * xsums/sigs OK
> > > * Can build and run from src OK
> > > * Loaded some data locally
> > >
> > >
> > > On 3/2/18 6:40 PM, Stack wrote:
> > >
> > >> The first release candidate for HBase 2.0.0-beta-2 is up at
> > >>
> > >>   https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-
> beta-2.RC0/
> > >>
> > >> Maven artifacts are available from a staging directory here:
> > >>
> > >>https://repository.apache.org/content/repositories/
> > orgapachehbase-1199
> > >>
> > >> All was signed with my key at 8ACC93D2 [1]
> > >>
> > >> I tagged the RC as 2.0.0-beta-2RC0.2 at
> > >> 9e9b347d667e1fc6165c9f8ae5ae7052147e8895
> > >>
> > >> hbase-2.0.0-beta-2 is a not-for-production preview of hbase-2.0.0. It
> is
> > >> meant for devs and downstreamers to test drive and flag us if we
> messed
> > up
> > >> on anything ahead of our rolling
> > >> actual 2.0.0 release candidates ("GAs").
> > >>
> > >> hbase-2.0.0-beta-2 is our second beta release. More than 200 fixes
> have
> > >> gone in since
> > >> beta-1. Unit tests generallly pass when run against hadoop2 and
> > >> hadoop3[5].
> > >> It includes
> > >> all that was in previous alphas and beta (new assignment manager,
> > offheap
> > >> read/write
> > >> path, in-memory compactions, etc).The list of features addressed in
> > 2.0.0
> > >> so far can be
> > >> found here [3]. There are thousands. The list of ~3k+ fixes in 2.0.0
> > >> exclusively can be
> > >> found here [4]. Our overview doc. on the state of 2.0.0 is at [6].
> > >>
> > >> This beta was supposed to have as its focus rolling upgrade from
> > 

[NOTICE] branch-2 open for commits again but please get my ok before commit; thanks.

[Stack]

On Tue, Mar 6, 2018 at 10:32 AM, Sean Busbey  wrote:

> Is branch-2 open again now? Sorry if I missed the all-clear but I
> could not find it.
>
>
I hadn't sent one.

Was nervous watching my nightlies:
https://builds.apache.org/job/HBase%20Nightly/job/branch-2/

Let me open it again.

Please ping me before committing on branch-2. Trying to keep the ship
upright.

Thanks for your patience,

St.Ack




> On Thu, Mar 1, 2018 at 11:59 AM, Stack  wrote:
> > Short Version: Please hold on making commits to branch-2. I am in the
> > process of cutting a hbase-2.0.0-beta-2 release candidate. It make take
> me
> > a day or so.
> >
> > Long Version:
> >
> > A miracle happened last night. Our nightly for branch-2 passed [1]. This
> > means all unit tests ran against hadoop2 and then subsequently, all unit
> > tests ran against hadoop3. It had been threatening for a while. A few
> > recent builds passed all unit tests only to fail on a few findbugs
> issues.
> >
> > Given this momentous event and that there are over 200 fixes since
> beta-1,
> > it is time to push out a beta-2.
> >
> > Originally beta-2 was supposed to be about making sure we could do a
> > rolling upgrade from branch-1 to branch-2. This was its theme. The work
> has
> > not been done. Also, a bunch of work is still to be done around testing,
> > compatibility checking, and performance. Perhaps folks are waiting on the
> > 2.0.0 release candidate before they dig in (smile)?
> >
> > I'll give the all clear once I've cut my release. Thank you for your
> > patience.
> > St.Ack
> >
> >
> > 1. See build #415 here
> > https://builds.apache.org/job/HBase%20Nightly/job/branch-2/
>

Re: [NOTICE] branch-2 open for commits again but please get my ok before commit; thanks.

[Stack]

After release, hopefully in morning. Thanks for reminder.
S

On Tue, Mar 6, 2018 at 4:48 PM, 张铎(Duo Zhang)  wrote:

> So is it the time to cut branch-2.0?
>
> 2018-03-07 8:04 GMT+08:00 Stack :
>
> > On Tue, Mar 6, 2018 at 10:32 AM, Sean Busbey  wrote:
> >
> > > Is branch-2 open again now? Sorry if I missed the all-clear but I
> > > could not find it.
> > >
> > >
> > I hadn't sent one.
> >
> > Was nervous watching my nightlies:
> > https://builds.apache.org/job/HBase%20Nightly/job/branch-2/
> >
> > Let me open it again.
> >
> > Please ping me before committing on branch-2. Trying to keep the ship
> > upright.
> >
> > Thanks for your patience,
> >
> > St.Ack
> >
> >
> >
> >
> > > On Thu, Mar 1, 2018 at 11:59 AM, Stack  wrote:
> > > > Short Version: Please hold on making commits to branch-2. I am in the
> > > > process of cutting a hbase-2.0.0-beta-2 release candidate. It make
> take
> > > me
> > > > a day or so.
> > > >
> > > > Long Version:
> > > >
> > > > A miracle happened last night. Our nightly for branch-2 passed [1].
> > This
> > > > means all unit tests ran against hadoop2 and then subsequently, all
> > unit
> > > > tests ran against hadoop3. It had been threatening for a while. A few
> > > > recent builds passed all unit tests only to fail on a few findbugs
> > > issues.
> > > >
> > > > Given this momentous event and that there are over 200 fixes since
> > > beta-1,
> > > > it is time to push out a beta-2.
> > > >
> > > > Originally beta-2 was supposed to be about making sure we could do a
> > > > rolling upgrade from branch-1 to branch-2. This was its theme. The
> work
> > > has
> > > > not been done. Also, a bunch of work is still to be done around
> > testing,
> > > > compatibility checking, and performance. Perhaps folks are waiting on
> > the
> > > > 2.0.0 release candidate before they dig in (smile)?
> > > >
> > > > I'll give the all clear once I've cut my release. Thank you for your
> > > > patience.
> > > > St.Ack
> > > >
> > > >
> > > > 1. See build #415 here
> > > > https://builds.apache.org/job/HBase%20Nightly/job/branch-2/
> > >
> >
>

Re: [VOTE] The first hbase-2.0.0-beta-2 Release Candidate is available for download

[Stack]

Thanks Mike. I'll remove the .md5 when I post the release.

I'm +1.

Ran 1B ITBLL and it passed (10B did not. Work to do).

St.Ack


On Tue, Mar 6, 2018 at 8:50 PM, Mike Drob  wrote:

> +0, I didn't have enough time on this RC to run a more complete suite of
> tests like I was hoping for.
>
> sign/sums ok
> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken). [1]
>
> compile src 8u151 ok
> compile src 8u151 w/ error-prone NOT OK - HBASE-19987
>
> [1]: http://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
> On Fri, Mar 2, 2018 at 5:40 PM, Stack  wrote:
>
> > The first release candidate for HBase 2.0.0-beta-2 is up at
> >
> >  https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-beta-2.RC0/
> >
> > Maven artifacts are available from a staging directory here:
> >
> >   https://repository.apache.org/content/repositories/orgapachehbase-1199
> >
> > All was signed with my key at 8ACC93D2 [1]
> >
> > I tagged the RC as 2.0.0-beta-2RC0.2 at
> > 9e9b347d667e1fc6165c9f8ae5ae7052147e8895
> >
> > hbase-2.0.0-beta-2 is a not-for-production preview of hbase-2.0.0. It is
> > meant for devs and downstreamers to test drive and flag us if we messed
> up
> > on anything ahead of our rolling
> > actual 2.0.0 release candidates ("GAs").
> >
> > hbase-2.0.0-beta-2 is our second beta release. More than 200 fixes have
> > gone in since
> > beta-1. Unit tests generallly pass when run against hadoop2 and
> hadoop3[5].
> > It includes
> > all that was in previous alphas and beta (new assignment manager, offheap
> > read/write
> > path, in-memory compactions, etc).The list of features addressed in 2.0.0
> > so far can be
> > found here [3]. There are thousands. The list of ~3k+ fixes in 2.0.0
> > exclusively can be
> > found here [4]. Our overview doc. on the state of 2.0.0 is at [6].
> >
> > This beta was supposed to have as its focus rolling upgrade from
> hbase-1.x
> > versions but
> > this is work not complete (At this late stage, it is looking like it will
> > be a post-2.0.0 project).
> >
> > This is our last hbase-2.0.0 beta release. Next up, we'll be rolling an
> > actual 2.0.0 release
> > candidate. Look for this in a week or two after beta-2 goes out, after
> > we've done more
> > testing and documentation (and we fix issues raised by you all against
> this
> > beta).
> >
> > One known issue, still unaddressed, is that the User API has not been
> > properly filtered
> > so it shows more than just InterfaceAudience Public content (HBASE-19663,
> > to be fixed
> > by release).
> >
> > Please take this beta for a spin. Please vote on whether it ok to put out
> > this RC as our second
> > beta (Note CHANGES has not yet been updated). Let the VOTE be open for at
> > least 72 hours
> > (Lets say Wednesday morning, March 7th).
> >
> > Thanks,
> > Your 2.0.0 Release Manager
> >
> > 1. http://pgp.mit.edu/pks/lookup?op=get&search=0x9816C7FC8ACC93D2
> > 3. https://goo.gl/scYjJr
> > 4. https://goo.gl/dFFT8b
> > 5. https://builds.apache.org/job/HBase%20Nightly/job/branch-2/
> > <https://goo.gl/dFFT8b>
> > 6. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> > z9iEu_ktczrlKHK8N4SZzs/
> >
>

[RESULT][VOTE] The first hbase-2.0.0-beta-2 Release Candidate is available for download

[Stack]

Vote passes with 4 binding +1s, 3 non-binding, and a +0.

Let me push out this release.

Thanks to all for the robust vote.

S


On Wed, Mar 7, 2018 at 9:12 AM, Stack  wrote:

> Thanks Mike. I'll remove the .md5 when I post the release.
>
> I'm +1.
>
> Ran 1B ITBLL and it passed (10B did not. Work to do).
>
> St.Ack
>
>
> On Tue, Mar 6, 2018 at 8:50 PM, Mike Drob  wrote:
>
>> +0, I didn't have enough time on this RC to run a more complete suite of
>> tests like I was hoping for.
>>
>> sign/sums ok
>> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken). [1]
>>
>> compile src 8u151 ok
>> compile src 8u151 w/ error-prone NOT OK - HBASE-19987
>>
>> [1]: http://www.apache.org/dev/release-distribution#sigs-and-sums
>>
>>
>>
>> On Fri, Mar 2, 2018 at 5:40 PM, Stack  wrote:
>>
>> > The first release candidate for HBase 2.0.0-beta-2 is up at
>> >
>> >  https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-beta-2.RC0/
>> >
>> > Maven artifacts are available from a staging directory here:
>> >
>> >   https://repository.apache.org/content/repositories/orgap
>> achehbase-1199
>> >
>> > All was signed with my key at 8ACC93D2 [1]
>> >
>> > I tagged the RC as 2.0.0-beta-2RC0.2 at
>> > 9e9b347d667e1fc6165c9f8ae5ae7052147e8895
>> >
>> > hbase-2.0.0-beta-2 is a not-for-production preview of hbase-2.0.0. It is
>> > meant for devs and downstreamers to test drive and flag us if we messed
>> up
>> > on anything ahead of our rolling
>> > actual 2.0.0 release candidates ("GAs").
>> >
>> > hbase-2.0.0-beta-2 is our second beta release. More than 200 fixes have
>> > gone in since
>> > beta-1. Unit tests generallly pass when run against hadoop2 and
>> hadoop3[5].
>> > It includes
>> > all that was in previous alphas and beta (new assignment manager,
>> offheap
>> > read/write
>> > path, in-memory compactions, etc).The list of features addressed in
>> 2.0.0
>> > so far can be
>> > found here [3]. There are thousands. The list of ~3k+ fixes in 2.0.0
>> > exclusively can be
>> > found here [4]. Our overview doc. on the state of 2.0.0 is at [6].
>> >
>> > This beta was supposed to have as its focus rolling upgrade from
>> hbase-1.x
>> > versions but
>> > this is work not complete (At this late stage, it is looking like it
>> will
>> > be a post-2.0.0 project).
>> >
>> > This is our last hbase-2.0.0 beta release. Next up, we'll be rolling an
>> > actual 2.0.0 release
>> > candidate. Look for this in a week or two after beta-2 goes out, after
>> > we've done more
>> > testing and documentation (and we fix issues raised by you all against
>> this
>> > beta).
>> >
>> > One known issue, still unaddressed, is that the User API has not been
>> > properly filtered
>> > so it shows more than just InterfaceAudience Public content
>> (HBASE-19663,
>> > to be fixed
>> > by release).
>> >
>> > Please take this beta for a spin. Please vote on whether it ok to put
>> out
>> > this RC as our second
>> > beta (Note CHANGES has not yet been updated). Let the VOTE be open for
>> at
>> > least 72 hours
>> > (Lets say Wednesday morning, March 7th).
>> >
>> > Thanks,
>> > Your 2.0.0 Release Manager
>> >
>> > 1. http://pgp.mit.edu/pks/lookup?op=get&search=0x9816C7FC8ACC93D2
>> > 3. https://goo.gl/scYjJr
>> > 4. https://goo.gl/dFFT8b
>> > 5. https://builds.apache.org/job/HBase%20Nightly/job/branch-2/
>> > <https://goo.gl/dFFT8b>
>> > 6. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
>> > z9iEu_ktczrlKHK8N4SZzs/
>> >
>>
>
>

[ANNOUNCE] Apache HBase 2.0.0-beta-2 is now available for download

[stack]

The HBase team is happy to announce the immediate availability of Apache
​HBase​ 2.0.0-beta-2!

Apache HBase is an open-source, distributed, versioned, non-relational
database. Apache HBase gives you low latency random access to billions of
rows with millions of columns atop non-specialized hardware. To learn more
about HBase, see https://hbase.apache.org/.

Download through an ASF mirror:

https://www.apache.org/dyn/closer.lua/hbase/

(It may take a while to show up on a particular mirror site)

hbase-2.0.0-beta-2 is our second beta release. It includes all that was in
previous alphas
and the previous beta (new assignment manager, offheap read/write path,
in-memory
compactions, etc.).

hbase-2.0.0-beta-2 is a not-for-production preview of hbase-2.0.0. It is
meant for devs and downstreamers to test drive and flag us if we messed up
on anything ahead of our rolling
2.0.0 release candidates ("GAs").

The list of features addressed in 2.0.0 so far can be found here [3]. There
are thousands.
The list of ~2k+ fixes in 2.0.0 exclusively can be found here [4].

I've updated our overview doc. on the state of 2.0.0 [6]. Next up are
actual 2.0.0 release
candidates. Look for them in a week or so after we do up a bit more doc and
test. Here is
the list of what is outstanding against 2.0.0 [5]. Only blockers will hold
up release.

One known issue is that the User API has not been properly filtered so it
shows more than
just InterfaceAudience Public content (HBASE-19663, to be fixed).

For instructions on verifying ASF release downloads, please see

https://www.apache.org/dyn/closer.cgi#verify

Project member signature keys can be found at

https://www.apache.org/dist/hbase/KEYS

This release was tagged with rel/2.0.0-beta-2.

Thanks to all the contributors who made this release possible!

Yours,
The HBase Dev Team

3. https://goo.gl/scYjJr
4. https://goo.gl/dFFT8b
5. *https://issues.apache.org/jira/projects/HBASE/versions/12327188
*
6. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
z9iEu_ktczrlKHK8N4SZzs/

NOTICE: Made branch-2.0 from tip of branch-2; hbase-2.0.0 will be spawned from branch-2.0

[Stack]

Please get my approval committing to branch-2.0. I'm trying to keep the
branch stable as we head toward our first 2.0.0 release candidate.

Related, please do not treat branch-2 as a dumping ground going forward.
I'll try and keep an eye on it too (per an old Andrew Purtell suggestion
that we do a left-shift on version numbers when it comes to RM
oversight[2]). Only bug-fixes and pre-discussed features should go on
branch-2[1] to ship in minor 2.x releases. Master and the our future hbase3
is where the new stuff should be targeted.

Thanks,
St.Ack

1. The only branch-2 'feature' I know of currently is the serial
replication work Duo and crew are currently working on.
2. I can't find his email. Please add to this thread if any of you know
what I'm on about (smile).

DISCUSSION: "branch RM" roles for non-released branches branch-1, branch-2, and master

[Stack]

(Moving discussion to DISCUSSION thread from "NOTICE: made branch-2.0
from..." -- my fault for starting it in wrong place)

A while back, Andrew made a PROPOSAL for 'branch RM's [1]. I like this
suggestion. I see it as a means of avoiding the hell that was 2.0.0 where
its taken near on a year to stabilize (first alpha was last year) because
it has over 5k JIRAs in it; RMs can help keep their branch tidy and free of
flotsam.

Andrew added more to his notion of 'branch RM' over in the NOTICE thread. I
repeat it below:

"​For a while leading up to 1.4.0 I was effectively the branch-1 RM in
practice. Slacked off a bit since, but I'd like to continue with that if
you're agreeable. I think that branch RM role involves informally:
- Keeping an eye on commits to branch. Periodic review of recent commits.
Not acting as a gate on commits and not needing to be pinged or in the loop
for every commit, except perhaps for short periods of time around branching
for new minors.
- Keeping an eye on test stability. Undertaking get well projects like
bisecting and reverting offending commits to resolve test suite decay.
- Periodically kicking off new minor releases. Depends on branch and need
for what's on deck."

Interested in what folks think.
St.Ack

1. 
*https://lists.apache.org/thread.html/247697bcfb97adeeec2d14241856ca7a77a167c9efe0dbc83b0a746f@%3Cdev.hbase.apache.org%3E
*

Re: [DISCUSS] our stable release pointer and EOM

[Stack]

On Thu, Mar 8, 2018 at 6:40 PM, Sean Busbey  wrote:

> Hi folks!
>
> I've been working to get the test suite back to green on branch-1.2;
> we have a lot of branches to track backport for and a non-trivial
> amount of tech debt across all of them from the nightlies being
> offline.
>
> After the stable pointer moves forward from branch-1.2 I'll keep doing
> RM duty for it for ~6 months. Ideally I'd be doing monthly releases,
> but that will largely depend on keeping the build green. After that,
> it'd go EOM.
>
> What do y'all think about moving the stable pointer to branch-1.4 as
> of the 1.4.2 release?
>
> If we move the stable pointer directly to 1.4 and skip over 1.3, what
> do folks think about marking it EOM after we get a 1.3.2 release out?
>


Makes sense to me. Interested in Andrew's take.
S

Re: [VOTE] Merge branch HBASE-19397 back to master and branch-2.

[Stack]

The intent is that this would come out in 2.1.0? This would be soon after a
2.0.0?

S

On Thu, Mar 8, 2018 at 11:41 PM, 张铎(Duo Zhang) 
wrote:

> Since branch-2.0 has been cut and branch-2 is now 2.1.0-SNAPSHOT, will
> merge branch HBASE-19397-branch-2 back to branch-2.
>
> 2018-01-10 9:20 GMT+08:00 张铎(Duo Zhang) :
>
> > If branch-2.0 will be out soon then let's target this to 2.1. No problem.
> >
> > Thanks.
> >
> > 2018-01-10 1:28 GMT+08:00 Stack :
> >
> >> On Mon, Jan 8, 2018 at 8:19 PM, 张铎(Duo Zhang) 
> >> wrote:
> >>
> >> > OK, let me merge it master first. And then create a
> HBASE-19397-branch-2
> >> > which will keep rebasing with the newest branch-2 to see if it is
> stable
> >> > enough. Since we can define this as a bug fix/refactoring rather than
> a
> >> big
> >> > new feature, it is OK to integrate it at any time. If we think it is
> >> stable
> >> > enough before cutting branch-2.0 then we can include it in the 2.0.0
> >> > release, else let's include it in 2.1(Maybe we can backport it to 2.0
> >> > later?).
> >> >
> >> >
> >>
> >> I need to cut the Appy-suggested branch-2.0. Shout if
> HBASE-19397-branch-2
> >> gets to be too much work and I'll do it sooner rather than later. Or, if
> >> easier on you, just say and I'll make the branch-2.0 now so you can just
> >> commit to branch-2 (branch-2.0 will become hbase2.0, branch-2 will
> become
> >> hbase2.1...).
> >>
> >> St.Ack
> >>
> >>
> >>
> >>
> >> > Thanks all here.
> >> >
> >> > 2018-01-09 12:06 GMT+08:00 ashish singhi :
> >> >
> >> > > +1 to merge on master and 2.1.
> >> > > Great work.
> >> > >
> >> > > Thanks,
> >> > > Ashish
> >> > >
> >> > > -Original Message-
> >> > > From: 张铎(Duo Zhang) [mailto:palomino...@gmail.com]
> >> > > Sent: Tuesday, January 09, 2018 6:53 AM
> >> > > To: dev@hbase.apache.org
> >> > > Subject: Re: [VOTE] Merge branch HBASE-19397 back to master and
> >> branch-2.
> >> > >
> >> > > Anyway, if no objections on merging this into master, let's do it?
> So
> >> > that
> >> > > we can start working on the follow-on features, such as table based
> >> > > replication storage, and synchronous replication, etc.
> >> > >
> >> > > Thanks.
> >> > >
> >> > > 2018-01-09 7:19 GMT+08:00 Stack :
> >> > >
> >> > > > On Mon, Jan 8, 2018 at 2:50 PM, 张铎(Duo Zhang) <
> >> palomino...@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > > This 'new' feature only changes DDL part, not the core part of
> >> > > > replication,
> >> > > > > i.e, how to read wal entries and how to replicate it to the
> remote
> >> > > > cluster,
> >> > > > > etc. And also there is no pb message/storage layout change, you
> >> can
> >> > > > > think of this as a big refactoring. Theoretically we even do not
> >> > > > > need to add
> >> > > > new
> >> > > > > UTs for this feature, i.e, no extra stability works. The only
> >> > > > > visible change to users is that it may require more time on
> >> > > > > modifying peers in shell. So in general I think it is less hurt
> to
> >> > > > > include it in the coming release?
> >> > > > >
> >> > > > > And why I think it SHOULD be included in our 2.0 release is
> that,
> >> > > > > the synchronous guarantee is really a good thing for our
> >> replication
> >> > > > > related UTs. The correctness of the current Test***Replication
> >> > > > > usually depends
> >> > > > on a
> >> > > > > flakey condition - we will not do a log rolling between the
> >> > > > > modification
> >> > > > on
> >> > > > > zk and the actual loading of the modification on RS. And we have
> >> > > > > also
> >> > > > done
> >> > > > > a hard work to cleanup the lockings in ReplicationSourceManager
> >> and
> >

Re: [NOTICE] enable error-prone checks on precommit

[Stack]

Very nice Mr. Drob.
S

On Fri, Mar 9, 2018 at 2:07 PM, Mike Drob  wrote:

> Hi devs,
>
> Over on HBASE-20153[1] I put up a patch to include error-prone[2] compiler
> checks on precommit. Wanted to send out a note to the dev list to make sure
> people saw it and understood what was happening, and so there would be less
> surprise if things start failing.
>
> Over the past several months, a few developers have been toiling at getting
> these additional bugs out of our code. Let's respect their efforts and help
> make sure bugs don't creep back in by adding them to precommit.
>
> Important things to be aware of:
> * We do more checking and analysis, so it slows compilation down ~5 minutes
> for the whole project.
> * The error messages usually have links to a wiki page explaining what is
> wrong, so they're not that scary.
> * You can run your build with -PerrorProne to get the same locally
>
> If there are concerns, please let me know before this is pushed!
>
> Mike
>
> [1]: https://issues.apache.org/jira/browse/HBASE-20153
> [2]: http://errorprone.info/
>

Re: [VOTE] Merge branch HBASE-19397 back to master and branch-2.

[Stack]

o re-think what is allowable for a Y release
> > in
> > > x.y.z...
> > >
> > > +1 for master (which already happened, maybe?)
> > > +0 for branch-2 (simply because I haven't looked closely enough at
> > > changes, can read through and try to change to +1 if you need the
> votes)
> > >
> > >
> > > On 3/9/18 2:41 AM, 张铎(Duo Zhang) wrote:
> > >
> > >> Since branch-2.0 has been cut and branch-2 is now 2.1.0-SNAPSHOT, will
> > >> merge branch HBASE-19397-branch-2 back to branch-2.
> > >>
> > >> 2018-01-10 9:20 GMT+08:00 张铎(Duo Zhang) :
> > >>
> > >> If branch-2.0 will be out soon then let's target this to 2.1. No
> > problem.
> > >>>
> > >>> Thanks.
> > >>>
> > >>> 2018-01-10 1:28 GMT+08:00 Stack :
> > >>>
> > >>> On Mon, Jan 8, 2018 at 8:19 PM, 张铎(Duo Zhang)  >
> > >>>> wrote:
> > >>>>
> > >>>> OK, let me merge it master first. And then create a
> > HBASE-19397-branch-2
> > >>>>> which will keep rebasing with the newest branch-2 to see if it is
> > >>>>> stable
> > >>>>> enough. Since we can define this as a bug fix/refactoring rather
> > than a
> > >>>>>
> > >>>> big
> > >>>>
> > >>>>> new feature, it is OK to integrate it at any time. If we think it
> is
> > >>>>>
> > >>>> stable
> > >>>>
> > >>>>> enough before cutting branch-2.0 then we can include it in the
> 2.0.0
> > >>>>> release, else let's include it in 2.1(Maybe we can backport it to
> 2.0
> > >>>>> later?).
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>> I need to cut the Appy-suggested branch-2.0. Shout if
> > >>>> HBASE-19397-branch-2
> > >>>> gets to be too much work and I'll do it sooner rather than later.
> Or,
> > if
> > >>>> easier on you, just say and I'll make the branch-2.0 now so you can
> > just
> > >>>> commit to branch-2 (branch-2.0 will become hbase2.0, branch-2 will
> > >>>> become
> > >>>> hbase2.1...).
> > >>>>
> > >>>> St.Ack
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> Thanks all here.
> > >>>>>
> > >>>>> 2018-01-09 12:06 GMT+08:00 ashish singhi  >:
> > >>>>>
> > >>>>> +1 to merge on master and 2.1.
> > >>>>>> Great work.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> Ashish
> > >>>>>>
> > >>>>>> -Original Message-
> > >>>>>> From: 张铎(Duo Zhang) [mailto:palomino...@gmail.com]
> > >>>>>> Sent: Tuesday, January 09, 2018 6:53 AM
> > >>>>>> To: dev@hbase.apache.org
> > >>>>>> Subject: Re: [VOTE] Merge branch HBASE-19397 back to master and
> > >>>>>>
> > >>>>> branch-2.
> > >>>>
> > >>>>>
> > >>>>>> Anyway, if no objections on merging this into master, let's do it?
> > So
> > >>>>>>
> > >>>>> that
> > >>>>>
> > >>>>>> we can start working on the follow-on features, such as table
> based
> > >>>>>> replication storage, and synchronous replication, etc.
> > >>>>>>
> > >>>>>> Thanks.
> > >>>>>>
> > >>>>>> 2018-01-09 7:19 GMT+08:00 Stack :
> > >>>>>>
> > >>>>>> On Mon, Jan 8, 2018 at 2:50 PM, 张铎(Duo Zhang) <
> > >>>>>>>
> > >>>>>> palomino...@gmail.com>
> > >>>>
> > >>>>> wrote:
> > >>>>>>>
> > >>>>>>> This 'new' feature only changes DDL part, not the core part of
> > >>>>>>>>
> > >>>>>>> replication,
> > >>>>>>>
> > >>>>>>>> i.e, how to read wal entr

Re: [VOTE] Merge branch HBASE-19397 back to master and branch-2.

[Stack]

On Tue, Mar 13, 2018 at 6:30 PM, 张铎(Duo Zhang) 
wrote:

> I would be interested to be the RM for 2.1 :)
>
>
I could help.
S



> Stack 于2018年3月14日 周三02:48写道：
>
> > I took a look. Its great. Our replication is in bad need of loving. The
> > patch does a refactor/revamp/fix of an internal facility putting our peer
> > management up on a Pv2 basis. It also moves forward the long-rumored
> > distributed procedure project.
> >
> > Its a big change though. Would have been great to have it make 2.0.0 but
> it
> > didn't. I'd be up for it in a minor release rather than wait on 3.0.0
> given
> > we allow ourselves some leeway adding facility on minors and that it is
> at
> > core a solidifying fix. Needs to be doc'd and tested, verified on a
> deploy
> > beyond unit test. I could help test. Is it proven compatible with
> existing
> > replication deploys?
> >
> > Who's the 3.0.0 RM? When is it going to roll? Having this in place is
> best
> > argument if folks propose back-ports. Without, we are doomed to repeat
> the
> > 2.0.0 experience.
> >
> > Thanks,
> > St.Ack
> >
> >
> >
> > On Tue, Mar 13, 2018 at 3:44 AM, 张铎(Duo Zhang) 
> > wrote:
> >
> > > I've already merged it to branch-2...
> > >
> > > And for the procedure based replication peer modification, the feature
> > has
> > > been finished, at least no critical TODOs. I will not say it has no
> bugs
> > > but I do not think it will block the 2.1 or 3.0 release too much.
> > >
> > > And please trust my judgement, I'm not a man who only want to show off.
> > For
> > > example I just reverted the serial replication feature from branch-2
> > before
> > > we release beta2 and tried to redo it on master because we found some
> > > critical problems. And since the basic idea has not been changed so we
> > > decided to do it on master because it will not spend too much time to
> > > finish. And for HBASE-19064 it is a big feature so we decided to do it
> > on a
> > > feature branch. We will select the branches which we want to merge back
> > at
> > > the time we finish the feature.
> > >
> > > For the CP cleanup, I think the problem is we started too late, and in
> > the
> > > past we made mistakes and let the related projects inject too much into
> > the
> > > internal of HBase. And for B&R, it is something like the serial
> > replication
> > > feature. For serial replication feature is that we tested it and found
> > some
> > > critical problems, and for B&R it was not well tested(IIRC) so we were
> > > afraid of there will be critical problems. And for the AMv2, I think
> the
> > > problem is premature optimization. We even implemented our own AvlTree,
> > and
> > > also a very complicated scheduler which always makes us dead lock...
> > >
> > > These are all very good cases for software engineering in the real
> > world...
> > > Should be avoided in the future... That's also my duty...
> > >
> > > Thanks.
> > >
> > > 2018-03-13 17:50 GMT+08:00 Apekshit Sharma :
> > >
> > > > I thought the problem with releasing 2.0 was that there were too many
> > > open
> > > > features dragging along not coming to finish- AMv2  (biggest one), CP
> > > > cleanup, B&R, etc.
> > > > If it was not the case, it wouldn't have been 1 year between first
> > alpha
> > > > and GA, rather just few months.
> > > >
> > > > That said, i don't mind new but hardened features which are already
> > ready
> > > > to ship (implementation only or not) going in minor version, but
> that's
> > > my
> > > > personal opinion. But going too aggressive on that can indeed lead to
> > the
> > > > trap Josh mentioned above.
> > > >
> > > > For this particular change, my +1 was based on following aspects:
> > > > -  it's internal
> > > > - moving ops procv2 framework (gives failure recovery, locking, etc)
> > > > - Duo's reasoning - "the synchronous guarantee is really a good
> > thing
> > > > for our replication related UTs..." and "For the replication peer
> > > tracking,
> > > > it is the same problem. It is hard to do fencing with zk watcher
> since
> > it
> > > > is asynchronous, so the UTs are always kind of flakey in

Re: [VOTE] Merge branch HBASE-19397 back to master and branch-2.

[Stack]

On Tue, Mar 13, 2018 at 8:44 PM, Stack  wrote:

> On Tue, Mar 13, 2018 at 6:30 PM, 张铎(Duo Zhang) 
> wrote:
>
>> I would be interested to be the RM for 2.1 :)
>>
>>
> I could help.
>

Or, I meant, good... We need more RMs. I volunteer to help smooth the
experience.

S

S
>
>
>
>> Stack 于2018年3月14日 周三02:48写道：
>>
>> > I took a look. Its great. Our replication is in bad need of loving. The
>> > patch does a refactor/revamp/fix of an internal facility putting our
>> peer
>> > management up on a Pv2 basis. It also moves forward the long-rumored
>> > distributed procedure project.
>> >
>> > Its a big change though. Would have been great to have it make 2.0.0
>> but it
>> > didn't. I'd be up for it in a minor release rather than wait on 3.0.0
>> given
>> > we allow ourselves some leeway adding facility on minors and that it is
>> at
>> > core a solidifying fix. Needs to be doc'd and tested, verified on a
>> deploy
>> > beyond unit test. I could help test. Is it proven compatible with
>> existing
>> > replication deploys?
>> >
>> > Who's the 3.0.0 RM? When is it going to roll? Having this in place is
>> best
>> > argument if folks propose back-ports. Without, we are doomed to repeat
>> the
>> > 2.0.0 experience.
>> >
>> > Thanks,
>> > St.Ack
>> >
>> >
>> >
>> > On Tue, Mar 13, 2018 at 3:44 AM, 张铎(Duo Zhang) 
>> > wrote:
>> >
>> > > I've already merged it to branch-2...
>> > >
>> > > And for the procedure based replication peer modification, the feature
>> > has
>> > > been finished, at least no critical TODOs. I will not say it has no
>> bugs
>> > > but I do not think it will block the 2.1 or 3.0 release too much.
>> > >
>> > > And please trust my judgement, I'm not a man who only want to show
>> off.
>> > For
>> > > example I just reverted the serial replication feature from branch-2
>> > before
>> > > we release beta2 and tried to redo it on master because we found some
>> > > critical problems. And since the basic idea has not been changed so we
>> > > decided to do it on master because it will not spend too much time to
>> > > finish. And for HBASE-19064 it is a big feature so we decided to do it
>> > on a
>> > > feature branch. We will select the branches which we want to merge
>> back
>> > at
>> > > the time we finish the feature.
>> > >
>> > > For the CP cleanup, I think the problem is we started too late, and in
>> > the
>> > > past we made mistakes and let the related projects inject too much
>> into
>> > the
>> > > internal of HBase. And for B&R, it is something like the serial
>> > replication
>> > > feature. For serial replication feature is that we tested it and found
>> > some
>> > > critical problems, and for B&R it was not well tested(IIRC) so we were
>> > > afraid of there will be critical problems. And for the AMv2, I think
>> the
>> > > problem is premature optimization. We even implemented our own
>> AvlTree,
>> > and
>> > > also a very complicated scheduler which always makes us dead lock...
>> > >
>> > > These are all very good cases for software engineering in the real
>> > world...
>> > > Should be avoided in the future... That's also my duty...
>> > >
>> > > Thanks.
>> > >
>> > > 2018-03-13 17:50 GMT+08:00 Apekshit Sharma :
>> > >
>> > > > I thought the problem with releasing 2.0 was that there were too
>> many
>> > > open
>> > > > features dragging along not coming to finish- AMv2  (biggest one),
>> CP
>> > > > cleanup, B&R, etc.
>> > > > If it was not the case, it wouldn't have been 1 year between first
>> > alpha
>> > > > and GA, rather just few months.
>> > > >
>> > > > That said, i don't mind new but hardened features which are already
>> > ready
>> > > > to ship (implementation only or not) going in minor version, but
>> that's
>> > > my
>> > > > personal opinion. But going too aggressive on that can indeed lead
>> to
>> > the
>> > > > trap Josh mentioned above.
>> > > >
>

Re: [DISCUSS] A Problem When Start HBase Cluster Using Table Based Replication

[Stack]

On Wed, Mar 14, 2018 at 11:55 PM, OpenInx  wrote:

> Hi :
>
> (Paste from https://issues.apache.org/jira/browse/HBASE-20166?
> focusedCommentId=16399886&page=com.atlassian.jira.
> plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16399886)
>
> There's a really big problem here if we use table based replication to
> start a hbase cluster:
>
> For HMaster process, it works as following:
> 1. Start active master initialization .
> 2. Master wait rs report in .
> 3. Master assign meta region to one of the region servers .
> 4. Master create hbase:replication table if not exist.
>
>
We have to have a new system table? Can't we add a column family on
hbase:meta that keeps offsets? We've done the work to make sure hbase:meta
is up before everything else. It has its own WALs so we can split these
ahead of user-space WALs, and so on. We've not done the work to for
hbase:replication or hbase:namespace, hbase:acl... etc.

Means more loading on hbase:meta and it is going to get bigger but I'd
rather work on splitting meta than on figuring how to preassign
miscellaneous system tables; one-per-feature.



> But the RS need to finish initialize the replication source & sink before
> finish startup( and the initialization of replication source & sink must
> finish before opening region, because we need to listen the wal event,
> otherwise our replication may lost data), and when initialize the source &
> sink , we need to read hbase:replication table which hasn't been avaiable
> because our master is waiting rs to be OK, and the rs is waiting
> hbase:replication to be OK ... a dead loop happen again ...
>
> After discussed with Guanghao Zhang offline, I'm considering that try to
> assign all system table to a rs which only accept regions of system table
> assignment (The rs will skip to initialize the replication source or sink
> )...
>
>
Can we avoid this sort of special-casing?

St.Ack



> I've tried to start a mini cluster by setting
> hbase.balancer.tablesOnMaster.systemTablesOnly=true
> & hbase.balancer.tablesOnMaster=true , it seems not work. because
> currently
> we initialize the master logic firstly, then region logic for the HMaster
> process, and it should be ...
>
>
> Any  suggestion  ?
>

Re: [DISCUSS] Compilation on $ARCH which protoc doesn't support

[Stack]

On Thu, Mar 15, 2018 at 12:03 PM, Josh Elser  wrote:

> I'm faced with what I think is a tricky issue. I'm trying to help folks to
> compile HBase on PPC. The protobuf community does not publish a PPC binary
> for protoc which means that the HBase build can't generate our protobuf
> files. Protobuf supports: x86/x86_64 for linux, windows, and osx in 3.3.0
> (an aarch_64 starting in 3.5.0).
>
> I'm trying to think through any/all solutions:
>
> 1. Get protobuf folks to build/publish protoc for PPC.
> 2. Commit our generated java files to the repository and provide an option
> to skip compilation on unsupported ARCHs.
> 3. Try to find/create some non-official PPC support ala
> https://github.com/os72/protoc-jar
> 4. Do nothing
>
> Evaluating these, #1 and #3 are a "later" solution. #2 puts some pain on
> developers to git-add these files if they are modified. Maybe not a lot of
> pain since protocol changes should be infrequent at this point?
>
>
We used to do this but after some work the nice protobuf-maven-plugin that
provides platform-appropriate protoc made it so we could skip having to do
this halving (IIRC) the size of our checkout.

Have a PPC profile that messes w/ the module list? (Claim on the profiles
doc page is that its possible[1]). Exclude hbase-protocol* modules if os
finds that the platform is PPC? (Activating a profile based off os is
common usage). Have the PPC profile depend on published versions of these
jars? Would something like this do Josh?

St.Ack
1. http://maven.apache.org/guides/introduction/introduction-to-profiles.html







> There seems to be some stalled work for #1 via
> https://github.com/google/protobuf/pull/3506
>
> What do people think? I'm probably between a rock and a hard place to get
> this work one way or another in the immediate. If there's support for this
> in the community, I'd prefer to work towards that.
>
> - Josh
>

Re: [DISCUSS] A Problem When Start HBase Cluster Using Table Based Replication

[Stack]

On Thu, Mar 15, 2018 at 5:39 PM, 张铎(Duo Zhang) 
wrote:

>
> But for other stuffs for replication, such as the WAL files we need to
> replicate, the row key will be a peer id, and multiplies the server name,
> and maybe also the file name? This is a completely different thing. If we
> put this in meta, the row key will be messed up...
>
>
Is there some artifice that would allow us shape this info so it fit the
meta schema: e.g. a table that we do not assign (special namespace? special
tablename? hbase:replication?). Then perhaps the regions in meta have peer
as the start row etc.

St.Ack



> 2018-03-16 3:01 GMT+08:00 Stack :
>
> > On Wed, Mar 14, 2018 at 11:55 PM, OpenInx  wrote:
> >
> > > Hi :
> > >
> > > (Paste from https://issues.apache.org/jira/browse/HBASE-20166?
> > > focusedCommentId=16399886&page=com.atlassian.jira.
> > > plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16399886)
> > >
> > > There's a really big problem here if we use table based replication to
> > > start a hbase cluster:
> > >
> > > For HMaster process, it works as following:
> > > 1. Start active master initialization .
> > > 2. Master wait rs report in .
> > > 3. Master assign meta region to one of the region servers .
> > > 4. Master create hbase:replication table if not exist.
> > >
> > >
> > We have to have a new system table? Can't we add a column family on
> > hbase:meta that keeps offsets? We've done the work to make sure
> hbase:meta
> > is up before everything else. It has its own WALs so we can split these
> > ahead of user-space WALs, and so on. We've not done the work to for
> > hbase:replication or hbase:namespace, hbase:acl... etc.
> >
> > Means more loading on hbase:meta and it is going to get bigger but I'd
> > rather work on splitting meta than on figuring how to preassign
> > miscellaneous system tables; one-per-feature.
> >
> >
> >
> > > But the RS need to finish initialize the replication source & sink
> before
> > > finish startup( and the initialization of replication source & sink
> must
> > > finish before opening region, because we need to listen the wal event,
> > > otherwise our replication may lost data), and when initialize the
> source
> > &
> > > sink , we need to read hbase:replication table which hasn't been
> avaiable
> > > because our master is waiting rs to be OK, and the rs is waiting
> > > hbase:replication to be OK ... a dead loop happen again ...
> > >
> > > After discussed with Guanghao Zhang offline, I'm considering that try
> to
> > > assign all system table to a rs which only accept regions of system
> table
> > > assignment (The rs will skip to initialize the replication source or
> sink
> > > )...
> > >
> > >
> > Can we avoid this sort of special-casing?
> >
> > St.Ack
> >
> >
> >
> > > I've tried to start a mini cluster by setting
> > > hbase.balancer.tablesOnMaster.systemTablesOnly=true
> > > & hbase.balancer.tablesOnMaster=true , it seems not work. because
> > > currently
> > > we initialize the master logic firstly, then region logic for the
> HMaster
> > > process, and it should be ...
> > >
> > >
> > > Any  suggestion  ?
> > >
> >
>

Re: [DISCUSS] A Problem When Start HBase Cluster Using Table Based Replication

[Stack]

On Thu, Mar 15, 2018 at 8:42 PM, Guanghao Zhang  wrote:

> >
> > We've done the work to make sure hbase:meta
> > is up before everything else. It has its own WALs so we can split these
> > ahead of user-space WALs, and so on. We've not done the work to for
> > hbase:replication or hbase:namespace, hbase:acl... etc.
>
> If we import a new SYSTEM-TABLE-ONLY state for region server startup, then
> it is necessary to own WALs for all system tables (not only hbase:meta).
>

WALs dedicated to system tables would be a new facility. Would be good to
have. We'd have a WAL per system table or they'd share a WAL? The meta-only
WAL was hacked in. Would probably take more work to get system-dedicated
WALs in the mix.



> All system tables have their own wal and split these ahead of user-space
> WALs. And the WAL of system tables no need replication. So we can start a
> region server without replication. After all system table online, region
> server can continue start from SYSTEM-TABLE-ONLY to STARTED.
>
>
A refactor of Master startup is definitely needed. Would like to get
considerations other than just assign order considered but I suppose that
can wait. Your suggested stepped assign sounds fine. What about shutdown
and when Master joins an existing cluster or a host goes down that had
system tables on it? How would stepping work then? Will there be a
hierarchy of assign amongst system tables? Or will it just be meta first,
then general system tables, and then user-space tables? We need to split
meta. How will that impinge on these plans?

My suggestion of overloading hbase:meta so it can carry the metadata for
replication is less pure but keeps our assign simple.

A Master startup refactor + WALs-per-system-table sounds like a lot of
change for a minor release.

Thanks Guanghao,
S



> Thanks.
>
> 2018-03-16 10:12 GMT+08:00 OpenInx :
>
> > The HBASE-15867 will not be introduced into 2.0.0,   I expect to
> introduce
> > in 2.1.0 release .
> >
> > Thanks.
> >
> > On Fri, Mar 16, 2018 at 12:45 AM, Mike Drob  wrote:
> >
> > > I'm also +1 for splitting RS startup into multiple steps.
> > >
> > > Looking at the linked JIRA and the parent issue it was not immediately
> > > apparent if this is an issue for 2.0 or not - can somebody clarify?
> > >
> > > On Thu, Mar 15, 2018 at 5:14 AM, 张铎(Duo Zhang) 
> > > wrote:
> > >
> > > > I'm +1 on the second solution.
> > > >
> > > > 2018-03-15 16:59 GMT+08:00 Guanghao Zhang :
> > > >
> > > > > From a more general perspective, this may be a general problem as
> we
> > > may
> > > > > move more and more data from zookeeper to system table. Or we may
> > have
> > > > more
> > > > > features to create new system table. But if the RS relays some
> system
> > > > table
> > > > > to start up, we will meet a dead lock...
> > > > >
> > > > > One solution is let master to serve system table only. So the
> cluster
> > > > > startup will have two step. First startup master to serve system
> > table.
> > > > > Then start region servers. But the problem is master will have
> > > > > more responsibility and may be a bottleneck.
> > > > >
> > > > > Another solution is break RS startup progress to two steps. First
> > step
> > > is
> > > > > "serve system table only". Second step is "totally startup and
> serve
> > > any
> > > > > tables". It means we will import a new state for RS startup. A RS's
> > > > startup
> > > > > progress will be STOPPED ==> SYSTEM-TABLE-ONLY ==> STARTED. But
> this
> > > may
> > > > > need more refactor for our RS code.
> > > > >
> > > > > Thanks.
> > > > >
> > > > > 2018-03-15 15:57 GMT+08:00 张铎(Duo Zhang) :
> > > > >
> > > > > > Oh, it should be 'The replication peer related data is small'.
> > > > > >
> > > > > > 2018-03-15 15:56 GMT+08:00 张铎(Duo Zhang)  >:
> > > > > >
> > > > > > > I think this is a bit awkward... A region server even does not
> > need
> > > > the
> > > > > > > meta table to be online when starting, but it needs another
> > system
> > > > > table
> > > > > > > when starting...
> > > > > > >
> > > > > > > I think unless we can make the regionserver start without
> > > > replication,
> > > > > > and
> > > > > > > initialize it later, otherwise we can not break the tie.
> Having a
> > > > > special
> > > > > > > 'region server' seems a bad smell to me. What's the advantage
> > > > comparing
> > > > > > to
> > > > > > > zk?
> > > > > > >
> > > > > > > BTW, I believe that we only need the ReplicationPeerStorage to
> be
> > > > > > > available when starting a region server, so we can keep this
> data
> > > in
> > > > > zk,
> > > > > > > and storage the queue related data to hbase:replication table?
> > The
> > > > > > > replication related data is small so I think this is OK.
> > > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > > 2018-03-15 14:55 GMT+08:00 OpenInx :
> > > > > > >
> > > > > > >> Hi :
> > > > > > >>
> > > > > > >> (Paste from https://issues.apache.org/jira/browse/HBASE-20166
> ?
> > > > > > >> focusedCommentId=16399886&page=com.atlass

Re: Looking at an hbase-thirdparty release

[Stack]

Probably too risky to do at this stage in the game but protobuf? We're on
3.3.1. Latest is 3.5.1 [1]: "Various performance optimizations." And Guava
is 24.1 now. We're on 22.

St.Ack
1. https://github.com/google/protobuf/releases


On Fri, Mar 16, 2018 at 1:47 PM, Josh Elser  wrote:

> Give a shout if there's anything that you all think would be good to
> include in a new hbase-thirdparty release for HBase 2.0.
>
> I'm pulling in commons-cli as per https://issues.apache.org/jira
> /browse/HBASE-20201.
>
> If there's more that should come in, please let me know!
>

Re: Looking at an hbase-thirdparty release

[Stack]

On Fri, Mar 16, 2018 at 3:48 PM, Josh Elser  wrote:

> No qualms from me. 3.5.1 the desired version?
>
>
Yeah. Hopefully just a pom version bump.



> The reason that brought me here was nothing that we do in our build
> (running MR jobs via `hadoop jar`), so I don't think we need to rush.
>
>
On the no rush, this release has to happen before I can put up an RC0. Was
hoping to do that in the next week or two. Just saying...

S



> On 3/16/18 5:46 PM, Andrew Purtell wrote:
>
>> +1 on protobuf. Any chance we can get a vote period long enough to do a
>> comparison perf / GC test? I can help.
>>
>> On Fri, Mar 16, 2018 at 2:41 PM, Stack  wrote:
>>
>> Probably too risky to do at this stage in the game but protobuf? We're on
>>> 3.3.1. Latest is 3.5.1 [1]: "Various performance optimizations." And
>>> Guava
>>> is 24.1 now. We're on 22.
>>>
>>> St.Ack
>>> 1. https://github.com/google/protobuf/releases
>>>
>>>
>>> On Fri, Mar 16, 2018 at 1:47 PM, Josh Elser  wrote:
>>>
>>> Give a shout if there's anything that you all think would be good to
>>>> include in a new hbase-thirdparty release for HBase 2.0.
>>>>
>>>> I'm pulling in commons-cli as per https://issues.apache.org/jira
>>>> /browse/HBASE-20201.
>>>>
>>>> If there's more that should come in, please let me know!
>>>>
>>>>
>>>
>>
>>
>>

Re: [VOTE] HBase Thirdparty 2.1.0 RC0

[Stack]

+1

Verified signature and hash.

I see it updates pb but not guava? No harm. Next time.

Built and installed jars locally from src tgz.

Applied the HBASE-20223 patch to branch-2 and built... all seems good.

Thanks for doing this,
S



On Mon, Mar 19, 2018 at 4:15 PM, Josh Elser  wrote:

> Hi,
>
> Please consider the following as the 2.1.0 release of Apache HBase
> Thirdparty.
>
> Source artifact, signatures, and checksums are available at
> https://dist.apache.org/repos/dist/dev/hbase/hbase-thirdpart
> y/hbase-thirdparty-2.1.0RC0/
>
> Git commit for the release candidate available at
> https://git-wip-us.apache.org/repos/asf?p=hbase-thirdparty.g
> it;a=commit;h=de125620dece453f49a56de5b328427dc534133f
>
> Maven repository available at https://repository.apache.org/
> content/repositories/orgapachehbase-1206
>
> This vote will remain open for at least 72 hours (~2018/03/22 2300 GMT).
> HBASE-20223 has a patch for branch-2.0 which supports this thirdparty
> update (obligatory: have not run a full test suite locally with this
> change  yet).
>
> Here's my +1
>
> - Josh
>

Re: Almost ready for HBaseCon+PhoenixCon 2018 SanJose CFP

[Stack]

On Tue, Mar 20, 2018 at 7:51 PM, Josh Elser  wrote:

> Hi all,
>
> I've published a new website for the upcoming event in June in California
> at [1][2] for the HBase and Phoenix websites, respectively. 1 & 2 are
> identical.
>
> I've not yet updated any links on either website to link to the new page.
> I'd appreciate if folks can give their feedback on anything outwardly
> wrong, incorrect, etc. If folks are happy, then I'll work on linking from
> the main websites, and coordinating an official announcement via mail
> lists, social media, etc.
>
> The website is generated from [3]. If you really want to be my
> best-friend, let me know about the above things which are wrong via
> pull-request ;)
>
> - Josh
>
> [1] https://hbase.apache.org/hbasecon-phoenixcon-2018/
> [2] https://phoenix.apache.org/hbasecon-phoenixcon-2018/
> [3] https://github.com/joshelser/hbasecon-jekyll
>


Thanks Josh for doing this.

Do they have to be conflated so? Each community is doing their own
conference. This page/announcement makes it look like they have been
squashed together.

Thanks,
S

Re: Almost ready for HBaseCon+PhoenixCon 2018 SanJose CFP

[Stack]

On Wed, Mar 21, 2018 at 9:26 AM, Josh Elser  wrote:

>
>
> On 3/21/18 2:59 AM, Stack wrote:
>
>> On Tue, Mar 20, 2018 at 7:51 PM, Josh Elser  wrote:
>>
>> Hi all,
>>>
>>> I've published a new website for the upcoming event in June in California
>>> at [1][2] for the HBase and Phoenix websites, respectively. 1 & 2 are
>>> identical.
>>>
>>> I've not yet updated any links on either website to link to the new page.
>>> I'd appreciate if folks can give their feedback on anything outwardly
>>> wrong, incorrect, etc. If folks are happy, then I'll work on linking from
>>> the main websites, and coordinating an official announcement via mail
>>> lists, social media, etc.
>>>
>>> The website is generated from [3]. If you really want to be my
>>> best-friend, let me know about the above things which are wrong via
>>> pull-request ;)
>>>
>>> - Josh
>>>
>>> [1] https://hbase.apache.org/hbasecon-phoenixcon-2018/
>>> [2] https://phoenix.apache.org/hbasecon-phoenixcon-2018/
>>> [3] https://github.com/joshelser/hbasecon-jekyll
>>>
>>>
>>
>> Thanks Josh for doing this.
>>
>> Do they have to be conflated so? Each community is doing their own
>> conference. This page/announcement makes it look like they have been
>> squashed together.
>>
>> Thanks,
>> S
>>
>
> You have any concrete suggestions I can change?



You have hbasecon+phoenixcon which to me reads as a combined conference. I
thought we wanted to avoid this sort of messaging. Probably best to have
separate announcement pages.


> Was trying to write this to respect the concerns you gave early on. My
> intent was to use a theme: "two events, one physical location".
>
> Using the same CFP and website content is mostly in hopes of saving myself
> time (as I wanted to get this done this past weekend and failed..)
>


Understood.

If a CFP is wanted, can use the easychair system for CFPs.

Thanks,
S

Sorry for JIRA spam... forgot to click 'Do Not Send Mail' during a few bulk edits.....

[Stack]

[HEADSUP] First 2.0.0 Release Candidate start of next week!

[Stack]

Shout if there is anything you need in it. Meantime help w/ docs, perf,
test, metrics, etc., appreciated.

Thanks.
Your RM,
S

P.S. Current list of 2.0.0 targeted items:
https://issues.apache.org/jira/projects/HBASE/versions/12327188#release-report-tab-body

Re: [HEADSUP] First 2.0.0 Release Candidate start of next week!

[Stack]

On Thu, Mar 22, 2018 at 5:53 AM, 张铎(Duo Zhang) 
wrote:

> I'd like to fix HBASE-19952. Before beta2 it is a bit difficult to do it
> because the 'mvn test' always fails...
>
>
Sounds good Duo. Thanks.
S



> 2018-03-22 13:44 GMT+08:00 Stack :
>
> > Shout if there is anything you need in it. Meantime help w/ docs, perf,
> > test, metrics, etc., appreciated.
> >
> > Thanks.
> > Your RM,
> > S
> >
> > P.S. Current list of 2.0.0 targeted items:
> > https://issues.apache.org/jira/projects/HBASE/versions/
> > 12327188#release-report-tab-body
> >
>

Re: [DISCUSS] purge the 'component owner' experiment?

[Stack]

Purge.

My assessment is that the OWNER project failed; valiants signed-up with the
best of intentions but because of tooling, consistency, attrition, unknowns
(pick one), the project faded soon after launch.

Thanks,
St.Ack

On Wed, Mar 28, 2018 at 7:42 AM, Sean Busbey  wrote:

> In late 2012 we tried an experiment where folks volunteered to act as
> stewards of particular functional areas. The basic idea was we'd
> require scrutiny from more folks if we didn't have anyone available
> who had self selected as familiar available at the time of review.
>
> Described like this in the ref guide currently
>
> 
> h3. Patch +1 Policy
>
> The below policy is something we put in place 09/2012. It is a
> suggested policy rather than a hard requirement. We want to try it
> first to see if it works before we cast it in stone.
>
> Apache HBase is made of components. Components have one or more
> OWNERs. See the 'Description' field on the components JIRA page for
> who the current owners are by component.
>
> Patches that fit within the scope of a single Apache HBase component
> require, at least, a +1 by one of the component’s owners before
> commit. If owners are absent — busy or otherwise — two +1s by
> non-owners will suffice.
>
> Patches that span components need at least two +1s before they can be
> committed, preferably +1s by owners of components touched by the
> x-component patch (TODO: This needs tightening up but I think fine for
> first pass).
>
> Any -1 on a patch by anyone vetoes a patch; it cannot be committed
> until the justification for the -1 is addressed.
>
> h3. Component Owner/Lieutenant
>
> Component owners are listed in the description field on this Apache
> HBase JIRA components page. The owners are listed in the 'Description'
> field rather than in the 'Component Lead' field because the latter
> only allows us list one individual whereas it is encouraged that
> components have multiple owners.
>
> Owners or component lieutenants are volunteers who are (usually, but
> not necessarily) expert in their component domain and may have an
> agenda on how they think their Apache HBase component should evolve.
>
> Owners will try and review patches that land within their component’s
> scope.
>
> If applicable, if an owner has an agenda, they will publish their
> goals or the design toward which they are driving their component
>
> If you would like to be volunteer as a component owner, just write the
> dev list and we’ll sign you up. Owners do not need to be committers.
>
> 
>
> This was still around enough when I joined the project in ~2014 one of
> the things I did was sign up for a few. I would be surprised if anyone
> with < 1 year in the project (a) knows about this approach or (b) what
> areas (as defined by this effort) specifically to reach out to me
> about. I certainly haven't updated the list for which ones I feel most
> comfortable talking about now that I've been here a bit.
>
> Should we drop this effort? Seems counter to several of our community
> trends; it adds one more thing to maintain, discourages sense of
> shared ownership, requires even more reviewer bandwidth that we don't
> have.
>
> Or should we revitalize it? Would provide a better idea of our
> "reviewer coverage" and give easier pointers for e.g. folks looking
> for mentorship on learning about particular parts of our HBase.
>

Re: Can we support seek for a reverse scan?

[Stack]

On Wed, Mar 28, 2018 at 11:31 PM, Toshihiro Suzuki 
wrote:

> Hi folks,
>
> We are facing “IllegalStateException: requestSeek cannot be called on
> ReversedKeyValueHeap" when using a reverse scan and the
> loadColumnFamiliesOnDemand optimization is enabled. As ReversedKeyValueHeap
> doesn't support requestSeek, it seems like the IllegalStateException
> occurs. I have already filed this issue to HBASE-20219, please see it for
> the details.
>
> Yeah, that seems right given how reverse is 'implemented' (or, to be more
correct, not implemented).



> I tried to add a server-side check to forbid such scans, but as James
> mentioned in the Jira, this would have negative implications on performance.
>
>  Which 'performance' I wonder?



> I think the best solution is to make ReversedKeyValueHeap support seek if
> we can. What do you guys think about the feasibility of this?
>


It'd be a big job given plumbing for going in reverse is just absent
(Reverse Scan is just a fake on top of forward Scanning; it is not even
complete just reversing on rows, not on columns in a row...). You'd add
reverse on indices in blocks and the notion of 'previous' to Scanners and
Streams. It'd be slow given buffering and seeks 'naturally' are aimed in
the forward direction.

Thanks,

S



> Thanks,
> Toshihiro Suzuki
>
>

Re: Looking for AsyncWAL docs

[Stack]

On Mon, Apr 2, 2018 at 1:27 PM, Mike Drob  wrote:

> Duo, can you include content from
> https://issues.apache.org/jira/browse/HBASE-16689 when you are writing up
> the docs as well? Not sure if that is talking about same or different Async
> Wal option, actually.
>
>
I'll do the above.

Just realized this morning that this will be a font of confusion going
forward.

You are the second one today to conflate ASYNC_WAL (i.e. not holding up
client writes until their the sync of their write to the WAL has succeeded)
and asycnfswal, our new asynchronous dfs writer used writing WALs.

Let me try and distinguish the two better in the refguide.

St.Ack




> Mike
>
> On Wed, Mar 28, 2018 at 12:58 PM, 张铎(Duo Zhang) 
> wrote:
>
> > Got it. Let me read the doc for HFile v3.
> >
> > Will go out in the coming week so.. Hope it will not be too late...
> >
> > 2018-03-28 20:28 GMT+08:00 Sean Busbey :
> >
> > > HFile v3 is probably a good example.
> > >
> > > An appendix entry that has details that would be relevant if debugging
> or
> > > troublehsooting.
> > >
> > > And a note for the upgrade section about it being in place by default
> and
> > > how an operator would avoid that if they need to for some reason (like
> an
> > > esoteric Hadoop FS or just conservative ops position).
> > >
> > > I've seen conflicting statements already about the impact on
> durability,
> > > for example, and we should be proactive in setting expectations since
> > folks
> > > will be concerned as soon as they hear we have our own DFS client.
> > >
> > >
> > > On Wed, Mar 28, 2018, 01:49 张铎(Duo Zhang) 
> wrote:
> > >
> > > > Any examples which we could follow? This is not a user visible
> feature,
> > > so
> > > > not sure what is the best way to mention it in the ref guide.
> > > >
> > > > 2018-03-27 23:47 GMT+08:00 Sean Busbey :
> > > >
> > > > > Could y'all get some of this into the reference guide? Talks and
> > > > > release notes are great, but I really want us to make sure
> operators
> > > > > have a nice place to figure out all the stuff we're landing in 2.0.
> > > > >
> > > > > On Tue, Mar 27, 2018 at 10:13 AM, Yu Li  wrote:
> > > > > > @Mike
> > > > > > FWIW, besides checking the JIRAs and codes, the talk Duo gave in
> > our
> > > > > > HBaseCon 2016 may help you better understand the whole picture,
> > > please
> > > > > > check page 14 to 20 of this presentation
> > > > > >  > > > > improvements-and-practices-at-xiaomi>
> > > > > > on
> > > > > > slideshare.
> > > > > >
> > > > > > Best Regards,
> > > > > > Yu
> > > > > >
> > > > > > On 27 March 2018 at 14:26, 张铎(Duo Zhang) 
> > > > wrote:
> > > > > >
> > > > > >> 2018-03-27 12:35 GMT+08:00 Mike Drob :
> > > > > >>
> > > > > >> > Hi folks,
> > > > > >> >
> > > > > >> > I've been working on some of the docs relating to the upcoming
> > 2.0
> > > > > >> release
> > > > > >> > and have struggled to find content around AsyncWAL. My
> > impression
> > > is
> > > > > that
> > > > > >> > this is a pretty important new feature, yet there's nothing in
> > the
> > > > ref
> > > > > >> > guide about it.
> > > > > >> >
> > > > > >> > Does it have a different name that I'm not familiar with?
> > > > > >> >
> > > > > >> > If it's not in the ref guide, should I file a JIRA issue for
> > > > somebody
> > > > > to
> > > > > >> > generate that content? Specific things that I'd be looking for
> > > are:
> > > > > >> > - How to enable/disable
> > > > > >> >
> > > > > >> See HBASE-15536, just like the old way, config
> hbase.wal.provider
> > > > > >>
> > > > > >> > - How does this impact data durability, MTTR, failover
> > scenarios,
> > > > etc.
> > > > > >> >
> > > > > >> Does not impact these things.
> > > > > >>
> > > > > >> > - How does this impact replication
> > > > > >> >
> > > > > >> Ditto.
> > > > > >>
> > > > > >> > - Which configuration knobs exist and when would I want to
> tune
> > > them
> > > > > >> >
> > > > > >> Usually you do not need to tune anything...
> > > > > >> Before committing HBASE-15536 we have done a lot of performance
> > > > > testings.
> > > > > >> There are two configs which may effect performance, one
> > > > > >> is hbase.wal.batch.size, and the other
> > > > > >> is hbase.wal.async.use-shared-event-loop. But it is hard to say
> > > how to
> > > > > >> tune
> > > > > >> them...
> > > > > >> And another thing is that, with AsyncFSWAL we can set a lower
> > > timeout
> > > > > when
> > > > > >> writing WAL, but now it just shares the common dfs
> configuration.
> > > > Maybe
> > > > > we
> > > > > >> should file an issue for it.
> > > > > >>
> > > > > >> >
> > > > > >> > As a last resort, I can try to dig through RNs in existing
> > issues,
> > > > but
> > > > > >> > that's been pretty hit or miss (mostly miss) for me so far
> too.
> > > > > >> >
> > > > > >> > I think at least we need to mention the reason why we
> introduce
> > > > > >> AsyncFSWAL
> > > > > >> and make it default for 2.0 in our refguide.
> > > >

Re: Looking for AsyncWAL docs

[Stack]

HBASE-20329 is first cut.
S

On Mon, Apr 2, 2018 at 2:11 PM, Stack  wrote:

> On Mon, Apr 2, 2018 at 1:27 PM, Mike Drob  wrote:
>
>> Duo, can you include content from
>> https://issues.apache.org/jira/browse/HBASE-16689 when you are writing up
>> the docs as well? Not sure if that is talking about same or different
>> Async
>> Wal option, actually.
>>
>>
> I'll do the above.
>
> Just realized this morning that this will be a font of confusion going
> forward.
>
> You are the second one today to conflate ASYNC_WAL (i.e. not holding up
> client writes until their the sync of their write to the WAL has succeeded)
> and asycnfswal, our new asynchronous dfs writer used writing WALs.
>
> Let me try and distinguish the two better in the refguide.
>
> St.Ack
>
>
>
>
>> Mike
>>
>> On Wed, Mar 28, 2018 at 12:58 PM, 张铎(Duo Zhang) 
>> wrote:
>>
>> > Got it. Let me read the doc for HFile v3.
>> >
>> > Will go out in the coming week so.. Hope it will not be too late...
>> >
>> > 2018-03-28 20:28 GMT+08:00 Sean Busbey :
>> >
>> > > HFile v3 is probably a good example.
>> > >
>> > > An appendix entry that has details that would be relevant if
>> debugging or
>> > > troublehsooting.
>> > >
>> > > And a note for the upgrade section about it being in place by default
>> and
>> > > how an operator would avoid that if they need to for some reason
>> (like an
>> > > esoteric Hadoop FS or just conservative ops position).
>> > >
>> > > I've seen conflicting statements already about the impact on
>> durability,
>> > > for example, and we should be proactive in setting expectations since
>> > folks
>> > > will be concerned as soon as they hear we have our own DFS client.
>> > >
>> > >
>> > > On Wed, Mar 28, 2018, 01:49 张铎(Duo Zhang) 
>> wrote:
>> > >
>> > > > Any examples which we could follow? This is not a user visible
>> feature,
>> > > so
>> > > > not sure what is the best way to mention it in the ref guide.
>> > > >
>> > > > 2018-03-27 23:47 GMT+08:00 Sean Busbey :
>> > > >
>> > > > > Could y'all get some of this into the reference guide? Talks and
>> > > > > release notes are great, but I really want us to make sure
>> operators
>> > > > > have a nice place to figure out all the stuff we're landing in
>> 2.0.
>> > > > >
>> > > > > On Tue, Mar 27, 2018 at 10:13 AM, Yu Li  wrote:
>> > > > > > @Mike
>> > > > > > FWIW, besides checking the JIRAs and codes, the talk Duo gave in
>> > our
>> > > > > > HBaseCon 2016 may help you better understand the whole picture,
>> > > please
>> > > > > > check page 14 to 20 of this presentation
>> > > > > > <https://www.slideshare.net/HBaseCon/apache-hbase-
>> > > > > improvements-and-practices-at-xiaomi>
>> > > > > > on
>> > > > > > slideshare.
>> > > > > >
>> > > > > > Best Regards,
>> > > > > > Yu
>> > > > > >
>> > > > > > On 27 March 2018 at 14:26, 张铎(Duo Zhang) > >
>> > > > wrote:
>> > > > > >
>> > > > > >> 2018-03-27 12:35 GMT+08:00 Mike Drob :
>> > > > > >>
>> > > > > >> > Hi folks,
>> > > > > >> >
>> > > > > >> > I've been working on some of the docs relating to the
>> upcoming
>> > 2.0
>> > > > > >> release
>> > > > > >> > and have struggled to find content around AsyncWAL. My
>> > impression
>> > > is
>> > > > > that
>> > > > > >> > this is a pretty important new feature, yet there's nothing
>> in
>> > the
>> > > > ref
>> > > > > >> > guide about it.
>> > > > > >> >
>> > > > > >> > Does it have a different name that I'm not familiar with?
>> > > > > >> >
>> > > > > >> > If it's not in the ref guide, should I file a JIRA issue for
>> > > > somebody
>> > > > > to

[VOTE] First release candidate for HBase 2.0.0 (RC0) is available

[Stack]

The first release candidate for Apache HBase 2.0.0 is available for
downloading and testing.

Artifacts are available here:

 https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0RC0/

Maven artifacts are available in the staging repository at:

 https://repository.apache.org/content/repositories/orgapachehbase-1209

All artifacts are signed with my signing key 8ACC93D2, which is also
in the project KEYS file at

 http://www.apache.org/dist/hbase/KEYS

These artifacts were tagged 2.0.0RC0 at
hash 011dd2dae33456b3a2bcc2513e9fdd29de23be46

Please review 'Upgrading from 1.x to 2.x' in the bundled HBase 2.0.0
Reference Guide before installing or upgrading for a list of
incompatibilities, major changes, and notable new features. Be aware that
according to our adopted Semantic Versioning guidelines[1], we've allow
ourselves to make breaking changes in this major version release. For
example, Coprocessors will need to be recast to fit more constrained CP
APIs and a rolling upgrade of an hbase-1.x install to hbase-2.x without
downtime is (currently) not possible. That said, a bunch of effort has been
expended mitigating differences; a hbase-1.x client can perform DML against
an hbase-2 cluster.

For the full list of ~6k issues addressed, see [2]. There are also
CHANGES.md and RELEASENOTES.md in the root directory of the source tarball.

Please take a few minutes to verify the release and vote on releasing it:

[ ] +1 Release this package as Apache HBase 2.0.0
[ ] +0 no opinion
[ ] -1 Do not release this package because...

This VOTE will run for one week and close Tuesday, April 17, 2018 @ 13:00
PST.

Thanks to the myriad who have helped out with this release,
Your 2.0.0 Release Manager

1. http://hbase.apache.org/2.0/book.html#hbase.versioning.post10
2.  https://s.apache.org/zwS9

Re: [VOTE] First release candidate for HBase 2.0.0 (RC0) is available

[Stack]

On Tue, Apr 10, 2018 at 2:50 PM, Sean Busbey  wrote:

> no compat report in the RC directory. does that mean we won't have one
> in the dist area?
>
>
> not a blocker; we've been inconsistent on it in prior releases, but
> the trend seemed to be towards including it.
>
>

HBASE-18622 has current state of compatibility compare. Let me do a new run
and add it to the RC dir.
St.Ack



>
> On Tue, Apr 10, 2018 at 3:47 PM, Stack  wrote:
> > The first release candidate for Apache HBase 2.0.0 is available for
> > downloading and testing.
> >
> > Artifacts are available here:
> >
> >  https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0RC0/
> >
> > Maven artifacts are available in the staging repository at:
> >
> >  https://repository.apache.org/content/repositories/orgapachehbase-1209
> >
> > All artifacts are signed with my signing key 8ACC93D2, which is also
> > in the project KEYS file at
> >
> >  http://www.apache.org/dist/hbase/KEYS
> >
> > These artifacts were tagged 2.0.0RC0 at
> > hash 011dd2dae33456b3a2bcc2513e9fdd29de23be46
> >
> > Please review 'Upgrading from 1.x to 2.x' in the bundled HBase 2.0.0
> > Reference Guide before installing or upgrading for a list of
> > incompatibilities, major changes, and notable new features. Be aware that
> > according to our adopted Semantic Versioning guidelines[1], we've allow
> > ourselves to make breaking changes in this major version release. For
> > example, Coprocessors will need to be recast to fit more constrained CP
> > APIs and a rolling upgrade of an hbase-1.x install to hbase-2.x without
> > downtime is (currently) not possible. That said, a bunch of effort has
> been
> > expended mitigating differences; a hbase-1.x client can perform DML
> against
> > an hbase-2 cluster.
> >
> > For the full list of ~6k issues addressed, see [2]. There are also
> > CHANGES.md and RELEASENOTES.md in the root directory of the source
> tarball.
> >
> > Please take a few minutes to verify the release and vote on releasing it:
> >
> > [ ] +1 Release this package as Apache HBase 2.0.0
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because...
> >
> > This VOTE will run for one week and close Tuesday, April 17, 2018 @ 13:00
> > PST.
> >
> > Thanks to the myriad who have helped out with this release,
> > Your 2.0.0 Release Manager
> >
> > 1. http://hbase.apache.org/2.0/book.html#hbase.versioning.post10
> > 2.  https://s.apache.org/zwS9
>

Re: [VOTE] First release candidate for HBase 2.0.0 (RC0) is available

[Stack]

On Tue, Apr 10, 2018 at 8:58 PM, Yu Li  wrote:

> HBASE-20188 is comparing latest 2.0 to our stable pointer 1.2.7, and there
> seems to be no conclusion yet, JFYI. While yes I believe more numbers could
> give us a broader view :-)
>
>
Yeah, what Yu Li says above. As is, hbase2 is much better than hbase1 under
mixed loading but does worse when pure read or write workloads.



> And I'm not sure but is performance regression some kind of a blocker for
> the release? Thanks.
>
>
I'd imagine that if the difference were large, then yes, it should be a
blocker -- or we as a community can decide to work on it in a follow-on
release making perf a priority (say, 2.1.0).

Thanks,
S




> Best Regards,
> Yu
>
> On 11 April 2018 at 08:39, 张铎(Duo Zhang)  wrote:
>
> > Do we have a ITBLL result and also some performance numbers? I could help
> > getting some performance numbers comparing to the version we use...
> >
> > 2018-04-11 5:50 GMT+08:00 Sean Busbey :
> >
> > > no compat report in the RC directory. does that mean we won't have one
> > > in the dist area?
> > >
> > >
> > > not a blocker; we've been inconsistent on it in prior releases, but
> > > the trend seemed to be towards including it.
> > >
> > >
> > > On Tue, Apr 10, 2018 at 3:47 PM, Stack  wrote:
> > > > The first release candidate for Apache HBase 2.0.0 is available for
> > > > downloading and testing.
> > > >
> > > > Artifacts are available here:
> > > >
> > > >  https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0RC0/
> > > >
> > > > Maven artifacts are available in the staging repository at:
> > > >
> > > >  https://repository.apache.org/content/repositories/
> > orgapachehbase-1209
> > > >
> > > > All artifacts are signed with my signing key 8ACC93D2, which is also
> > > > in the project KEYS file at
> > > >
> > > >  http://www.apache.org/dist/hbase/KEYS
> > > >
> > > > These artifacts were tagged 2.0.0RC0 at
> > > > hash 011dd2dae33456b3a2bcc2513e9fdd29de23be46
> > > >
> > > > Please review 'Upgrading from 1.x to 2.x' in the bundled HBase 2.0.0
> > > > Reference Guide before installing or upgrading for a list of
> > > > incompatibilities, major changes, and notable new features. Be aware
> > that
> > > > according to our adopted Semantic Versioning guidelines[1], we've
> allow
> > > > ourselves to make breaking changes in this major version release. For
> > > > example, Coprocessors will need to be recast to fit more constrained
> CP
> > > > APIs and a rolling upgrade of an hbase-1.x install to hbase-2.x
> without
> > > > downtime is (currently) not possible. That said, a bunch of effort
> has
> > > been
> > > > expended mitigating differences; a hbase-1.x client can perform DML
> > > against
> > > > an hbase-2 cluster.
> > > >
> > > > For the full list of ~6k issues addressed, see [2]. There are also
> > > > CHANGES.md and RELEASENOTES.md in the root directory of the source
> > > tarball.
> > > >
> > > > Please take a few minutes to verify the release and vote on releasing
> > it:
> > > >
> > > > [ ] +1 Release this package as Apache HBase 2.0.0
> > > > [ ] +0 no opinion
> > > > [ ] -1 Do not release this package because...
> > > >
> > > > This VOTE will run for one week and close Tuesday, April 17, 2018 @
> > 13:00
> > > > PST.
> > > >
> > > > Thanks to the myriad who have helped out with this release,
> > > > Your 2.0.0 Release Manager
> > > >
> > > > 1. http://hbase.apache.org/2.0/book.html#hbase.versioning.post10
> > > > 2.  https://s.apache.org/zwS9
> > >
> >
>

Re: [VOTE] First release candidate for HBase 2.0.0 (RC0) is available

[Stack]

On Thu, Apr 12, 2018 at 1:47 PM, Stack  wrote:

> On Tue, Apr 10, 2018 at 2:50 PM, Sean Busbey  wrote:
>
>> no compat report in the RC directory. does that mean we won't have one
>> in the dist area?
>>
>>
>> not a blocker; we've been inconsistent on it in prior releases, but
>> the trend seemed to be towards including it.
>>
>>
>
> HBASE-18622 has current state of compatibility compare. Let me do a new
> run and add it to the RC dir.
>

I just added
https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0RC0/compat-check-v1.2.6-v2.0.0.report.html

Thanks,
S



> St.Ack
>
>
>
>>
>> On Tue, Apr 10, 2018 at 3:47 PM, Stack  wrote:
>> > The first release candidate for Apache HBase 2.0.0 is available for
>> > downloading and testing.
>> >
>> > Artifacts are available here:
>> >
>> >  https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0RC0/
>> >
>> > Maven artifacts are available in the staging repository at:
>> >
>> >  https://repository.apache.org/content/repositories/orgapachehbase-1209
>> >
>> > All artifacts are signed with my signing key 8ACC93D2, which is also
>> > in the project KEYS file at
>> >
>> >  http://www.apache.org/dist/hbase/KEYS
>> >
>> > These artifacts were tagged 2.0.0RC0 at
>> > hash 011dd2dae33456b3a2bcc2513e9fdd29de23be46
>> >
>> > Please review 'Upgrading from 1.x to 2.x' in the bundled HBase 2.0.0
>> > Reference Guide before installing or upgrading for a list of
>> > incompatibilities, major changes, and notable new features. Be aware
>> that
>> > according to our adopted Semantic Versioning guidelines[1], we've allow
>> > ourselves to make breaking changes in this major version release. For
>> > example, Coprocessors will need to be recast to fit more constrained CP
>> > APIs and a rolling upgrade of an hbase-1.x install to hbase-2.x without
>> > downtime is (currently) not possible. That said, a bunch of effort has
>> been
>> > expended mitigating differences; a hbase-1.x client can perform DML
>> against
>> > an hbase-2 cluster.
>> >
>> > For the full list of ~6k issues addressed, see [2]. There are also
>> > CHANGES.md and RELEASENOTES.md in the root directory of the source
>> tarball.
>> >
>> > Please take a few minutes to verify the release and vote on releasing
>> it:
>> >
>> > [ ] +1 Release this package as Apache HBase 2.0.0
>> > [ ] +0 no opinion
>> > [ ] -1 Do not release this package because...
>> >
>> > This VOTE will run for one week and close Tuesday, April 17, 2018 @
>> 13:00
>> > PST.
>> >
>> > Thanks to the myriad who have helped out with this release,
>> > Your 2.0.0 Release Manager
>> >
>> > 1. http://hbase.apache.org/2.0/book.html#hbase.versioning.post10
>> > 2.  https://s.apache.org/zwS9
>>
>
>

Re: [ANNOUNCE] Please welcome Francis Liu to the HBase PMC

[Stack]

Hot dog!

On Wed, Apr 11, 2018 at 1:03 PM, Andrew Purtell  wrote:

> On behalf of the Apache HBase PMC I am pleased to announce that Francis
> Liu has accepted our invitation to become a PMC member on the Apache
> HBase project. We appreciate Francis stepping up to take more
> responsibility in the HBase project. He has been an active contributor to
> HBase for many years and recently took over responsibilities as branch RM
> for branch-1.3.
>
> Please join me in welcoming Francis to the HBase PMC!
>
> --
> Best regards,
> Andrew
>

Re: [VOTE] First release candidate for HBase 2.0.0 (RC0) is available

[Stack]

On Fri, Apr 13, 2018 at 5:40 AM, Jean-Marc Spaggiari <
jean-m...@spaggiari.org> wrote:

> Do we have a list of tests that we know will not pass this release?
>
> I got those failures so far, but since I want to run multiple runs, I want
> to make sure to exclude the un-stable tests.
>
> TestAssignmentManagerMetrics.testRITAssignmentManagerMetrics:152 Metrics
> Should be equal expected:<1> but was:<0>
> TestReplicationSmallTests.testDisableEnable:198 Replication wasn't
> disabled
> TestReplicationSmallTests.testSimplePutDelete:168->TestReplicationBase.
> runSimplePutDeleteTest:266
> Waited too much time for put replication
> TestClientOperationTimeout.setUp:99 » SocketTimeout callTimeout=500,
> callDurat...
>
>
Sorry about that JMS. We just had a nice set of stable runs but a little
bit of rot seems to have set in in the last day or so. Probably needs a
little attention to shut down the new set of flakies.

Thanks,
St.Ack



> Thanks,
>
> JMS
>
>
> 2018-04-13 0:19 GMT-04:00 Yu Li :
>
> > bq. I'd imagine that if the difference were large, then yes, it should
> > be a blocker
> > -- or we as a community can decide to work on it in a follow-on release
> > making perf a priority (say, 2.1.0).
> > I see, thanks for the clarification boss, makes sense.
> >
> > Best Regards,
> > Yu
> >
> > On 13 April 2018 at 06:02, Stack  wrote:
> >
> > > On Thu, Apr 12, 2018 at 1:47 PM, Stack  wrote:
> > >
> > > > On Tue, Apr 10, 2018 at 2:50 PM, Sean Busbey 
> > wrote:
> > > >
> > > >> no compat report in the RC directory. does that mean we won't have
> one
> > > >> in the dist area?
> > > >>
> > > >>
> > > >> not a blocker; we've been inconsistent on it in prior releases, but
> > > >> the trend seemed to be towards including it.
> > > >>
> > > >>
> > > >
> > > > HBASE-18622 has current state of compatibility compare. Let me do a
> new
> > > > run and add it to the RC dir.
> > > >
> > >
> > > I just added
> > > https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0RC0/
> > > compat-check-v1.2.6-v2.0.0.report.html
> > >
> > > Thanks,
> > > S
> > >
> > >
> > >
> > > > St.Ack
> > > >
> > > >
> > > >
> > > >>
> > > >> On Tue, Apr 10, 2018 at 3:47 PM, Stack  wrote:
> > > >> > The first release candidate for Apache HBase 2.0.0 is available
> for
> > > >> > downloading and testing.
> > > >> >
> > > >> > Artifacts are available here:
> > > >> >
> > > >> >  https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0RC0/
> > > >> >
> > > >> > Maven artifacts are available in the staging repository at:
> > > >> >
> > > >> >  https://repository.apache.org/content/repositories/
> > > orgapachehbase-1209
> > > >> >
> > > >> > All artifacts are signed with my signing key 8ACC93D2, which is
> also
> > > >> > in the project KEYS file at
> > > >> >
> > > >> >  http://www.apache.org/dist/hbase/KEYS
> > > >> >
> > > >> > These artifacts were tagged 2.0.0RC0 at
> > > >> > hash 011dd2dae33456b3a2bcc2513e9fdd29de23be46
> > > >> >
> > > >> > Please review 'Upgrading from 1.x to 2.x' in the bundled HBase
> 2.0.0
> > > >> > Reference Guide before installing or upgrading for a list of
> > > >> > incompatibilities, major changes, and notable new features. Be
> aware
> > > >> that
> > > >> > according to our adopted Semantic Versioning guidelines[1], we've
> > > allow
> > > >> > ourselves to make breaking changes in this major version release.
> > For
> > > >> > example, Coprocessors will need to be recast to fit more
> constrained
> > > CP
> > > >> > APIs and a rolling upgrade of an hbase-1.x install to hbase-2.x
> > > without
> > > >> > downtime is (currently) not possible. That said, a bunch of effort
> > has
> > > >> been
> > > >> > expended mitigating differences; a hbase-1.x client can perform
> DML
> > > >> against
> > > >> > an hbase-2 cluster.
> > > >> >
> > > >> > For the full list of ~6k issues addressed, see [2]. There are also
> > > >> > CHANGES.md and RELEASENOTES.md in the root directory of the source
> > > >> tarball.
> > > >> >
> > > >> > Please take a few minutes to verify the release and vote on
> > releasing
> > > >> it:
> > > >> >
> > > >> > [ ] +1 Release this package as Apache HBase 2.0.0
> > > >> > [ ] +0 no opinion
> > > >> > [ ] -1 Do not release this package because...
> > > >> >
> > > >> > This VOTE will run for one week and close Tuesday, April 17, 2018
> @
> > > >> 13:00
> > > >> > PST.
> > > >> >
> > > >> > Thanks to the myriad who have helped out with this release,
> > > >> > Your 2.0.0 Release Manager
> > > >> >
> > > >> > 1. http://hbase.apache.org/2.0/book.html#hbase.versioning.post10
> > > >> > 2.  https://s.apache.org/zwS9
> > > >>
> > > >
> > > >
> > >
> >
>

Re: [VOTE] First release candidate for HBase 2.0.0 (RC0) is available

[Stack]

On Fri, Apr 13, 2018 at 8:07 AM, Peter Somogyi  wrote:

> -1
>
> Tests under hbase-rsgroup fail in setup phase. It is related to
> HBASE-20224, some addendums were not backported to branch-2.0 which causes
> the failure. I'm reopening HBASE-20224 and backport 2 missing commits.
>
>
Thanks Peter. Yeah, go for the backports. That issue was a bit of a mess.

I'm trying to see why we didn't catch this in the nightlies.

Lets not shut down the RC yet. I'd like to see if there are other issues
out there before cutting new RC.

Good stuff,
St.Ack





> On Fri, Apr 13, 2018 at 3:54 PM, Jean-Marc Spaggiari <
> jean-m...@spaggiari.org> wrote:
>
> > Exactly what I was looking for! Thanks Sean!
> >
> > 2018-04-13 9:37 GMT-04:00 Sean Busbey :
> >
> > > Hi JMS!
> > >
> > > Current flaky results for branch-2.0, generated today AFAICT:
> > >
> > > https://builds.apache.org/view/H-L/view/HBase/job/HBase-
> > > Find-Flaky-Tests-branch2.0/lastSuccessfulBuild/artifact/
> > >
> > > you can look at the "dashboard.html" artifact for a human usable idea
> > > of what we think is broken.
> > >
> > > you can fetch the excludes file to get a pattern you can pass to Maven.
> > >
> > >
> > > On Fri, Apr 13, 2018 at 7:40 AM, Jean-Marc Spaggiari
> > >  wrote:
> > > > Do we have a list of tests that we know will not pass this release?
> > > >
> > > > I got those failures so far, but since I want to run multiple runs, I
> > > want
> > > > to make sure to exclude the un-stable tests.
> > > >
> > > > TestAssignmentManagerMetrics.testRITAssignmentManagerMetrics:152
> > Metrics
> > > > Should be equal expected:<1> but was:<0>
> > > > TestReplicationSmallTests.testDisableEnable:198 Replication wasn't
> > > disabled
> > > > TestReplicationSmallTests.testSimplePutDelete:168->
> > TestReplicationBase.
> > > runSimplePutDeleteTest:266
> > > > Waited too much time for put replication
> > > > TestClientOperationTimeout.setUp:99 » SocketTimeout callTimeout=500,
> > > > callDurat...
> > > >
> > > > Thanks,
> > > >
> > > > JMS
> > > >
> > > >
> > > > 2018-04-13 0:19 GMT-04:00 Yu Li :
> > > >
> > > >> bq. I'd imagine that if the difference were large, then yes, it
> should
> > > >> be a blocker
> > > >> -- or we as a community can decide to work on it in a follow-on
> > release
> > > >> making perf a priority (say, 2.1.0).
> > > >> I see, thanks for the clarification boss, makes sense.
> > > >>
> > > >> Best Regards,
> > > >> Yu
> > > >>
> > > >> On 13 April 2018 at 06:02, Stack  wrote:
> > > >>
> > > >> > On Thu, Apr 12, 2018 at 1:47 PM, Stack  wrote:
> > > >> >
> > > >> > > On Tue, Apr 10, 2018 at 2:50 PM, Sean Busbey  >
> > > >> wrote:
> > > >> > >
> > > >> > >> no compat report in the RC directory. does that mean we won't
> > have
> > > one
> > > >> > >> in the dist area?
> > > >> > >>
> > > >> > >>
> > > >> > >> not a blocker; we've been inconsistent on it in prior releases,
> > but
> > > >> > >> the trend seemed to be towards including it.
> > > >> > >>
> > > >> > >>
> > > >> > >
> > > >> > > HBASE-18622 has current state of compatibility compare. Let me
> do
> > a
> > > new
> > > >> > > run and add it to the RC dir.
> > > >> > >
> > > >> >
> > > >> > I just added
> > > >> > https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0RC0/
> > > >> > compat-check-v1.2.6-v2.0.0.report.html
> > > >> >
> > > >> > Thanks,
> > > >> > S
> > > >> >
> > > >> >
> > > >> >
> > > >> > > St.Ack
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >>
> > > >> > >> On Tue, Apr 10, 2018 at 3:47 PM, Stack 
> wrote:
> > > >>

Re: [VOTE] First release candidate for HBase 2.0.0 (RC0) is available

[Stack]

On Fri, Apr 13, 2018 at 10:53 AM, Josh Elser  wrote:

> Was poking around with PE on a few nodes (I forget the exact
> circumstances, need to look back at this), and ran into a case where ~35
> regions were left as RIT
>
> 2018-04-12 22:05:24,431 ERROR 
> [master/ctr-e138-1518143905142-221855-01-02:16000]
> procedure2.ProcedureExecutor: Corrupt pid=3580, ppid=3534,
> state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable,
> region=71fef  ffe6b5b3cf1cb6d3328a5a58690
>
> Saw entries like this (I think) for each region which was stuck. A simple
> `assign` in the shell brought them back, but I need to dig in some more to
> understand what went wrong.
>
>
Log?

HBASE-18152?

Thanks Josh,
S




>
> On 4/10/18 4:47 PM, Stack wrote:
>
>> The first release candidate for Apache HBase 2.0.0 is available for
>> downloading and testing.
>>
>> Artifacts are available here:
>>
>>   https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0RC0/
>>
>> Maven artifacts are available in the staging repository at:
>>
>>   https://repository.apache.org/content/repositories/orgapachehbase-1209
>>
>> All artifacts are signed with my signing key 8ACC93D2, which is also
>> in the project KEYS file at
>>
>>   http://www.apache.org/dist/hbase/KEYS
>>
>> These artifacts were tagged 2.0.0RC0 at
>> hash 011dd2dae33456b3a2bcc2513e9fdd29de23be46
>>
>> Please review 'Upgrading from 1.x to 2.x' in the bundled HBase 2.0.0
>> Reference Guide before installing or upgrading for a list of
>> incompatibilities, major changes, and notable new features. Be aware that
>> according to our adopted Semantic Versioning guidelines[1], we've allow
>> ourselves to make breaking changes in this major version release. For
>> example, Coprocessors will need to be recast to fit more constrained CP
>> APIs and a rolling upgrade of an hbase-1.x install to hbase-2.x without
>> downtime is (currently) not possible. That said, a bunch of effort has
>> been
>> expended mitigating differences; a hbase-1.x client can perform DML
>> against
>> an hbase-2 cluster.
>>
>> For the full list of ~6k issues addressed, see [2]. There are also
>> CHANGES.md and RELEASENOTES.md in the root directory of the source
>> tarball.
>>
>> Please take a few minutes to verify the release and vote on releasing it:
>>
>> [ ] +1 Release this package as Apache HBase 2.0.0
>> [ ] +0 no opinion
>> [ ] -1 Do not release this package because...
>>
>> This VOTE will run for one week and close Tuesday, April 17, 2018 @ 13:00
>> PST.
>>
>> Thanks to the myriad who have helped out with this release,
>> Your 2.0.0 Release Manager
>>
>> 1. http://hbase.apache.org/2.0/book.html#hbase.versioning.post10
>> 2.  https://s.apache.org/zwS9
>>
>>

Re: [VOTE] First release candidate for HBase 2.0.0 (RC0) is available

[Stack]

Yeah, what Ashish says Umesh (and yeah, checkout HBASE-20385 for the why
sir).

Any one else given the RC a try? Its seven days now. Time is about up. I
have two -1s, one of which I think I can overturn. Any other feedback on
the RC? Any PMCers tried it?

Thanks,

St.Ack


On Mon, Apr 16, 2018 at 9:41 PM, ashish singhi 
wrote:

> bq. signatures & sums- NOT
> OK
> (md5 checksums missing)
>
> This is intentional I think, check HBASE-20385.
>
> Regards,
> Ashish
>
> -Original Message-
> From: Umesh Agashe [mailto:uaga...@cloudera.com]
> Sent: Tuesday, April 17, 2018 4:01 AM
> To: dev@hbase.apache.org
> Subject: Re: [VOTE] First release candidate for HBase 2.0.0 (RC0) is
> available
>
> -1 non-binding (hbck with write operations disabled not included)
>
> download src & bin tar ball   - OK
> signatures & sums- NOT OK
> (md5 checksums missing)
> build from source (openjdk version "1.8.0_151")  - OK
> rat check   -
> OK
> start local instance from bin & CRUD from shell  - OK
> LTT write, read1 million rows, 2 cols/row  - OK
> check logs - OK
>
>
> On Fri, Apr 13, 2018 at 10:55 AM, Stack  wrote:
>
> > On Fri, Apr 13, 2018 at 10:53 AM, Josh Elser  wrote:
> >
> > > Was poking around with PE on a few nodes (I forget the exact
> > > circumstances, need to look back at this), and ran into a case where
> > > ~35 regions were left as RIT
> > >
> > > 2018-04-12 22:05:24,431 ERROR
> > > [master/ctr-e138-1518143905142-221855-01-
> > 02:16000]
> > > procedure2.ProcedureExecutor: Corrupt pid=3580, ppid=3534,
> > > state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
> table=TestTable,
> > > region=71fef  ffe6b5b3cf1cb6d3328a5a58690
> > >
> > > Saw entries like this (I think) for each region which was stuck. A
> > > simple `assign` in the shell brought them back, but I need to dig in
> > > some more
> > to
> > > understand what went wrong.
> > >
> > >
> > Log?
> >
> > HBASE-18152?
> >
> > Thanks Josh,
> > S
> >
> >
> >
> >
> > >
> > > On 4/10/18 4:47 PM, Stack wrote:
> > >
> > >> The first release candidate for Apache HBase 2.0.0 is available for
> > >> downloading and testing.
> > >>
> > >> Artifacts are available here:
> > >>
> > >>   https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0RC0/
> > >>
> > >> Maven artifacts are available in the staging repository at:
> > >>
> > >>   https://repository.apache.org/content/repositories/
> > orgapachehbase-1209
> > >>
> > >> All artifacts are signed with my signing key 8ACC93D2, which is
> > >> also in the project KEYS file at
> > >>
> > >>   http://www.apache.org/dist/hbase/KEYS
> > >>
> > >> These artifacts were tagged 2.0.0RC0 at hash
> > >> 011dd2dae33456b3a2bcc2513e9fdd29de23be46
> > >>
> > >> Please review 'Upgrading from 1.x to 2.x' in the bundled HBase
> > >> 2.0.0 Reference Guide before installing or upgrading for a list of
> > >> incompatibilities, major changes, and notable new features. Be
> > >> aware
> > that
> > >> according to our adopted Semantic Versioning guidelines[1], we've
> > >> allow ourselves to make breaking changes in this major version
> > >> release. For example, Coprocessors will need to be recast to fit
> > >> more constrained CP APIs and a rolling upgrade of an hbase-1.x
> > >> install to hbase-2.x without downtime is (currently) not possible.
> > >> That said, a bunch of effort has been expended mitigating
> > >> differences; a hbase-1.x client can perform DML against an hbase-2
> > >> cluster.
> > >>
> > >> For the full list of ~6k issues addressed, see [2]. There are also
> > >> CHANGES.md and RELEASENOTES.md in the root directory of the source
> > >> tarball.
> > >>
> > >> Please take a few minutes to verify the release and vote on
> > >> releasing
> > it:
> > >>
> > >> [ ] +1 Release this package as Apache HBase 2.0.0 [ ] +0 no opinion
> > >> [ ] -1 Do not release this package because...
> > >>
> > >> This VOTE will run for one week and close Tuesday, April 17, 2018 @
> > 13:00
> > >> PST.
> > >>
> > >> Thanks to the myriad who have helped out with this release, Your
> > >> 2.0.0 Release Manager
> > >>
> > >> 1. http://hbase.apache.org/2.0/book.html#hbase.versioning.post10
> > >> 2.  https://s.apache.org/zwS9
> > >>
> > >>
> >
>

Re: [VOTE] First release candidate for HBase 2.0.0 (RC0) is available

[Stack]

Thanks for voting Wei-Chiu Chuang.

We should run on hadoop3, yeah, but I wouldn't call it a blocker given
current state of hadoop3 (3.0.1, 3.0.2, 3.1.0, not-yet-production-ready...
What do others think?). Perhaps by hbase2.1 it'd be a blocker?

(Thanks for the help over on the likes of HBASE-20244).

St.Ack

On Tue, Apr 17, 2018 at 4:14 PM, Wei-Chiu Chuang 
wrote:

> -1 (non-binding)
> Hbase 2 will break on Hadoop 3.0.1 or above, due to HBASE-20244 caused by
> the refactor in HDFS-12574.
> Is Hbase 2 on Hadoop 3 a requirement for the release?
>
> On Tue, Apr 17, 2018 at 2:59 PM, Stack  wrote:
>
> > Yeah, what Ashish says Umesh (and yeah, checkout HBASE-20385 for the why
> > sir).
> >
> > Any one else given the RC a try? Its seven days now. Time is about up. I
> > have two -1s, one of which I think I can overturn. Any other feedback on
> > the RC? Any PMCers tried it?
> >
> > Thanks,
> >
> > St.Ack
> >
> >
> > On Mon, Apr 16, 2018 at 9:41 PM, ashish singhi  >
> > wrote:
> >
> > > bq. signatures & sums-
> > NOT
> > > OK
> > > (md5 checksums missing)
> > >
> > > This is intentional I think, check HBASE-20385.
> > >
> > > Regards,
> > > Ashish
> > >
> > > -Original Message-
> > > From: Umesh Agashe [mailto:uaga...@cloudera.com]
> > > Sent: Tuesday, April 17, 2018 4:01 AM
> > > To: dev@hbase.apache.org
> > > Subject: Re: [VOTE] First release candidate for HBase 2.0.0 (RC0) is
> > > available
> > >
> > > -1 non-binding (hbck with write operations disabled not included)
> > >
> > > download src & bin tar ball   - OK
> > > signatures & sums- NOT
> OK
> > > (md5 checksums missing)
> > > build from source (openjdk version "1.8.0_151")  - OK
> > > rat check
>  -
> > > OK
> > > start local instance from bin & CRUD from shell  - OK
> > > LTT write, read1 million rows, 2 cols/row  - OK
> > > check logs
>  -
> > OK
> > >
> > >
> > > On Fri, Apr 13, 2018 at 10:55 AM, Stack  wrote:
> > >
> > > > On Fri, Apr 13, 2018 at 10:53 AM, Josh Elser 
> > wrote:
> > > >
> > > > > Was poking around with PE on a few nodes (I forget the exact
> > > > > circumstances, need to look back at this), and ran into a case
> where
> > > > > ~35 regions were left as RIT
> > > > >
> > > > > 2018-04-12 22:05:24,431 ERROR
> > > > > [master/ctr-e138-1518143905142-221855-01-
> > > > 02:16000]
> > > > > procedure2.ProcedureExecutor: Corrupt pid=3580, ppid=3534,
> > > > > state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
> > > table=TestTable,
> > > > > region=71fef  ffe6b5b3cf1cb6d3328a5a58690
> > > > >
> > > > > Saw entries like this (I think) for each region which was stuck. A
> > > > > simple `assign` in the shell brought them back, but I need to dig
> in
> > > > > some more
> > > > to
> > > > > understand what went wrong.
> > > > >
> > > > >
> > > > Log?
> > > >
> > > > HBASE-18152?
> > > >
> > > > Thanks Josh,
> > > > S
> > > >
> > > >
> > > >
> > > >
> > > > >
> > > > > On 4/10/18 4:47 PM, Stack wrote:
> > > > >
> > > > >> The first release candidate for Apache HBase 2.0.0 is available
> for
> > > > >> downloading and testing.
> > > > >>
> > > > >> Artifacts are available here:
> > > > >>
> > > > >>   https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0RC0/
> > > > >>
> > > > >> Maven artifacts are available in the staging repository at:
> > > > >>
> > > > >>   https://repository.apache.org/content/repositories/
> > > > orgapachehbase-1209
> > > > >>
> > > > >> All artifacts are signed with my signing key 8ACC93D2, which is
> > > > >> also in the project KEYS file at
> > > > >>
> > > > >>   http://www.apache.org/dist/hbase/KEYS
> > > > >>
> > > > >> These ar

Re: [VOTE] First release candidate for HBase 2.0.0 (RC0) is available

[Stack]

On Tue, Apr 17, 2018 at 6:25 PM, Sean Busbey  wrote:

> I've been using it and have some concerns wrt our shaded clients
> (tracking in HBASE-20331). I've been hesitant to vote -1 due to them
> because the work isn't done yet. But I hope to have it wrapped by end
> of week.
>
>
Should HBASE-20331 be a blocker?


> Reviewing  HBASE-18792, I'm also concerned that it isn't included in
> the current RC. But again, hasn't landed yet.
>
> HBASE-20244 looks bad, but I agree that running on Hadoop 3 shouldn't
> be a blocker. Yay for the asyncfs log message looking reasonable in
> the face of failure though!
>
> I've only been kind of half-following along for the perf saga in
> HBASE-20188. We have enough info yet to get some guidance into the
> upgrade section of the ref guide?
>
>
Perf is taking time. There are a few of us at it. Calibrating
understanding, expectations, and tooling has taken a bunch of time. We're
making progress though. For the upgrade section, I was thinking of adding a
general note that perf profile changes in hbase2 -- it will likely run
faster but it may be slower in some instances -- and then refer readers to
the perf chapter which we can fill in findings as we go (post hbase2
release even).

Thanks,
St.Ack


>
> On Tue, Apr 17, 2018 at 4:59 PM, Stack  wrote:
> > Yeah, what Ashish says Umesh (and yeah, checkout HBASE-20385 for the why
> > sir).
> >
> > Any one else given the RC a try? Its seven days now. Time is about up. I
> > have two -1s, one of which I think I can overturn. Any other feedback on
> > the RC? Any PMCers tried it?
> >
> > Thanks,
> >
> > St.Ack
> >
> >
> > On Mon, Apr 16, 2018 at 9:41 PM, ashish singhi  >
> > wrote:
> >
> >> bq. signatures & sums-
> NOT
> >> OK
> >> (md5 checksums missing)
> >>
> >> This is intentional I think, check HBASE-20385.
> >>
> >> Regards,
> >> Ashish
> >>
> >> -Original Message-
> >> From: Umesh Agashe [mailto:uaga...@cloudera.com]
> >> Sent: Tuesday, April 17, 2018 4:01 AM
> >> To: dev@hbase.apache.org
> >> Subject: Re: [VOTE] First release candidate for HBase 2.0.0 (RC0) is
> >> available
> >>
> >> -1 non-binding (hbck with write operations disabled not included)
> >>
> >> download src & bin tar ball   - OK
> >> signatures & sums- NOT
> OK
> >> (md5 checksums missing)
> >> build from source (openjdk version "1.8.0_151")  - OK
> >> rat check
>  -
> >> OK
> >> start local instance from bin & CRUD from shell  - OK
> >> LTT write, read1 million rows, 2 cols/row  - OK
> >> check logs
>  - OK
> >>
> >>
> >> On Fri, Apr 13, 2018 at 10:55 AM, Stack  wrote:
> >>
> >> > On Fri, Apr 13, 2018 at 10:53 AM, Josh Elser 
> wrote:
> >> >
> >> > > Was poking around with PE on a few nodes (I forget the exact
> >> > > circumstances, need to look back at this), and ran into a case where
> >> > > ~35 regions were left as RIT
> >> > >
> >> > > 2018-04-12 22:05:24,431 ERROR
> >> > > [master/ctr-e138-1518143905142-221855-01-
> >> > 02:16000]
> >> > > procedure2.ProcedureExecutor: Corrupt pid=3580, ppid=3534,
> >> > > state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
> >> table=TestTable,
> >> > > region=71fef  ffe6b5b3cf1cb6d3328a5a58690
> >> > >
> >> > > Saw entries like this (I think) for each region which was stuck. A
> >> > > simple `assign` in the shell brought them back, but I need to dig in
> >> > > some more
> >> > to
> >> > > understand what went wrong.
> >> > >
> >> > >
> >> > Log?
> >> >
> >> > HBASE-18152?
> >> >
> >> > Thanks Josh,
> >> > S
> >> >
> >> >
> >> >
> >> >
> >> > >
> >> > > On 4/10/18 4:47 PM, Stack wrote:
> >> > >
> >> > >> The first release candidate for Apache HBase 2.0.0 is available for
> >> > >> downloading and testing.
> >> > >>
> >> > >> Artifacts are available here:
> >> > >>
> >> > >>   https://dist.a

Re: [VOTE] First release candidate for HBase 2.0.0 (RC0) is available

[Stack]

On Wed, Apr 18, 2018 at 12:58 PM, Josh Elser  wrote:

> +1 (binding)
>
> There are rough edges, of course, but I think this is more than enough
> quality for a 2.0.0. Putting a line in the sand for 2.0 will help us
> continue to make changes that improve the product going forward with more
> focus and get those changes into the hands of folks downstream.
>
>
I'm in line with this sort of thinking. The diff between RC0 and current
tip is below. There's some pretty important changes in the list I'd say.
How about I put up another RC tonight/tomorrow with some more doc fixes,
the HBCK, and if you have something for test random ports, that too... Then
you show up again w/ the above philosophizing (girded by the accompanying
+1)?

St.Ack


f168689 HBASE-19994 Create a new class for RPC throttling exception, make
it retryable
10f74df HBASE-20404 Fixes to CleanChore correctness and operability.
c7c0861 HBASE-20398 Redirect doesn't work on web UI
acc73b1 HBASE-20399 Fix merge layout
10a2905 HBASE-19963 TestFSHDFSUtils assumes wrong default port for Hadoop
3.0.1+
bbe1551 HBASE-20409 Set hbase.client.meta.operation.timeout in
TestClientOperationTimeout
fb01cc4 HBASE-20364 ensure jira comment from nightly reflects results of
run comment comes from.
a825bf1 HBASE-20415 branches-2 don't need scala tooling.
f5f03f9 HBASE-20233 Remove redundant region server metric
4de3d7d HBASE-20389 Move website building flags into a profile.
4176237 HBASE-20335 ensure each stage of the nightly job gathers machine
information.
7704d3b HBASE-20410 update protoc to 3.5.1-1 for rhel6
8ca2d0f HBASE-20394 HBase over rides the value of HBASE_OPTS (if any) set
by client
ed3a656 HBASE-20379 shadedjars yetus plugin should add a footer link
0d8262f HBASE-20112 register nightly junit over hadoop3 results with
jenkins.
cba88d1 HBASE-20224 Web UI is broken in standalone mode - addendum for
hbase-endpoint and hbase-rsgroup modules
ee3c430 HBASE-20224 Web UI is broken in standalone mode - addendum for
hbase-archetypes module
0eacb3e HBASE-20338 WALProcedureStore#recoverLease() should have fixed
sleeps for retrying rollWriter()
263cc8d HBASE-20376 RowCounter and CellCounter documentations are incorrect
e72a1c6 HBASE-20330 ProcedureExecutor.start() gets stuck in recover lease
on store
0d2b18b HBASE-20350 NullPointerException in Scanner during close()
6a081d3 HBASE-20310 Fixed false inconsistency shown by hbck -metaonly
option on HBase 2
0e8d42b HBASE-20219 An error occurs when scanning with reversed=true and
loadColumnFamiliesOnDemand=true
907b264 HBASE-20358 Fix bin/hbase thrift usage text
200afa1 HBASE-20382 If RSGroups not enabled, rsgroup.jsp prints stack trace
e05b318 HBASE-20386 [DOC] Align WALPlayer help text and refguide
63125ea HBASE-20385 Purge md5-making from our little make_rc.sh script
47128f5 HBASE-20384 [AMv2] Logging format improvements; use encoded name
rather than full region name marking transitions
9320c5d HBASE-20253. Error message is missing for restore_snapshot
b3ec5f0 HBASE-15291 FileSystem not closed in secure bulkLoad
2e6eff0 HBASE-20182 Addendum throw IOException instead of
NoServerForRegionException because it is a DoNotRetryRegionException
3887648 HBASE-20335 nightly job bash cleanup.
86f4686 HBASE-20068 personality tests for Apache Yetus should use the maven
plugin to exec maven.





> Well done, Stack.
>
> On 4/10/18 4:47 PM, Stack wrote:
>
>> The first release candidate for Apache HBase 2.0.0 is available for
>> downloading and testing.
>>
>> Artifacts are available here:
>>
>>   https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0RC0/
>>
>> Maven artifacts are available in the staging repository at:
>>
>>   https://repository.apache.org/content/repositories/orgapachehbase-1209
>>
>> All artifacts are signed with my signing key 8ACC93D2, which is also
>> in the project KEYS file at
>>
>>   http://www.apache.org/dist/hbase/KEYS
>>
>> These artifacts were tagged 2.0.0RC0 at
>> hash 011dd2dae33456b3a2bcc2513e9fdd29de23be46
>>
>> Please review 'Upgrading from 1.x to 2.x' in the bundled HBase 2.0.0
>> Reference Guide before installing or upgrading for a list of
>> incompatibilities, major changes, and notable new features. Be aware that
>> according to our adopted Semantic Versioning guidelines[1], we've allow
>> ourselves to make breaking changes in this major version release. For
>> example, Coprocessors will need to be recast to fit more constrained CP
>> APIs and a rolling upgrade of an hbase-1.x install to hbase-2.x without
>> downtime is (currently) not possible. That said, a bunch of effort has
>> been
>> expended mitigating differences; a hbase-1.x client can perform DML
>> against
>> an hbase-2 cluster.
>>
>> For the full list of ~6k issues

[RESULT] [VOTE] First release candidate for HBase 2.0.0 (RC0) is available

[Stack]

This RC failed. It had a binding +1 and two -1s inside the voting allotment
but discussion says a new RC will get more attention.

S

On Wed, Apr 18, 2018 at 2:37 PM, Josh Elser  wrote:

> I like that plan. I'll try to dig into what a "better" fix might be for
> the random ports stuff tonight.
>
>
> On 4/18/18 5:35 PM, Stack wrote:
>
>> On Wed, Apr 18, 2018 at 12:58 PM, Josh Elser  wrote:
>>
>> +1 (binding)
>>>
>>> There are rough edges, of course, but I think this is more than enough
>>> quality for a 2.0.0. Putting a line in the sand for 2.0 will help us
>>> continue to make changes that improve the product going forward with more
>>> focus and get those changes into the hands of folks downstream.
>>>
>>>
>>> I'm in line with this sort of thinking. The diff between RC0 and current
>> tip is below. There's some pretty important changes in the list I'd say.
>> How about I put up another RC tonight/tomorrow with some more doc fixes,
>> the HBCK, and if you have something for test random ports, that too...
>> Then
>> you show up again w/ the above philosophizing (girded by the accompanying
>> +1)?
>>
>> St.Ack
>>
>>
>> f168689 HBASE-19994 Create a new class for RPC throttling exception, make
>> it retryable
>> 10f74df HBASE-20404 Fixes to CleanChore correctness and operability.
>> c7c0861 HBASE-20398 Redirect doesn't work on web UI
>> acc73b1 HBASE-20399 Fix merge layout
>> 10a2905 HBASE-19963 TestFSHDFSUtils assumes wrong default port for Hadoop
>> 3.0.1+
>> bbe1551 HBASE-20409 Set hbase.client.meta.operation.timeout in
>> TestClientOperationTimeout
>> fb01cc4 HBASE-20364 ensure jira comment from nightly reflects results of
>> run comment comes from.
>> a825bf1 HBASE-20415 branches-2 don't need scala tooling.
>> f5f03f9 HBASE-20233 Remove redundant region server metric
>> 4de3d7d HBASE-20389 Move website building flags into a profile.
>> 4176237 HBASE-20335 ensure each stage of the nightly job gathers machine
>> information.
>> 7704d3b HBASE-20410 update protoc to 3.5.1-1 for rhel6
>> 8ca2d0f HBASE-20394 HBase over rides the value of HBASE_OPTS (if any) set
>> by client
>> ed3a656 HBASE-20379 shadedjars yetus plugin should add a footer link
>> 0d8262f HBASE-20112 register nightly junit over hadoop3 results with
>> jenkins.
>> cba88d1 HBASE-20224 Web UI is broken in standalone mode - addendum for
>> hbase-endpoint and hbase-rsgroup modules
>> ee3c430 HBASE-20224 Web UI is broken in standalone mode - addendum for
>> hbase-archetypes module
>> 0eacb3e HBASE-20338 WALProcedureStore#recoverLease() should have fixed
>> sleeps for retrying rollWriter()
>> 263cc8d HBASE-20376 RowCounter and CellCounter documentations are
>> incorrect
>> e72a1c6 HBASE-20330 ProcedureExecutor.start() gets stuck in recover lease
>> on store
>> 0d2b18b HBASE-20350 NullPointerException in Scanner during close()
>> 6a081d3 HBASE-20310 Fixed false inconsistency shown by hbck -metaonly
>> option on HBase 2
>> 0e8d42b HBASE-20219 An error occurs when scanning with reversed=true and
>> loadColumnFamiliesOnDemand=true
>> 907b264 HBASE-20358 Fix bin/hbase thrift usage text
>> 200afa1 HBASE-20382 If RSGroups not enabled, rsgroup.jsp prints stack
>> trace
>> e05b318 HBASE-20386 [DOC] Align WALPlayer help text and refguide
>> 63125ea HBASE-20385 Purge md5-making from our little make_rc.sh script
>> 47128f5 HBASE-20384 [AMv2] Logging format improvements; use encoded name
>> rather than full region name marking transitions
>> 9320c5d HBASE-20253. Error message is missing for restore_snapshot
>> b3ec5f0 HBASE-15291 FileSystem not closed in secure bulkLoad
>> 2e6eff0 HBASE-20182 Addendum throw IOException instead of
>> NoServerForRegionException because it is a DoNotRetryRegionException
>> 3887648 HBASE-20335 nightly job bash cleanup.
>> 86f4686 HBASE-20068 personality tests for Apache Yetus should use the
>> maven
>> plugin to exec maven.
>>
>>
>>
>>
>>
>> Well done, Stack.
>>>
>>> On 4/10/18 4:47 PM, Stack wrote:
>>>
>>> The first release candidate for Apache HBase 2.0.0 is available for
>>>> downloading and testing.
>>>>
>>>> Artifacts are available here:
>>>>
>>>>https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0RC0/
>>>>
>>>> Maven artifacts are available in the staging repository at:
>>>>
>>>>https://repository.apache.org/content/repositori

[VOTE] The second release candidate for HBASE 2.0.0 (RC1) is available

[Stack]

The second release candidate for Apache HBase 2.0.0 is available for
downloading and testing.

Artifacts are available here:

 https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0RC1/

Maven artifacts are available in the staging repository at:

 https://repository.apache.org/content/repositories/orgapachehbase-1211

All artifacts are signed with my signing key 8ACC93D2, which is also
in the project KEYS file at

 http://www.apache.org/dist/hbase/KEYS

These artifacts were tagged 2.0.0RC1 at
hash d8985e04bf6adfc3d00660c3e04e27a5f5bacb4b

Please review 'Upgrading from 1.x to 2.x' in the bundled HBase 2.0.0
Reference Guide before installing or upgrading for a list of
incompatibilities, major changes, and notable new features. Be aware that
according to our adopted Semantic Versioning guidelines[1], we've allow
ourselves to make breaking changes in this major version release. For
example, Coprocessors will need to be recast to fit more constrained CP
APIs and a rolling upgrade of an hbase-1.x install to hbase-2.x without
downtime is (currently) not possible. That said, a bunch of effort has been
expended mitigating differences; a hbase-1.x client can perform DML against
an hbase-2 cluster. A
bundled compatibility report showing difference from 1.2.6 may be of help
[3].

For the full list of ~6k issues addressed, see [2]. There are also
CHANGES.md and RELEASENOTES.md in the root directory of the source tarball.

Please take a few minutes to verify the release and vote on releasing it:

[ ] +1 Release this package as Apache HBase 2.0.0
[ ] +0 no opinion
[ ] -1 Do not release this package because...

This VOTE will run for one week and close Friday, April 27, 2018 @ 11:00
PST.

Thanks to the myriad who have helped out with this release,
Your 2.0.0 Release Manager

1. http://hbase.apache.org/2.0/book.html#hbase.versioning.post10
2.  https://s.apache.org/zwS9
3.
https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0RC1/compatibiliity_report_1.2.6vs2.0.0RC1.html

Re: [VOTE] The second release candidate for HBASE 2.0.0 (RC1) is available

[Stack]

Yeah, I have same thing local. I disabled in-memory-compaction in RC1. Some
tests must depend on it. My mess. Let me fix and put up a new RC quick.
Thanks for finding this Umesh,
S

On Fri, Apr 20, 2018 at 3:14 PM, Umesh Agashe  wrote:

> -1 non-binding
>
> download src & bin tar ball- OK
> signatures & sums - OK
> build from source (openjdk version "1.8.0_151")   - OK
> rat check-
> OK
> Unit tests   -
> NOT OK
>
> Following unit tests fails with:
> [INFO] ---
> [INFO]  T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.hbase.regionserver.TestHStore
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 2.327 s <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestHStore
> [ERROR] testFlushSizeSizing(org.apache.hadoop.hbase.
> regionserver.TestHStore)
> Time elapsed: 1.548 s  <<< FAILURE!
> java.lang.AssertionError: expected: org.apache.hadoop.hbase.
> regionserver.MemStoreSizing
> but
> was: org.apache.hadoop.hbase.regionserver.MemStoreSize heapSize=312 , offHeapSize=0>
> at org.apache.hadoop.hbase.regionserver.TestHStore.testFlushSizeSizing(
> TestHStore.java:251)
>
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Failures:
> [ERROR]   TestHStore.testFlushSizeSizing:251 expected:
> org.apache.hadoop.hbase.regionserver.MemStoreSizing heapSize=312 , offHeapSize=0> but was: org.apache.hadoop.hbase.
> regionserver.MemStoreSize
> [INFO]
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
>
> From surefire report:
> 2018-04-20 15:01:55,386 INFO  [Time-limited test]
> hbase.ResourceChecker(172): after: regionserver.TestHStore#
> testFlushSizeSizing
> Thread=18 (was 8)
> Potentially hanging thread: Time-limited test-MemStoreChunkPool Statistics
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.parkNanos(
> LockSupport.java:215)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> java.util.concurrent.ScheduledThreadPoolExecutor$
> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> java.util.concurrent.ScheduledThreadPoolExecutor$
> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1074)
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1134)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
>
> Potentially hanging thread: Monitor thread for TaskMonitor
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hbase.monitoring.TaskMonitor$
> MonitorRunnable.run(
> TaskMonitor.java:302)
> java.lang.Thread.run(Thread.java:748)
>
> ...
> Potentially hanging thread: AsyncFSWAL-0
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
> java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1074)
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1134)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
>  - Thread LEAK? -, OpenFileDescriptor=237 (was 232) - OpenFileDescriptor
> LEAK? -, MaxFileDescriptor=1048576 (was 1048576), SystemLoadAverage=185
> (was 167) - SystemLoadAverage LEAK? -, ProcessCount=123 (was 123),
> AvailableMemoryMB=3609 (was 3728)
>
> On Fri, Apr 20, 2018 at 11:04 AM, Stack  wrote:
>
> > The second release candidate for Apache HBase 2.0.0 is available for
> > downloading and testing.
> >
> > Artifacts are available here:
> >
> >  https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0RC1/
> >
> > Maven artifacts are available in the staging repository at:
> >
> >  https://repository.apache.org/content/repositories/orgapachehbase-1211
> >
> > All artifacts are signed with my signing key 8ACC93D2, which is also
> > in the project KEYS file at
> &

[VOTE] The third release candidate for HBASE 2.0.0 (RC2) is available

[Stack]

The third release candidate for Apache HBase 2.0.0 is available for
downloading and testing.

Artifacts are available here:

 https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0RC2/

Maven artifacts are available in the staging repository at:

https://repository.apache.org/content/repositories/orgapachehbase-1213

All artifacts are signed with my signing key 8ACC93D2, which is also
in the project KEYS file at

 http://www.apache.org/dist/hbase/KEYS

These artifacts were tagged 2.0.0RC2 at hash 7483b111e4da77adbfc8062b3b22cb
e7c2cb91c1

Please review 'Upgrading from 1.x to 2.x' in the bundled HBase 2.0.0
Reference Guide before installing or upgrading for a list of
incompatibilities, major changes, and notable new features. Be aware that
according to our adopted Semantic Versioning guidelines[1], we've allow
ourselves to make breaking changes in this major version release. For
example, Coprocessors will need to be recast to fit more constrained CP
APIs and a rolling upgrade of an hbase-1.x install to hbase-2.x without
downtime is (currently) not possible. That said, a bunch of effort has been
expended mitigating differences; a hbase-1.x client can perform DML against
an hbase-2 cluster. A
bundled compatibility report showing difference from 1.2.6 may be of help
[3].

For the full list of ~6k issues addressed, see [2]. There are also
CHANGES.md and RELEASENOTES.md in the root directory of the source tarball.

Please take a few minutes to verify the release and vote on releasing it:

[ ] +1 Release this package as Apache HBase 2.0.0
[ ] +0 no opinion
[ ] -1 Do not release this package because...

This VOTE will run for one week and close Sunday, April 29, 2018 @ 21:00
PST.

Thanks to the myriad who have helped out with this release,
Your 2.0.0 Release Manager

1. http://hbase.apache.org/2.0/book.html#hbase.versioning.post10
2. https://s.apache.org/zwS9
3.
https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0RC2/compatibiliity_report_1.2.6vs2.0.0RC2.html

Re: [VOTE] The third release candidate for HBASE 2.0.0 (RC2) is available

[Stack]

On Tue, Apr 24, 2018 at 1:25 AM, Lars Francke 
wrote:

> Stack,
>
> I'm sure you'll want to slap me for bringing this up so late (I only found
> it recently) but could you take a look at the recent discussions in <
> https://issues.apache.org/jira/browse/HBASE-19746>?
> Is there any need for me to work on this or would it be too late to merge
> this change anyway?
>
>
Good discussion between you and Chia-Ping Tsai up on HBASE-19746.

We at least need the API extra-doc on getTypeByte (with maybe pointer from
getType) and note in upgrade section of refguide. Your @deprecation works
for me too as a way to flag users to take a deeper look.

Its too late for this RC. Transferring your -0 on the patch up on
HBASE-19746 here as vote on this release (say...), this RC could still go
out as 2.0.0.

If it fails for another reason, lets get your suggested patch in sir. Mark
the 'doc' issue a blocker on 2.0.0 please. If it fails to make 2.0.0, lets
get it into 2.0.1.

Thanks to you (and Chia-Ping) for the finding.

S




> Cheers,
> Lars
>
> On Tue, Apr 24, 2018 at 4:13 PM, Balazs Meszaros <
> balazs.mesza...@cloudera.com> wrote:
>
> > +1
> >
> > - signatures and checksums: OK
> > - rat check: OK
> > - built from source (8u112): OK
> > - basic shell functionalities: OK
> > - web UI: OK
> > - LTT with 1M rows: OK
> >
> >
> > On Mon, Apr 23, 2018 at 7:10 AM, Stack  wrote:
> >
> > > The third release candidate for Apache HBase 2.0.0 is available for
> > > downloading and testing.
> > >
> > > Artifacts are available here:
> > >
> > >  https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0RC2/
> > >
> > > Maven artifacts are available in the staging repository at:
> > >
> > > https://repository.apache.org/content/repositories/orgapachehbase-1213
> > >
> > > All artifacts are signed with my signing key 8ACC93D2, which is also
> > > in the project KEYS file at
> > >
> > >  http://www.apache.org/dist/hbase/KEYS
> > >
> > > These artifacts were tagged 2.0.0RC2 at hash
> > 7483b111e4da77adbfc8062b3b22cb
> > > e7c2cb91c1
> > >
> > > Please review 'Upgrading from 1.x to 2.x' in the bundled HBase 2.0.0
> > > Reference Guide before installing or upgrading for a list of
> > > incompatibilities, major changes, and notable new features. Be aware
> that
> > > according to our adopted Semantic Versioning guidelines[1], we've allow
> > > ourselves to make breaking changes in this major version release. For
> > > example, Coprocessors will need to be recast to fit more constrained CP
> > > APIs and a rolling upgrade of an hbase-1.x install to hbase-2.x without
> > > downtime is (currently) not possible. That said, a bunch of effort has
> > been
> > > expended mitigating differences; a hbase-1.x client can perform DML
> > against
> > > an hbase-2 cluster. A
> > > bundled compatibility report showing difference from 1.2.6 may be of
> help
> > > [3].
> > >
> > > For the full list of ~6k issues addressed, see [2]. There are also
> > > CHANGES.md and RELEASENOTES.md in the root directory of the source
> > tarball.
> > >
> > > Please take a few minutes to verify the release and vote on releasing
> it:
> > >
> > > [ ] +1 Release this package as Apache HBase 2.0.0
> > > [ ] +0 no opinion
> > > [ ] -1 Do not release this package because...
> > >
> > > This VOTE will run for one week and close Sunday, April 29, 2018 @
> 21:00
> > > PST.
> > >
> > > Thanks to the myriad who have helped out with this release,
> > > Your 2.0.0 Release Manager
> > >
> > > 1. http://hbase.apache.org/2.0/book.html#hbase.versioning.post10
> > > 2. https://s.apache.org/zwS9
> > > 3.
> > > https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0RC2/
> > > compatibiliity_report_1.2.6vs2.0.0RC2.html
> > >
> >
>

Re: [VOTE] The third release candidate for HBASE 2.0.0 (RC2) is available

[Stack]

Thanks Lars.
S

On Wed, Apr 25, 2018 at 1:28 AM, Lars Francke 
wrote:

> Thanks Stack.
>
> In that case I'm -0 on this RC.
> I'd really like to either revert that patch or at least document the
> deprecation so we can remove it in 3.0. The way it is now we can't remove
> getTypeByte() in 3.0 because we'll need it for getType() which we can't
> change because it's not documented as deprecated.
>
> I will try to get a patch up until the weekend but don't really want to
> hold up a release either knowing how much work each RC is.
>
> Cheers,
> Lars
>
> On Wed, Apr 25, 2018 at 1:12 PM, Andrew Purtell 
> wrote:
>
> > +1
> >
> > Checked sums and signatures: ok
> > Built from source: ok (8u152)
> > RAT check passes: ok (8u152)
> > LTT 1M rows 20% updates: ok (8u152)
> > ITBLL 100M rows: ok (8u152)
> >
> >
> > On Sun, Apr 22, 2018 at 10:10 PM, Stack  wrote:
> >
> > > The third release candidate for Apache HBase 2.0.0 is available for
> > > downloading and testing.
> > >
> > > Artifacts are available here:
> > >
> > >  https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0RC2/
> > >
> > > Maven artifacts are available in the staging repository at:
> > >
> > > https://repository.apache.org/content/repositories/orgapachehbase-1213
> > >
> > > All artifacts are signed with my signing key 8ACC93D2, which is also
> > > in the project KEYS file at
> > >
> > >  http://www.apache.org/dist/hbase/KEYS
> > >
> > > These artifacts were tagged 2.0.0RC2 at hash
> > 7483b111e4da77adbfc8062b3b22cb
> > > e7c2cb91c1
> > >
> > > Please review 'Upgrading from 1.x to 2.x' in the bundled HBase 2.0.0
> > > Reference Guide before installing or upgrading for a list of
> > > incompatibilities, major changes, and notable new features. Be aware
> that
> > > according to our adopted Semantic Versioning guidelines[1], we've allow
> > > ourselves to make breaking changes in this major version release. For
> > > example, Coprocessors will need to be recast to fit more constrained CP
> > > APIs and a rolling upgrade of an hbase-1.x install to hbase-2.x without
> > > downtime is (currently) not possible. That said, a bunch of effort has
> > been
> > > expended mitigating differences; a hbase-1.x client can perform DML
> > against
> > > an hbase-2 cluster. A
> > > bundled compatibility report showing difference from 1.2.6 may be of
> help
> > > [3].
> > >
> > > For the full list of ~6k issues addressed, see [2]. There are also
> > > CHANGES.md and RELEASENOTES.md in the root directory of the source
> > tarball.
> > >
> > > Please take a few minutes to verify the release and vote on releasing
> it:
> > >
> > > [ ] +1 Release this package as Apache HBase 2.0.0
> > > [ ] +0 no opinion
> > > [ ] -1 Do not release this package because...
> > >
> > > This VOTE will run for one week and close Sunday, April 29, 2018 @
> 21:00
> > > PST.
> > >
> > > Thanks to the myriad who have helped out with this release,
> > > Your 2.0.0 Release Manager
> > >
> > > 1. http://hbase.apache.org/2.0/book.html#hbase.versioning.post10
> > > 2. https://s.apache.org/zwS9
> > > 3.
> > > https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0RC2/
> > > compatibiliity_report_1.2.6vs2.0.0RC2.html
> > >
> >
>

Re: [DISCUSS] ruby-lint in precommit

[Stack]

Agree.

If we want to torture a contributor, we can set them on fixing all
complaints.

St.Ack

On Wed, Apr 25, 2018 at 7:28 AM, Sean Busbey  wrote:

> Hi folks!
>
> Given how often I see us ignoring ruby-lint feedback in precommit,
> would anyone be opposed to me changing its vote from -1 to -0?
>
> If someone eventually does the work to make it deal with e.g.
> jruby-isms and maven directory structure, we could then turn it back
> on. But I definitely don't personally have the bandwidth to do that
> right now. I could file a JIRA explaining the issue, so we'd have some
> chance of it getting remembered.
>