Re: [DISCUSS] Add Ozone as dependency to hbase-asyncfs ?

[Zach York]

With S3, we use reflection when loading the filesystem. As long as it is on
the CP, we're okay. However, I think the difference here is the APIs that
Stephen wants to use is not in the hadoop filesystem API/class, but instead
in DFS I believe. Others in this thread have mentioned ways to work around
this both in the short term and long term.

On Thu, Mar 16, 2023, 11:06 PM 张铎(Duo Zhang)  wrote:

> I just mean how do we manage the dependencies. We do not ship hbase with
> aws-sdk, but I believe users can put the related jars under lib and then
> use S3AFileSystem. I think we can support ozone in the same way?
>
> Viraj Jasani  于2023年3月17日周五 13:33写道：
>
> > WAL is still kept on hdfs only, even when hfiles are kept in s3 AFAIK.
> But
> > here it seems, both WAL and HFiles can be kept in Ozone IIUC.
> >
> >
> > On Thu, Mar 16, 2023 at 8:46 PM 张铎(Duo Zhang) 
> > wrote:
> >
> > > How do we support S3 as HFile storage currently? I do not think we have
> > > added aws-sdk as a direct dependency in HBase now?
> > >
> > > Viraj Jasani  于2023年3月17日周五 04:37写道：
> > >
> > > > +1, similar to what was done in the past for using
> > > > HdfsDataOutputStreamBuilder that was available since hadoop 2.9 or
> so I
> > > > think.
> > > >
> > > >
> > > > On Thu, Mar 16, 2023 at 1:04 PM Andrew Purtell <
> > andrew.purt...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > It should be done with reflection rather than take a direct
> > dependency,
> > > > > until Hadoop common interfaces are available in what we consider
> the
> > > > lowest
> > > > > supported version.
> > > > >
> > > > > > On Mar 16, 2023, at 12:35 PM, Viraj Jasani 
> > > wrote:
> > > > > >
> > > > > > It would be nice using PathCapabilities to determine lease
> > recovery
> > > > as a
> > > > > > feature flag.
> > > > > > In fact, s3a and abfs have lots of feature flags being derived
> from
> > > > this
> > > > > > API already. It would be good for dfs and ozone to recognize
> lease
> > > > > recovery
> > > > > > as a capability.
> > > > > >
> > > > > > However, this alone might not be sufficient and something like
> > > > > > RecoverableFileSystem interface would be helpful as long as we
> can
> > > > > abstract
> > > > > > out lease recovery (and safe mode etc) options as hbase anyways
> > need
> > > to
> > > > > > perform them.
> > > > > >
> > > > > > Hence, having both: a) path capability to identify if lease
> > recovery
> > > > etc
> > > > > > features are available and b) a new FileSystem interface that
> both
> > > dfs
> > > > > and
> > > > > > ozone can implement, would be great IMHO. Because even if we just
> > > have
> > > > > path
> > > > > > capability for the feature flag, we would still end up adding
> ozone
> > > > > > dependency (unless done with reflection as Andrew mentioned) to
> > > perform
> > > > > > lease recovery unless lease recovery is abstracted out somewhere
> in
> > > > > hadoop.
> > > > > >
> > > > > >> One of the original worries is if the Hadoop/HDFS community
> > > > > >> would reject our proposal when we change the base
> > interface/abstract
> > > > > class
> > > > > >> in FileSystem (if it's non-backward compatible).
> > > > > >
> > > > > > I believe, new IA.Public interface in hadoop that can abstract
> out
> > > > lease
> > > > > > recovery etc would have less likelihood of getting rejected than
> > > > "making
> > > > > > changes in FileSystem directly".
> > > > > >
> > > > > >
> > > > > >> On Thu, Mar 16, 2023 at 2:07 AM Tak Lon (Stephen) Wu <
> > > > tak...@apache.org
> > > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> In addition, I'm yet confirm but based on another search in the
> > > hadoop
> > > > > >> code, we may be able to add recover lease as a feature flag in
> > > > > >> CommonPathCapabilities [3] and can be used by the interface of
> > > > > >> PathCapabilities#hasPathCapability [4]. (this is similar to
> > > > > >> StreamCapabilities as mentioned by Viraj)
> > > > > >>
> > > > > >> 3.
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/hadoop/blob/branch-3.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonPathCapabilities.java
> > > > > >> 4.
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/hadoop/blob/branch-3.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathCapabilities.java
> > > > > >>
> > > > > >> -Stephen
> > > > > >>
> > > > > >>> On Thu, Mar 16, 2023 at 12:00 AM Tak Lon (Stephen) Wu <
> > > > > tak...@apache.org>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>> Thanks everyone ! Sean helped to clarify that something like
> DFS
> > > > > specific
> > > > > >>> APIs used by HBase has been in-place in many HBase modules as
> the
> > > > > feature
> > > > > >>> implementation but yet standardized in hadoop general
> FileSystem
> > > API,
> > > > > >> e.g.
> > > > > >>> lease recovery. One of the original worries is if the
> Hadoop/HDFS
> > > > > >> community
> > > > > >>> would reject our proposal when we 

Re: [DISCUSS] Replace log4j with reload4j for branch-2.x

[Zach York]

+1 on the original discussion

I think that all makes sense, we can't know whether one dependency is more
vulnerable/going to be more work to maintain or not. However, I think
perhaps the more interesting question is whether we should be okay with
using EOL dependencies in any active release line. CVEs happen... I think
it's important to be able to react quickly. By depending on any EOL code in
our active release lines, we severely limit our ability to quickly deal
with CVEs. I understand it's not always easy (the backwards incompatibility
of log4j2 and the properties files are good examples), but it's also been
6+ years since log4j 1.x has become EOL.

On Fri, Jan 21, 2022 at 11:47 AM Andrew Purtell  wrote:

> I will offer a guess as answer to the question originally proposed, which
> is, no the Logging PMC is not going to behave in a cooperative manner to
> Reload4J. The best we can hope for, and I personally doubt it, is they will
> not be actively hostile. Decisions made by Apache PMC can unfortunately be
> made on the basis of years-long running arguments and personality
> conflicts, and Logging is unfortunately one of them. Just my opinion. You
> won't change it. Fortunately that is neither here nor there. We aren't here
> to judge up front if Reload4J can respond adequately to hypothetical future
> security issues. That is not knowable and remains to be seen. It also
> remains to be seen if Log4J 2 is going to respond adequately to future
> security issues. Or Netty. Or Jersey. Or Jackson. Or any of our other risky
> third party dependencies (as measured by having "interesting" CVEs). What
> we can do is look at the current track record, which has been adequate so
> far.
>
>
> On Fri, Jan 21, 2022 at 11:35 AM Wei-Chiu Chuang
>  wrote:
>
> > The reload4j is maintained by one of the Apache Logging PMC. And a
> release
> > was made to address the latest log4j 1.x CVEs announced only a day after.
> >
> > There is a thread in the private@logging that mentioned the “new”
> log4j1.x
> > CVEs were actually not new. They were announced years before for 2.x
> which
> > happens to also impact 1.x. But because 1.x was considered EOL, they
> didn’t
> > bother to mention 1.x were affected. It is only now that 1.x’s getting
> > interest that they did this.
> >
> > Sean Busbey 於 2022年1月22日 週六，上午2:47寫道：
> >
> > > It's relevant to what kind of mitigation this is. The effectiveness of
> > > reload4j to deal with "the critical CVEs of log4j 1" is limited by how
> > > likely it is that they know about them.
> > >
> > > Otherwise at the next CVE we're back in the same place where downstream
> > > users aren't meaningfully more protected. And in that case perhaps we
> > would
> > > do better for our users by e.g. putting more emphasis upgrading across
> > our
> > > releases or providing breaking changes to get the pain over with.
> > >
> > > On Fri, Jan 21, 2022 at 11:03 AM Andrew Purtell <
> > andrew.purt...@gmail.com>
> > > wrote:
> > >
> > > > That does not seem germane to this discussion. We don’t investigate
> and
> > > > attempt to manage the security reporting arrangement of any of our
> > other
> > > > third party dependencies.
> > > >
> > > > > On Jan 21, 2022, at 7:59 AM, Sean Busbey 
> wrote:
> > > > >
> > > > > Has anyone asked the ASF Logging PMC if they'll forward security
> > > reports
> > > > > against log4j 1 to the reload4j project?
> > > > >
> > > > >> On Fri, Jan 21, 2022 at 3:33 AM Pankaj Kumar <
> > pankajku...@apache.org>
> > > > wrote:
> > > > >>
> > > > >> +1 for reload4j.
> > > > >>
> > > > >> Regards,
> > > > >> Pankaj
> > > > >>
> > > > >>> On Fri, Jan 21, 2022, 2:39 PM 张铎(Duo Zhang) <
> palomino...@gmail.com
> > >
> > > > wrote:
> > > > >>>
> > > > >>> Already filed HBASE-26691.
> > > > >>>
> > > > >>> Wei-Chiu Chuang  于2022年1月21日周五 16:53写道：
> > > > >>>
> > > >  +1 I am doing the same in Hadoop.
> > > > 
> > > >  On Fri, Jan 21, 2022 at 4:51 PM Viraj Jasani <
> vjas...@apache.org>
> > > > >> wrote:
> > > > 
> > > > > +1 for Reload4J migration in active release branches.
> > > > >
> > > > >
> > > > > On Fri, 21 Jan 2022 at 12:52 PM, Andrew Purtell <
> > > >  andrew.purt...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> +1 for migrating to Reload4J. It is binary and configuration
> > > > >>> compatible
> > > > >> with log4j 1 so meets our compatibility guidelines.
> > > > >>
> > > > >> If this is an agreeable plan I can make the changes in a PR
> and
> > we
> > > > >>> can
> > > >  do
> > > > >> a round of new releases.
> > > > >>
> > > > >>> On Jan 20, 2022, at 10:16 PM, Duo Zhang  >
> > > > >>> wrote:
> > > > >>>
> > > > >>> On master we have already migrated to log4j2, but for all
> > other
> > > > > release
> > > > >>> lines we are still on log4j1.
> > > > >>>
> > > > >>> Recently there are several new CVEs for log4j1, so I think we
> > > > >>> should
> > > > > also
> > > > >>> 

Re: [VOTE] Second release candidate for hbase-operator-tools 1.2.0 is available for download

[Zach York]

Hey Guangxu,

I noticed recently that there is no rel/1.2.0 tag on github for
hbase-operator tools. Was that step missed while releasing?

Thanks,
Zach

On Thu, Dec 23, 2021 at 11:44 PM Guangxu Cheng  wrote:

> With 3 binding +1 votes, 1 non-binding +1 votes, and no 0 or -1 votes, the
> vote passes.
>
> Thanks to all who voted on the release candidate!
>
> Let me push the release out.
> --
> Best Regards,
> Guangxu
>
>
> Guangxu Cheng  于2021年12月24日周五 15:39写道：
>
> > +1 from me
> > --
> > Best Regards,
> > Guangxu
> >
> >
> > Josh Elser  于2021年12月23日周四 05:00写道：
> >
> >> +1 (binding)
> >>
> >> * xsums/sigs OK
> >> * apache-rat:check OK
> >> * Can build from src
> >> * Ran all UT
> >> * Log4j 2.17 in use
> >>
> >> Looks great, Guangxu!
> >>
> >> On 12/20/21 1:15 AM, Guangxu Cheng wrote:
> >> > Please vote on this Apache hbase operator tools release candidate,
> >> > hbase-operator-tools-1.2.0RC1
> >> >
> >> > The VOTE will remain open for at least 72 hours.
> >> >
> >> > [ ] +1 Release this package as Apache hbase operator tools 1.2.0
> >> > [ ] -1 Do not release this package because ...
> >> >
> >> > The tag to be voted on is 1.2.0RC1:
> >> >
> >> >https://github.com/apache/hbase-operator-tools/tree/1.2.0RC1
> >> >
> >> > This tag currently points to git reference
> >> >
> >> >478af00af79f82624264fd2bb447b97fecc8e790
> >> >
> >> > The release files, including signatures, digests, as well as
> CHANGES.md
> >> > and RELEASENOTES.md included in this RC can be found at:
> >> >
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hbase/hbase-operator-tools-1.2.0RC1
> >> >
> >> > Maven artifacts are available in a staging repository at:
> >> >
> >> >https://repository.apache.org/content/repositories/org apache
> >> hbase-1479
> >> > <
> https://repository.apache.org/content/repositories/orgapachehbase-1479
> >> >
> >> > Artifacts were signed with the 5EF3A66D57EC647A key which can be found
> >> in:
> >> >
> >> >https://downloads.apache.org/hbase/KEYS
> >> >
> >> > hbase-operator-tools 1.2.0 contains a critical security fix for
> >> addressing
> >> > the log4j2
> >> > CVE-2021-44228 and CVE-2021-45105. All users who use
> >> hbase-operator-tools
> >> > should upgrade to hbase-operator-tools 1.2.0 ASAP.
> >> >
> >> > To learn more about Apache hbase operator tools, please see
> >> >
> >> >http://hbase.apache.org/
> >> >
> >> > Thanks,
> >> > Your HBase Release Manager
> >> > --
> >> > Best Regards,
> >> > Guangxu
> >> >
> >>
> >
>

Re: [ANNOUNCE] New HBase committer Zhuoyue Huang(GeorryHuang)

[Zach York]

Congratulations and welcome!

On Thu, Oct 14, 2021 at 10:44 AM Jan Hentschel <
jan.hentsc...@ultratendency.com> wrote:

> Congratulations and welcome!
>
> From: Guanghao Zhang 
> Date: Thursday, October 14, 2021 at 9:14 AM
> To: HBase Dev List , Hbase-User <
> u...@hbase.apache.org>
> Subject: [ANNOUNCE] New HBase committer Zhuoyue Huang(GeorryHuang)
> [CAUTION] The sender of this email is outside our organization. Please DO
> NOT CLICK links, download attachments or respond unless you recognize the
> sender and know the content is safe. Please contact IT immediately in case
> you find it suspicious.
>
> Folks,
>
> On behalf of the Apache HBase PMC I am pleased to announce that Zhuoyue
> Huang has accepted the PMC's invitation to become a committer on the
> project.
>
> We appreciate all of the great contributions Zhuoyue Huang has made to the
> community thus far and we look forward to his continued involvement.
>
> Allow me to be the first to congratulate Zhuoyue Huang on his new role!
>
> Thanks.
>

[jira] [Created] (HBASE-26320) Separate Log Cleaner DirScanPool to prevent the OLDWALs from filling up the disk when archive is large

[Zach York (Jira)]

Zach York created HBASE-26320:
-

 Summary: Separate Log Cleaner DirScanPool to prevent the OLDWALs 
from filling up the disk when archive is large
 Key: HBASE-26320
 URL: https://issues.apache.org/jira/browse/HBASE-26320
 Project: HBase
  Issue Type: Improvement
  Components: Operability
Affects Versions: 2.4.6, 1.7.1
Reporter: Zach York
Assignee: Zach York


We currently share the DirScanPool (threadpool for scanning for files to delete 
in the OldLogs and archive directories) between the LogCleaner and 
HFileCleaner. This means that if the archive directory is large/has lots of 
files/directories, the threads can get stuck scanning through the archive 
directory, starving the LogCleaner. This is especially apparent on S3 where 
list can be slower than on HDFS.

This JIRA creates separate DirScanPools for the LogCleaner and HFileCleaner



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-26257) Improve Performance of HBCK when specifying a subset of tables

[Zach York (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zach York resolved HBASE-26257.
---
Fix Version/s: hbase-operator-tools-1.2.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

Pushed to hbase-operator-tools. I might backport to branch-1, but I will open a 
new JIRA.

> Improve Performance of HBCK when specifying a subset of tables
> --
>
> Key: HBASE-26257
> URL: https://issues.apache.org/jira/browse/HBASE-26257
> Project: HBase
>  Issue Type: Improvement
>  Components: hbase-operator-tools, hbck2
>Affects Versions: hbase-operator-tools-1.1.0
>    Reporter: Zach York
>Assignee: Zach York
>Priority: Major
> Fix For: hbase-operator-tools-1.2.0
>
>
> On clusters with a large number of tables, listing all table directories from 
> the filesystem can take a lot of time (especially on S3). We should avoid 
> listing all the tables if we pass in a subset of the tables



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26257) Improve Performance of HBCK when specifying a subset of tables

[Zach York (Jira)]

Zach York created HBASE-26257:
-

 Summary: Improve Performance of HBCK when specifying a subset of 
tables
 Key: HBASE-26257
 URL: https://issues.apache.org/jira/browse/HBASE-26257
 Project: HBase
  Issue Type: Improvement
  Components: hbase-operator-tools, hbck2
Affects Versions: hbase-operator-tools-1.1.0
Reporter: Zach York
Assignee: Zach York


On clusters with a large number of tables, listing all table directories from 
the filesystem can take a lot of time (especially on S3). We should avoid 
listing all the tables if we pass in a subset of the tables



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New HBase committer Baiqiang Zhao

[Zach York]

Congratulations and welcome Baiqiang!

On Mon, Jul 12, 2021 at 4:59 AM Baiqiang Zhao  wrote:

> Thank you all for your warm welcome!
>
> Peter Somogyi  于2021年7月12日周一 下午7:46写道：
>
> > Congratulations!
> >
> > On Mon, Jul 12, 2021 at 12:53 PM Yu Li  wrote:
> >
> > > Congrats and welcome, Baiqiang.
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Mon, 12 Jul 2021 at 16:56, 哈晓琳  wrote:
> > >
> > > > Congratulations Baiqiang!
> > > >
> > > > Balazs Meszaros  于2021年7月12日周一
> > > > 下午4:35写道：
> > > >
> > > > > Congratulations Baiqiang!
> > > > >
> > > > > On Mon, Jul 12, 2021 at 9:22 AM Pankaj Kumar <
> pankajku...@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > Congratulations Baiqiang.!!
> > > > > >
> > > > > > Regards,
> > > > > > Pankaj
> > > > > >
> > > > > > On Sun, Jul 11, 2021, 12:48 AM Nick Dimiduk  >
> > > > wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > On behalf of the Apache HBase PMC I am pleased to announce that
> > > > > Baiqiang
> > > > > > > Zhao has accepted the PMC's invitation to become a committer on
> > the
> > > > > > > project!
> > > > > > >
> > > > > > > We appreciate all of the great contributions Baiqiang has made
> to
> > > > > > > the community thus far and we look forward to his continued
> > > > > involvement.
> > > > > > >
> > > > > > > Allow me to be the first to congratulate Baiqiang on his new
> > role!
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Nick
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[jira] [Resolved] (HBASE-26072) Upgrade hbase version in hbase-operator-tools to 2.4.4

[Zach York (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zach York resolved HBASE-26072.
---
Fix Version/s: hbase-operator-tools-1.2.0
   Resolution: Fixed

> Upgrade hbase version in hbase-operator-tools to 2.4.4
> --
>
> Key: HBASE-26072
> URL: https://issues.apache.org/jira/browse/HBASE-26072
> Project: HBase
>  Issue Type: Bug
>  Components: hbase-operator-tools
>Affects Versions: hbase-operator-tools-1.2.0
>    Reporter: Zach York
>Assignee: Zach York
>Priority: Trivial
> Fix For: hbase-operator-tools-1.2.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26072) Upgrade hbase version in hbase-operator-tools to 2.4.4

[Zach York (Jira)]

Zach York created HBASE-26072:
-

 Summary: Upgrade hbase version in hbase-operator-tools to 2.4.4
 Key: HBASE-26072
 URL: https://issues.apache.org/jira/browse/HBASE-26072
 Project: HBase
  Issue Type: Bug
  Components: hbase-operator-tools
Affects Versions: hbase-operator-tools-1.2.0
Reporter: Zach York
Assignee: Zach York






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-26054) Fix hbase-operator-tools build with HBase 2.4.4

[Zach York (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zach York resolved HBASE-26054.
---
Resolution: Fixed

> Fix hbase-operator-tools build with HBase 2.4.4
> ---
>
> Key: HBASE-26054
> URL: https://issues.apache.org/jira/browse/HBASE-26054
> Project: HBase
>  Issue Type: Bug
>  Components: hbase-operator-tools
>Affects Versions: hbase-operator-tools-1.2.0
>    Reporter: Zach York
>Assignee: Zach York
>Priority: Minor
> Fix For: hbase-operator-tools-1.2.0
>
>
> I tried building hbase-operator-tools with HBase 2.4.4 and encountered a 
> number of issues. Opening this for the fix.
> {noformat}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile 
> (default-compile) on project hbase-hbck2: Compilation failure: Compilation 
> failure:
> [ERROR] 
> /Users/zyork/workspace/apache/hbase-operator-tools/hbase-hbck2/src/main/java/org/apache/hbase/HBCKAbstractFileStatusFilter.java:[20,39]
>  package edu.umd.cs.findbugs.annotations does not exist
> [ERROR] 
> /Users/zyork/workspace/apache/hbase-operator-tools/hbase-hbck2/src/main/java/org/apache/hbase/HBCKAbstractFileStatusFilter.java:[52,46]
>  cannot find symbol
> [ERROR]   symbol:   class CheckForNull
> [ERROR]   location: class org.apache.hbase.HBCKAbstractFileStatusFilter
> [ERROR] 
> /Users/zyork/workspace/apache/hbase-operator-tools/hbase-hbck2/src/main/java/org/apache/hbase/HBCKAbstractFileStatusFilter.java:[64,44]
>  cannot find symbol
> [ERROR]   symbol:   class CheckForNull
> [ERROR]   location: class org.apache.hbase.HBCKAbstractFileStatusFilter
> [ERROR] 
> /Users/zyork/workspace/apache/hbase-operator-tools/hbase-hbck2/src/main/java/org/apache/hbase/HBCKAbstractFileStatusFilter.java:[68,49]
>  cannot find symbol
> [ERROR]   symbol:   class CheckForNull
> [ERROR]   location: class org.apache.hbase.HBCKAbstractFileStatusFilter
> [ERROR] 
> /Users/zyork/workspace/apache/hbase-operator-tools/hbase-hbck2/src/main/java/org/apache/hbase/HBCKFsUtils.java:[367,39]
>  cannot find symbol
> [ERROR]   symbol:   class CheckForNull
> [ERROR]   location: class org.apache.hbase.HBCKFsUtils.FamilyDirFilter
> [ERROR] 
> /Users/zyork/workspace/apache/hbase-operator-tools/hbase-hbck2/src/main/java/org/apache/hbase/HBCKFsUtils.java:[402,39]
>  cannot find symbol
> [ERROR]   symbol:   class CheckForNull
> [ERROR]   location: class org.apache.hbase.HBCKFsUtils.RegionDirFilter
> [ERROR] -> [Help 1]
> {noformat}
> The Threads change is masked by these changes, but the method is clearly 
> removed in HBASE-24750
> `



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26055) Master local region should be considered as a system table

[Zach York (Jira)]

Zach York created HBASE-26055:
-

 Summary: Master local region should be considered as a system table
 Key: HBASE-26055
 URL: https://issues.apache.org/jira/browse/HBASE-26055
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 2.4.4, 3.0.0-alpha-1
Reporter: Zach York
Assignee: Zach York


The title says most of it. The master local region is not a typical 
Table/Region, however, we should consider it a system table when creating the 
TableName so that if we call isSystemTable() it returns true.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26054) Fix hbase-operator-tools build with HBase 2.4.4

[Zach York (Jira)]

Zach York created HBASE-26054:
-

 Summary: Fix hbase-operator-tools build with HBase 2.4.4
 Key: HBASE-26054
 URL: https://issues.apache.org/jira/browse/HBASE-26054
 Project: HBase
  Issue Type: Bug
  Components: hbase-operator-tools
Affects Versions: hbase-operator-tools-1.2.0
Reporter: Zach York
Assignee: Zach York


I tried building hbase-operator-tools with HBase 2.4.4 and encountered a number 
of issues. Opening this for the fix.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New HBase PMC Huaxiang Sun

[Zach York]

Congrats Huaxiang!

On Tue, Apr 13, 2021 at 12:01 PM Stack  wrote:

> Hurray Huaxiang!
> S
>
> On Tue, Apr 13, 2021 at 12:39 AM Viraj Jasani  wrote:
>
> > On behalf of the Apache HBase PMC I am pleased to announce that Huaxiang
> > Sun has accepted our invitation to become a PMC member on the HBase
> > project. We appreciate Huaxiang stepping up to take more responsibility
> for
> > the project.
> >
> > Congratulations and welcome, Huaxiang!
> >
>

Re: [ANNOUNCE] New HBase committer Geoffrey Jacoby

[Zach York]

Congrats Geoffrey!

On Tue, Apr 13, 2021 at 9:21 AM Esteban Gutierrez
 wrote:

> Congrats and welcome, Geoffrey!
>
> --
> Cloudera, Inc.
>
>
>
> On Tue, Apr 13, 2021 at 8:18 AM zheng wang <18031...@qq.com> wrote:
>
> > Congratulations!
> >
> >
> >
> >
> > --原始邮件--
> > 发件人:
> >   "user"
> > <
> > vjas...@apache.org;
> > 发送时间:2021年4月9日(星期五) 晚上7:53
> > 收件人:"hbase-dev" > u...@hbase.apache.org;
> >
> > 主题:[ANNOUNCE] New HBase committer Geoffrey Jacoby
> >
> >
> >
> > On behalf of the Apache HBase PMC I am pleased to announce that Geoffrey
> > Jacoby has accepted the PMC's invitation to become a committer on the
> > project.
> >
> > Thanks so much for the work you've been contributing. We look forward to
> > your continued involvement.
> >
> > Congratulations and welcome, Geoffrey!
>

Re: Project management (JIRA fix version tracking) is in crisis

[Zach York]

Awesome Viraj! Also if we have confidence in the script we could probably
add it to run during a nightly run to try to catch issues earlier and
remove the burden on the RM (if there isn't too much noise from open
issues).

On Fri, Jan 15, 2021 at 10:55 AM Andrew Purtell  wrote:

> Thanks, Viraj. A script like that will be super helpful.
>
> On Fri, Jan 15, 2021 at 10:50 AM Viraj Jasani  wrote:
>
> > > This is not something RMs have had to do
> > > in the past and it asks a lot of them.
> >
> > +1. It is too much for RM to do if RM alone is responsible for
> > maintaining Git/Jira in sync.
> >
> > The reason why I was able to get the diffs for 2.3.4 and 2.4.1
> > RCs is because I used a small script that a) goes through all
> > commits, b) gets Jira number, c) search for the Jira fields using it's
> > API and d) finds the diff. And similarly using JQL, it finds all Jiras
> > with expected fixVersion but has no git commit present. Mostly I
> > will raise PR for this script tomorrow early as part of HBASE-25514.
> >
> > If we have enough trust on this, when RM given heads up on
> > dev mailing list for a new release, more committers can be
> > encouraged to start using this script to find any discrepancy, resolve
> > it if possible and keep in mind the upcoming release version while
> > resolving a Jira.
> > However, this is just a script that can help specifically just before
> > spinning RCs. More precautionary steps would be bringing
> > awareness of all active branches and how they relate to active
> > releases as mentioned by Andrew.
> >
> >
> > On 2021/01/15 18:39:18, Andrew Purtell  wrote:
> > > We do not have a single source of truth as to what are active branches
> > and
> > > what was the most recent release by version number made from each
> branch.
> > > If we had such a resource, then committers could refer to it whenever
> > > merging PRs or cherry picking commits backward. Perhaps something can
> be
> > > created that takes the union of git branch listing and reporter.a.o
> > release
> > > version history. The downside of course it this would be something new
> to
> > > maintain. Committers would also need to be educated about the existence
> > of
> > > this tool and the requirement to use it, but that could be solved with
> an
> > > update to the How To Release section of the online book. I filed
> > > HBASE-25515 for this idea.
> > >
> > >
> > >
> > > On Fri, Jan 15, 2021 at 10:05 AM Andrew Purtell 
> > wrote:
> > >
> > > > I have now had to sink two 2.4.1 RCs because of errors in the change
> > log.
> > > >
> > > > I made a pass over git history and ensured every commit was
> included. I
> > > > had also made a pass over JIRA to move out any unresolved issues or
> > > > complete the resolution of same. What I did not do is check that
> every
> > > > resolved JIRA corresponded to an actual commit. This is not something
> > RMs
> > > > have had to do in the past and it asks a lot of them.
> > > >
> > > > I know NOW that as RM I cannot currently trust committers to get fix
> > > > versions right or care about this.
> > > >
> > > > That's right... Commtters cannot be trusted to correctly maintain
> issue
> > > > metadata in JIRA.
> > > >
> > > > That is not a good situation for the project to be in. Up until now
> it
> > has
> > > > not been the responsibility of the RM to check each and every JIRA
> > status.
> > > > It has been the collective responsibility of committers to care about
> > the
> > > > project's release tracking insofar as to correctly update fix
> versions
> > in
> > > > JIRA. For releases containing relatively few changes, like 2.4.1,
> with
> > ~50
> > > > changes, I suppose it is possible for the RM to remove all 2.4.1 fix
> > > > versions, walk the commit history, and set back fix versions on JIRA
> to
> > > > actually correspond with what was truly committed. However, for minor
> > > > releases, with hundreds of commits, this will not be possible.
> > > >
> > > > I think the root cause is GitHub and JIRA are two separate change
> > tracking
> > > > systems with only a minimal amount of integration. It requires manual
> > > > effort. More and more, new committers are familiar with GitHub and
> PRs
> > and
> > > > are not familiar with JIRA and the Apache way of using JIRA to build
> > change
> > > > logs. We need to better educate new and existing committers on their
> > > > responsibilities with regards to maintaining JIRA metadata correctly.
> > > >
> > > > --
> > > > Best regards,
> > > > Andrew
> > > >
> > > > Words like orphans lost among the crosstalk, meaning torn from
> truth's
> > > > decrepit hands
> > > >- A23, Crosstalk
> > > >
> > >
> > >
> > > --
> > > Best regards,
> > > Andrew
> > >
> > > Words like orphans lost among the crosstalk, meaning torn from truth's
> > > decrepit hands
> > >- A23, Crosstalk
> > >
> >
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, 

[jira] [Resolved] (HBASE-25362) Quoting in hbase-vote script fails build with empty ""

[Zach York (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zach York resolved HBASE-25362.
---
Fix Version/s: 3.0.0-alpha-1
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Quoting in hbase-vote script fails build with empty ""
> --
>
> Key: HBASE-25362
> URL: https://issues.apache.org/jira/browse/HBASE-25362
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.4.1
>Reporter: Zach York
>Assignee: Zach York
>Priority: Minor
> Fix For: 3.0.0-alpha-1
>
>
> HBASE-25085 added support for maven options to be passed into the hbase-vote 
> script. However, with the quoting on my machine it fails if I do not pass in 
> any maven opts:
>   * Signature: ok
> * Checksum : ok
> * Rat check (1.8.0_92): failed
>  - mvn clean apache-rat:check ""
> * Built from source (1.8.0_92): failed
>  - mvn clean install -DskipTests ""
> * Unit tests pass (1.8.0_92): failed
>  - mvn package -P runAllTests "" -Dsurefire.rerunFailingTestsCount=3
> Notice the "" and the build failed due to unknown lifecycle ""



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] First release candidate for HBase 2.4.0 (RC1) is available

[Zach York]

Seems reasonable to at least have an easy option for running the
permutations even if it's not the default.

On Mon, Dec 7, 2020 at 3:23 PM Andrew Purtell  wrote:

> > The dev-support/hbase-vote.sh has not been updated to match that logic.
>
> Ok, shall we at least do that? Seems like a good idea, although of course
> the voter's resources will be consumed for a longer period of time.
>
>
> On Mon, Dec 7, 2020 at 2:45 PM Nick Dimiduk  wrote:
>
> > On Mon, Dec 7, 2020 at 2:30 PM Andrew Purtell 
> wrote:
> >
> > > Do we have a list of optional build profiles that are required or
> > > recommended for release or release testing? I might have missed it.
> > >
> > > If we do not have one, shall we start one?
> > >
> > > If we should start a list of optional build profiles that the RM and
> > > release testers should ensure are successful, what should be on it?
> > Should
> > > Java 11 and Hadoop 3 be on it?
> > >
> > > I am in favor of starting this practice with this release and this RC.
> > >
> >
> > These permutations are codified on a per-branch basis in our Jenkinsfile.
> > The dev-support/hbase-vote.sh has not been updated to match that logic.
> >
> > Should we break this question out into a DISCUSS thread?
> > >
> >
> > I don't think it's necessary, though I suppose the details are not well
> > socialized.
> >
> > On Mon, Dec 7, 2020 at 1:52 PM Nick Dimiduk  wrote:
> > >
> > > > Has anyone successfully built/run this RC with JDK11 and Hadoop3
> > profile?
> > >
> >
> > Presumably this test does run with JDK11 + Hadoop3, given HBase Nightly /
> > branch-2.4 / #3 [0] was green.
> >
> > [0]:
> >
> >
> https://ci-hadoop.apache.org/blue/organizations/jenkins/HBase%2FHBase%20Nightly/detail/branch-2.4/3/pipeline/102
> >
> > > I'm seeing test failures locally in the hbase-asyncfs module.
> > > > Reproducible with:
> > > >
> > > > $
> > > >
> > > >
> > >
> >
> JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home
> > > > mvn clean install -Dhadoop.profile=3.0
> > > >
> > >
> >
> -Dtest=org.apache.hadoop.hbase.io.asyncfs.TestFanOutOneBlockAsyncDFSOutput
> > > > ...
> > > > [INFO] Running
> > > > org.apache.hadoop.hbase.io.asyncfs.TestFanOutOneBlockAsyncDFSOutput
> > > >
> > > >
> > > > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time
> elapsed:
> > > > 1.785 s <<< FAILURE! - in
> > > > org.apache.hadoop.hbase.io.asyncfs.TestFanOutOneBlockAsyncDFSOutput
> > > >
> > > > [ERROR] org.apache.hadoop.hbase.io
> > > > .asyncfs.TestFanOutOneBlockAsyncDFSOutput
> > > >  Time elapsed: 1.775 s  <<< ERROR!
> > > >
> > > > java.lang.ExceptionInInitializerError
> > > >
> > > >
> > > > at
> > > > org.apache.hadoop.hbase.io
> > > >
> > >
> >
> .asyncfs.TestFanOutOneBlockAsyncDFSOutput.setUp(TestFanOutOneBlockAsyncDFSOutput.java:87)
> > > >
> > > > Caused by: java.lang.IllegalArgumentException: Invalid Java version
> > > > 11.0.9.1
> > > >
> > > > at
> > > > org.apache.hadoop.hbase.io
> > > >
> > >
> >
> .asyncfs.TestFanOutOneBlockAsyncDFSOutput.setUp(TestFanOutOneBlockAsyncDFSOutput.java:87)
> > > >
> > > > On Thu, Dec 3, 2020 at 4:05 PM Andrew Purtell 
> > > wrote:
> > > >
> > > > > Please vote on this Apache hbase release candidate, hbase-2.4.0RC1
> > > > >
> > > > > The VOTE will remain open for at least 72 hours.
> > > > >
> > > > > [ ] +1 Release this package as Apache hbase 2.4.0
> > > > > [ ] -1 Do not release this package because ...
> > > > >
> > > > > The tag to be voted on is 2.4.0RC1:
> > > > >
> > > > > https://github.com/apache/hbase/tree/2.4.0RC1
> > > > >
> > > > > The release files, including signatures, digests, as well as
> > CHANGES.md
> > > > > and RELEASENOTES.md included in this RC can be found at:
> > > > >
> > > > > https://dist.apache.org/repos/dist/dev/hbase/2.4.0RC1/
> > > > >
> > > > > Customarily Maven artifacts would be available in a staging
> > repository.
> > > > > Unfortunately I was forced to terminate the Maven deploy step after
> > > > > the upload ran for more than four hours and my build equipment
> > > > > needed to be relocated, with loss of network connectivity. This RC
> > has
> > > > > been delayed long enough. A temporary Maven repository is not a
> > > > > requirement for a vote. I will retry Maven deploy tomorrow. I can
> > > > > promise the artifacts for this RC will be staged in Apache Nexus
> and
> > > > > ready for release well ahead of the earliest possible time this
> vote
> > > > > can complete.
> > > > >
> > > > > Artifacts were signed with the apurt...@apache.org key which can
> be
> > > > found
> > > > > in:
> > > > >
> > > > > https://dist.apache.org/repos/dist/release/hbase/KEYS
> > > > >
> > > > > The API compatibility report for this RC can be found at:
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hbase/2.4.0RC1/api_compare_2.4.0RC1_to_2.3.0.html
> > > > >
> > > > > The changes are mostly added methods, which conform to the
> > > compatibility
> > > > 

Re: [ANNOUNCE] New HBase committer Xin Sun

[Zach York]

Congratulations Xin Sun!

On Fri, Dec 4, 2020 at 4:51 PM Andrew Purtell  wrote:

> Congratulations and welcome!
>
> On Thu, Dec 3, 2020 at 1:13 AM Guanghao Zhang  wrote:
>
> > Folks,
> >
> > On behalf of the Apache HBase PMC I am pleased to announce that Xin Sun
> has
> > accepted the PMC's invitation to become a committer on the project.
> >
> > We appreciate all of the great contributions Xin Sun has made to the
> > community thus far and we look forward to his continued involvement.
> >
> > Allow me to be the first to congratulate Xin Sun on his new role!
> >
> > Thanks.
> >
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

Re: [ANNOUNCE] New HBase committer Yulin Niu

[Zach York]

Congratulations Yulin!

On Fri, Dec 4, 2020 at 4:51 PM Andrew Purtell  wrote:

> Congratulations and welcome!
>
> On Thu, Dec 3, 2020 at 1:11 AM Guanghao Zhang  wrote:
>
> > Folks,
> >
> > On behalf of the Apache HBase PMC I am pleased to announce that Yulin Niu
> > has accepted the PMC's invitation to become a committer on the project.
> >
> > We appreciate all of the great contributions Yulin has made to the
> > community thus far and we look forward to his continued involvement.
> >
> > Allow me to be the first to congratulate Yulin on his new role!
> >
> > Thanks.
> >
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

[jira] [Created] (HBASE-25362) Quoting in hbase-vote script fails build with empty ""

[Zach York (Jira)]

Zach York created HBASE-25362:
-

 Summary: Quoting in hbase-vote script fails build with empty ""
 Key: HBASE-25362
 URL: https://issues.apache.org/jira/browse/HBASE-25362
 Project: HBase
  Issue Type: Bug
Affects Versions: 3.0.0-alpha-1, 2.4.1
Reporter: Zach York
Assignee: Zach York


HBASE-25085 added support for maven options to be passed into the hbase-vote 
script. However, with the quoting on my machine it fails if I do not pass in 
any maven opts:

  * Signature: ok
* Checksum : ok
* Rat check (1.8.0_92): failed
 - mvn clean apache-rat:check ""
* Built from source (1.8.0_92): failed
 - mvn clean install -DskipTests ""
* Unit tests pass (1.8.0_92): failed
 - mvn package -P runAllTests "" -Dsurefire.rerunFailingTestsCount=3

Notice the "" and the build failed due to unknown lifecycle ""



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24923) Running InitMetaProcedures are missed if Hmaster startup happens after Meta assign

[Zach York (Jira)]

Zach York created HBASE-24923:
-

 Summary: Running InitMetaProcedures are missed if Hmaster startup 
happens after Meta assign
 Key: HBASE-24923
 URL: https://issues.apache.org/jira/browse/HBASE-24923
 Project: HBase
  Issue Type: Bug
Affects Versions: 3.0.0-alpha-1
Reporter: Zach York


In HMaster startup, if the InitMetaProcedure was stopped after the meta assign 
stage, it will have a RegionState assigned to it and will not fall into this if 
statement: 
https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java#L1044.
 

This means if an InitMetaProcedure is running (in the 
INIT_META_CREATE_NAMESPACES state), we won't look for any instances of running 
InitMetaProcedures and won't correctly await until they are done: 
https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java#L1065

The check for running InitMetaProcedures should be moved outside of the if 
statement and only if it is null should it try to add a new InitMetaProcedure 
(if there is no RegionState for meta or if meta is offline). This way we will 
correctly wait for procedures that are running that have passed the 
INIT_META_ASSIGN_META state (which sets the RegionStates that HMaster is 
looking at).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24922) InitMetaProcedure create Namespace is not Idempotent

[Zach York (Jira)]

Zach York created HBASE-24922:
-

 Summary: InitMetaProcedure create Namespace is not Idempotent
 Key: HBASE-24922
 URL: https://issues.apache.org/jira/browse/HBASE-24922
 Project: HBase
  Issue Type: Bug
  Components: meta, Region Assignment
Affects Versions: 3.0.0-alpha-1
Reporter: Zach York


If InitMetaProcedure is retried after creating at least one namespace 
directory: 
https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/InitMetaProcedure.java#L115,
 it will not be able to recover as the procedure does not check for existence 
of the directories before attempting to create them.

We should check for existence of the directory first and only execute 
createDirectories if they do not exist. The rest of the procedure looks like it 
is idempotent. We can additionally optimize to not overwrite the column 
families.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: EOL 1.3.x?

[Zach York]

+1

Do we typically do a final release or can we just EOL as is?

On Tue, Aug 11, 2020 at 9:43 AM Geoffrey Jacoby  wrote:

> +1 (non-binding)
>
> Geoffrey
>
> On Tue, Aug 11, 2020 at 8:19 AM Josh Elser  wrote:
>
> > Sounds good to me.
> >
> > On 8/11/20 4:01 AM, 张铎(Duo Zhang) wrote:
> > > The last release for 1.3.x is 2019.10.20, which means we do not have a
> > > release for this release line for about 10 months.
> > >
> > > Let's make it EOL and tell users to at least upgrade to 1.4.x?
> > >
> > > Thanks.
> > >
> >
>

Re: [DISCUSS] Removing problematic terms from our project

[Zach York]

While reading the definitions, I think it is pretty clear that the
definition HBase is intending is somewhere under the #2 definition link.
HMaster is not a teacher (which implies learning on the "student" side),
but rather orders the RS to do a task.
I think master in this context is still worth changing to coordinator since
in my mind this is actually a more clear definition of what HMaster does.

On Mon, Jun 22, 2020 at 2:09 PM Geoffrey Jacoby  wrote:

> For most of the proposals (slave -> worker, blacklist -> denylist,
> whitelist-> allowlist), I'm +1 (nonbinding). Denylist and acceptlist even
> have the advantage of being clearer than the terms they're replacing.
>
> However, I'm not convinced about changing "master" to "coordinator", or
> something similar. Unlike "slave", which is negative in any context,
> "master" has many definitions, including some common ones which do not
> appear problematic. See https://www.merriam-webster.com/dictionary/master
> for
> examples. In particular, the progression of an artisan was from
> "apprentice" to "journeyman" to "master". A master smith, carpenter, or
> artist would run a shop managing lots of workers and apprentices who would
> hope to become masters of their own someday. So "master" and "worker" can
> still go together.
>
> Since it's the least problematic term, and by far the hardest term to
> change (both within HBase and with effects on downstream projects such as
> Ambari), I'm -0 (nonbinding) on changing "master".
>
> Geoffrey
>
> On Mon, Jun 22, 2020 at 1:32 PM Rushabh Shah
>  wrote:
>
> > +1 to renaming.
> >
> >
> > Rushabh Shah
> >
> >- Software Engineering SMTS | Salesforce
> >-
> >   - Mobile: 213 422 9052
> >
> >
> >
> > On Mon, Jun 22, 2020 at 1:18 PM Josh Elser  wrote:
> >
> > > +1
> > >
> > > On 6/22/20 4:03 PM, Sean Busbey wrote:
> > > > We should change our use of these terms. We can be equally or more
> > clear
> > > in
> > > > what we are trying to convey where they are present.
> > > >
> > > > That they have been used historically is only useful if the advantage
> > we
> > > > gain from using them through that shared context outweighs the
> > potential
> > > > friction they add. They make me personally less enthusiastic about
> > > > contributing. That's enough friction for me to advocate removing
> them.
> > > >
> > > > AFAICT reworking our replication stuff in terms of "active" and
> > "passive"
> > > > clusters did not result in a big spike of folks asking new questions
> > > about
> > > > where authority for state was.
> > > >
> > > > On Mon, Jun 22, 2020, 13:39 Andrew Purtell 
> > wrote:
> > > >
> > > >> In response to renewed attention at the Foundation toward addressing
> > > >> culturally problematic language and terms often used in technical
> > > >> documentation and discussion, several projects have begun
> discussions,
> > > or
> > > >> made proposals, or started work along these lines.
> > > >>
> > > >> The HBase PMC began its own discussion on private@ on June 9, 2020
> > > with an
> > > >> observation of this activity and this suggestion:
> > > >>
> > > >> There is a renewed push back against classic technology industry
> terms
> > > that
> > > >> have negative modern connotations.
> > > >>
> > > >> In the case of HBase, the following substitutions might be proposed:
> > > >>
> > > >> - Coordinator instead of master
> > > >>
> > > >> - Worker instead of slave
> > > >>
> > > >> Recommendations for these additional substitutions also come up in
> > this
> > > >> type of discussion:
> > > >>
> > > >> - Accept list instead of white list
> > > >>
> > > >> - Deny list instead of black list
> > > >>
> > > >> Unfortunately we have Master all over our code base, baked into
> > various
> > > >> APIs and configuration variable names, so for us the necessary
> changes
> > > >> amount to a new major release and deprecation cycle. It could well
> be
> > > worth
> > > >> it in the long run. We exist only as long as we draw a willing and
> > > >> sufficient contributor community. It also wouldn’t be great to have
> an
> > > >> activist fork appear somewhere, even if unlikely to be successful.
> > > >>
> > > >> Relevant JIRAs are:
> > > >>
> > > >> - HBASE-12677 <
> https://issues.apache.org/jira/browse/HBASE-12677
> > >:
> > > >> Update replication docs to clarify terminology
> > > >> - HBASE-13852 <
> https://issues.apache.org/jira/browse/HBASE-13852
> > >:
> > > >> Replace master-slave terminology in book, site, and javadoc
> with a
> > > more
> > > >> modern vocabulary
> > > >> - HBASE-24576 <
> https://issues.apache.org/jira/browse/HBASE-24576
> > >:
> > > >> Changing "whitelist" and "blacklist" in our docs and project
> > > >>
> > > >> In response to this proposal, a member of the PMC asked if the term
> > > >> 'master' used by itself would be fine, because we only have use of
> > > 'slave'
> > > >> in replication documentation and that is easily addressed. In
> response
> > > to
> > > >> 

Re: [ANNOUNCE] New HBase committer Wei-Chiu Chuang

[Zach York]

Congratulations Wei-Chiu!

On Wed, May 13, 2020 at 12:15 PM Bharath Vissapragada 
wrote:

> Congrats, Wei-Chiu.
>
> On Wed, May 13, 2020 at 12:13 PM Andrew Purtell 
> wrote:
>
> > Congratulations and welcome Wei-Chiu!
> >
> > On Wed, May 13, 2020 at 12:10 PM Sean Busbey  wrote:
> >
> > > Folks,
> > >
> > > On behalf of the Apache HBase PMC I am pleased to announce that
> Wei-Chiu
> > > Chuang has accepted the PMC's invitation to become a committer on the
> > > project.
> > >
> > > We appreciate all of the great contributions Wei-Chiu has made to the
> > > community thus far and we look forward to his continued involvement.
> > >
> > > Allow me to be the first to congratulate Wei-Chiu on his new role!
> > >
> > > thanks,
> > > busbey
> > >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> >- A23, Crosstalk
> >
>

Re: Recent experience with Chaos Monkey?

[Zach York]

I should note that I was using HBase 2.2.3 to test.

On Thu, May 7, 2020 at 5:26 PM Zach York 
wrote:

> I recently ran ITBLL with Chaos monkey[1] against a real HBase
> installation (EMR). I initially tried to run it locally, but couldn't get
> it working and eventually gave up.
>
> > So I'm curious if this matches others' experience running the monkey. For
> example, do you have an environment more resilient than mine, one where an
> external actor is restarting downed processed without the monkey action's
> involvement?
>
> It actually performs even worse in this case in my experience since Chaos
> monkey can consider the failure mechanism to have failed (and eventually
> times out)
> because the process is too quick to recover (or the recovery fails because
> the process is already running). The only way I was able to get it to run
> was to disable
> the process that automatically restarts killed processes in my system.
>
> One other thing I hit was the validation for a suspended process was
> incorrect so if chaos monkey tried to suspend the process the run would
> fail. I'll put up a JIRA for that.
>
> This brings up a discussion on whether the ITBLL (or whatever process)
> should even continue if either a killing or recovering action failed. I
> would argue that invalidates the entire test,
> but it might not be obvious it failed unless you were watching the logs as
> it went.
>
> Thanks,
> Zach
>
>
> [1] sudo -u hbase hbase
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList -m serverKilling
> loop 4 2 100 ${RANDOM} 10
>
> On Thu, May 7, 2020 at 5:05 PM Nick Dimiduk  wrote:
>
>> Hello,
>>
>> Does anyone have recent experience running Chaos Monkey? Are you running
>> against an external cluster, or one of the other modes? What monkey
>> factory
>> are you using? Any property overrides? A non-default ClusterManager?
>>
>> I'm trying to run ITBLL with chaos against branch-2.3 and I'm not having
>> much luck. My environment is an "external" cluster, 4 racks of 4 hosts
>> each, the relatively simple "serverKilling" factory with
>> `rolling.batch.suspend.rs.ratio = 0.0`. So, randomly kill various hosts
>> on
>> various scheduled, plus some balancer play mixed in; no process
>> suspension.
>>
>> Running for any length of time (~30 minutes) the chaos monkey eventually
>> terminates between a majority and all of the hosts in the cluster. My logs
>> are peppered with warnings such as the below. There are other variants. As
>> far as I can tell, actions are intended to cause some harm and then
>> restore
>> state after themselves. In practice, the harm is successful but
>> restoration
>> rarely succeeds. Mostly these actions are "safeguarded" but this 60-sec
>> timeout. The result is a methodical termination of the cluster.
>>
>> So I'm curious if this matches others' experience running the monkey. For
>> example, do you have an environment more resilient than mine, one where an
>> external actor is restarting downed processed without the monkey action's
>> involvement? Is the monkey designed to run only in such an environment?
>> These timeouts are configurable; are you cranking them way up?
>>
>> Any input you have would be greatly appreciated. This is my last major
>> action item blocking initial 2.3.0 release candidates.
>>
>> Thanks,
>> Nick
>>
>> 20/05/05 21:19:29 WARN policies.Policy: Exception occurred during
>> performing action: java.io.IOException: did timeout 6ms waiting for
>> region server to start: host-a.example.com
>> at
>>
>> org.apache.hadoop.hbase.HBaseCluster.waitForRegionServerToStart(HBaseCluster.java:163)
>> at
>> org.apache.hadoop.hbase.chaos.actions.Action.startRs(Action.java:228)
>> at
>>
>> org.apache.hadoop.hbase.chaos.actions.RestartActionBaseAction.gracefulRestartRs(RestartActionBaseAction.java:70)
>> at
>>
>> org.apache.hadoop.hbase.chaos.actions.GracefulRollingRestartRsAction.perform(GracefulRollingRestartRsAction.java:61)
>> at
>>
>> org.apache.hadoop.hbase.chaos.policies.DoActionsOncePolicy.runOneIteration(DoActionsOncePolicy.java:50)
>> at
>>
>> org.apache.hadoop.hbase.chaos.policies.PeriodicPolicy.run(PeriodicPolicy.java:41)
>> at
>>
>> org.apache.hadoop.hbase.chaos.policies.CompositeSequentialPolicy.run(CompositeSequentialPolicy.java:42)
>> at java.base/java.lang.Thread.run(Thread.java:834)
>>
>

Re: Recent experience with Chaos Monkey?

[Zach York]

I recently ran ITBLL with Chaos monkey[1] against a real HBase installation
(EMR). I initially tried to run it locally, but couldn't get it working and
eventually gave up.

> So I'm curious if this matches others' experience running the monkey. For
example, do you have an environment more resilient than mine, one where an
external actor is restarting downed processed without the monkey action's
involvement?

It actually performs even worse in this case in my experience since Chaos
monkey can consider the failure mechanism to have failed (and eventually
times out)
because the process is too quick to recover (or the recovery fails because
the process is already running). The only way I was able to get it to run
was to disable
the process that automatically restarts killed processes in my system.

One other thing I hit was the validation for a suspended process was
incorrect so if chaos monkey tried to suspend the process the run would
fail. I'll put up a JIRA for that.

This brings up a discussion on whether the ITBLL (or whatever process)
should even continue if either a killing or recovering action failed. I
would argue that invalidates the entire test,
but it might not be obvious it failed unless you were watching the logs as
it went.

Thanks,
Zach


[1] sudo -u hbase hbase
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList -m serverKilling
loop 4 2 100 ${RANDOM} 10

On Thu, May 7, 2020 at 5:05 PM Nick Dimiduk  wrote:

> Hello,
>
> Does anyone have recent experience running Chaos Monkey? Are you running
> against an external cluster, or one of the other modes? What monkey factory
> are you using? Any property overrides? A non-default ClusterManager?
>
> I'm trying to run ITBLL with chaos against branch-2.3 and I'm not having
> much luck. My environment is an "external" cluster, 4 racks of 4 hosts
> each, the relatively simple "serverKilling" factory with
> `rolling.batch.suspend.rs.ratio = 0.0`. So, randomly kill various hosts on
> various scheduled, plus some balancer play mixed in; no process suspension.
>
> Running for any length of time (~30 minutes) the chaos monkey eventually
> terminates between a majority and all of the hosts in the cluster. My logs
> are peppered with warnings such as the below. There are other variants. As
> far as I can tell, actions are intended to cause some harm and then restore
> state after themselves. In practice, the harm is successful but restoration
> rarely succeeds. Mostly these actions are "safeguarded" but this 60-sec
> timeout. The result is a methodical termination of the cluster.
>
> So I'm curious if this matches others' experience running the monkey. For
> example, do you have an environment more resilient than mine, one where an
> external actor is restarting downed processed without the monkey action's
> involvement? Is the monkey designed to run only in such an environment?
> These timeouts are configurable; are you cranking them way up?
>
> Any input you have would be greatly appreciated. This is my last major
> action item blocking initial 2.3.0 release candidates.
>
> Thanks,
> Nick
>
> 20/05/05 21:19:29 WARN policies.Policy: Exception occurred during
> performing action: java.io.IOException: did timeout 6ms waiting for
> region server to start: host-a.example.com
> at
>
> org.apache.hadoop.hbase.HBaseCluster.waitForRegionServerToStart(HBaseCluster.java:163)
> at
> org.apache.hadoop.hbase.chaos.actions.Action.startRs(Action.java:228)
> at
>
> org.apache.hadoop.hbase.chaos.actions.RestartActionBaseAction.gracefulRestartRs(RestartActionBaseAction.java:70)
> at
>
> org.apache.hadoop.hbase.chaos.actions.GracefulRollingRestartRsAction.perform(GracefulRollingRestartRsAction.java:61)
> at
>
> org.apache.hadoop.hbase.chaos.policies.DoActionsOncePolicy.runOneIteration(DoActionsOncePolicy.java:50)
> at
>
> org.apache.hadoop.hbase.chaos.policies.PeriodicPolicy.run(PeriodicPolicy.java:41)
> at
>
> org.apache.hadoop.hbase.chaos.policies.CompositeSequentialPolicy.run(CompositeSequentialPolicy.java:42)
> at java.base/java.lang.Thread.run(Thread.java:834)
>

Re: [VOTE] First Release Candidate for HBase 1.6.0 is available for download

[Zach York]

+1 (binding)

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_92): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_92): ok
 - mvn clean install -DskipTests
* Unit tests pass (1.8.0_92): ok
 - mvn package -P runAllTests

On Tue, Feb 25, 2020 at 8:31 AM Balazs Meszaros
 wrote:

> +1
>
> - signatures, checksums: OK
> - rat check: OK
> - built from source: OK (8u242)
> - unit tests: OK (8u242)
> - ltt with 1M rows: OK
> - web UI: OK
> - shell and basic commands: OK
>
> On Tue, Feb 25, 2020 at 3:29 PM Viraj Jasani  wrote:
>
> > +1 (non-binding)
> >
> > Signature: ok
> > Checksum : ok
> > Rat check : ok
> >   - mvn clean apache-rat:check
> > Built from source: ok
> >   - mvn clean install -DskipTests
> > Unit tests pass: ok
> >   - mvn package -P runAllTests
> > CRUD with LTT (1 million rows): ok
> > Flush/Compaction with busy cluster: ok
> >
> > Below 3 tests failed in the first run but succeeded in all subsequent
> > multiple runs.
> >
> > [ERROR] Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> > 15.687 s <<< FAILURE! - in
> > org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy
> > [ERROR]
> >
> testFIFOCompactionPolicyExpiredEmptyHFiles(org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy)
> > Time elapsed: 4.139 s  <<< ERROR!
> >
> > [WARNING]
> >
> org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy.testFIFOCompactionPolicyExpiredEmptyHFiles(org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy)
> > [ERROR]   Run 1:
> > TestFIFOCompactionPolicy.testFIFOCompactionPolicyExpiredEmptyHFiles:236 »
> > IO j...
> > [INFO]   Run 2: PASS
> >
> >
> > [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> > 51.953 s <<< FAILURE! - in
> > org.apache.hadoop.hbase.client.TestClientOperationInterrupt
> > [ERROR]
> >
> testInterrupt50Percent(org.apache.hadoop.hbase.client.TestClientOperationInterrupt)
> > Time elapsed: 50.094 s  <<< FAILURE!
> > java.lang.AssertionError:  noEx: 64, badEx=0, noInt=0
> > at
> >
> org.apache.hadoop.hbase.client.TestClientOperationInterrupt.testInterrupt50Percent(TestClientOperationInterrupt.java:152)
> >
> >
> > [ERROR] Tests run: 29, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> > 295.629 s <<< FAILURE! - in org.apache.hadoop.hbase.client.TestAdmin1
> > [ERROR]
> >
> testRegionStateCleanupFromAMMemoryAfterRegionSplitAndMerge(org.apache.hadoop.hbase.client.TestAdmin1)
> > Time elapsed: 9.313 s  <<< FAILURE!
> > java.lang.AssertionError: expected:<1> but was:<3>
> > at
> >
> org.apache.hadoop.hbase.client.TestAdmin1.testRegionStateCleanupFromAMMemoryAfterRegionSplitAndMerge(TestAdmin1.java:1288)
> >
> >
> > On 2020/02/24 17:04:43, Andrew Purtell  wrote:
> > > Please vote on this release candidate (RC0) for Apache HBase 1.6.0.
> > >
> > > The VOTE will remain open for at least 72 hours.
> > >
> > > The tag to be voted on is '1.6.0RC0' (5ec5a5b115), available for
> > download at
> > > https://dist.apache.org/repos/dist/dev/hbase/1.6.0RC0/
> > >
> > > Maven artifacts are available in a staging repository at
> > > https://repository.apache.org/content/repositories/orgapachehbase-1383
> > >
> > > A list of the 73 issues resolved in this release can be found at
> > > https://s.apache.org/b7eml
> > >
> > > A list of exclude patters for the known flaky unit tests can be found
> at:
> > >
> >
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-1/
> > >
> > > A detailed source and binary compatibility report for this release is
> > > available at
> > >
> >
> https://dist.apache.org/repos/dist/dev/hbase/1.6.0RC0/api_compare_1.5.0_to_1.6.0RC0.html
> > > .
> > >
> > > There are no binary or source compatibility issues reported.
> > >
> > > Prior to making this announcement I made the following preflight
> checks:
> > >
> > > RAT check passes (7u80)
> > > Unit test suite passes (7u80, 8u232)
> > >
> > >
> > > --
> > > Best regards,
> > > Andrew
> > >
> >
>

Re: [VOTE] First Release Candidate for HBase 1.4.13 is available for download

[Zach York]

+1 (binding)

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_92): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_92): ok
 - mvn clean install -DskipTests
* Unit tests pass (1.8.0_92): ok
 - mvn package -P runAllTests

On Tue, Feb 25, 2020 at 4:39 AM Peter Somogyi  wrote:

> +1 (binding)
>
> Signatures: ok
> Checksums: ok
> Rat check: ok
> Built from source: ok
> Unit tests: ok
> Failed TestMasterBalanceThrottling, TestAdmin2,
> TestReplicationKillSlaveRS but multiple reruns were successful
> Compatibility report: ok
> Start in standalone mode: ok
> Web UI: ok
> LTT 1M rows: ok
>
> Peter
>
> On Tue, Feb 25, 2020 at 10:26 AM Viraj Jasani  wrote:
>
> > +1 (non-binding)
> >
> > Signature: ok
> > Checksum: ok
> > Rat check: ok
> > Built from source: ok
> > Unit tests pass: ok
> > Compatibility report: ok
> >
> >
> > On 2020/02/24 11:19:54, Sakthi  wrote:
> > > Please vote on this release candidate (RC0) for Apache HBase 1.4.13.
> > >
> > > The VOTE will remain open for at least 72 hours.
> > >
> > > [ ] +1 Release this package as Apache HBase 1.4.13
> > > [ ] -1 Do not release this package because ...
> > >
> > > The tag to be voted on is '1.4.13RC0' (38bf65a
> > > <
> >
> https://github.com/apache/hbase/commit/38bf65a22b7e9320f07aeb27677e4533b9a77ef4
> > >
> > > ):
> > > https://github.com/apache/hbase/tree/1.4.13RC0
> > >
> > > It's available for download at:
> > > https://dist.apache.org/repos/dist/dev/hbase/1.4.13RC0/
> > >
> > > Maven artifacts are available in a staging repository at:
> > >
> https://repository.apache.org/content/repositories/orgapachehbase-1384/
> > >
> > > Artifacts are signed with my key (851528A6) published in our KEYS file
> > at:
> > > https://dist.apache.org/repos/dist/release/hbase/KEYS
> > >
> > > A list of the 13 issues resolved in this release can be found at:
> > > https://s.apache.org/s8i5w
> > >
> > > A list of exclude patters for the known flakey tests can be found at:
> > >
> >
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-1.4/
> > >
> > > A detailed source and binary compatibility report for this release is
> > > available at:
> > >
> >
> https://dist.apache.org/repos/dist/dev/hbase/1.4.13RC0/compat-check-report.html
> > >
> > >
> > > *PREFLIGHT CHECKS DONE:*
> > > Java Version - Zulu 1.7.0_232-b6
> > >
> > > Fully-Distributed Mode (3 Nodes)
> > >
> > > Hadoop Version - 2.7.7
> > >
> > >
> > >
> > >-
> > >
> > >Built bin tarball tarball from Source
> > >-
> > >
> > >Checksums & Signatures - OK
> > >-
> > >
> > >RAT check - OK
> > >-
> > >
> > >ITBLL (8M) rows (& MR Tasks using Resource Manager [Verified]) - OK
> > >-
> > >
> > >LTT (5M) rows - OK
> > >-
> > >
> > >Shell [CRUD Operations] - OK
> > >-
> > >
> > >Web UI, Logs - OK
> > >-
> > >
> > >REST Server - OK
> > >-
> > >
> > >   CRUD Operations, Web UI, Logs - OK
> > >   -
> > >
> > >Thrift Server - OK
> > >-
> > >
> > >   CRUD Operations, Web UI, Logs - OK
> > >   -
> > >
> > >Compatibility Report - OK
> > >-
> > >
> > >HDFS directory structure - OK
> > >
> > > Java Version - Corretto 1.8.0_222-b10
> > > Standalone mode
> > >
> > >- Built bin tarball from Source
> > >-
> > >
> > >Shell [CRUD Operations] - OK
> > >-
> > >
> > >Web UI, Logs - OK
> > >
> > > To learn more about Apache HBase, please see http://hbase.apache.org/
> > >
> > > Thanks,
> > > Sakthi
> > >
> >
>

Re: [VOTE] First release candidate for HBase 2.1.9 is available for download

[Zach York]

+1 (binding)

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_92): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_92): ok
 - mvn clean install -DskipTests
* Unit tests pass (1.8.0_92): ok
 - mvn test -P runAllTests



On Tue, Feb 11, 2020 at 6:53 PM Duo Zhang  wrote:

> Please vote on this Apache hbase release candidate,
> hbase-2.1.9RC0
>
> The VOTE will remain open for at least 72 hours.
>
> [ ] +1 Release this package as Apache hbase 2.1.9
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 2.1.9RC0:
>
> https://github.com/apache/hbase/tree/2.1.9RC0
>
> The release files, including signatures, digests, as well as CHANGES.md
> and RELEASENOTES.md included in this RC can be found at:
>
>  https://dist.apache.org/repos/dist/dev/hbase/2.1.9RC0/
>
> Maven artifacts are available in a staging repository at:
>
>  https://repository.apache.org/content/repositories/orgapachehbase-1382/
>
> Artifacts were signed with the 9AD2AE49 key which can be found in:
>
>  https://dist.apache.org/repos/dist/release/hbase/KEYS
>
> 2.1.9 includes ~62 bug and improvement fixes done since the 2.1.8.
>
>  To learn more about apache hbase, please see
> http://hbase.apache.org/
>
> Thanks,
> Your HBase Release Manager
>

Re: [DISCUSS] HBASE-18095 - Remove zookeeper dependency in client codepaths

[Zach York]

I have reviewed the doc and it looks like a good feature!

+1

On Thu, Feb 6, 2020 at 11:47 AM Wellington Chevreuil <
wellington.chevre...@gmail.com> wrote:

> Thanks for clarifying on this, Bharath and Andrew. Sorry for the late
> reply, +1 for adding it into branch-2 as well as non-default registry.
>
> Em qui., 6 de fev. de 2020 às 00:59, Bharath Vissapragada <
> bhara...@apache.org> escreveu:
>
> > Whatever Andrew said. Wellington, I also addressed your comments in the
> > doc. Thanks for taking your time and going through it. Appreciate it.
> >
> > On Wed, Feb 5, 2020 at 2:29 PM Andrew Purtell 
> > wrote:
> >
> > > Which registry to use is a client side configuration so old clients are
> > > unaffected. (At some future time an operator might want to foreclose on
> > > direct ZK access with network or host ACLs but of course this is
> totally
> > up
> > > to them and they are in complete control.)
> > >
> > > > On Feb 5, 2020, at 2:12 PM, Wellington Chevreuil <
> > > wellington.chevre...@gmail.com> wrote:
> > > >
> > > > Thanks for the detailed summary, Bharath. I'm +1 for master.
> > > >
> > > > Just additional question I have, it wasn't clear for me on the
> > > doc/summary:
> > > > does it consider fallback to ZK based registry, in case of clients
> > > running
> > > > old versions on clusters with this feature enabled?
> > > >
> > > >> Em qua., 5 de fev. de 2020 às 07:37, Bharath Vissapragada <
> > > >> bhara...@apache.org> escreveu:
> > > >>
> > > >> Thanks everyone for chiming in. Sean, regarding your comments.
> > > >>
> > > >>> I don't see the current design doc in the feature branch
> > > >> (i.e.dev-support/design-docs) please include it there
> > > >>
> > > >> Of course, HBASE-23331 <
> > > https://issues.apache.org/jira/browse/HBASE-23331>
> > > >> is
> > > >> the subtask for this. The plan is to update the ref guide with all
> the
> > > >> details once the branch is merged in the master. I'll make sure to
> add
> > > the
> > > >> design doc too.
> > > >>
> > > >>> the current design doc has comments open still, should I assume
> those
> > > >> things haven't been addressed in the branch? or should I assume they
> > > have
> > > >> but it hasn't been updated yet?
> > > >>
> > > >> I addressed most of them already, forgot to resolve the comments.
> > There
> > > >> were some new comments since this email, so I addressed them and
> > > cleaned up
> > > >> the doc. Thanks for pointing it out.
> > > >>
> > > >>> On Tue, Feb 4, 2020 at 10:15 PM Stack  wrote:
> > > >>>
> > > >>> I'm +1 for merge to master with it default enabled and to branch-2
> > with
> > > >> it
> > > >>> off by default.
> > > >>>
> > > >>> Nice work Bharath.
> > > >>>
> > > >>> S
> > > >>>
> > > >>> On Tue, Feb 4, 2020 at 8:37 AM Bharath Vissapragada <
> > > bhara...@apache.org
> > > >>>
> > > >>> wrote:
> > > >>>
> > >  Hello everyone,
> > > 
> > >  I'd like to kickoff a discuss thread on dev@ to see what folks
> > think
> > > >>> about
> > >  merging the feature branch for HBASE-18095
> > >   into the
> > master.
> > > >> For
> > >  those of you who aren't following this work, over the last few
> > months,
> > > >> a
> > >  lot of effort went into a feature branch
> > >  <
> > > 
> > > >>>
> > > >>
> > >
> >
> https://github.com/apache/hbase/tree/HBASE-18095/client-locate-meta-no-zookeeper
> > > >
> > >  to
> > >  remove the ZK dependency in the client.
> > > 
> > >  *Please refer to the design doc
> > >  <
> > > 
> > > >>>
> > > >>
> > >
> >
> https://docs.google.com/document/d/1JAJdM7eUxg5b417f0xWS4NztKCx1f2b6wZrudPtiXF4/edit
> > > >
> > >  attached to the parent jira and go through the subtasks for all
> the
> > >  technical details and design considerations*.
> > > 
> > >  *TL;DR*: With this feature, the client connection implementation
> > *does
> > > >>> not*
> > >  need access to zookeeper to fetch the connection metadata.
> Instead,
> > a
> > >  predefined set of master end points in the configuration are used
> by
> > >  clients to fetch the required metadata.
> > > 
> > >  This new feature is *enabled by default on the feature branch* and
> > > >> passes
> > >  the entire nightly test suite (modulo some known flakes not
> specific
> > > to
> > > >>> the
> > >  branch). At this point, I'm not aware of any performance concerns
> /
> > > >>> feature
> > >  gaps compared to original default implementation. The original
> > > registry
> > >  implementation is still retained and can be used by setting the
> > > >> following
> > >  client configuration. This kill switch gives the users more
> > > flexibility
> > >  since they have a fallback incase they run into any issues.
> > > 
> > >  
> > >  
> > > hbase.client.registry.impl
> > > 
> > org.apache.hadoop.hbase.client.ZKConnectionRegistry
> > >  
> > > 
> 

Re: [ANNOUNCE] New HBase committer Bharath Vissapragada

[Zach York]

Congratulations and welcome Bharath! :)

On Thu, Feb 6, 2020 at 11:50 AM Wellington Chevreuil <
wellington.chevre...@gmail.com> wrote:

> Congratulations and welcome to the team, Bharath!
>
> Em qui., 6 de fev. de 2020 às 19:25, Esteban Gutierrez
>  escreveu:
>
> > Yay! Congratulations Bharath!
> >
> > esteban.
> > --
> > Cloudera, Inc.
> >
> >
> >
> > On Thu, Feb 6, 2020 at 12:07 PM Andrew Purtell 
> > wrote:
> >
> > > Congratulations and welcome, Bharath!
> > >
> > > On Wed, Feb 5, 2020 at 7:36 PM Nick Dimiduk 
> wrote:
> > >
> > > > On behalf of the Apache HBase PMC I am pleased to announce that
> Bharath
> > > > Vissapragada has accepted the PMC's invitation to become a commiter
> on
> > > the
> > > > project. We appreciate all of Bharath's generous contributions thus
> far
> > > and
> > > > look forward to his continued involvement.
> > > >
> > > > Allow me to be the first to congratulate and welcome Bharath into his
> > new
> > > > role!
> > > >
> > > > Thanks,
> > > > Nick
> > > >
> > >
> > >
> > > --
> > > Best regards,
> > > Andrew
> > >
> > > Words like orphans lost among the crosstalk, meaning torn from truth's
> > > decrepit hands
> > >- A23, Crosstalk
> > >
> >
>

[jira] [Created] (HBASE-23732) TestHFileCleaner can fail after addition of ManualMobMaintHFileCleaner

[Zach York (Jira)]

Zach York created HBASE-23732:
-

 Summary: TestHFileCleaner can fail after addition of 
ManualMobMaintHFileCleaner
 Key: HBASE-23732
 URL: https://issues.apache.org/jira/browse/HBASE-23732
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 2.2.3
Reporter: Zach York


After 
https://github.com/apache/hbase/commit/a881678ef4e0ffcd94add935729424c6e2567356 
testHFileCleaning can fail due to some lazy directory creation in the test. 
ManualMobMaintHFileCleaner needs to get the table directory (and indirectly 
namespace) which can fail since the test itself only creates the file within 
the archive directory. It does not contain the 
archive/data/namespace/table/region/cf/ which the ManualMobMaintHFileCleaner 
expects.

This is a relatively minor issue, but we saw this test fail when running on EC2 
because we had an 'invalid namespace' (because it had a '-' in the path).

FYI [~busbey]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] Please welcome Guangxu Cheng the HBase PMC

[Zach York]

Congratulations and welcome Guangxu!!

On Mon, Dec 9, 2019 at 10:25 AM Esteban Gutierrez
 wrote:

> Congrats Guangxu!
> --
> Cloudera, Inc.
>
>
>
> On Mon, Dec 9, 2019 at 12:03 PM Karthik Palanisamy <
> karthikshva...@gmail.com>
> wrote:
>
> > Superb!! Congratulations Guangxu Cheng!!!
> >
> > On Mon, Dec 9, 2019 at 5:58 AM Jan Hentschel <
> > jan.hentsc...@ultratendency.com> wrote:
> >
> > > Congratulations and welcome!
> > >
> > > From: Duo Zhang 
> > > Reply-To: "dev@hbase.apache.org" 
> > > Date: Monday, December 9, 2019 at 10:47 AM
> > > To: HBase Dev List , hbase-user <
> > > u...@hbase.apache.org>
> > > Subject: [ANNOUNCE] Please welcome Guangxu Cheng the HBase PMC
> > >
> > > On behalf of the Apache HBase PMC I am pleased to announce that Guangxu
> > > Cheng has accepted our invitation to become a PMC member on the Apache
> > > HBase project. We appreciate Guangxu Cheng stepping up to take more
> > > responsibility in the HBase project.
> > >
> > > Please join me in welcoming Guangxu Cheng to the HBase PMC!
> > >
> > >
> >
>

Re: [DISCUSS] Deprecate Review Board in favor of Github reviews

[Zach York]

+1, much easier and available.

On Thu, Nov 14, 2019 at 9:56 AM Geoffrey Jacoby  wrote:

> +1 (non-binding), GitHub is a much better user experience, for both
> reviewers and contributors
>
> Geoffrey.
>
> On Thu, Nov 14, 2019 at 5:48 AM Guangxu Cheng 
> wrote:
>
> > +1
> > It will be more convenient to use Github PR.
> >
> > 张铎(Duo Zhang)  于2019年11月14日周四 下午6:17写道：
> >
> > > +1,  ReviewBoard is almost dead now as it is only available to
> > > committers...
> > >
> > > Jan Hentschel  于2019年11月14日周四
> 下午6:11写道：
> > >
> > > > +1
> > > >
> > > > I also like the GitHub way much more compared to ReviewBoard.
> > > >
> > > > From: Peter Somogyi 
> > > > Reply-To: "dev@hbase.apache.org" 
> > > > Date: Wednesday, November 13, 2019 at 6:23 PM
> > > > To: HBase Dev List 
> > > > Subject: Re: [DISCUSS] Deprecate Review Board in favor of Github
> > reviews
> > > >
> > > > +1
> > > >
> > > > Another issue with ReviewBoard is that it requires Apache ID so only
> > > > committers are able to create new reviews or even comment.
> > > >
> > > > On Wed, Nov 13, 2019 at 5:21 PM Nick Dimiduk  > >  > > > ndimi...@apache.org>> wrote:
> > > >
> > > > Heya,
> > > >
> > > > Seems in the old days we were explicitly non-strict about where code
> > > review
> > > > were happening. I remember bouncing between Review Board and a
> > > Phabricator
> > > > instance (in addition to in-line patch reviews on JIRA). Now that we
> > have
> > > > this fancy Gitbox and integration with GitHub, it seems we're making
> a
> > > > strong statement toward using Github PRs (in addition to in-line
> patch
> > > > reviews on JIRA) for our code review system. Is it worth while
> > supporting
> > > > those older tools? I think maintaining the developer support tooling
> > > around
> > > > just these two mechanisms is plenty to keep up with.
> > > >
> > > > I propose we make the move to Github PR's "official". This
> > > > basically involves updating the tome (here [0], here [1], probably
> > > others)
> > > > accordingly and sweeping the `dev-support` dir for old scripts.
> > > >
> > > > Thoughts?
> > > >
> > > > Thanks,
> > > > Nick
> > > >
> > > > [0]: http://hbase.apache.org/book.html#developing
> > > > [1]: http://hbase.apache.org/book.html#reviewboard
> > > >
> > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] Please welcome Sakthi to the Apache HBase PMC

[Zach York]

Welcome Sakthi! Congratulations!

On Wed, Oct 23, 2019 at 1:15 PM Sean Busbey  wrote:

> Please join me in welcoming Sakthi to the HBase PMC, of course.
>
> Welcome Sakthi! Well deserved!
>
> On Wed, Oct 23, 2019 at 3:14 PM Sean Busbey  wrote:
> >
> > On behalf of the Apache HBase PMC I am pleased to announce that
> > Sakthi has accepted our invitation to become a PMC member on the
> > HBase project. We appreciate Sakthi stepping up to take more
> > responsibility in the HBase project.
> >
> > Please join me in welcoming Jan to the HBase PMC!
> >
> >
> >
> > As a reminder, if anyone would like to nominate another person as a
> > committer or PMC member, even if you are not currently a committer or
> > PMC member, you can always drop a note to priv...@hbase.apache.org to
> > let us know.
>

Re: [ANNOUNCE] Please welcome Wellington Chevreuil to the Apache HBase PMC

[Zach York]

Welcome Wellington! Congratulations!

On Wed, Oct 23, 2019 at 1:19 PM Salvatore LaMendola (BLOOMBERG/ 731 LEX) <
slamendo...@bloomberg.net> wrote:

> Congrats Sakthi and Wellington!
>
> From: dev@hbase.apache.org At: 10/23/19 16:17:58To:  dev@hbase.apache.org
> Cc:  u...@hbase.apache.org
> Subject: Re: [ANNOUNCE] Please welcome Wellington Chevreuil to the Apache
> HBase PMC
>
> Congrats Wellington!
>
> Sakthi
>
> On Wed, Oct 23, 2019 at 1:16 PM Sean Busbey  wrote:
>
> > On behalf of the Apache HBase PMC I am pleased to announce that
> > Wellington Chevreuil has accepted our invitation to become a PMC member
> on
> > the
> > HBase project. We appreciate Wellington stepping up to take more
> > responsibility in the HBase project.
> >
> > Please join me in welcoming Wellington to the HBase PMC!
> >
> >
> >
> > As a reminder, if anyone would like to nominate another person as a
> > committer or PMC member, even if you are not currently a committer or
> > PMC member, you can always drop a note to priv...@hbase.apache.org to
> > let us know.
> >
>
>
>

Re: [VOTE] First release candidate for Apache HBase 1.4.11

[Zach York]

+1 (binding)

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_92): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_92): ok
 - mvn clean install -DskipTests
* Unit tests pass (1.8.0_92): ok
 - mvn test -P runAllTests

On Tue, Oct 22, 2019 at 9:28 AM Sean Busbey  wrote:

> +1
>
> On Fri, Oct 18, 2019 at 1:56 AM Sean Busbey  wrote:
> >
> > Hi folks!
> >
> > The first release candidate for HBase 1.4.11 is available for download:
> >
> > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.4.11RC0/
> >
> > Maven artifacts are also available in a staging repository at:
> >
> > https://repository.apache.org/content/repositories/orgapachehbase-1369/
> >
> > Artifacts are signed with my key (0D80DB7C) published in our KEYS
> > file at http://www.apache.org/dist/hbase/KEYS
> >
> > The RC corresponds to the signed tag 1.4.11RC0, which currently points
> > to commit ref
> >
> > 927981c8a0367b6d0ec2e6c94f12d57592dfa5e3
> >
> > HBase 1.4.11 is the next maintenance release in the HBase 1.4 line,
> > continuing on the theme of bringing a stable, reliable database to the
> > Hadoop and NoSQL communities. This release includes approximately one
> > hundred fixes and improvements done by the HBase community in the four
> > months since the prior release.
> >
> > The detailed source and binary compatibility report vs 1.4.10 has been
> > published for your review, at:
> >
> > https://s.apache.org/hbase-1.4.11-rc0-compat-report
> >
> > The report shows some binary incompatible changes to two interfaces
> > marked as LimitedPrivate. One set is to interfaces that are marked for
> > the CONFIG audience, which should mean only their class names needs to
> > remain the same (and they do) . The other change is to the Region
> > interface in order to solve a critical error in the way bulkloaded files
> > are handled by the replication system.
> >
> > Critical changes include:
> >
> > * HBASE-22784 Clusters in a cyclical replication topology that only have
> >   incoming writes from replication never clean out WALs.
> > * HBASE-23101 replication of bulkloaded files must handle cyclical
> >   topologies. Includes binary incompatible change to Region
> >   interface.
> > * HBASE-23015 Stop including Jackson libraries in classpaths except when
> >   needed by the HBase REST Proxy or by Apache Hadoop.
> > * HBASE-22728, HBASE-23174 update HBase REST Proxy to use Jackson 2 for
> >   serialization (CVEs).
> > * HBASE-22627 Recovered WAL directories not getting cleaned up when
> >   running different WAL and base filesystems.
> > * HBASE-15666 New utility artifact for testing shaded client:
> >   hbase-shaded-testing-util.
> > * HBASE-22874 Define a public interface for Canary and move existing
> >   implementation to LimitedPrivate
> >
> > The full set of fixes in this release can be found here:
> >
> > https://s.apache.org/hbase-1.4.11-jira-release-notes
> >
> > and in the CHANGES.txt file included in the distribution.
> >
> > Please try out this candidate and vote +1/-1 on whether we should
> > release these artifacts as Apache HBase 1.4.11.
> >
> > The VOTE will remain open for at least 72 hours.
> >
> > Prior to posting this RC:
> > * Untrusted test list looks normal
> >   https://s.apache.org/hbase-1.4.11-rc0-untrusted-test-list
> > * local build with Zulu JDK 1.7.0_232 and unit tests with Zulu
> >   JDK 1.8.0_222 passes, when using the above untrusted test list.
> > * using WIP pseudo distributed test from HBASE-23162 passes for use of
> >   shaded client and MR artifacts
> > * ltt 1m keys write and then read works. Zulu JDK 1.8.0_222.
> >
> > Thanks!
> >
> > --busbey
> >
> > as of this email the posted artifacts have the following SHA512:
> >
> > hbase-1.4.11-src.tar.gz:
> >
> > 2B45368F 5D0A109D F3101E9F 05FE0ADE 7AE634D9 AFC83AB0
> > B540AFCF 2C64CE9A 7122649E E3CCD6FF 72EDCBDD 2350D4C4
> > 45F96BF2 292BE810 F957A791 581DB1C8
> >
> > hbase-1.4.11-bin.tar.gz:
> >
> > D9006D31 D3B4641B 061F908D D7F4AD13 D041C454 E0FA3383
> > 5109B878 81DB591D 69DDA6DB DC22A88D 16E6C45A FEF69245
> > CF28D4AF 87B83F7C 12578042 0BFB2236
>

[jira] [Created] (HBASE-23202) ExportSnapshot (import) will fail if copying files to root directory takes longer than cleaner TTL

[Zach York (Jira)]

Zach York created HBASE-23202:
-

 Summary: ExportSnapshot (import) will fail if copying files to 
root directory takes longer than cleaner TTL
 Key: HBASE-23202
 URL: https://issues.apache.org/jira/browse/HBASE-23202
 Project: HBase
  Issue Type: Bug
  Components: snapshots
Affects Versions: 1.5.0, 1.4.11
Reporter: Zach York


HBASE-17330 removed the checking of the snapshot .tmp directory when 
determining which files are candidates for deletes. It appears that in the 
latest branches, this isn't an issue for taking a snapshot as it checks whether 
a snapshot is in progress via the SnapshotManager.

However, when using the ExportSnapshot tool to import a snapshot into a 
cluster, it will first copy the snapshot manifest into 
/.snapshot/.tmp/ [1], copies the files, and then renames the snapshot 
manifest to the final snapshot directory. If the copyFiles job takes longer 
than the cleaner TTL, the ExportSnapshot job will fail because HFiles will get 
deleted before the snapshot is committed to the final directory. 

The ExportSnapshot tool already has a functionality to skipTmp and write the 
manifest directly to the final location. However, this has unintended 
consequences such as the snapshot appearing to the user before it is usable. So 
it looks like we will have to bring back the tmp directory check to avoid this 
situation.

[1] 
https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java#L1029



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Second release candidate for HBase 2.1.6 is available for download

[Zach York]

+1

  * Signature: ok
  * Checksum : ok
  * Rat check (1.8.0_92): ok
   - mvn clean apache-rat:check
  * Built from source (1.8.0_92): ok
   - mvn clean install -DskipTests
  * Unit tests pass (1.8.0_92): ok
   - mvn test -P runAllTests

The test script is so set and forget that I forgot to post back on the
thread until today :)

On Wed, Aug 28, 2019 at 9:58 AM Nick Dimiduk  wrote:

> +1
>
> using this fancy new hbase-vote.sh:
> * Signature: ok
> * Checksum : ok
> * Rat check (1.8.0_222): ok
>  - mvn clean apache-rat:check
> * Built from source (1.8.0_222): ok
>  - mvn clean install -DskipTests
> * Unit tests pass (1.8.0_222): failed
>  - mvn package -P runAllTests
>
> I have only been able to get about 1/10 full test runs to complete, largely
> due to timeouts in test setup.
>
> On Mon, Aug 26, 2019 at 7:54 AM Duo Zhang  wrote:
>
> > The second release candidate for HBase 2.1.6 is available for download:
> >
> >   https://dist.apache.org/repos/dist/dev/hbase/2.1.6RC1
> >
> > Maven artifacts are also available in a staging repository at:
> >
> >
> https://repository.apache.org/content/repositories/orgapachehbase-1331/
> >
> > Artifacts are signed with my key (9AD2AE49) published in our KEYS file at
> >
> >   http://www.apache.org/dist/hbase/KEYS
> >
> > The tag to be voted on is 2.1.6RC1:
> >
> >   https://github.com/apache/hbase/tree/2.1.6RC1
> >
> > HBase 2.1.6 is the seventh maintenance release in the HBase 2.1 line,
> > continuing on the theme of bringing a stable, reliable database to the
> > Hadoop and NoSQL communities. It fixes several CVEs related to jackson by
> > upgrading jackson-databind dependency to 2.9.9.2, all hbase users are
> > highly recommended to upgrade, especially the one who uses hbase-rest.
> >
> > 2.1.6 includes ~152 bug and improvement fixes done since the 2.1.5. There
> > are several critical fixes around WAL, which may cause WAL corruption or
> > hang the region server. Please see HBASE-22539, HBASE-22681, HBASE-22684
> > for more details.
> >
> > The detailed source and binary compatibility report vs 2.1.5 has been
> > published for your review, at:
> >
> >
> >
> >
> https://dist.apache.org/repos/dist/dev/hbase/2.1.6RC1/compatibility_report_2.1.5vs2.1.6RC1.html
> >
> > The report shows no incompatibilities.
> >
> > The full list of fixes included in this release is available in the
> > CHANGES.md that ships as part of the release also available here:
> >
> >   https://dist.apache.org/repos/dist/dev/hbase/2.1.6RC1/CHANGES.md
> >
> > The RELEASENOTES.md are here:
> >
> >   https://dist.apache.org/repos/dist/dev/hbase/2.1.6RC1/RELEASENOTES.md
> >
> > Please try out this candidate and vote +1/-1 on whether we should release
> > these artifacts as HBase 2.1.6.
> >
> > The VOTE will remain open for at least 72 hours.
> >
> > Thanks
> >
>

Re: Considering immediate EOL of branch-1.3 and branch-1.4 (was: Re: HBASE-22728 consequences)

[Zach York]

I'm fine with eventually EOLing 1.3 and 1.4, but I don't think we can do it
until we know 1.5.0 is for sure coming out within a reasonable time and
will be stable (the current stable pointer is 1.4.10 so what would we move
that to?).

I'm always a fan of reducing maintenance burden, but let's hold off on
officially EOLing until we know users have something to move to.

On Wed, Aug 7, 2019 at 11:51 AM Andrew Purtell  wrote:

> Changing subject line for visibility.
>
> On Wed, Aug 7, 2019 at 11:48 AM Stack  wrote:
>
> > EOL'ing 1.3+1.4 sounds good to me.
> > S
> >
> > On Wed, Aug 7, 2019 at 10:46 AM Andrew Purtell 
> > wrote:
> >
> > > HBASE-22728 addresses theoretical exposure to a Jackson CVE by us (via
> > > hbase-rest) or to our downstream by removing Jackson artifacts from our
> > > exported transitive dependencies, and by updating hbase-rest to use a
> > safe
> > > Jackson version. These changes are arguably not suitable for patch
> > releases
> > > because they can cause a transitive binary compatibility problem. For
> > this
> > > reason I would like us to consider immediate EOL of 1.3 and 1.4 with a
> > > recommendation to upgrade to 1.5.0.
> > >
> > > In order for that to happen, we need to commit HBASE-22728 to branch-1,
> > > then release 1.5.0 from head of branch-1, which I will do. Assuming
> test
> > > results are good I will propose a 1.5.0 release candidate in the next
> few
> > > days.
> > >
> > > Or would you find the HBASE-22728 change acceptable for a patch
> release?
> > >
> > > There are other good reasons to move on from 1.3 and 1.4, foremost a
> nice
> > > reduction in maintenance burden keeping up these old code lines.
> > >
> > > Are there any objections or concerns to this plan?
> > >
> > > --
> > > Best regards,
> > > Andrew
> > >
> > > Words like orphans lost among the crosstalk, meaning torn from truth's
> > > decrepit hands
> > >- A23, Crosstalk
> > >
> >
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

Re: [ANNOUNCE] New HBase committer Tak-Lon (Stephen) Wu

[Zach York]

Congratulations and welcome Stephen!

On Mon, Aug 5, 2019 at 1:00 PM Andrew Purtell  wrote:

> Congratulations and welcome, Stephen.
>
> On Mon, Aug 5, 2019 at 11:53 AM Sean Busbey  wrote:
>
> > On behalf of the Apache HBase PMC I am super pleased to announce that
> > Tak-Lon (Stephen) Wu has accepted the PMC's invitation to become a
> > commiter on the project.
> >
> > Thanks so much for the work you've been contributing. We look forward
> > to your continued involvement.
> >
> > Congratulations and welcome!
> >
> > -busbey
> >
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

Re: [ANNOUNCE] Please welcome Zheng Hu to the HBase PMC

[Zach York]

Congratulations and welcome Zheng!

On Mon, Aug 5, 2019 at 9:54 AM Xu Cang  wrote:

> Congratulations, Zheng Hu!
>
> On Mon, Aug 5, 2019 at 6:07 AM Guangxu Cheng 
> wrote:
>
> > Congratulations, Zheng!
> >
> > Allan Yang  于2019年8月5日周一 下午5:13写道：
> >
> > > Congratulations, Hu！
> > > Best Regards
> > > Allan Yang
> > >
> > >
> > > Peter Somogyi  于2019年8月5日周一 下午4:47写道：
> > >
> > > > Congratulations!
> > > >
> > > > On Mon, Aug 5, 2019 at 8:57 AM Pankaj kr 
> wrote:
> > > >
> > > > > Congratulations Zheng..!!
> > > > >
> > > > > Regards,
> > > > > Pankaj
> > > > >
> > > > > -Original Message-
> > > > > From: Duo Zhang [mailto:zhang...@apache.org]
> > > > > Sent: 05 August 2019 07:38
> > > > > To: HBase Dev List ; hbase-user <
> > > > > u...@hbase.apache.org>
> > > > > Subject: [ANNOUNCE] Please welcome Zheng Hu to the HBase PMC
> > > > >
> > > > > On behalf of the Apache HBase PMC I am pleased to announce that
> Zheng
> > > Hu
> > > > > has accepted our invitation to become a PMC member on the Apache
> > HBase
> > > > > project. We appreciate Zheng Hu stepping up to take more
> > responsibility
> > > > in
> > > > > the HBase project.
> > > > >
> > > > > Please join me in welcoming Zheng Hu to the HBase PMC!
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] new HBase committer Sakthi

[Zach York]

Congrats Sakthi! Welcome!

On Thu, Aug 1, 2019 at 11:07 AM Jesse Yates  wrote:

> Congrats sakthi!
>
>
>
> On Thu, Aug 1, 2019 at 10:02 AM Ankit Singhal 
> wrote:
>
> > Congratulations Sakthi !!
> >
> > On Thu, Aug 1, 2019 at 7:15 AM Toshihiro Suzuki 
> > wrote:
> >
> > > Congratulations! Sakthi
> > >
> > > On Thu, Aug 1, 2019 at 9:49 PM Guanghao Zhang 
> > wrote:
> > >
> > > > Congratulations!
> > > >
> > > > Allan Yang  于2019年8月1日周四 下午8:46写道：
> > > >
> > > > > Congratulations Sakthi !
> > > > >
> > > > > Best Regards
> > > > > Allan Yang
> > > > >
> > > > >
> > > > > Nihal Jain  于2019年8月1日周四 下午8:40写道：
> > > > >
> > > > > > Congrats Sakthi. More power to you!
> > > > > >
> > > > > > On Thu, 1 Aug, 2019, 4:42 PM ramkrishna vasudevan, <
> > > > > > ramkrishna.s.vasude...@gmail.com> wrote:
> > > > > >
> > > > > > > Congratulations Sakthi !!!
> > > > > > >
> > > > > > > On Thu, Aug 1, 2019 at 3:34 PM 张铎(Duo Zhang) <
> > > palomino...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Congratulations!
> > > > > > > >
> > > > > > > > Pankaj kr  于2019年8月1日周四 下午5:56写道：
> > > > > > > >
> > > > > > > > > Congratulation Sakthi..!!
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Pankaj
> > > > > > > > >
> > > > > > > > > -Original Message-
> > > > > > > > > From: Sean Busbey [mailto:bus...@apache.org]
> > > > > > > > > Sent: 01 August 2019 05:35
> > > > > > > > > To: u...@hbase.apache.org; dev 
> > > > > > > > > Subject: [ANNOUNCE] new HBase committer Sakthi
> > > > > > > > >
> > > > > > > > > On behalf of the HBase PMC, I'm pleased to announce that
> > Sakthi
> > > > has
> > > > > > > > > accepted our invitation to become an HBase committer.
> > > > > > > > >
> > > > > > > > > We'd like to thank Sakthi for all of his diligent
> > contributions
> > > > to
> > > > > > the
> > > > > > > > > project thus far. We look forward to his continued
> > > participation
> > > > in
> > > > > > our
> > > > > > > > > community.
> > > > > > > > >
> > > > > > > > > Congrats and welcome Sakthi!
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [Announce] 张铎 (Duo Zhang) is Apache HBase PMC chair

[Zach York]

Congratulations Duo! Thanks for offering to take on the additional work!

On Fri, Jul 19, 2019 at 10:34 AM Stack  wrote:

> Thank you Misty for your years of service (FYI, for non-PMCers, the reports
> Misty wrote to the Apache Board on our behalf were repeatedly called out
> for their quality and thoughtfulness).
>
> Duo Zhang, thank you for taking on the mantle.
>
> S
>
> On Thu, Jul 18, 2019 at 10:46 AM Misty Linville  wrote:
>
> > Each Apache project has a project management committee (PMC) that
> oversees
> > governance of the project, votes on new committers and PMC members, and
> > ensures that the software we produce adheres to the standards of the
> > Foundation. One of the roles on the PMC is the PMC chair. The PMC chair
> > represents the project as a Vice President of the Foundation and
> > communicates to the board about the project's health, once per quarter
> and
> > at other times as needed.
> >
> > It's been my honor to serve as your PMC chair since 2017, when I took
> over
> > from Andrew Purtell. I've decided to step back from my volunteer ASF
> > activities to leave room in my life for other things. The HBase PMC
> > nominated Duo for this role, and Duo has kindly agreed! The board passed
> > this resolution in its meeting yesterday[1] and it is already
> official[2].
> > Congratulations, Duo, and thank you for continuing to honor the project
> > with your dedication.
> >
> > Misty
> >
> > [1] The minutes have not yet posted at the time of this email, but will
> be
> > available at http://www.apache.org/foundation/records/minutes/2019/.
> > [2] https://www.apache.org/foundation/#who-runs-the-asf
> >
>

Re: Thank you Misty

[Zach York]

Thank you for all you have done as our Chair, Misty! Best of luck!

On Thu, Jul 18, 2019 at 10:54 AM Andrew Purtell  wrote:

> Thank you for serving as our Chair, Misty. Your reports were some of the
> best I've ever seen in my ten or so years at the ASF. Best of luck to you
> and yours in your future endeavors.
>
> On Thu, Jul 18, 2019 at 10:46 AM Misty Linville  wrote:
>
> > It's been my honor to serve as your PMC chair since 2017, when I took
> over
> > from Andrew Purtell. I've decided to step back from my volunteer ASF
> > activities to leave room in my life for other things.
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

Re: [DISCUSS] What to do with branches for EOL versions

[Zach York]

I agree that deleting EOM branches is the best approach (with tags).
Having a branch for an EOM line doesn't make sense since a branch signifies
(and/or promotes) active development.
Having a tag signifies that this line is static and is a point-in-time
pointer to the branch. In the worst case scenario,
we can still create a branch off of the tag if we want to 'revive' the
branch.

This also simplifies life for committers as it will be easier to know which
branches are 'active' since those will be the only
branches that exist.

On Mon, Jul 15, 2019 at 8:12 AM Wellington Chevreuil <
wellington.chevre...@gmail.com> wrote:

> I think delete EOM branches and keeping only tags sounds reasonable, but
> ain't much experienced on releases management, honestly. Do we know what's
> the standard among most apache projects? Maybe we could follow those?
>
> Em seg, 15 de jul de 2019 às 15:27, Josh Elser 
> escreveu:
>
> > (Sending this note for Busbey as he's chasing other stuff)
> >
> > He had sent a note to private asking what had happened to branch-1.2. In
> > my cleanup of old branches to try to reduce our Jenkins usage at the
> > request of Infra, I created a git tag instead of the branch: branch-1.2
> > was deleted remotely, and the tag branch-1.2-EOM was created.
> >
> > Busbey said he would put back branch-1.2 (at least temporarily), but
> > that we should discuss what we want the "normal" to be going forward.
> >
> > I can see the confusion in folks who are "used to" the "branch-x[.y]"
> > notation being confused when the upstream branch is no longer present.
> > However, I can also see the "EOM" tag for the same "branch-x[.y]" being
> > self-explanatory (or at least a good indication that maybe a user is
> > trying to interact with a development line that we no longer maintain).
> >
> > How do others think we should handle branches which we voted on
> > no-longer maintaining releases for?
> >
> > - Josh
> >
>

Re: [VOTE] The sixth HBase 2.2.0 release candidate (RC6) is available

[Zach York]

+1

  * Signature: ok
  * Checksum : ok
  * Rat check (1.8.0_92): ok
   - mvn clean apache-rat:check
  * Built from source (1.8.0_92): ok
   - mvn clean install -DskipTests
  * Unit tests pass (1.8.0_92): failed
   - mvn test -P runAllTests

Output of the failed tests:
 [ERROR] Failures:
  [ERROR]
TestClusterRestartFailoverSplitWithoutZk>TestClusterRestartFailover.test:89
serverNode should not be null when restart whole cluster
  [ERROR] Errors:
  [ERROR]
org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster.org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster
  [ERROR]   Run 1:
TestRegionMergeTransactionOnCluster.afterAllTests:132->Object.wait:-2 »
TestTimedOut
  [ERROR]   Run 2:
TestRegionMergeTransactionOnCluster.org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster
»

Both of these tests are flaky per the dashboard (although
TestClusterRestartFailoverSplitWithoutZk seems to fail consistently when
all ran together - passes separately).

On Wed, Jun 12, 2019 at 2:33 PM Artem Ervits  wrote:

> +1 (non-binding)
>
> * Signature: ok
> * Checksum : ok
> * Rat check (1.8.0_212): ok
>  - mvn clean apache-rat:check
> * Built from source (1.8.0_212): ok
>  - mvn clean install -DskipTests
> * Unit tests pass (1.8.0_212): ok
>  - mvn test -fn -P runAllTests
>
> used a modified version of hbase-vote to prevent failure (-fn). In my env,
> this script fails due to hbase-http and I'm working to identify issues
> local to my environment so not -1.
>
> [hbase-server]
> org.apache.hadoop.hbase.client.TestSnapshotCloneIndependence failed as part
> of script but passed individually.
> org.apache.hadoop.hbase.replication.TestReplicationDroppedTables failed as
> part of script but passed individually.
>
> [hbase-mapreduce]
> org.apache.hadoop.hbase.snapshot.TestExportSnapshot timed out and failed
> for various test class setUp reasons as part of script and passed
> individually.
> org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot failed as part of
> script but passed individually.
> org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot failed as part
> of script but passed individually.
> org.apache.hadoop.hbase.snapshot.TestExportSnapshotWithTemporaryDirectory
> failed as part of script but passed individually.
>
> [hbase-thrift]
> org.apache.hadoop.hbase.thrift.TestThriftSpnegoHttpServer failed both via
> script and individually
>
> [INFO] Running org.apache.hadoop.hbase.thrift.TestThriftSpnegoHttpServer
> [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 16.962 s <<< FAILURE! - in
> org.apache.hadoop.hbase.thrift.TestThriftSpnegoHttpServer
> [ERROR]
>
> testRunThriftServer(org.apache.hadoop.hbase.thrift.TestThriftSpnegoHttpServer)
>  Time elapsed: 1.419 s  <<< ERROR!
> org.apache.thrift.transport.TTransportException: HTTP Response code: 401
> at
>
> org.apache.hadoop.hbase.thrift.TestThriftSpnegoHttpServer.talkToThriftServer(TestThriftSpnegoHttpServer.java:207)
>
> possibly local to my system again
>
> [hbase-rest]
> org.apache.hadoop.hbase.rest.TestSecureRESTServer failed both via script
> and individually, again possibly due to my environment
>
> [ERROR] Tests run: 2, Failures: 2, Errors: 0, Skipped: 0, Time elapsed:
> 23.819 s <<< FAILURE! - in
> org.apache.hadoop.hbase.rest.TestSecureRESTServer
> [ERROR]
>
> testNegativeAuthorization(org.apache.hadoop.hbase.rest.TestSecureRESTServer)
>  Time elapsed: 0.452 s  <<< FAILURE!
> java.lang.AssertionError:
> Got response: 
> 
> 
> Error 401 
> 
> 
> HTTP ERROR: 401
> Problem accessing /test/schema. Reason:
> Authentication required
> 
> 
> 
>  expected:<403> but was:<401>
> at
>
> org.apache.hadoop.hbase.rest.TestSecureRESTServer.testNegativeAuthorization(TestSecureRESTServer.java:381)
>
> [ERROR]
>
> testPositiveAuthorization(org.apache.hadoop.hbase.rest.TestSecureRESTServer)
>  Time elapsed: 1.497 s  <<< FAILURE!
> java.lang.AssertionError: HTTP/1.1 401 Authentication required
> expected:<200> but was:<401>
> at
>
> org.apache.hadoop.hbase.rest.TestSecureRESTServer.testPositiveAuthorization(TestSecureRESTServer.java:345)
>
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Failures:
> [ERROR]   TestSecureRESTServer.testNegativeAuthorization:381 Got response:
> 
> 
> 
> Error 401 
> 
> 
> HTTP ERROR: 401
> Problem accessing /test/schema. Reason:
> Authentication required
> 
> 
> 
>  expected:<403> but was:<401>
> [ERROR]   TestSecureRESTServer.testPositiveAuthorization:345 HTTP/1.1 401
> Authentication required expected:<200> but was:<401>
> [INFO]
> [ERROR] Tests run: 6, Failures: 2, Errors: 0, Skipped: 0
>
> On Wed, Jun 12, 2019 at 4:40 PM Tak-Lon (Stephen) Wu 
> wrote:
>
> > +1 (non-binding)
> >
> > * Signature: ok
> > * Checksum : ok
> > * Rat check (1.8.0_212): ok
> >  - mvn clean apache-rat:check
> > * Built from source 

Re: DISCUSS: Heads-up 2.1.5 RC in next few days and suggest EOL'ing 2.1 branch

[Zach York]

I like the proactive approach to EOLing branches, but I don't think we can
quite EOL a branch when there is no newer branch (2.2.0) out. If that's
2.1.6, that's fine.

On Thu, May 16, 2019 at 3:18 PM Stack  wrote:

> Was going to put up an RC for 2.1.5 in next day or so after a review of
> unresolved JIRAs that have 2.1.5 as fix version. It is coming up on two
> months since 2.1.4 and by the time we're done, there'll be 90+ changes.
> Branch-2.1 nightlies have started to stabilize again.
>
> Was also thinking of EOL'ing the 2.1 branch. We EOL'd 2.0 branch at 2.0.5.
> We have too many branches as it is. What do folks think? 2.2.0 isn't out
> yet so maybe wait on 2.1.6? That'd be fine too.
>
> Thanks,
> S
>

[jira] [Created] (HBASE-22401) Document which branches are 'live' in the ref guide

[Zach York (JIRA)]

Zach York created HBASE-22401:
-

 Summary: Document which branches are 'live' in the ref guide
 Key: HBASE-22401
 URL: https://issues.apache.org/jira/browse/HBASE-22401
 Project: HBase
  Issue Type: Improvement
Reporter: Zach York


As a committer it is hard to keep up with all of the branches that are active. 
This JIRA is to add a definitive list of active branches to the 
contributing/committer section of the ref guide so committers know where bug 
fixes should be pushed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HBASE-22389) Revert HBASE-19275 TestSnapshotFileCache never worked properly on branch-1

[Zach York (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-22389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zach York resolved HBASE-22389.
---
   Resolution: Fixed
Fix Version/s: 1.4.11
   1.5.0

> Revert HBASE-19275 TestSnapshotFileCache never worked properly on branch-1
> --
>
> Key: HBASE-22389
> URL: https://issues.apache.org/jira/browse/HBASE-22389
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.5.0
>Reporter: Zach York
>    Assignee: Zach York
>Priority: Major
> Fix For: 1.5.0, 1.4.11
>
>
> This commit seems broken on branch-1. 
> https://github.com/apache/hbase/commit/d80d3fa45487fedf203c7144e8267753e4d53529#diff-ae046012dd310fc116a729bf36c7df81
>  seems to have fixed the issue that the other HBASE-19275 patches fixed, but 
> when HBASE-19275 was applied to branch-1 it appears to have reverted this 
> code: 
> https://github.com/apache/hbase/commit/e8e4beacb039299a9e56cd355e3ada08784a7acb#diff-ae046012dd310fc116a729bf36c7df81
>  meaning the tests always pass again.
> [~apurtell] [~xucang]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-22390) backport HBASE-22190 to branch-1

[Zach York (JIRA)]

Zach York created HBASE-22390:
-

 Summary: backport HBASE-22190 to branch-1
 Key: HBASE-22390
 URL: https://issues.apache.org/jira/browse/HBASE-22390
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.4.10, 1.5.0
Reporter: Zach York
Assignee: Zach York


HBASE-22190 was only applied to branch-2+ while the bug exists in branch-1 as 
well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-22389) Revert HBASE-19275 TestSnapshotFileCache never worked properly on branch-1

[Zach York (JIRA)]

Zach York created HBASE-22389:
-

 Summary: Revert HBASE-19275 TestSnapshotFileCache never worked 
properly on branch-1
 Key: HBASE-22389
 URL: https://issues.apache.org/jira/browse/HBASE-22389
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.5.0
Reporter: Zach York
Assignee: Zach York


This commit seems broken on branch-1. 
https://github.com/apache/hbase/commit/d80d3fa45487fedf203c7144e8267753e4d53529#diff-ae046012dd310fc116a729bf36c7df81
 seems to have fixed the issue that the other HBASE-19275 patches fixed, but 
when HBASE-19275 was applied to branch-1 it appears to have reverted this code: 
https://github.com/apache/hbase/commit/e8e4beacb039299a9e56cd355e3ada08784a7acb#diff-ae046012dd310fc116a729bf36c7df81
 meaning the tests always pass again.

[~apurtell] [~xucang]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] lazy consensus on "hbase-filesystem" repository

[Zach York]

+1

On Fri, Apr 26, 2019 at 2:48 PM Andrew Purtell  wrote:

> +1
>
> On Fri, Apr 26, 2019 at 7:33 AM Sean Busbey  wrote:
>
> > Hi folks!
> >
> > HBASE-22149 "HBOSS: A FileSystem implementation to provide HBase's
> > required semantics" has an approach to reliably running HBase on top
> > of non-HDFS Filesystem implementations.
> >
> > Copying from my comment there:
> >
> > > Given the combination of this a) needing hadoop 3 only and b) being an
> > experimental approach that we're not sure on sustainability in production
> > I'd much prefer a different repository.
> > >
> > > Is anyone opposed to landing this in a new repository, i.e.
> > `hbase-filesystem`? Provided it includes instructions for installation /
> > set up we wouldn't even need to add the artifacts from that repository
> as a
> > dependency for the main repo's binary artifacts.
> >
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

Re: [DISCUSS] Handling removal of deprecated methods and classes

[Zach York]

Always good to see the history of the discussion :) It looks like nothing
really was decided last time (use caution if removing before a full major
version), hopefully we can come up with something more descriptive this
time.

I think the idea of having a tag/annotation for detailing when the
method/feature is set for removal is a cool idea that we should investigate
further. If every deprecated method/feature had such an annotation, it
would be easy to build automation around generating release
notes/documentation on what was removed in a particular version (instead of
relying on someone to manually update the release note/docs).

I agree that major releases are taking a long time to release and that
causes deprecated features to hang around for a long time, but I also feel
like moving it to >= 1 minor release is a bit too quick and would add to
upgrade burden. Not everyone uses every minor version so they might be
taken unaware by the change or be less likely to upgrade as quickly.
I'm not necessarily against shortening the time. Maybe there is some sort
of middle ground (>=2 minor releases), but that gets a little muddy and
would need to be documented heavily.

Thanks,
Zach

On Tue, Apr 23, 2019 at 9:37 AM Lars Francke  wrote:

> We had this exact same discussion four years ago (I thought it was last
> year :O ): <
>
> https://lists.apache.org/thread.html/bbdbf135abe7d8a0a4e7e9563f0bdc3dd6b555a0d3cdc1b1fdd688c2@1427759561@%3Cdev.hbase.apache.org%3E
> >
>
> On Tue, Apr 23, 2019 at 6:32 PM Sean Busbey  wrote:
>
> > I have the same interpretation as stack. However, I think the
> > expectation back when 1.0.0 was rolling out was that we'd have a
> > regular cadence of major releases to use for removing things that had
> > been deprecated. I don't think that cadence has actually happened.
> >
> > For example, here's a set of major releases and how long it had been
> > since the prior major release at the time it came out:
> >
> > * 0.94 - 4 months
> > * 0.96 - 1.5 years
> > * 0.98 - 4 months
> > * 1.0 - 1 year
> > * 2.0 - 3.25 years
> >
> > If we started a vote today 3.0 would be at 1 year. I suspect it's more
> > likely to end up landing around the 2 year mark.
> >
> > Given this rate and the fact that we've made exceptions in the past
> > (e.g. removing things in 2.0 that were deprecated after 1.0 was
> > already out), should we change our behavior or change what we document
> > to allow removal at a major version without such a long lead time?
> >
> > On Tue, Apr 23, 2019 at 10:17 AM Jan Hentschel
> >  wrote:
> > >
> > > Hi everyone,
> > >
> > > in the PR for HBASE-22262 we
> > noticed that there could be easily a misunderstanding on when to remove
> > deprecated methods. In the book<
> > http://hbase.apache.org/book.html#hbase.versioning.post10> it is
> > documented “an API needs to be deprecated for a major version before we
> > will change/remove it.”. As the discussion on the PR shows it leaves some
> > room for interpretation and should be made more specific, including some
> > examples. I think we all agree that removing a method in 3.0.0 is ok when
> > it was deprecated back in 2.0.0, but what do we do with methods or
> classes
> > which were deprecated in a minor or patch version? As an example, if we
> > deprecate a method in 2.3.0, should it be removed in 3.0.0 or 4.0.0?
> > >
> > > Best, Jan
> >
>

[jira] [Created] (HBASE-22291) Fix recovery of recovered.edits files under root dir

[Zach York (JIRA)]

Zach York created HBASE-22291:
-

 Summary: Fix recovery of recovered.edits files under root dir
 Key: HBASE-22291
 URL: https://issues.apache.org/jira/browse/HBASE-22291
 Project: HBase
  Issue Type: Improvement
Affects Versions: 1.4.9
Reporter: Zach York
Assignee: Zach York


It looks like a few places are using incorrect FS instances in the 
replayRecoveredEditsForPath method that was introduced in HBASE-20734.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Send GitHub notification to issues@hbase

[Zach York]

Will dev@ still get notified when new issues are opened? Or not at all?

+1 if dev@ still gets notified when new issues are opened (how it currently
works with JIRA)

On Wed, Mar 27, 2019 at 5:34 PM Andrew Purtell  wrote:

> +1
>
>
> On Wed, Mar 27, 2019 at 12:29 PM Peter Somogyi 
> wrote:
>
> > Hi,
> >
> > Right now notification emails from GitHub activities are sent to
> dev@hbase
> > for https://github.com/apache/hbase repository. Nick Dimiduk had a
> > suggestion that these should be forwarded to a different hbase mailing
> > list.
> >
> > Since these kinds of emails are already sent to issues@hbase for the
> rest
> > of our repositories (hbase-operator-tools, hbase-connectors and
> hbase-site)
> > I recommend doing the same for hbase as well.
> >
> > Please vote whether you agree to make the change or you're against it.
> >
> > Thanks,
> > Peter
> >
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

Re: [DISCUSS] what's our biggest source of friction for getting new contributors going?

[Zach York]

I think the complexities of actually using HBase is a fairly large
impediment since it limits our user base (where most of the contributors
are likely coming from). HBase is a complex machine and requires someone to
have a significant amount of knowledge of the internal workings to manage.
I think making HBase simpler to use/configure will help. That being said...
HBase isn't worthwhile for every use case and so we won't ever have the
user/dev base of a more universal tool.

On Tue, Mar 19, 2019 at 8:59 PM Misty Linville  wrote:

> #1 this is difficult code. That’s probably the one we can’t surmount but I
> wanted to put it out there.
>
> On Tue, Mar 19, 2019 at 5:03 PM Sean Busbey  wrote:
>
> > I have my own opinions, obvs. But I'm curious what other folks think are
> > the biggest impediments to new contributors?
> >
>

Re: [VOTE] First release candidate for HBase 1.2.11 is available

[Zach York]

+1

- checked sums and sigs: Ok
- Rat check: Ok
- built from source: Ok (8u92)
- Ran unit tests: Ok

On Tue, Feb 19, 2019 at 9:08 AM Wellington Chevreuil <
wellington.chevre...@gmail.com> wrote:

> +1
>
> Signature, checksum: OK
> Build from source: OK
> Unit tests: OK
> Rat check: OK
> Start in standalone mode: OK
> LTT 1M rows: OK
> PE sequentialWrite 1M rows: OK
>
> Em seg, 18 de fev de 2019 às 14:09, Peter Somogyi 
> escreveu:
>
> > +1
> >
> > Signature, checksum: OK
> > Build from source: OK
> > Unit tests: OK
> > Rat check: OK
> > Start in standalone mode: OK
> > LTT 1M rows: OK
> > Shell basic commands: OK
> >
> > On Sat, Feb 16, 2019 at 3:32 AM Sean Busbey  wrote:
> >
> > > The first release candidate for HBase 1.2.11 is available for download:
> > >
> > > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.2.11RC0/
> > >
> > > Maven artifacts are also available in a staging repository at:
> > >
> > >
> https://repository.apache.org/content/repositories/orgapachehbase-1257/
> > >
> > > Artifacts are signed with my key (0D80DB7C) published in our KEYS
> > > file at http://www.apache.org/dist/hbase/KEYS
> > >
> > > The RC corresponds to the signed tag 1.2.11RC0, which currently points
> > > to commit ref
> > >
> > > ca53d58f5b7abde0c189c9f78baf4246bddffac3
> > >
> > > HBase 1.2.11 is the eleventh maintenance release in the HBase 1.2 line,
> > > continuing on the theme of bringing a stable, reliable database to the
> > > Hadoop and NoSQL communities. This release includes two bug fixes done
> > > in the month since 1.2.10.
> > >
> > > The detailed source and binary compatibility report vs 1.2.10 has been
> > > published for your review, at:
> > >
> > > https://s.apache.org/hbase-1.2.11-rc0-compat-report
> > >
> > > The report shows no issues.
> > >
> > > Critical fixes include:
> > >
> > > * HBASE-21196 - HTableMultiplexer clears the meta cache after every put
> > > operation
> > > * HBASE-21374 - FileSystem in use may get closed by other bulk load
> call
> > > in secure bulkLoad
> > >
> > > The full list of fixes included in this release is available at:
> > >
> > > https://s.apache.org/hbase-1.2.11-jira-release-notes
> > >
> > > and in the CHANGES.txt file included in the distribution.
> > >
> > > Please try out this candidate and vote +1/-1 on whether we should
> > > release these artifacts as HBase 1.2.11.
> > >
> > > The VOTE will remain open for at least 72 hours.
> > >
> > > thanks!
> > >
> > > -busbey
> > >
> > > as of this email the posted artifacts have the following SHA512:
> > >
> > > hbase-1.2.11-src.tar.gz:
> > >
> > > 07B09CFC FD331381 6D03CDF3 A91ED896 A212F08F 0AE22B23
> > > AC61F092 96A852CE B5892F66 8F169D47 D877DABD 14862D09
> > > D4B643D1 AA0B09CA DC2A4870 3F9A2891
> > >
> > >
> > > hbase-1.2.11-bin.tar.gz:
> > >
> > > AA781CFC B094DEED BE5BE36B 485B219B 4ECEEBB2 3E5F67A9
> > > 656AFC10 2DEFCBAE AE9DAD5C 9C4984B2 092A3A6E 9CE77EB2
> > > 5DA1159C 4810D65C 3EB6E92D AA782E27
> > >
> >
>

Re: [VOTE] Second release candidate for HBase 2.1.3 is available for download

[Zach York]

Sorry, I just saw this. I think the tool is great and can help make it
easier for PMCs to vote and/or unburden them to do further testing beyond
the basic checks.

I think it makes sense to add this under dev-support/ to make it easy to
access and maintain. Would you mind opening a JIRA to add it?

Thanks,
Zach



On Wed, Feb 13, 2019 at 4:55 PM 张铎(Duo Zhang)  wrote:

> With 3 binding +1s and 2 non-binding +1s, the vote passes. Will push 2.1.3
> out soon.
>
> And thanks Stephen, the tool is great. But we still need 3 PMCs to vote +1
> if we want to make a release, this is required by the ASF policy, an
> automatic tool can not be counted as a binding vote...
>
> Anyway, it will take less time for PMCs to vote on a release if we have a
> good tool for verifying a release, which means we will likely to have more
> PMCs to vote, so thanks for your sharing:)
>
> 张铎(Duo Zhang)  于2019年2月14日周四 上午8:48写道：
>
> > +1(binding)
> >
> > The RC is generated by me so most things such as rat check, signature,
> > etc, are all fine.
> >
> > Run UTs with openjdk 8u202: all passed
> > Run ITBLL with deterministic, 3 loops, 1B rows per each loop: passed.
> >
> > Tak-Lon (Stephen) Wu  于2019年2月14日周四 上午2:15写道：
> >
> >> +1 (non-bindling)
> >>
> >>- checked sums and signatures: ok
> >>- Rat check (8u192):  ok
> >>   - mvn apache-rat:check
> >>- built from source: ok (8u192)
> >>   - mvn clean install -DskipTests
> >>- Unit tests pass: ok (8u92)
> >>   - mvn test -P runAllTests
> >>
> >>
> >> I have been using this script
> >> ,
> >> e.g. bash
> >> hbase-vote.sh 2.1.3 hbase-2.1.3RC1
> >> https://dist.apache.org/repos/dist/dev/hbase/hbase-2.1.3RC1 to confirm
> >> above, hopefully this will be handy for others.
> >>
> >> And I'm assuming the community has an automation running for these
> >> checking
> >> in somewhere, if not, IMO should we introduce an automation and simplify
> >> few steps of checking the new release candidate?
> >>
> >> -Stephen
> >>
> >> On Wed, Feb 13, 2019 at 9:21 AM Norbert Kalmar
> >> 
> >> wrote:
> >>
> >> > +1 (non-binding)
> >> >
> >> > Checksums: ok
> >> > Build from source: ok
> >> > Unit tests: ok
> >> > Start hbase in standalone: ok
> >> > Basic shell commands: ok
> >> >
> >> > On Tue, Feb 12, 2019 at 5:49 PM Peter Somogyi 
> >> wrote:
> >> >
> >> > > +1 (binding)
> >> > >
> >> > > Checked signatures, checksums: ok
> >> > > Apache Rat check: ok
> >> > > Build source: ok (jdk1.8.0_201)
> >> > > Unit tests: ok
> >> > > LTT 1M rows: ok
> >> > > Run in pseudo distributed mode: ok (Hadoop 2.7.7)
> >> > > Shell basic commands: ok
> >> > > Web UI: ok
> >> > > Tested RS Group: ok
> >> > >
> >> > > On Tue, Feb 12, 2019 at 7:48 AM Guanghao Zhang 
> >> > wrote:
> >> > >
> >> > > > +1 (binding)
> >> > > >
> >> > > > hbase-2.1.3-bin.tar.gz (jdk1.8.0_171)
> >> > > > - Verified sha512: ok
> >> > > > - Start HBase in standalone mode: ok
> >> > > > - Verified with shell,
> >> create/disable/enable/drop/get/put/scan/delete:
> >> > ok
> >> > > > - Checked master/regionserver/region Web UI: ok
> >> > > > - PE write/read 100K rows: good
> >> > > >
> >> > > > hbase-2.1.3-src.tar.gz (jdk1.8.0_171)
> >> > > > - Verified sha512: ok
> >> > > > - Build tarball: ok
> >> > > > - Start HBase in standalone mode: ok
> >> > > > - Verified with shell,
> >> create/disable/enable/drop/get/put/scan/delete:
> >> > ok
> >> > > > - Checked master/regionserver/region Web UI: ok
> >> > > > - PE write/read 100K rows: good
> >> > > >
> >> > > > 张铎(Duo Zhang)  于2019年2月11日周一 下午8:15写道：
> >> > > >
> >> > > > > The second release candidate for HBase 2.1.3 is available for
> >> > download:
> >> > > > >
> >> > > > >   https://dist.apache.org/repos/dist/dev/hbase/hbase-2.1.3RC1
> >> > > > >
> >> > > > > Maven artifacts are also available in a staging repository at:
> >> > > > >
> >> > > > >
> >> > > >
> >> >
> https://repository.apache.org/content/repositories/orgapachehbase-1254/
> >> > > > >
> >> > > > > Artifacts are signed with my key (9AD2AE49) published in our
> KEYS
> >> > file
> >> > > at
> >> > > > >
> >> > > > >  http://www.apache.org/dist/hbase/KEYS
> >> > > > >
> >> > > > > The RC corresponds to the signed tag 2.1.3RC1, which currently
> >> points
> >> > > to
> >> > > > > commit
> >> > > > >
> >> > > > >   da5ec9e4c06c537213883cca8f3cc9a7c19daf67
> >> > > > >
> >> > > > > HBase 2.1.3 is the fourth maintenance release in the HBase 2.1
> >> line,
> >> > > > > continuing on the theme of bringing a stable, reliable database
> to
> >> > the
> >> > > > > Hadoop and NoSQL communities. It fixes CVE-2018-1320 by
> upgrading
> >> > > thrift
> >> > > > > dependency from 0.9.3 to 0.12.0, all hbase users who use thrift
> >> are
> >> > > > highly
> >> > > > > recommended to upgrade.
> >> > > > >
> >> > > > > 2.1.3 includes ~60 bug and improvement fixes done since the
> 2.1.2.
> >> > > There
> >> > > > is
> >> > > > > an incompatible change, HBASE-21684, where we 

Re: [DISCUSS] Moving towards a branch-2 line that can get the 'stable' pointer.

[Zach York]

I agree with the sentiment around HBCK2. I think these kind of recovery
tools are essential before marking something stable.

I also remember when we did testing around HBase 2.x/2.1 that we were
getting perf degradations and couldn't seem to get performance to be as
good as we were getting in the 1.x line.

- Zach

On Thu, Jan 17, 2019 at 11:06 PM Pankaj kr  wrote:

> Yeah, HBCK2/ OfflineMetaRepair tools are really required to migrate old
> version data to HBase-2. We have use cases where we are using these tools
> to rebuild the meta for further region assignment.
> Similar discussion is going on HBASE-21665, after fixing the NPE and
> rebuilding the meta, master don't assign the regions as we skip the empty
> regions while loading meta during master startup.
>
> A big +1 from my side on this...
>
> Regards,
> Pankaj
>
> -Original Message-
> From: 张铎(Duo Zhang) [mailto:palomino...@gmail.com]
> Sent: 18 January 2019 11:55
> To: HBase Dev List 
> Subject: Re: [DISCUSS] Moving towards a branch-2 line that can get the
> 'stable' pointer.
>
> So the first priority is to make progress on HBCK2? If we all agree, let's
> start to work.
>
> Andrew Purtell  于2019年1月18日周五 下午12:31写道：
>
> > Sorry, let me add... Check all the boxes on that list and I'm +1 for
> > moving the stable pointer (modulo some time to pound on the candidate
> > to really put it through its paces, like two weeks of chaos...)
> >
> > On Thu, Jan 17, 2019 at 8:28 PM Andrew Purtell 
> > wrote:
> >
> > > I do not believe we should move the stable pointer to any 2.x until
> > > HBCK2 is feature complete. We can discuss what that milestone should
> look like.
> > > At a minimum, I think we need:
> > >
> > >- Rebuild meta from region metadata in the filesystem, aka offline
> > >meta rebuild.
> > >- Fix assignment errors (undeployed regions, double assignments
> (yes,
> > >should not be possible), etc)
> > >- Fix region holes, overlaps, and other errors in the region chain
> > >- Fix failed split and merge transactions that have failed to roll
> > >back due to some bug (related to previous)
> > >- Enumerate store files to determine file level corruption and
> > >sideline corrupt files
> > >- Fix hfile link problems (dangling / broken)
> > >
> > > This is a list of the real problems I have had to fix in production
> > > at least once (in the past 10 years...).
> > >
> > > On Thu, Jan 17, 2019 at 8:19 PM 张铎(Duo Zhang)
> > > 
> > > wrote:
> > >
> > >> There are still lots of small new features which we want to
> > >> integrate
> > into
> > >> branch-2 so I'm -1 on making release directly from branch-2.
> > >> Backporting at once before release is a pain I'd say, I've tried
> > >> this many times recently, as we have to follow up the community
> > >> version...Let's make a branch-2.2 when we want to release 2.2.0,
> > >> and maybe also retire the branch-2.0?
> > >>
> > >> For the stable pointer, I think 2.1.x maybe a good candidate?
> > >> Though we know that we may still have some bugs for the AMv2, but
> > >> actually we all know that the AMv1 for all the branch-1.x also has
> > >> lots of bugs, that's why hbck is very important.
> > >>
> > >> And also +! on making progress on HBCK2, we need to port he useful
> > >> features of HBCK1 to HBCK2. There is no software can guarantee that
> > >> there is no bug, so FWIW we should have a way to fix broken
> > >> clusters.
> > >>
> > >> Sean Busbey  于2019年1月18日周五 上午11:47写道：
> > >>
> > >> > There are a few related topics I'd like to discuss and I figured
> > >> > this subject line is the most likely to get a bit of attention.
> > >> > :)
> > >> >
> > >> > First, I'd like us all to get on the same page wrt the current
> > >> > state of branch-2. Personally, I don't think it can be released
> > >> > as-is with a 2.y version because folks can't rolling upgrade from
> > >> > 2.0 or 2.1 to it due to the current implementation of
> > >> > HBASE-20881. As Duo has mentioned a couple of times, folks have
> > >> > to ensure there are no region transitions around during the
> > >> > upgrade. I think that will be prohibitive for folks looking to
> upgrade. What do other folks think?
> > >> >
> > >> > Second, I think our recent discussions around the need for
> > >> > shifting to more minor releases for HBase 1.y also applies to the
> 2.y branches.
> > >> > branch-2 hasn't had a release since 2.1.0 came out in July 2018.
> > >> > That's a scary long amount of time. I think it contributes to us
> > >> > ending up with changes like the above since it's easy to think
> > >> > about the branch as something that has a lot of time before the
> > >> > next release.
> > >> >
> > >> > Personally, I'd like to see us skip making minor-release specific
> > >> > branches for a bit unless a CVE fix or something comes up.
> > >> > Ideally, that would mean we work towards a 2.2.0 release directly
> > >> > from branch-2 and then 2.2.1, etc. When we have a feature that's
> > >> > ready to 

Re: Seek for help to create an account on review.apache

[Zach York]

It looks like it is now much easier to close PRs so I think it makes sense
to use a PR as a substitute for ReviewBoard in the short term. Note that it
would be the committer's responsibility to close the PR.

I think this is a good intermediate step before we figure out the process
to commit using PR.

On Wed, Jan 16, 2019 at 8:15 AM Sean Busbey  wrote:

> Unfortunately it looks like review board posting is now only available
> to ASF committers. We'll have to update our community docs to
> de-emphasize using it. :/
>
> Wang Huan it looks like you have a patch file attached to the JIRA for
> HBASE-21699. Until a reviewer asks for it in a different format that
> ought to be fine. If the change is big enough that review board is
> needed then we'll have to get a committer to upload it.
>
> Misty is correct that you could put up a GitHub PR, but keep in mind
> we (the project) haven't gotten much process expectations in place for
> those so how well it goes will depend on the specific reviewers
> looking at your contribution.
>
> On Tue, Jan 15, 2019 at 10:31 PM Misty Linville  wrote:
> >
> > I think you may need to contact infra directly. I'm sorry you're facing
> > this trouble and I appreciate the extra effort you're going through to
> > contribute. I wonder if you can try creating a pull request on GitHub
> > instead of using Reviewboard until you get a resolution. I confess I am
> not
> > as familiar with the Gitbox functionality as I should be. Can anyone else
> > help out here? I've added the PMC list just in case it helps speed things
> > along.
> >
> > On Tue, Jan 15, 2019, 6:59 PM wang huan  >
> > > Hi, Zhang: https://reviews.apache.org have enabled LDAP,  can not log
> in
> > > with account on jira.
> > >
> > > On 16/1/19, 10:17 AM, "张铎(Duo Zhang)"  wrote:
> > >
> > > I think it uses the same account with jira? i.e, issues.apache.org
> .
> > >
> > > wang huan  于2019年1月16日周三 上午10:13写道：
> > >
> > > > Hi, All:
> > > > I want to login https://reviews.apache.org to upload my
> patch
> > > > onhttps://jira.apache.org/jira/browse/HBASE-21699,  all apache
> > > projects
> > > > told me go to https://reviews.apache.org and create an account,
> but
> > > no
> > > > register page on there !
> > > >  My colleague told me he got his account from dev@mesos, so
> I
> > > want to
> > > > know if someone can
> > > > help me create an account
> > > >
> > > > Best Regards,
> > > > huan
> > > >
> > > >
> > >
> > >
>

Re: [VOTE] The second release candidate for HBase 2.0.4 is available

[Zach York]

Ankit,

This seems related to HBASE-21548.

On Thu, Jan 3, 2019 at 6:04 PM Ankit Singhal 
wrote:

> +1
>
> -- Build from src - Ok
> -- Started a local cluster of 2.0.4 on existing 2.0 deployment - Ok
> -- Basic shell commands - Ok
> -- UI - Ok
> -- Ran rat check on source - Success
> -- SHA and Signatures - Good
> -- Ran selected Integration tests on cluster - ok
> -- HBase local setup used to work with local filesystem but this time I
> have to set these properties to false (hbase.procedure.store.wal.use.hsync
> and hbase.unsafe.stream.capability.enforce), Not sure if there is any
> change in defaults but calling out just in case developers sees following
> error (and know what to change) "java.lang.IllegalStateException: The
> procedure WAL relies on the ability to hflush for proper operation during
> component failures, but the underlying filesystem does not support doing
> so. Please check the config value of 'hbase.procedure.store.wal.use.hsync'
> to set the desired level of robustness and ensure the config value of
> 'hbase.wal.dir' points to a FileSystem mount that can provide it."
>
> Regards,
> Ankit Singhal
>
> On Thu, Jan 3, 2019 at 6:01 AM Peter Somogyi  wrote:
>
> > +1
> >
> > Checksum, signature: OK
> > Build from source: OK
> > Unit tests: OK (TestVerifyReplication fails sporadically)
> > Apache Rat check: OK
> > Web UI: OK
> > LTT 1M rows: OK
> > Shell: OK
> >
> > Thanks,
> > Peter
> >
> > On Thu, Jan 3, 2019 at 7:29 AM Stack  wrote:
> >
> > > Thanks for the ITBLL Allan.
> > >
> > > +1 from me too. Thats three binding +1s. Will leave it open another
> while
> > > in case anyone else wants to take a looksee.
> > >
> > > S
> > >
> > > On Sat, Dec 29, 2018 at 8:44 AM Stack  wrote:
> > >
> > > > The second release candidate for HBase 2.0.4 is available for
> download:
> > > >
> > > >  *https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.4RC1/
> > > > *
> > > >
> > > > Maven artifacts are also available in a staging repository at:
> > > >
> > > >
> > https://repository.apache.org/content/repositories/orgapachehbase-1246
> > > >
> > > > Artifacts are signed with my key (DB9D313DA7874F29) published in our
> > > > KEYS file at http://www.apache.org/dist/hbase/KEYS
> > > >
> > > > The RC corresponds to the signed tag 2.0.4RC1, which currently points
> > > > to commit
> > > >
> > > >   205e39c5704bf38568b34926dde9f1ee76e6b5d0
> > > >
> > > > HBase 2.0.4 is the fourth maintenance release in the HBase 2.0 line,
> > > > continuing on the theme of bringing a stable, reliable database to
> > > > the Hadoop and NoSQL communities. It fixes a critical issue found
> > > > in the recent 2.0.3 and 2.1.1 releases (only), HBASE-21551. 2.0.4
> > > > includes ~31 bug and improvement fixes done since the 2.0.3,
> > > > ~ a month ago.
> > > >
> > > > The detailed source and binary compatibility report vs 2.0.3 has been
> > > > published for your review, at:
> > > >
> > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.4RC1/compatibility_report_2.0.3vs2.0.4.html
> > > >
> > > > The report shows no incompatibilities.
> > > >
> > > > The full list of fixes included in this release is available in
> > > > the CHANGES.md that ships as part of the release also available
> > > > here:
> > > >
> > > >
> > https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.4RC1/CHANGES.md
> > > >
> > > > The RELEASENOTES.md are here:
> > > >
> > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.4RC1/RELEASENOTES.md
> > > >
> > > > Please try out this candidate and vote +1/-1 on whether we should
> > > > release these artifacts as HBase 2.0.4.
> > > >
> > > > The VOTE will remain open for at least 72 hours. It'll probably take
> > > > longer given
> > > > its the holidays. I'll close the vote when sufficient votes after
> > > > Tuesday, January
> > > > 1st, 2019.
> > > >
> > > > Thanks,
> > > > S
> > > >
> > >
> >
>

Re: [VOTE] The second HBase 1.4.9 release candidate (RC1) is available

[Zach York]

+1

- checked sums and sigs: Ok
- Rat check: Ok
- built from source: Ok (8u92)
- Ran unit tests: Ok
- Messed around with some basic shell commands: Ok
- Loaded and read 1M rows with LTT: Ok
- Sanity checked Web UI: Ok

On Tue, Dec 11, 2018 at 12:28 AM Peter Somogyi  wrote:

> +1 (non-binding)
>
> * Signature, checksum: OK
> * Apache Rat check: OK
> * Build from source: OK
> * CHANGES.txt: OK
> * Unit tests (8u191, Maven 3.5.3): OK
> * LTT 1M rows: OK
> * Shell commands: OK
> * Web UI: OK
>
> On Sat, Dec 8, 2018 at 7:01 AM Andrew Purtell 
> wrote:
>
> > YCSB workloads are not that interesting. I think it’s still useful as a
> > “standard” tool though that indicates we aren’t doing worse. Also, there
> > may be more difference under overload conditions. Next time.
> >
> > Using Phoenix and some of our internal test tools we can generate
> > workloads that turn up interesting differences at the day job, but mostly
> > between 0.98 and any 1.x, and the changes were small and additive over
> > time. Some of our folks contributed perf fixes that return some of the
> > difference, 10%, 20% here and there. There are opportunities for more.
> >
> > For 1.5 branch-1 has some interesting changes, like pressure aware
> > compaction rate limiting on by default, which may change things (via
> > influence on GC). TinyLRU if it goes in might also be interesting.
> >
> >
> > > On Dec 7, 2018, at 8:02 PM, Stack  wrote:
> > >
> > > Thanks for the test runs.
> > >
> > > The diffs are miniscule. After so many releases, would have expected a
> > > tendency up or down but not constant (smile).
> > >
> > > S
> > >
> > >> On Fri, Dec 7, 2018 at 5:34 PM Andrew Purtell 
> > wrote:
> > >>
> > >> Today I did a comparison between 1.2.6.1 and 1.4.9RC1 with YCSB. The
> > >> results are close. Overall runtimes are almost the same. In the
> average
> > and
> > >> high percentile measures there is a general upward trend but nothing
> > that
> > >> looks like a significant regression. Still for 1.5.0 I think we should
> > see
> > >> if it's possible to lower measures made by YCSB closer to those
> observed
> > >> with 1.2.
> > >>
> > >> Linux version 4.14.55-62.37.amzn1.x86_64
> > >> c3.8xlarge x 5
> > >> OpenJDK Runtime Environment (build 1.8.0_181-shenandoah-b13)
> > >> -Xms20g -Xmx20g -XX:+UseG1GC -XX:+AlwaysPreTouch -XX:+UseNUMA
> > >> -XX:-UseBiasedLocking -XX:+ParallelRefProcEnabled
> > >> Hadoop 2.8.5
> > >> Init: Load 100 M rows and snapshot
> > >> Run: Delete table, clone and redeploy from snapshot, run 10 M
> operations
> > >> (except 1M for workload E)
> > >> Args: -threads 100 -target 5 (except 1 for workload E)
> > >>
> > >> YCSB Workload A
> > >>
> > >> target 50k/op/s 1.2.6.1 1.4.9
> > >>
> > >>
> > >>
> > >> [OVERALL], RunTime(ms) 200581 200605
> > >> [OVERALL], Throughput(ops/sec) 49855 49849
> > >> [READ], AverageLatency(us) 491 517
> > >> [READ], MinLatency(us) 233 273
> > >> [READ], MaxLatency(us) 140287 165503
> > >> [READ], 95thPercentileLatency(us) 605 647
> > >> [READ], 99thPercentileLatency(us), 880 1031
> > >> [UPDATE], AverageLatency(us) 1332 1327
> > >> [UPDATE], MinLatency(us) 711 692
> > >> [UPDATE], MaxLatency(us) 137215 160383
> > >> [UPDATE], 95thPercentileLatency(us) 1790 1800
> > >> [UPDATE], 99thPercentileLatency(us) 2333 2495
> > >>
> > >> YCSB Workload B
> > >>
> > >> target 50k/op/s 1.2.6.1 1.4.9
> > >>
> > >>
> > >>
> > >> [OVERALL], RunTime(ms) 200569 200566
> > >> [OVERALL], Throughput(ops/sec) 49858 49859
> > >> [READ], AverageLatency(us),  438 445
> > >> [READ], MinLatency(us) 198 216
> > >> [READ], MaxLatency(us) 143615 153343
> > >> [READ], 95thPercentileLatency(us) 539 552
> > >> [READ], 99thPercentileLatency(us) 806 957
> > >> [UPDATE], AverageLatency(us) 1098 1064
> > >> [UPDATE], MinLatency(us) 746 762
> > >> [UPDATE], MaxLatency(us) 141183 149503
> > >> [UPDATE], 95thPercentileLatency(us) 1334 1336
> > >> [UPDATE], 99thPercentileLatency(us) 1695 1777
> > >>
> > >> YCSB Workload C
> > >>
> > >> target 50k/op/s 1.2.6.1 1.4.9
> > >>
> > >>
> > >>
> > >> [OVERALL], RunTime(ms) 200543 200559
> > >> [OVERALL], Throughput(ops/sec) 49865 49861
> > >> [READ], AverageLatency(us) 348 340
> > >> [READ], MinLatency(us) 174 182
> > >> [READ], MaxLatency(us) 140287 161279
> > >> [READ], 95thPercentileLatency(us) 428 421
> > >> [READ], 99thPercentileLatency(us) 793 841
> > >>
> > >> YCSB Workload D
> > >>
> > >> target 50k/op/s 1.2.6.1 1.4.9
> > >>
> > >>
> > >>
> > >> [OVERALL], RunTime(ms) 200557 200577
> > >> [OVERALL], Throughput(ops/sec) 49861 49856
> > >> [READ], AverageLatency(us) 528 503
> > >> [READ], MinLatency(us) 187 213
> > >> [READ], MaxLatency(us) 134655 147199
> > >> [READ], 95thPercentileLatency(us) 1395 991
> > >> [READ], 99thPercentileLatency(us) 1880 1721
> > >> [INSERT], AverageLatency(us) 1272 1245
> > >> [INSERT], MinLatency(us) 830 827
> > >> [INSERT], MaxLatency(us) 124479 140671
> > >> [INSERT], 95thPercentileLatency(us) 1518 1505
> > >> [INSERT], 99thPercentileLatency(us) 

Re: Rolling 2.1.2 and 2.0.4

[Zach York]

Do we have a process for deprecating releases with critical bugs? I'm not
sure this particular bug would warrant that, but I'm just curious if such a
process is in place.

On Wed, Dec 5, 2018, 9:46 PM Stack  Let me put notice on download page too (Let me write the user-list).
>
> Wondering if anyone has a bit of time to take a look at the flakeys? For
> 2.1, they cause a fail every other time:
>
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.0/lastSuccessfulBuild/artifact/dashboard.html
>
> Otherwise I'll try and take a look.
>
> Thanks,
> St.Ack
>
> On Wed, Dec 5, 2018 at 9:34 PM Allan Yang  wrote:
>
> > According to @Zheng Hu, this memory leak was introduced by HBASE-20704
> > , which will only
> > affect
> > the recently released version 2.0.3 and 2.1.1. I also think we need to
> > notice those users.
> > Best Regards
> > Allan Yang
> >
> >
> > Yu Li  于2018年12月6日周四 下午12:46写道：
> >
> > > Memory leak is critical enough to roll a new release, and maybe we
> should
> > > add a notice somewhere for 2.x users, like our ref guide and/or user
> > > mailing list? Thanks.
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Thu, 6 Dec 2018 at 12:39, Reid Chan  wrote:
> > >
> > > > +1 for rolling.
> > > >
> > > > Nice found, Zheng Hu. (y)
> > > >
> > > >
> > > > --
> > > >
> > > > Best regards,
> > > > R.C
> > > >
> > > >
> > > >
> > > > 
> > > > From: Stack 
> > > > Sent: 06 December 2018 12:03
> > > > To: HBase Dev List
> > > > Subject: Rolling 2.1.2 and 2.0.4
> > > >
> > > > 2.1.1 and 2.0.3 have an ugly bug found by our Zheng Hu. See
> HBASE-21551
> > > > Memory leak when use scan with STREAM at server side.
> > > >
> > > > S
> > > >
> > >
> >
>

Re: [DISCUSS] JDK8 for >=1.5.x?

[Zach York]

Isn't Phoenix's compatibility guarantees separate from HBase's? If Phoenix
only works with a Java8 environment, then couldn't Phoenix only support
Java8 environments in releases after the Java 8 Avatica issue without
requiring HBase's compatibility to drop support for Java7?

While it's nice to have Phoenix and HBase compatibility guarantees remain
in sync, I don't think that's a requirement for either project.
I do feel like dropping support for a java version in a minor release is
very questionable as a Java upgrade isn't always trivial. Unfortunately,
this is a by product of our long lived release lines.

On Tue, Dec 4, 2018 at 4:21 PM Andrew Purtell  wrote:

> We have the current compatibility promise that we will build with Java 7 so
> folks using a Java 7 JRE can consume and deploy them.
>
> If we will be assuming Java 8 runtimes predominate, then we can build with
> Java 8.
>
>
> On Tue, Dec 4, 2018 at 4:15 PM Sean Busbey  wrote:
>
> > What about installing Phoenix coprocessors built with Java 8 into a Java
> 8
> > runtime that's running the currently built for jdk7 HBase? Shouldn't that
> > work fine?
> >
> > On Tue, Dec 4, 2018, 16:16 Andrew Purtell  >
> > > I thought only the Avatica module aka PQS must be compiled using Java
> 8?
> > > Maybe you can script a Phoenix build that uses the Java 7 compiler for
> > the
> > > coprocessor modules and a Java 8 compiler only for the PQS?
> > >
> > > Otherwise, if all Phoenix modules including the coprocessor code are
> > > compiled with Java 8, then HBase must also be compiled with Java 8. If
> > you
> > > attempt to install Phoenix coprocessors compiled with Java 8 into a
> Java
> > > 7 + HBase runtime then you will see fatal runtime errors related to
> > linkage
> > > errors in the concurrent data types.
> > >
> > >
> > > On Tue, Dec 4, 2018 at 1:54 PM Josh Elser  wrote:
> > >
> > > > On 12/4/18 3:28 PM, Sean Busbey wrote:
> > > > > On Tue, Dec 4, 2018 at 2:14 PM Josh Elser 
> wrote:
> > > > >>
> > > > >> Phoenix depends on Avatica for its Phoenix Query Server. Avatica
> > > > >> requires Java8. Thus, the impasse.
> > > > >>
> > > > >
> > > > > Generally our maintaining JDK7 support shouldn't limit the ability
> of
> > > > > any downstream user to require a newer JDK. Is there some context
> I'm
> > > > > missing? Are they manually recompiling some part of our codebase
> that
> > > > > doesn't work with JDK8? Or do they not want to have folks
> complaining
> > > > > to them for requiring a JDK update that we don't require?
> > > >
> > > > Phoenix isn't doing anything exceptional.
> > > >
> > > > I'm struggling to find the Phoenix thread where Andrew P suggested
> that
> > > > I start this discussion in HBase. I don't remember if Phoenix is
> using
> > > > JDK7 purely to build solely because HBase does, or if there is a
> > > > technical reason that this is required. I'll have to keep looking.
> > > >
> > > > >> I'd pose the question: are we really doing our users a service for
> > > > >> continuing to allow them to use Java7? Not trying to be
> contentious
> > in
> > > > >> asking that -- I am just (happily) seeing fewer and fewer folks
> > still
> > > on
> > > > >> Java7.
> > > > >
> > > > > Yes. Making it so downstream folks don't have to worry about
> certain
> > > > > categories of changes is the entire advantage of us declaring
> > backward
> > > > > compatibility promises at all. JDK changes are a big deal in any
> > > > > deployment.
> > > > >
> > > > > As a project we already have a path forward for no longer worrying
> > > > > about maintaining Java 7 support; it's Apache HBase 2+. Providing
> > > > > advantages for that move and reducing the risk of updating to it
> is a
> > > > > straight forward way to remove our limitation on JDK features.
> IMHO,
> > > > > making it riskier to upgrade to newer 1.y minor versions is an
> > > > > undesirable way to drive that change.
> > > >
> > > > I disagree with you on this point but that is fine.
> > > >
> > >
> > >
> > > --
> > > Best regards,
> > > Andrew
> > >
> > > Words like orphans lost among the crosstalk, meaning torn from truth's
> > > decrepit hands
> > >- A23, Crosstalk
> > >
> >
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

Re: [VOTE] First release candidate for HBase 2.0.3 is available

[Zach York]

+1

- checked sums and sigs: Ok
- Rat check: Ok
- built from source: Ok (8u92)
- Ran unit tests: Ok
- Messed around with some basic shell commands: Ok
- Loaded and read 1M rows with LTT: Ok
- Sanity checked Web UI: Ok

On Fri, Nov 30, 2018 at 4:16 PM Zach York 
wrote:

> I'm in the process of testing it. I'll be back in an hour or less
> (hopefully :))
>
> On Fri, Nov 30, 2018 at 3:27 PM Stack  wrote:
>
>> I love these non-binding votes. Thanks to you all for trying the RC. Need
>> one more binding vote to release after my +1 here (Allan Yang just made
>> PMC! Yah! for Allan!).
>> Thanks,
>> S
>>
>> On Sun, Nov 25, 2018 at 4:22 PM Stack  wrote:
>>
>> > The first release candidate for HBase 2.0.3 is available for download:
>> >
>> >  https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.3RC0/
>> >
>> > Maven artifacts are also available in a staging repository at:
>> >
>> >  https://repository.apache.org/content/repositories/orgapachehbase-1237
>> >
>> > Artifacts are signed with my key (DB9D313DA7874F29) published in our
>> > KEYS file at http://www.apache.org/dist/hbase/KEYS
>> >
>> > The RC corresponds to the signed tag 2.0.3RC0, which currently points
>> > to commit
>> >
>> >   87a3aea8ee2d284807f7d4fbdac1f6d9dfedbc17
>> >
>> > HBase 2.0.3 is the third maintenance release in the HBase 2.0 line,
>> > continuing on the theme of bringing a stable, reliable database to
>> > the Hadoop and NoSQL communities. This release includes ~120 bug
>> > and improvements fixes done since the 2.0.2 release almost 3 months
>> > ago.
>> >
>> > The detailed source and binary compatibility report vs 2.0.2 has been
>> > published for your review, at:
>> >
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.3RC0/compat-check-report-2.0.2-vs-2.0.3.html
>> >
>> > The report shows no incompatibilities.
>> >
>> > The full list of fixes included in this release is available in
>> > the CHANGES.md that ships as part of the release also available
>> > here:
>> >
>> >  https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.3RC0/CHANGES.md
>> >
>> > The RELEASENOTES.md are here:
>> >
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.3RC0/RELEASENOTES.md
>> >
>> > Please try out this candidate and vote +1/-1 on whether we should
>> > release these artifacts as HBase 2.0.3.
>> >
>> > The VOTE will remain open for at least 72 hours. Given sufficient votes
>> > I would like to close it on Thursday, November 29th, 2018.
>> >
>> > Thanks,
>> > S
>> >
>>
>

Re: [VOTE] First release candidate for HBase 2.0.3 is available

[Zach York]

I'm in the process of testing it. I'll be back in an hour or less
(hopefully :))

On Fri, Nov 30, 2018 at 3:27 PM Stack  wrote:

> I love these non-binding votes. Thanks to you all for trying the RC. Need
> one more binding vote to release after my +1 here (Allan Yang just made
> PMC! Yah! for Allan!).
> Thanks,
> S
>
> On Sun, Nov 25, 2018 at 4:22 PM Stack  wrote:
>
> > The first release candidate for HBase 2.0.3 is available for download:
> >
> >  https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.3RC0/
> >
> > Maven artifacts are also available in a staging repository at:
> >
> >  https://repository.apache.org/content/repositories/orgapachehbase-1237
> >
> > Artifacts are signed with my key (DB9D313DA7874F29) published in our
> > KEYS file at http://www.apache.org/dist/hbase/KEYS
> >
> > The RC corresponds to the signed tag 2.0.3RC0, which currently points
> > to commit
> >
> >   87a3aea8ee2d284807f7d4fbdac1f6d9dfedbc17
> >
> > HBase 2.0.3 is the third maintenance release in the HBase 2.0 line,
> > continuing on the theme of bringing a stable, reliable database to
> > the Hadoop and NoSQL communities. This release includes ~120 bug
> > and improvements fixes done since the 2.0.2 release almost 3 months
> > ago.
> >
> > The detailed source and binary compatibility report vs 2.0.2 has been
> > published for your review, at:
> >
> >
> >
> https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.3RC0/compat-check-report-2.0.2-vs-2.0.3.html
> >
> > The report shows no incompatibilities.
> >
> > The full list of fixes included in this release is available in
> > the CHANGES.md that ships as part of the release also available
> > here:
> >
> >  https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.3RC0/CHANGES.md
> >
> > The RELEASENOTES.md are here:
> >
> >
> >
> https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.3RC0/RELEASENOTES.md
> >
> > Please try out this candidate and vote +1/-1 on whether we should
> > release these artifacts as HBase 2.0.3.
> >
> > The VOTE will remain open for at least 72 hours. Given sufficient votes
> > I would like to close it on Thursday, November 29th, 2018.
> >
> > Thanks,
> > S
> >
>

Re: [ANNOUNCE] Allan Yang joins the Apache HBase PMC

[Zach York]

Congratulations and welcome Allan!


On Wed, Nov 28, 2018 at 10:21 AM Esteban Gutierrez
 wrote:

> Congratulations, Allan!
>
> --
> Cloudera, Inc.
>
>
>
> On Wed, Nov 28, 2018 at 10:11 AM Yu Li  wrote:
>
> > On behalf of the Apache HBase PMC I am pleased to announce that Allan
> Yang
> > has accepted our invitation to become a PMC member on the Apache HBase
> > project. We appreciate Allan stepping up to take more responsibility in
> the
> > HBase project.
> >
> > Please join me in welcoming Allan to the HBase PMC!
> >
> > Best Regards,
> > Yu
> >
>

Site build depends on hbase-annotation:test jar

[Zach York]

Hey All,

I was recently trying to build a HBase RC using Apache Bigtop (In my setup
it's easier to have it build with bigtop to test on cluster) and I ran into
a rather curious error. The build was failing because it was trying to
download the hbase-annotations:test jar from maven central instead of
locally. This is only an issue when building with Bigtop or building with
the site option (mvn install site). I did a bit of research and it seems
like it is due to the fact that maven executes both goals in order for each
module instead of completely executing one goal before the other. Obviously
I can remove the site option from the build command, but that causes a
whole bunch of other errors since the spec files expect the docs to be
present.

So my question is whether we can improve this or is the best I can do to
run the install and site as two separate commands? What does site need from
the annotations test jar?

I understand that this is a bit of an edge case, but since it is used by
Apache Bigtop, it would be nice to improve it.

Thanks,
Zach

Re: [DISCUSS] Gathering metrics on HBase versions in use

[Zach York]

Can we have metrics around the downloads from Apache/archives? I'm not sure
how that is all set up, but might be a low cost way to get some metrics.

On Wed, Nov 14, 2018, 12:12 PM Andrew Purtell  While it seems you are proposing some kind of autonomous ongoing usage
> metrics collection, please note I ran an anonymous version usage survey via
> surveymonkey for 1.x last year. It was opt in and there were no PII
> concerns by its nature. All of the issues around data collection, storage,
> and processing were also handled (by surveymonkey). Unfortunately I
> recently cancelled my account.
>
> For occasional surveys something like that might work. Otherwise there are
> a ton of questions: How do we generate the data? How do we get per-site
> opt-in permission? How do we collect the data? Store it? Process it? Audit
> it? Seems more trouble than it's worth and requires ongoing volunteer
> hosting and effort to maintain.
>
>
> On Wed, Nov 14, 2018 at 11:47 AM Misty Linville  wrote:
>
> > When discussing the 2.0.x branch in another thread, it came up that we
> > don’t have a good way to understand the version skew of HBase across the
> > user base. Metrics gathering can be tricky. You don’t want to capture
> > personally identifiable information (PII) and you need to be transparent
> > about what you gather, for what purpose, how long the data will be
> > retained, etc. The data can also be sensitive, for instance if a large
> > number of installations are running a version with a CVE or known
> > vulnerability against it. If you gather metrics, it really needs to be
> > opt-out rather than opt-in so that you actually get a reasonable amount
> of
> > data. You also need to stand up some kind of metrics-gathering service
> and
> > run it somewhere, and some kind of reporting / visualization tooling. The
> > flip side of all these difficulties is a more intelligent way to decide
> > when to retire a branch or when to communicate more broadly / loudly
> asking
> > people in a certain version stream to upgrade, as well as where to
> > concentrate our efforts.
> >
> > I’m not sticking my hand up to implement such a monster. I only wanted to
> > open a discussion and see what y’all think. It seems to me that a few
> > must-haves are:
> >
> > - Transparency: Release notes, logging about the status of
> > metrics-gathering (on or off) at master or RS start-up, logging about
> > exactly when and what metrics are sent
> > - Low frequency: Would we really need to wake up and send metrics more
> > often than weekly?
> > - Conservative approach: Only collect what we can find useful today,
> don’t
> > collect the world.
> > - Minimize PII: This probably means not trying to group together
> > time-series results for a given server or cluster at all, but could make
> > the data look like there were a lot more clusters running in the world
> than
> > really are.
> > - Who has access to the data? Do we make it public or limit access to the
> > PMC? Making it public would bolster our discipline about transparency and
> > minimizing PII.
> >
> > I’m sure I’m missing a ton so I leave the discussion to y’all.
> >
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

Re: [ANNOUNCE] New HBase committer Jingyun Tian

[Zach York]

Congratulations, Jingyun! Welcome!

On Tue, Nov 13, 2018 at 10:59 AM Peter Somogyi  wrote:

> Congratulations Jingyun!
>
> On Tue, Nov 13, 2018 at 8:54 AM 张铎(Duo Zhang) 
> wrote:
>
> > On behalf of the Apache HBase PMC, I am pleased to announce that Jingyun
> > Tian has accepted the PMC's invitation to become a committer on the
> > project. We appreciate all of Jingyun's generous contributions thus far
> and
> > look forward to his continued involvement.
> >
> > Congratulations and welcome, Jingyun!
> >
>

Re: [VOTE] First release candidate for hbase-2.1.1 (RC0) is available for download and test

[Zach York]

+1

- checked sums and sigs: Ok
- Rat check: Ok
- built from source: Ok (8u92)
- Ran unit tests: Ok
- Messed around with some basic shell commands: Ok
- Loaded and read 1M rows with LTT: Ok
- Sanity checked Web UI: Ok

On Mon, Oct 29, 2018 at 9:11 PM Tak-Lon (Stephen) Wu 
wrote:

> +1 (non-binding)
>
>
>
>- checked sums and signatures: ok
>- Rat check (8u181):  ok
>   - mvn apache-rat:check
>- built from source: ok (8u181)
>   - mvn clean install -DskipTests
>- Unit tests pass: ok (8u181)
>   - mvn test -P runAllTests
>   - [nit] TestRegionInfoDisplay.testRegionDetailsForDisplay failed
>   once, but the second time it passed. the error was:
>  -   [ERROR]   TestRegionInfoDisplay.testRegionDetailsForDisplay:78
>  expected:<...:31 UTC 2018 (PT0.00[5]S ago), server=null> but
> was:<...:31
>  UTC 2018 (PT0.00[6]S ago), server=null>
>   - 1 node HBase cluster test on Hadoop-2.8.4
>   - WEB UI (8u181): checked Procedures/Process Metrics/Table
>   Details/Home looks correct
>   - basic HBase shell command
>   [create/put/get/scan/flush/list/disable/drop] (8u181) : ok
>   - Loaded 1M rows with LTT (8u181):  ok
>  - hbase ltt -tn TestTable -write 10:16 -num_keys 100
>  - hbase ltt -tn TestTable -read 100 -num_keys 100
>
> -Stephen
>
> On Mon, Oct 29, 2018 at 2:25 PM Stack  wrote:
>
> > I'm +1
> >
> > 10B ITBLL w/ slow deterministic just completed and verified on my little
> > 6node cluster.
> >
> > Nightlies are pretty good. #540 is when the last commits for 2.1.1 went
> in
> > and ignoring the cancelled, ~50% passed:
> >
> >
> https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-2.1/
> >
> > S
> >
> >
> >
> > On Sat, Oct 27, 2018 at 11:29 AM Stack  wrote:
> >
> > > The first release candidate for Apache HBase 2.1.1 (RC0) is available
> > > for download and testing. This is a bug fix release with 180+ commits
> [1]
> > > since hbase-2.1.0. Release notes are available here [2]. The
> > compatibility
> > > report is at [3].
> > >
> > > Artifacts are available here:
> > >
> > >   https://dist.apache.org/repos/dist/dev/hbase/hbase-2.1.1RC0/
> > >
> > > Corresponding convenience artifacts for maven use are in the staging
> > > repository:
> > >
> > >
> https://repository.apache.org/content/repositories/orgapachehbase-1235
> > >
> > > All artifacts are signed with my code signing key, 8ACC93D2, which is
> > > also in the project KEYS file:
> > >
> > > http://www.apache.org/dist/hbase/KEYS
> > >
> > > These artifacts correspond to commit ref
> > >
> > >  b60a92d6864ef27295027f5961cb46f9162d7637
> > >
> > > which has been tagged as 2.1.1RC0.
> > >
> > > Nightlies are looking pretty good:
> > >
> > >
> > >
> >
> https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-2.1/
> > >
> > > Please take a few minutes to verify the release and vote on releasing
> it:
> > >
> > > [ ] +1 Release these artifacts as Apache HBase 2.1.1
> > > [ ] -1 Do not release this package because ...
> > >
> > > This VOTE thread will remain open for at least 72 hours. It will close
> > > for sure
> > > Wednesday night (10/31).
> > >
> > > Thanks,
> > > S
> > >
> > > 1.
> > https://dist.apache.org/repos/dist/dev/hbase/hbase-2.1.1RC0/CHANGES.md
> > > 2.
> > >
> >
> https://dist.apache.org/repos/dist/dev/hbase/hbase-2.1.1RC0/RELEASENOTES.md
> > >
> > > P.S. Compatibility report is taking time. Will be back with it later
> > today.
> > >
> >
>
>
> --
> Stephen Wu | Indiana University, Bloomington
>

Re: [DISCUSS] Changing hadoop check versions in our hbase-personality?

[Zach York]

Thanks for the context Duo!

Yes I agree, if we know about a CVE for a dependency it is our duty to
either upgrade the dependency or disclose the vulnerability (ReleaseNotes?)
so that users can make the necessary decision.
However, this obviously does not apply for any CVE that hasn't been brought
to our attention.

Just my 2 cents.

Zach

P.S. I am fine with the removal now that I have more context.

On Tue, Oct 23, 2018 at 9:44 AM Sean Busbey  wrote:

> I can get behind more aggressively updating our dependencies to avoid
> CVEs. I don't think we should do this in maintenance releases though.
> Maintenance releases are meant to be extremely low risk for downstream
> users. Despite our efforts to date upgrading a dependency is still
> disruptive, especially when it's Hadoop. CVEs carry with them a needed
> context for something to be an issue. That context almost never covers
> all possible deployment scenarios and we should leave it to downstream
> users to decide if the risk / reward trade off of justifies the
> dependency update. Asking folks who think the risk is worth it to bump
> up a minor HBase version or patch their deployment locally is a
> reasonable trade off IMHO.
>
> I think I have the Hadoop PMC on board for publicizing impacted
> versions on CVE-2018-8009 specifically. Give me a couple of days to
> get that out in whatever form so that everyone in this discussion has
> a more level field?
> On Mon, Oct 22, 2018 at 9:07 PM 张铎(Duo Zhang) 
> wrote:
> >
> > I believe if there is a CVE for any of the dependencies we should try to
> > upgrade it, and IIRC there is an issue about finding these dependencies
> out
> > automatically. We haven't done this before does not mean ignoring a CVE
> is
> > the correct way, it is just because no one takes care of it...
> >
> > And the hadoop team has stated the versions which are vulnerable, all
> > versions before 2.7.7, 2.8.5, 2.9.2(not released yet?), 3.0.3 and 3.1.1.
> > Not sure if they have published this out to the public, but as you can
> see
> > the url provided by me above, it is already public, so it does not matter
> > whether the hadoop team has published or not...
> >
> > Sean Busbey  于2018年10月23日周二 上午9:50写道：
> >
> > > Has the Hadoop PMC put out a public notice on the impact of that CVE
> yet?
> > > Specifically have they stated what versions are vulnerable? Are we
> flagging
> > > all versions impacted by it as "HBase says keep away"?
> > >
> > > Is there some reason this particular CVE especially impacts users of
> HBase?
> > > I presume not since we're talking about this on dev@ and in JIRA
> instead
> > > of
> > > on private@.
> > >
> > > Why are we reacting to this CVE when we don't seem to react to any
> other
> > > Hadoop CVEs? Or is this the start of a change wrt that?
> > >
> > > What about other dependencies with open CVEs?
> > >
> > > On Mon, Oct 22, 2018, 20:33 张铎(Duo Zhang) 
> wrote:
> > >
> > > > See here:
> > > >
> > > > https://access.redhat.com/security/cve/cve-2018-8009
> > > >
> > > > All 2.7.x releases before 2.7.7 have the problem. And for 2.6.x, the
> > > hadoop
> > > > team seems to drop the support as there is no release about two
> years, so
> > > > either we keep the original support versions, or we just drop the
> support
> > > > for the 2.6.x release line.
> > > >
> > > > Zach York  于2018年10月23日周二 上午8:51写道：
> > > >
> > > > > What is the main reason for the change? Build time speedup?
> > > > >
> > > > > Any reason for testing all of the 2.6.x line, but not the 2.7.x
> line?
> > > We
> > > > > don't check at all for 2.8.x?
> > > > >
> > > > > Can we be more consistent with how we test compatibility? (Do we
> only
> > > > care
> > > > > about the latest patch release in a line?)
> > > > >
> > > > > Sorry If I'm missing some of the reasoning, but at a surface level
> it
> > > > seems
> > > > > fairly arbitrary which releases we are cutting.
> > > > >
> > > > > On Mon, Oct 22, 2018 at 5:44 PM Sean Busbey 
> wrote:
> > > > >
> > > > > > Please leave me time to review before it is committed.
> > > > > >
> > > > > > On Mon, Oct 22, 2018, 13:58 Stack  wrote:
> > > > > >
> > > > > > > Duo has a patch up on HBASE-20970 that changes the Hadoop
> versions
> > > we
> > > > > > check
> > > > > > > at build time. Any objections to committing to branch-2.1+?
> > > > > > >
> > > > > > > It makes following changes:
> > > > > > >
> > > > > > > 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4
> > > > > > >
> > > > > > > becomes
> > > > > > >
> > > > > > > 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.7
> > > > > > >
> > > > > > > And...
> > > > > > >
> > > > > > > 3.0.0
> > > > > > >
> > > > > > > goes to
> > > > > > >
> > > > > > > 3.0.3
> > > > > > >
> > > > > > > Shout if you are against the change else will commit tomorrow.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > S
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

Re: [DISCUSS] Changing hadoop check versions in our hbase-personality?

[Zach York]

What is the main reason for the change? Build time speedup?

Any reason for testing all of the 2.6.x line, but not the 2.7.x line? We
don't check at all for 2.8.x?

Can we be more consistent with how we test compatibility? (Do we only care
about the latest patch release in a line?)

Sorry If I'm missing some of the reasoning, but at a surface level it seems
fairly arbitrary which releases we are cutting.

On Mon, Oct 22, 2018 at 5:44 PM Sean Busbey  wrote:

> Please leave me time to review before it is committed.
>
> On Mon, Oct 22, 2018, 13:58 Stack  wrote:
>
> > Duo has a patch up on HBASE-20970 that changes the Hadoop versions we
> check
> > at build time. Any objections to committing to branch-2.1+?
> >
> > It makes following changes:
> >
> > 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4
> >
> > becomes
> >
> > 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.7
> >
> > And...
> >
> > 3.0.0
> >
> > goes to
> >
> > 3.0.3
> >
> > Shout if you are against the change else will commit tomorrow.
> >
> > Thanks,
> > S
> >
>

Re: [ANNOUNCE] New Committer: Balazs Meszaros

[Zach York]

Congratulations, Balazs!! Welcome!

On Fri, Oct 12, 2018 at 3:30 AM Balazs Meszaros
 wrote:

> Thank you very much!
>
> Balazs
>
> On Fri, Oct 12, 2018 at 9:01 AM Peter Somogyi  wrote:
>
> > Congratulations, Balázs!
> >
> > On Thu, Oct 11, 2018 at 9:49 PM Sean Busbey  wrote:
> >
> >> On behalf of the HBase PMC, I'm pleased to announce that Balazs
> >> Meszaros has accepted our invitation to become an HBase committer.
> >>
> >> Thanks for all your hard work Balazs; we look forward to more
> >> contributions!
> >>
> >> Please join me in extending congratulations to Balazs!
> >>
> >
>

Re: [ANNOUNCE] Please welcome Zach York to the HBase PMC

[Zach York]

Thank you all!!

On Fri, Oct 12, 2018 at 3:31 AM Balazs Meszaros
 wrote:

> Congratulations Zach!
>
> On Fri, Oct 12, 2018 at 10:13 AM Peter Somogyi 
> wrote:
>
> > Congratulations!
> >
> > On Thu, Oct 11, 2018 at 10:01 PM Sean Busbey  wrote:
> >
> > > On behalf of the Apache HBase PMC I am pleased to announce that Zach
> > > York has accepted our invitation to become a PMC member on the Apache
> > > HBase project. We appreciate Zach stepping up to take more
> > > responsibility in the HBase project.
> > >
> > > Please join me in welcoming Zach to the HBase PMC!
> > >
> > > As a reminder, if anyone would like to nominate another person as a
> > > committer or PMC member, even if you are not currently a committer or
> > > PMC member, you can always drop a note to priv...@hbase.apache.org to
> > > let us know.
> > >
> >
>

Re: [VOTE] The first HBase 1.4.8 release candidate (RC0) is available

[Zach York]

+1

Checked sums and sigs: OK
RAT check: OK
Built from src: OK (8u92)
Unit tests pass: OK (8u92)

Thanks,
Zach


On Thu, Oct 4, 2018 at 10:40 AM Ted Yu  wrote:

> +1
>
>  - verified checksums and signatures: good
>  - basic checking on Web UI : good
>  - ran test suite with : good
>
> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
> 2018-06-17T18:33:14Z)
> Maven home: /apache-maven-3.5.4
> Java version: 1.8.0_161, vendor: Oracle Corporation, runtime:
> /mnt/disk2/a/jdk1.8.0_161/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.10.0-327.28.3.el7.x86_64", arch: "amd64",
> family: "unix"
>
>  - Ran LTT with 1M rows : good
>
> On Wed, Oct 3, 2018 at 9:34 AM Andrew Purtell  wrote:
>
> > RC errata:
> >
> > TestRSGroups will pass in isolation but may fail when run as part of the
> > suite. There have been several JIRAs filed against this unit like
> > HBASE-19444, HBASE-19461, HBASE-20137, and mentioned on HBASE-21187. It's
> > running time is far too long. I have filed HBASE-21265 to split it up.
> Test
> > stabilization work would be a part of that. I don't think  this rises to
> > the level of failing the RC vote because TestRSGroups will pass
> > consistently, at least for me, when run in isolation. I do agree that
> > without work to improve the test it doesn't offer the kind of functional
> > assurance we'd like to derive from a unit test.
> >
> >
> > On Tue, Oct 2, 2018 at 5:57 PM Andrew Purtell 
> wrote:
> >
> > > The first HBase 1.4.8 release candidate (RC0) is available for download
> > at
> > > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.4.8RC0/ and Maven
> > > artifacts are available in the temporary repository
> > >
> https://repository.apache.org/content/repositories/orgapachehbase-1233/
> > >
> > > The git tag corresponding to the candidate is '1.4.8RC0' (91118ce5f1).
> > >
> > > A detailed source and binary compatibility report for this release is
> > > available for your review at
> > >
> >
> https://dist.apache.org/repos/dist/dev/hbase/hbase-1.4.8RC0/compat-check-report.html
> > > . There are no reported compatibility issues.
> > >
> > > A list of the 33 issues resolved in this release can be found at
> > > https://s.apache.org/xpxo .
> > >
> > > Please try out the candidate and vote +1/0/-1.
> > >
> > > The vote will be open for at least 72 hours. Unless objection I will
> try
> > > to close it Monday October 8, 2018 if we have sufficient votes.
> > >
> > > Prior to making this announcement I made the following preflight
> checks:
> > >
> > > RAT check passes (7u80)
> > > Unit test suite passes (7u80, 8u181)
> > > Opened the UI in a browser, poked around
> > > LTT load 1M rows with 100% verification and 20% updates (8u181)
> > > ITBLL 500M rows with serverKilling monkey (8u181)
> > >
> > >
> > > --
> > > Best regards,
> > > Andrew
> > >
> > > Words like orphans lost among the crosstalk, meaning torn from truth's
> > > decrepit hands
> > >- A23, Crosstalk
> > >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> >- A23, Crosstalk
> >
>

Re: [VOTE] First release candidate for HBase 1.2.7 is available

[Zach York]

+1 (non-binding)


Checked sums and sigs: OK
RAT check: OK
Built from src: OK (8u92)
Unit tests pass: OK (8u92)

Thanks,
Zach

On Tue, Sep 11, 2018 at 9:06 AM Andrew Purtell  wrote:

> +1
>
> Checked sums and signatures: ok
> Checked compat report: ok, removed methods in Base64 are allowed by
> consensus exception
> RAT check: ok
> Built from source: ok (7u80)
> Unit tests pass: ok (8u172), note TestMutateRowsRecovery and
> TestCompactionWithThroughputController failed when running the suite but
> passed by themselves
> 1M row LTT: ok (8u172)
>
>
> On Fri, Sep 7, 2018 at 8:54 PM Sean Busbey  wrote:
>
> > The first release candidate for HBase 1.2.7 is available for download:
> >
> > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.2.7RC0/
> >
> > Maven artifacts are also available in a staging repository at:
> >
> > https://repository.apache.org/content/repositories/orgapachehbase-1232/
> >
> > Artifacts are signed with my key (0D80DB7C) published in our KEYS
> > file at http://www.apache.org/dist/hbase/KEYS
> >
> > The RC corresponds to the signed tag 1.2.7RC0, which currently points
> >  to commit ref
> >
> > ac57c51f7ad25e312b4275665d62b34a5945422f
> >
> > HBase 1.2.7 is the seventh maintenance release in the HBase 1.2 line,
> > continuing on the theme of bringing a stable, reliable database to
> > the Hadoop and NoSQL communities. This release includes over 250
> > bug fixes done in the 15 months since 1.2.6.
> >
> > This release candidate contains the following incompatible changes,
> > details in the release notes for the specific issue:
> >
> > * HBASE-20884 Replace usage of our Base64 implementation with java's
> > * HBASE-18577 shaded client should not include non-relocated third party
> >   dependencies
> > * HBASE-18142 delete in HBase shell should not delete previous versions
> >   of a cell
> > * HBASE-18731 some protected methods of QuotaSettings have been marked
> >   IA.Private and deprecated
> > * HBASE-16459 HBase shell no longer recognizes the --format option
> >
> > The detailed source and binary compatibility report vs 1.2.6 has been
> > published for your review, at:
> >
> > https://s.apache.org/hbase-1.2.7-rc0-compat-report
> >
> > The report shows some expected incompatibilities and one false positive.
> > Details on HBASE-18276.
> >
> > Critical fixes include:
> >
> > * HBASE-18036 Data locality is not maintained after cluster restart
> > * HBASE-18233 We shouldn't wait for readlock in doMiniBatchMutation in
> >   case of deadlock
> > * HBASE-9393  Region Server fails to properly close socket resulting in
> >   many CLOSE_WAIT to Data Nodes
> > * HBASE-19924 RPC throttling does not work for multi() with request
> >   count rater.
> > * HBASE-18192 Replication drops recovered queues on Region Server
> >   shutdown
> > * HBASE-18282 ReplicationLogCleaner can delete WALs not yet replicated
> >   in case of a KeeperException
> > * HBASE-19796 ReplicationSynUp tool is not replicating data if a WAL is
> >   moved to splitting directory
> > * HBASE-17648 HBase table-level synchronization fails between two
> >   secured(kerberized) clusters
> > * HBASE-18137 Replication gets stuck for empty WALs
> > * HBASE-18577 shaded client should not include non-relocated third party
> >   dependencies
> > * HBASE-19900 Region level exceptions destroy the result of batch client
> >   operations
> > * HBASE-21007 Memory leak in HBase REST server
> >
> > The full list of fixes included in this release is available at:
> >
> > https://s.apache.org/hbase-1.2.7-jira-release-notes
> >
> > and in the CHANGES.txt file included in the distribution.
> >
> > Please try out this candidate and vote +1/-1 on whether we should
> > release these artifacts as HBase 1.2.7.
> >
> > The VOTE will remain open for at least 72 hours. Given sufficient votes
> > I would like to close it on September 12th, 2018.
> >
> > thanks!
> >
> > -busbey
> >
> > as of this email the posted artifacts have the following SHA512:
> >
> > hbase-1.2.7-bin.tar.gz:
> > 00FC806A 335DFBDD 30720411 9FC0DD19 01AAC2E4 C8220B90
> > D54263F4 4F49D49A 111C30D0 6E4CC6D3 249C4F5A 7A66064B
> > 9EF97A92 B8E559F9 11705137 C3B652F2
> >
> > hbase-1.2.7-src.tar.gz:
> > F142B8E2 4F615D32 2B3816B1 71A61D9A 618FDD99 1EE69772
> > E3D52226 D30E34B8 0F9A469E 5AA00F6F 12AB77D6 C7FC6ADE
> > 8CAB7254 8B2AF8E6 6D00E9EC D0E93AB4
> >
>

Re: [VOTE] Second release candidate for hbase-2.0.2 (RC1) is available for download

[Zach York]

+1 (non-binding)

- checked sigs and sums: OK
- built from source: OK
- Ran UTs: OK
- RAT check: OK

Zach

On Thu, Aug 30, 2018 at 5:42 AM 张铎(Duo Zhang)  wrote:

> When running UT the TestMergeRegionsProcedure failed for me...
>
> Let me try again...
>
> Tak-Lon (Stephen) Wu  于2018年8月30日周四 下午1:07写道：
>
> > +1 (non-binding)
> >
> >- checked sums and signatures: ok
> >- built from source: ok (8u121)
> >- Unit tests pass: ok (8u121)
> >- Rat check:  ok (8u121)
> >- WEB UI (8u181): looks correct
> >- basic HBase shell command
> >[put/get/scan/list/disable/drop/create/flush] (8u181) : ok
> >- Loaded 1M rows with LTT (8u181):  ok
> >- ITBLL *Loop 1* 1M rows with *serverKilling* monkey (8u181): ok
> >   - used command `hbase
> >   org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList -m
> > serverKilling
> >   loop 1 10 10 ${RANDOM} 10`
> >
> > -Stephen
> >
> > On Wed, Aug 29, 2018 at 8:14 PM Guanghao Zhang 
> wrote:
> >
> > > +1 (binding)
> > >
> > > hbase-2.0.2-bin.tar.gz (jdk1.8.0_171)
> > > - Verified sha512: ok
> > > - Start HBase in standalone mode: ok
> > > - Verified with shell, create/disable/enable/drop/get/put/scan/delete:
> ok
> > > - Checked master/regionserver/table/region Web UI: ok
> > >
> > > hbase-2.0.2-src.tar.gz (jdk1.8.0_171)
> > > - Verified sha512: ok
> > > - Build tarball: ok
> > > - Start HBase in standalone mode: ok
> > > - Verified with shell, create/disable/enable/drop/get/put/scan/delete:
> ok
> > > - Checked master/regionserver/table/region Web UI: ok
> > >
> > > Andrew Purtell  于2018年8月30日周四 上午7:29写道：
> > >
> > > > +1
> > > >
> > > > Checked sums and signatures, ok
> > > > Built from source, ok (8u172)
> > > > RAT check passes (8u172)
> > > > Unit test suite passes (8u181)
> > > > Loaded 1M rows with ITBLL, looks fine (8u172)
> > > >
> > > >
> > > > On Tue, Aug 28, 2018 at 9:38 PM Stack  wrote:
> > > >
> > > > > The second release candidate for Apache HBase 2.0.2 (RC1) is
> > available
> > > > > for download and testing. This is a bug fix release with 100+
> commits
> > > > [1].
> > > > > Release notes are available here [2]. The compatibility report is
> at
> > > [3].
> > > > >
> > > > > Artifacts are available here:
> > > > >
> > > > >  https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.2RC1/
> > > > >
> > > > > Corresponding convenience artifacts for maven use are in the
> staging
> > > > > repository:
> > > > >
> > > > >
> > > https://repository.apache.org/content/repositories/orgapachehbase-1231
> > > > >
> > > > > All artifacts are signed with my code signing key, 8ACC93D2, which
> is
> > > > > also in the project KEYS file:
> > > > >
> > > > > http://www.apache.org/dist/hbase/KEYS
> > > > >
> > > > > These artifacts correspond to commit ref
> > > > >
> > > > >  1cfab033e779df840d5612a85277f42a6a4e8172
> > > > >
> > > > > which has been tagged as 2.0.2RC1.
> > > > >
> > > > > Please take a few minutes to verify the release and vote on
> releasing
> > > it:
> > > > >
> > > > > [ ] +1 Release these artifacts as Apache HBase 2.0.2
> > > > > [ ] -1 Do not release this package because ...
> > > > >
> > > > > This VOTE thread will remain open for at least 72 hours. It will
> > close
> > > > for
> > > > > sure
> > > > > Saturday morning.
> > > > >
> > > > > Thanks,
> > > > > S
> > > > >
> > > > > 1.
> > > >
> https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.2RC1/CHANGES.md
> > > > > 2.
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.2RC1/RELEASENOTES.md
> > > > > 3.
> > > > >
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.2RC1/compatibility_report_2.0.1_2.0.2.html
> > > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Andrew
> > > >
> > > > Words like orphans lost among the crosstalk, meaning torn from
> truth's
> > > > decrepit hands
> > > >- A23, Crosstalk
> > > >
> > >
> >
> >
> > --
> > Stephen Wu | Indiana University, Bloomington
> >
>

[jira] [Created] (HBASE-21070) SnapshotFileCache won't update for snapshots stored in S3

[Zach York (JIRA)]

Zach York created HBASE-21070:
-

 Summary: SnapshotFileCache won't update for snapshots stored in S3
 Key: HBASE-21070
 URL: https://issues.apache.org/jira/browse/HBASE-21070
 Project: HBase
  Issue Type: Bug
  Components: snapshots
Affects Versions: 3.0.0, 2.1.1, 1.4.7
Reporter: Zach York
Assignee: Zach York


The SnapshotFileCache depends on last modified time to determine whether to 
update the Snapshot HFile cache. However, in S3, real 'folders' don't exist. S3 
filesystems create a dummy file in place of a folder, but the dummy file last 
modified time is not updated when files are changed 'under' it. This means that 
the SnapshotFileCache doesn't pick up new snapshot HFiles and these files 
aren't removed from the HFileCleaner and can be eligible for deletion.

 

My patch removes the lastmodified assumption.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Hbase slack invite

[Zach York]

I'll jump on here as well while it's on my mind. Thanks!

On Fri, Aug 10, 2018 at 8:36 AM, Manjeet Singh 
wrote:

> Request you to please add me as well
>
> Thanks
>  Manjeet Singh
>
> On Fri, 10 Aug 2018, 20:50 Reid Chan,  wrote:
>
> > Sent, please check.
> >
> >
> > R.C
> >
> > 
> > From: James Moore 
> > Sent: 10 August 2018 21:35:06
> > To: dev@hbase.apache.org
> > Subject: Hbase slack invite
> >
> > May I join the slack channel as well?
> >
>

Re: [DISCUSS] Expanded "work items" for HBase-in-the-Cloud doc

[Zach York]

I would REALLY hope that the WAL interface/API changes would go into master
even if the feature work for Ratis is going in a feature branch. Not only
would this enable other backends to be developed in parallel with the Ratis
solution if there are other good fits for a non-HDFS WAL, but also it would
save the burden of having to rebase these core changes onto the latest
master to maintain compatibility. I'm assuming the Ratis portion of the
code would be mostly new files so these would be less of a concern.

On Thu, Jul 26, 2018 at 9:24 AM, Josh Elser  wrote:

> On 7/26/18 1:00 AM, Stack wrote:
>
>> All this said, I'd like to start moving toward the point where we start
>>> breaking out this work into a feature-branch off of master and start
>>> building code. My hope is that this is amenable to everyone, with the
>>> acknowledge that the Ratis work is considered "experimental" and not an
>>> attempt to make all of HBase use Ratis-backed WALs.
>>>
>>>
>>> Go for it.
>>
>> The branch would have WAL API changes only or would it include Ratis WAL
>> dev? (If the latter, would that be better done over on Ratis project?).
>>
>
> I think we would start with WAL API changes, get those "blessed", and then
> continue Ratis WAL dev after that.
>

Re: [DISCUSS] Expanded "work items" for HBase-in-the-Cloud doc

[Zach York]

+1 to starting the work. I think most of the concerns can be figured out on
the JIRAs and we can have a project update every X weeks if enough people
are interested.

I also agree to frame the feature correctly. Decoupling from a HDFS WAL or
WAL on Ratis would be more appropriate names that would better convey the
scope. I think there are a number of projects necessary to complete "HBase
on Cloud" with this being one of those.


Thanks for driving this initiative!

Zach


On Wed, Jul 25, 2018 at 11:55 AM, Josh Elser  wrote:

> Let me give an update on-list for everyone:
>
> First and foremost, thank you very much to everyone who took the time to
> read this, with an extra thanks to those who participated in discussion.
> There were lots of great points raised. Some about things that were unclear
> in the doc, and others shining light onto subjects I hadn't considered yet.
>
> My biggest take-away is that I complicated this document by tying it too
> closely with "HBase on Cloud", treating the WAL+Ratis LogService as the
> only/biggest thing to figure out. This was inaccurate and overly bold of
> me: I apologize. I think this complicated discussion on a number of points,
> and ate a good bit of some of your's time.
>
> My goal was to present this as an important part of a transition to the
> "cloud", giving justification to what WAL+Ratis helps HBase achieve. I did
> not want this document to be a step-by-step guide to a perfect HBase on
> Cloud design. I need to do a better job with this in the future; sorry.
>
> That said, my feeling is that, on the whole, folks are in support of the
> proposed changes/architecture described for the WAL+Ratis work (tl;dr
> revisit WAL API, plug in current WAL implementation to any API
> modification, build new Ratis-backed WAL impl). There were some concerns
> which still need immediate action that I am aware of:
>
> * Sync with Ram and Anoop re: in-memory WAL [1]
> * Where is Ratis LogService metadata kept? How do we know what LogStreams
> were being used/maintained by a RS? How does this tie into recovery?
>
> There are also long-term concerns which I don't think I have an answer for
> yet (for either reasons out of my control or a lack of technical
> understanding):
>
> * Maturity of the Ratis community
> * Required performance by HBase and the ability of the LogService to
> provide that perf (Areas already mentioned: gRPC perf, fsyncs bogging down
> disks, ability to scale RAFT quorums).
> * Continue with WAL-per-RS or move to WAL-per-Region? Related to perf,
> dependent upon Ratis scalability.
> * I/O amplification on WAL retention for backup and replication
> ("logstream export")
> * Ensure that LogStreams can be exported to a dist-filesystem in a manner
> which requires no additional metadata/handling (avoid more storage/mgmt
> complexity)
> * Ability to build krb5 authn into Ratis (really, gRPC)
>
> I will continue the two immediate action items. I think the latter
> concerns are some that will require fingers-on-keyboard -- I don't know
> enough about runtime characteristics without seeing it for myself.
>
> All this said, I'd like to start moving toward the point where we start
> breaking out this work into a feature-branch off of master and start
> building code. My hope is that this is amenable to everyone, with the
> acknowledge that the Ratis work is considered "experimental" and not an
> attempt to make all of HBase use Ratis-backed WALs.
>
> Finally, I do *not* want this message to be interpreted as me squashing
> anyone's concerns. My honest opinion is that discussion has died down, but
> I will be the first to apologize if I have missed any outstanding concerns.
> Please, please, please ping me if I am negligent.
>
> Thanks once again for everyone's participation.
>
> [1] https://docs.google.com/document/d/1Su5py_T5Ytfh9RoTTX2s20Kb
> SJwBHVxbO7ge5ORqbCk/edit?disco=CBm3RLM
>
> On 2018/07/13 20:15:45, Josh Elser  wrote: > Hi all,
>
>>
>> A long time ago, I shared a document about a (I'll call it..) "vision"
>> where we make some steps towards decoupling HBase from HDFS in an effort to
>> make deploying HBase on Cloud IaaS providers a bit easier (operational
>> simplicity, effective use of common IaaS paradigms, etc).
>>
>> https://docs.google.com/document/d/1Su5py_T5Ytfh9RoTTX2s20Kb
>> SJwBHVxbO7ge5ORqbCk/edit?usp=sharing
>>
>> A good ask from our Stack back then was: "[can you break down this
>> work]?" The original document was very high-level, and asking for some more
>> details make a lot of sense. Months later, I'd like to share that I've
>> updated the original document with some new content at the bottom (as well
>> as addressed some comments which went unanswered by me -- sorry!)
>>
>> Based on a discussion I had earlier this week (and some discussions
>> during HBaseCon in California in June), I've tried to add a brief
>> "refresher" on what some of the big goals for this effort are. Please check
>> it out at your leisure and let me know what 

[jira] [Created] (HBASE-20868) Fix TestCheckTestClasses on HBASE-18477

[Zach York (JIRA)]

Zach York created HBASE-20868:
-

 Summary: Fix TestCheckTestClasses on HBASE-18477
 Key: HBASE-20868
 URL: https://issues.apache.org/jira/browse/HBASE-20868
 Project: HBase
  Issue Type: Sub-task
Affects Versions: HBASE-18477
Reporter: Zach York
Assignee: Zach York
 Fix For: HBASE-18477






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HBASE-20787) Rebase the HBASE-18477 onto the current master to continue dev

[Zach York (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zach York resolved HBASE-20787.
---
Resolution: Fixed

> Rebase the HBASE-18477 onto the current master to continue dev
> --
>
> Key: HBASE-20787
> URL: https://issues.apache.org/jira/browse/HBASE-20787
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: HBASE-18477
>Reporter: Zach York
>    Assignee: Zach York
>Priority: Minor
> Fix For: HBASE-18477
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (HBASE-20787) Rebase the HBASE-18477 onto the current master to continue dev

[Zach York (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zach York reopened HBASE-20787:
---

Rebasing again to pull in fixes for unit tests.

> Rebase the HBASE-18477 onto the current master to continue dev
> --
>
> Key: HBASE-20787
> URL: https://issues.apache.org/jira/browse/HBASE-20787
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: HBASE-18477
>Reporter: Zach York
>    Assignee: Zach York
>Priority: Minor
> Fix For: HBASE-18477
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20836) Add Yetus annotation for ReadReplicaClustersTableNameUtil

[Zach York (JIRA)]

Zach York created HBASE-20836:
-

 Summary: Add Yetus annotation for ReadReplicaClustersTableNameUtil
 Key: HBASE-20836
 URL: https://issues.apache.org/jira/browse/HBASE-20836
 Project: HBase
  Issue Type: Sub-task
Affects Versions: HBASE-18477
Reporter: Zach York
Assignee: Zach York
 Fix For: HBASE-18477


Found via nightly builds.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HBASE-20787) Rebase the HBASE-18477 onto the current master to continue dev

[Zach York (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zach York resolved HBASE-20787.
---
Resolution: Fixed

> Rebase the HBASE-18477 onto the current master to continue dev
> --
>
> Key: HBASE-20787
> URL: https://issues.apache.org/jira/browse/HBASE-20787
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: HBASE-18477
>Reporter: Zach York
>    Assignee: Zach York
>Priority: Minor
> Fix For: HBASE-18477
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20787) Rebase the HBASE-18477 onto the current master to continue dev

[Zach York (JIRA)]

Zach York created HBASE-20787:
-

 Summary: Rebase the HBASE-18477 onto the current master to 
continue dev
 Key: HBASE-20787
 URL: https://issues.apache.org/jira/browse/HBASE-20787
 Project: HBase
  Issue Type: Task
Affects Versions: HBASE-18477
Reporter: Zach York
Assignee: Zach York
 Fix For: HBASE-18477






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Apache HBase 1.4.5 RC1

[Zach York]

One potentially blocking issue that I can see is
https://issues.apache.org/jira/browse/HBASE-20723 which causes potential
dataloss when recovering from a RS crash when a custom WAL directory is set
(hbase.wal.dir).

Josh and I had some discussion on the issue and since this is not a default
config, I believe it is okay to let this slip to a 1.4.6 as long as we
release 1.4.6 as quickly as we have been releasing the other 1.4.x line :)

Just wanted to call this out to get greater awareness.

Thanks,
Zach

On Wed, Jun 13, 2018 at 10:18 PM, Chia-Ping Tsai 
wrote:

> +1 (binding)
>
> jdk_1.8.0_171
>
> unit tests - all pass
> web UI - LGTM
> CRUD by shell - LGTM
>
> On 2018/06/13 19:53:33, Josh Elser  wrote:
> > Hi,
> >
> > Please vote to approve the following as Apache HBase 1.4.5. The only
> > change in RC1 over RC0 is the updated CHANGES.txt.
> >
> > https://dist.apache.org/repos/dist/dev/hbase/1.4.5RC1/
> >
> > Per usual, there is a source release as well as a convenience binary
> >
> > This is built with JDK7 from the commit:
> > https://git-wip-us.apache.org/repos/asf?p=hbase.git;a=commit;h=
> ca99a9466415dc4cfc095df33efb45cb82fe5480
> > (there is a corresponding tag "1.4.5RC1" for convenience). Please ignore
> > the incorrect commit message (forgot to update that text after rc0).
> >
> > hbase-1.4.5-bin.tar.gz: 1C34A448 DF4E102E 7964C6D8 84C2B1E5 35DD0CAA
> > E67E6B15
> >  92DB2B10 8A6D3A0F B2D841F7 677EB5C3 8EB6D78F
> > 342CB42F
> >  8CE90AA6 D62A5362 3C727023 4E8C95EE
> > hbase-1.4.5-src.tar.gz: 61B83966 952D334A FDAE7E3A 11FD8529 90583302
> > 4AC186C4
> >  2B7BDA11 5CB472A4 34C71466 4DCF90BB 8735F658
> > 20975292
> >  35319D5C 2287B0DE 64F484C7 42F635D6
> >
> > There is also a Maven staging repository for this release:
> > https://repository.apache.org/content/repositories/orgapachehbase-1221
> >
> > This vote will be open for at least 72 hours (2018/06/16 2000 UTC).
> >
> > - Josh (on behalf of the HBase PMC)
> >
> >
>

Re: HBase not replaying WAL file

[Zach York]

Nandakishore,

Are the edits persisted to the WAL? The RS will only be able to recover
edits that were persisted to the WAL.
Therefore, if writes happened that had not been synced/flushed to the WAL
when the node was terminated, it is expected that there could be a loss of
that data.
However, from the client side, I don't believe the write would have
succeeded (someone can correct me on this as I don't have concrete
evidence) if the data hadn't been fsynced/flushed to the WAL.

Thanks,
Zach

On Sun, May 20, 2018 at 11:27 PM, Nandakishore Arvapaly <
nandak.arvap...@bigcommerce.com> wrote:

> Hi Team,
>
> I have a issue with HBase cluster.
>
> I have hosted an HBase cluster with Phoenix on EMR emr-5.8.0. I have 1
> master and 5 slaves 4.x large nodes. I’m losing the data while querying a
> table after a region server dies. Here are the steps I followed.
>
> 1. Launched the cluster with hfs replication factory as 3.
> 2. Created the tables and loaded the data using Phoenix.
> 3. Cross checked the data I loaded into tables and I see the data.
> 4. Wantedly terminated a EC2 machine which is part of cluster, meaning
> killing region server.
> 5. I could see EMR resizing and bringing up the new node.
> 6. When I query the table after the whole cluster is stable, which usually
> took 5-10 minutes, I see losing some data which is on dead RS.
>
> I believe HBase replays the WAL once new node is brought up and I could
> also see the WAL file on HDFS new RS’s directory. But somehow I don’t see
> the complete data from the table.
>
> Could you please let me know what possibly could go wrong. Also please let
> me know if I have to set any properties.
>
> I would be happy to provide more details if you need.
>
> Thanks,
> Nandakishore.
>

Re: Snapshotted files after compactions

[Zach York]

Ah, of course. I missed that HFileLink handled the archive directory too.

Thanks Ted!

On Tue, May 1, 2018 at 4:35 PM, Ted Yu <yuzhih...@gmail.com> wrote:

> Please take a look at HFileLink#buildFromHFileLinkPattern where archive
> directory is taken into account.
>
>
>
> Cheers
>
> On Tue, May 1, 2018 at 4:23 PM, Zach York <zyork.contribut...@gmail.com>
> wrote:
>
> > Hello,
> >
> > This should be a fairly simple question, but I have been diving into docs
> > and code and haven't found anything obvious.
> >
> > I have been looking at snapshot code, but have been unable to
> > understand/find one part. When a file that was present in a snapshot is
> > compacted and moved to the archive directory, how does the snapshot know
> > where it is? Does a compaction update the data.manifest file?
> >
> > Thanks,
> > Zach
> >
>

Snapshotted files after compactions

[Zach York]

Hello,

This should be a fairly simple question, but I have been diving into docs
and code and haven't found anything obvious.

I have been looking at snapshot code, but have been unable to
understand/find one part. When a file that was present in a snapshot is
compacted and moved to the archive directory, how does the snapshot know
where it is? Does a compaction update the data.manifest file?

Thanks,
Zach

[jira] [Created] (HBASE-20447) Only fail cacheBlock if block collisions aren't related to next block metadata

[Zach York (JIRA)]

Zach York created HBASE-20447:
-

 Summary: Only fail cacheBlock if block collisions aren't related 
to next block metadata
 Key: HBASE-20447
 URL: https://issues.apache.org/jira/browse/HBASE-20447
 Project: HBase
  Issue Type: Bug
  Components: BlockCache, BucketCache
Affects Versions: 1.4.3, 2.0.0
Reporter: Zach York
Assignee: Zach York


This is the issue I was originally having here: 
[http://mail-archives.apache.org/mod_mbox/hbase-dev/201802.mbox/%3CCAN+qs_Pav=md_aoj4xji+kcnetubg2xou2ntxv1g6m8-5vn...@mail.gmail.com%3E]

 

When we pread, we don't force the read to read all of the next block header.
However, when we get into a race condition where two opener threads try to
cache the same block and one thread read all of the next block header and the 
other one didn't, it will fail the open process. This is especially important
in a splitting case where it will potentially fail the split process.
Instead, in the caches, we should only fail if the required blocks are 
different.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20280) Fix possibility of deadlocking in refreshFileConnections when prefetch is enabled

[Zach York (JIRA)]

Zach York created HBASE-20280:
-

 Summary: Fix possibility of deadlocking in refreshFileConnections 
when prefetch is enabled
 Key: HBASE-20280
 URL: https://issues.apache.org/jira/browse/HBASE-20280
 Project: HBase
  Issue Type: Bug
  Components: BucketCache
Affects Versions: 1.4.2, 2.0.0-beta-2
Reporter: Zach York
Assignee: Zach York


When prefetch on open is specified, there is a deadlocking case
where if the prefetch is cancelled, the PrefetchExecutor interrupts
the threads if necessary, when that happens in FileIOEngine, it
causes an ClosedByInterruptException which is a subclass of
ClosedChannelException. If we retry all ClosedChannelExceptions,
this will lock as this access is expected to be interrupted.
This change removes calling refreshFileConnections for
ClosedByInterruptExceptions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20204) Add locking to RefreshFileConnections in BucketCache

[Zach York (JIRA)]

Zach York created HBASE-20204:
-

 Summary: Add locking to RefreshFileConnections in BucketCache
 Key: HBASE-20204
 URL: https://issues.apache.org/jira/browse/HBASE-20204
 Project: HBase
  Issue Type: Bug
  Components: BucketCache
Reporter: Zach York
Assignee: Zach York


This is a follow-up to HBASE-20141 where [~anoop.hbase] suggested adding 
locking for refreshing channels.

I have also seen this become an issue when a RS has to abort and it locks on 
trying to flush out the remaining data to the cache (since cache on write was 
turned on).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] New HBase committer Zach York

[Zach York]

Thanks everyone!

I am excited to continue my work in HBase!

Thanks,
Zach

On Thu, Mar 8, 2018 at 9:03 AM, ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com> wrote:

> Congratulations Zach !!!
>
> On Thu, Mar 8, 2018 at 11:03 AM, Yu Li <car...@gmail.com> wrote:
>
> > Congratulations, Zach!
> >
> > Best Regards,
> > Yu
> >
> > On 8 March 2018 at 06:13, Mike Drob <md...@apache.org> wrote:
> >
> > > Congratulations, Zach!
> > >
> > > On Wed, Mar 7, 2018 at 4:03 PM, Andrew Purtell <apurt...@apache.org>
> > > wrote:
> > >
> > > > Congratulations and welcome Zach!
> > > >
> > > >
> > > > On Wed, Mar 7, 2018 at 8:27 AM, Sean Busbey <bus...@apache.org>
> wrote:
> > > >
> > > > > On behalf of the Apache HBase PMC, I am pleased to announce that
> Zach
> > > > > York has accepted the PMC's invitation to become a committer on the
> > > > > project.
> > > > >
> > > > > We appreciate all of Zach's great work thus far and look forward to
> > > > > continued involvement.
> > > > >
> > > > > Please join me in congratulating Zach!
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Andrew
> > > >
> > > > Words like orphans lost among the crosstalk, meaning torn from
> truth's
> > > > decrepit hands
> > > >- A23, Crosstalk
> > > >
> > >
> >
>

Re: [DISCUSS] strategy on Java versions

[Zach York]

> -- "Recommended" for running
>
> I think we should suggest folks rely on LTS releases of Java, wether
> from the OpenJDK project (which looks like it will be equivalent to
> the Oracle version) or a vender.
>
> That would mean updating our support matrix to call out Java 9 and
> Java 10 as "Not Supported" for current releases.
>

+1 for at least calling out Java 9 and Java 10 as "Not Tested" (Not
Supported if we know for sure it won't work).

As for recommending folks rely on LTS releases, I think this will require a
shift in thinking
from most people in the Hadoop ecosystem to upgrade the Java versions more
frequently.
Otherwise it will become a pain for users to use multiple open source
projects in the same
environment (for example if Hadoop is using Java 8, Spark - Java 9, and
HBase - Java 11).
Not saying that we should hold ourselves back because of other projects,
but we should
at least consider the burden.

I do, however, think that long term we should only recommend running with
the current LTS release.
Getting there might be a lot of fun though :)



>
> -- Development changes
>
> The next LTS version of Java is slated to be Java 11, which won't be
> out until September 2018. I think we should aim for a minor release on
> whatever major branches are still active within a month of the GA date
> where we feel comfortable calling out Java 11 as supported.
>
> To that end, I think we would should do 2 things:
>
> 1) Work to make sure we can run through our test suite with either the
> release prior to the next LTS or an early access version of the
> upcoming LTS release before the GA date, ideally ~3 months before.
> That'd be a target of June 2018 for the planned GA date of Java 11. I
> expect that means abandoning our Java 9 specific efforts and starting
> in April/May to work on updates related to Java 10 (or 11 depending on
> when bits start shipping).
>
>
+1, doesn't make sense doing work that will be already be obsolete by this
desired direction.


> 2) In the process of getting the above to work if we find we can't get
> a major release branch ready to go without violating our compatibility
> guidelines (especially wrt the existing supported java versions), we
> should cease making new minor releases for that major release branch.
> This could mean either just going to bugfix releases or it could mean
> declaring the branch EOM entirely.
>
> Note that the above doesn't necessarily require that we be able to
> build with a newer JDK version. That would certainly be simpler
> though; I don't think our current tooling makes it easy to run tests
> with a different JDK than is used to compile.
>
> -- Dropping of older versions
>
> I don't think we've codified anywhere when we drop Java version
> supports? (our guide merely says when we _could_.) I'm also not sure
> if we should write it down somewhere? If we can stick to the above and
> proactively add new versions as supported in minor releases, I think
> we'll be in a good position to drop everything but the latest LTS Java
> release with each HBase major release. Since this notion of LTS and
> non-LTS releases for Java is still relatively new, maybe that's a
> discussion we're better off having once HBase 3 is closer.
>

I don't see this happening before 3.0 anyways. Let's work on getting 1)
done for Java 9/10.
I think we should wait until we see how this is all going to pan out (and
how much work each upgrade will be)
before making anything public, but I think this is a good goal.

Thanks for writing this up,
Zach