Re: [DISCUSS] How to deal with the disabling of public sign ups for jira.a.o(enable github issues?)

[Guanghao Zhang]

Did other projects have the same solution for this, sync github issues to
jira issues? Github issues will be useful to get more feedback.

张铎(Duo Zhang)  于2022年12月6日周二 00:13写道：

> The PR for HBASE-27513 is available
>
> https://github.com/apache/hbase/pull/4913
>
> Let's at least tell our users to send email to private@hbase for
> acquiring a jira account.
>
> Thanks.
>
> 张铎(Duo Zhang)  于2022年12月2日周五 12:46写道：
> >
> > Currently all the comment on github PR will be sent to issues@hbase,
> > like this one
> >
> > https://lists.apache.org/thread/jbfm269b4m24xl2r82l8b0t3pmqr44hr
> >
> > But I think this can only be used as an archive, to make sure that all
> > discussions are recorded on asf infrastructure.
> >
> > For github issues, I'm afraid we can only do the same thing. As the
> > format of github comment is different, it will be hard to read if we
> > just sync the message to jira...
> >
> > Thanks.
> >
> > Bryan Beaudreault  于2022年12月1日周四
> 21:30写道：
> > >
> > > Should we have them sent to private@? Just thinking in terms of
> reducing
> > > spam to users who put their email and full name on a public list.
> > >
> > > One thought I had about bug tracking is whether we could use some sort
> of
> > > github -> jira sync. I've seen them used before, where it automatically
> > > syncs issues and comments between the two systems. It's definitely not
> > > ideal, but maybe an option? I'm guessing it would require INFRA help.
> > >
> > > On Thu, Dec 1, 2022 at 5:47 AM 张铎(Duo Zhang) 
> wrote:
> > >
> > > > I've filed HBASE-27513 for changing the readme on github.
> > > >
> > > > At least let's reuse the existing mailing list for acquiring jira
> account.
> > > >
> > > > Thanks.
> > > >
> > > > 张铎(Duo Zhang)  于2022年11月29日周二 22:34写道：
> > > >
> > > > >
> > > > > Bump and also send this to user@hbase.
> > > > >
> > > > > We need to find a way to deal with the current situation where
> > > > > contributors can not create a Jira account on their own...
> > > > >
> > > > > At least, we need to change the readme on github page, web site and
> > > > > also the ref guide to tell users how to acquire a jira account...
> > > > >
> > > > > Thanks.
> > > > >
> > > > > 张铎(Duo Zhang)  于2022年11月27日周日 22:06写道：
> > > > > >
> > > > > > For me, I think most developers already have a github account, so
> > > > > > enabling it could help us get more feedback. For lots of younger
> > > > > > Chinese developers, they rarely use email in their daily life...
> > > > > > No doubt later we need to modify our readme on github. If we
> just let
> > > > > > users go to github issues on the readme, they will soon open an
> issue
> > > > > > there. But if we ask users to first send an email to a mailing
> list,
> > > > > > for acquiring a jira account, and then wait for a PMC member to
> submit
> > > > > > the request, and receive the email response, set up their
> account, and
> > > > > > then they can finally open an issue on jira. I'm afraid lots of
> users
> > > > > > will just give up, it is not very friendly...
> > > > > >
> > > > > > And I do not mean separate issue systems for users and devs.
> Users can
> > > > > > still open jira issues or ask in the mailing list if they want,
> github
> > > > > > issues is just another channel. If a user asks something in the
> > > > > > mailing list and we think it is a bug, we will ask the user to
> file an
> > > > > > issue or we will file an issue for it. It is just the same with
> github
> > > > > > issues.
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > Nick Dimiduk  于2022年11月24日周四 15:44写道：
> > > > > > >
> > > > > > > This new situation around JIRA seems very similar to the
> existing
> > > > situation
> > > > > > > around Slack. A new community member currently must acquire a
> Slack
> > > > invite
> > > > > > > somehow, usually by emailing one of the lists. Mailing lists
> > > > themselves
> > > > > > > involve a signup process, though it may be possible to email
> > > > user/-zh/dev
> > > > > > > without first subscribing to the list.
> > > > > > >
> > > > > > > I have a -0 opinion on using GitHub Issues to manage JIRA
> > > > subscription
> > > > > > > access. It seems like a comical cascade of complexity. I’d
> prefer to
> > > > keep
> > > > > > > GitHub Issues available to us as a future alternative to JIRA
> for
> > > > project
> > > > > > > issue tracking. I agree with you that migrating away from JIRA
> will
> > > > be
> > > > > > > painful.
> > > > > > >
> > > > > > > I’m not a big fan of having separate issue systems for users
> vs.
> > > > devs. It
> > > > > > > emphasizes the idea that users and devs are different groups of
> > > > people with
> > > > > > > unequal voice in the project direction. I suppose it could be
> done
> > > > well,
> > > > > > > but I think it is more likely to be done poorly.
> > > > > > >
> > > > > > > I follow the Infra list, but only casually. It seems there’s a
> plan
> > > > to
> > > > > > > eventually adopt some Atlassian Cloud service, which 

Re: [ANNOUNCE] New HBase Committer Liangjun He

[Guanghao Zhang]

Congratulations!

OpenInx  于2022年12月6日周二 19:03写道：

> Congrats and welcome !
>
> On Tue, Dec 6, 2022 at 2:21 AM Andrew Purtell  wrote:
>
> > Congratulations, and welcome!
> >
> > On Sat, Dec 3, 2022 at 5:51 AM Yu Li  wrote:
> >
> > > Hi All,
> > >
> > > On behalf of the Apache HBase PMC, I am pleased to announce that
> Liangjun
> > > He (heliangjun) has accepted the PMC's invitation to become a committer
> > on
> > > the project. We appreciate all of Liangjun's generous contributions
> thus
> > > far and look forward to his continued involvement.
> > >
> > > Congratulations and welcome, Liangjun!
> > >
> > > 我很高兴代表 Apache HBase PMC 宣布 Liangjun He (何良均) 已接受我们的邀请，成为 Apache HBase
> 项目的
> > > Committer。感谢何良均一直以来为 HBase 项目做出的贡献，并期待他在未来继续承担更多的责任。
> > >
> > > 欢迎良均！
> > >
> > > Best Regards,
> > > Yu
> > > --
> > > Best Regards,
> > > Yu
> > >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Unrest, ignorance distilled, nihilistic imbeciles -
> > It's what we’ve earned
> > Welcome, apocalypse, what’s taken you so long?
> > Bring us the fitting end that we’ve been counting on
> >- A23, Welcome, Apocalypse
> >
>

Re: [ANNOUNCE] New HBase Committer Liangjun He

[Guanghao Zhang]

Congratulations!

OpenInx  于2022年12月6日周二 19:03写道：

> Congrats and welcome !
>
> On Tue, Dec 6, 2022 at 2:21 AM Andrew Purtell  wrote:
>
> > Congratulations, and welcome!
> >
> > On Sat, Dec 3, 2022 at 5:51 AM Yu Li  wrote:
> >
> > > Hi All,
> > >
> > > On behalf of the Apache HBase PMC, I am pleased to announce that
> Liangjun
> > > He (heliangjun) has accepted the PMC's invitation to become a committer
> > on
> > > the project. We appreciate all of Liangjun's generous contributions
> thus
> > > far and look forward to his continued involvement.
> > >
> > > Congratulations and welcome, Liangjun!
> > >
> > > 我很高兴代表 Apache HBase PMC 宣布 Liangjun He (何良均) 已接受我们的邀请，成为 Apache HBase
> 项目的
> > > Committer。感谢何良均一直以来为 HBase 项目做出的贡献，并期待他在未来继续承担更多的责任。
> > >
> > > 欢迎良均！
> > >
> > > Best Regards,
> > > Yu
> > > --
> > > Best Regards,
> > > Yu
> > >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Unrest, ignorance distilled, nihilistic imbeciles -
> > It's what we’ve earned
> > Welcome, apocalypse, what’s taken you so long?
> > Bring us the fitting end that we’ve been counting on
> >- A23, Welcome, Apocalypse
> >
>

[ANNOUNCE] New HBase committer Zhuoyue Huang(GeorryHuang)

[Guanghao Zhang]

Folks,

On behalf of the Apache HBase PMC I am pleased to announce that Zhuoyue
Huang has accepted the PMC's invitation to become a committer on the
project.

We appreciate all of the great contributions Zhuoyue Huang has made to the
community thus far and we look forward to his continued involvement.

Allow me to be the first to congratulate Zhuoyue Huang on his new role!

Thanks.

[ANNOUNCE] Apache HBase 2.2.7 is now available for download

[Guanghao Zhang]

The HBase team is happy to announce the immediate availability of HBase
2.2.7.

Apache HBase™ is an open-source, distributed, versioned, non-relational
database. Apache HBase gives you low latency random access to billions of
rows with
millions of columns atop non-specialized hardware. To learn more about
HBase,
see https://hbase.apache.org/.
To download 2.2.7, visit our download page:
http://hbase.apache.org/downloads.html

HBase 2.2.7 is the final release of the HBase 2.2 release line, continuing
on the
theme of bringing a stable, reliable database to the Apache Big Data
ecosystem and beyond. 2.2.7 includes 138 bug fixes and improvements done
since the 2.2.6. The list of 138 changes made since 2.2.6 can be found here:
https://downloads.apache.org/hbase/2.2.7/CHANGES.md

For instructions on verifying ASF release downloads, please see
https://www.apache.org/dyn/closer.cgi#verify
Project member signature keys can be found at
https://www.apache.org/dist/hbase/KEYS
Question, comments, and problems are always welcome at:
d...@hbase.apache.org user@hbase.apache.org user...@hbase.apache.org

Thanks to all who contributed and made this release possible.

Cheers,
The HBase Dev Team

[VOTE] The first HBase 2.2.7 release candidate (RC0) is available

[Guanghao Zhang]

Please vote on this release candidate (RC) for Apache HBase 2.2.7.
Meanwhile, as branch-2.2 will be EOL, please don't push new commits to it.
And this will be the last one of the 2.2.x releases. Thanks.

The VOTE will remain open for at least 72 hours.

[ ] +1 Release this package as Apache HBase 2.2.7
[ ] -1 Do not release this package because ...

The tag to be voted on is 2.2.7RC0. The release files, including
signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/hbase/2.2.7RC0/

Maven artifacts are available in a staging repository at:
https://repository.apache.org/content/repositories/orgapachehbase-1440/

Signatures used for HBase RCs can be found in this file:
https://dist.apache.org/repos/dist/release/hbase/KEYS

The list of bug fixes going into 2.2.7 can be found in included
CHANGES.md and RELEASENOTES.md available here:
https://dist.apache.org/repos/dist/dev/hbase/2.2.7RC0/CHANGES.md
https://dist.apache.org/repos/dist/dev/hbase/2.2.7RC0/RELEASENOTES.md

A detailed source and binary compatibility report for this release is
available at:
https://dist.apache.org/repos/dist/dev/hbase/2.2.7RC0/api_compare_2.2.7RC0_to_2.2.6.html

To learn more about Apache HBase, please see http://hbase.apache.org/

Thanks,
Guanghao Zhang

[通知]欢迎Xin Sun成为HBase Committer

[Guanghao Zhang]

我很高兴代表Apache HBase PMC宣布Xin Sun已接受我们的邀请，成为Apache HBase项目的Committer。感谢Xin
Sun为HBase项目做出的贡献，期待未来在HBase项目中承担更多责任。

欢迎Xin Sun成为HBase Committer！

[通知]欢迎Yulin Niu成为HBase Committer

[Guanghao Zhang]

我很高兴代表Apache HBase PMC宣布Yulin Niu已接受我们的邀请，成为Apache
HBase项目的Committer。感谢Yulin Niu为HBase项目做出的贡献，期待未来在HBase项目中承担更多责任。

欢迎Yulin Niu成为HBase Committer！

[ANNOUNCE] New HBase committer Xin Sun

[Guanghao Zhang]

Folks,

On behalf of the Apache HBase PMC I am pleased to announce that Xin Sun has
accepted the PMC's invitation to become a committer on the project.

We appreciate all of the great contributions Xin Sun has made to the
community thus far and we look forward to his continued involvement.

Allow me to be the first to congratulate Xin Sun on his new role!

Thanks.

[ANNOUNCE] New HBase committer Yulin Niu

[Guanghao Zhang]

Folks,

On behalf of the Apache HBase PMC I am pleased to announce that Yulin Niu
has accepted the PMC's invitation to become a committer on the project.

We appreciate all of the great contributions Yulin has made to the
community thus far and we look forward to his continued involvement.

Allow me to be the first to congratulate Yulin on his new role!

Thanks.

Re: [DISCUSS] HBASE-25299 Scan#setRowPrefixFilter Unexpected behavior

[Guanghao Zhang]

I am +1 to deprecated setRowPrefixFilter method. This method name is
setRowPrefixFilter but not use filter and only set start row and end row. I
thought this could be done by user.

唐天航  于2020年11月19日周四 上午12:45写道：

> Hi,
>   I have opened an issue HBASE-25299
>  about
> Scan#setRowPrefixFilter
> Unexpected behavior.
>
> e.g.
>
> startRow : "112"
>
> rowPrefixFilter : "11"
>
> The Result of this scan might contain : "111", which is unexpected.
>
>   public Scan setRowPrefixFilter(byte[] rowPrefix) {
> if (rowPrefix == null) {
>   setStartRow(HConstants.EMPTY_START_ROW);
>   setStopRow(HConstants.EMPTY_END_ROW);
> } else {
>   this.setStartRow(rowPrefix);
>   this.setStopRow(calculateTheClosestNextRowKeyForPrefix(rowPrefix));
> }
> return this;
>   }
>
>  Scan#setRowPrefixFilter achieves this function by setting startRow and
> stopRow, ignoring the situation that startRow may have been set.
>
>
> I have discussed this issue with @infraio and he suggested to deprecate
> this method because modifying it may cause compatibility issues.
>
> Is this plan acceptable? Hope to get some suggestions.
>
>
> Thank you. Regards
>

[ANNOUNCE] Apache HBase 2.2.6 is now available for download

[Guanghao Zhang]

The HBase team is happy to announce the immediate availability of HBase
2.2.6.

Apache HBase™ is an open-source, distributed, versioned, non-relational
database. Apache HBase gives you low latency random access to billions of
rows with
millions of columns atop non-specialized hardware. To learn more about
HBase, see https://hbase.apache.org/.

To download 2.2.6, visit our download page:
http://hbase.apache.org/downloads.html

HBase 2.2.6 is the latest release of the HBase 2.2 line, continuing on the
theme of bringing a stable, reliable database to the Apache Big Data
ecosystem and beyond. 2.2.6 includes 124 bug fixes and improvements  done
since the 2.2.5.
The list of 124 changes made since 2.2.5 can be found here:
https://downloads.apache.org/hbase/2.2.6/CHANGES.md

For instructions on verifying ASF release downloads, please see
https://www.apache.org/dyn/closer.cgi#verify

Project member signature keys can be found at
https://www.apache.org/dist/hbase/KEYS

Question, comments, and problems are always welcome at:
d...@hbase.apache.org
user@hbase.apache.org
user...@hbase.apache.org

Thanks to all who contributed and made this release possible.

Cheers,
The HBase Dev Team

Re: [VOTE] The third HBase 2.2.6 release candidate (RC2) is available

[Guanghao Zhang]

Update the bin tar in https://dist.apache.org/repos/dist/dev/hbase/2.2.6RC2/ as
the revision and the UI language problem. But the
dev-support/create-release scripts still not worked for me. It always
failed in "gpg sign" step.. Sorry for take so long time to release
2.2.6.

For this vote, +1 from me. With 3 binding +1 and 1 non-binging +1. The vote
passed. Thanks all for voting.





Andrew Purtell  于2020年9月11日周五 上午1:07写道：

> [resending to dev@, sorry]
>
> +1 (binding)
>
> * Signature: ok
> * Checksum : ok
> * Rat check (1.8.0_272): ok
>  - mvn clean apache-rat:check
> * Built from source (1.8.0_272): ok
>  - mvn clean install -DskipTests
> * Unit tests pass (1.8.0_272): ok
>  - mvn package -P runAllTests
>
> On Thu, Sep 3, 2020 at 8:32 PM Guanghao Zhang  wrote:
> >
> >> Please vote on this release candidate (RC2) for Apache HBase 2.2.6.
> >>
> >> The VOTE will remain open for at least 72 hours.
> >>
> >> [ ] +1 Release this package as Apache HBase 2.2.6
> >> [ ] -1 Do not release this package because ...
> >>
> >> The tag to be voted on is 2.2.6RC2. The release files, including
> >> signatures, digests, etc. can be found at:
> >> https://dist.apache.org/repos/dist/dev/hbase/2.2.6RC2/
> >>
> >> Maven artifacts are available in a staging repository at:
> >> https://repository.apache.org/content/repositories/orgapachehbase-1407
> >>
> >> Signatures used for HBase RCs can be found in this file:
> >> https://dist.apache.org/repos/dist/release/hbase/KEYS
> >>
> >> The list of bug fixes going into 2.2.6 can be found in included
> >> CHANGES.md and RELEASENOTES.md available here:
> >> https://dist.apache.org/repos/dist/dev/hbase/2.2.6RC2/CHANGES.md
> >> https://dist.apache.org/repos/dist/dev/hbase/2.2.6RC2/RELEASENOTES.md
> >>
> >> A detailed source and binary compatibility report for this release is
> >> available at:
> >>
> >>
> https://dist.apache.org/repos/dist/dev/hbase/2.2.6RC2/api_compare_2.2.6RC2_to_2.2.5.html
> >>
> >> To learn more about Apache HBase, please see http://hbase.apache.org/
> >>
> >> Thanks,
> >> Guanghao Zhang
> >>
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> >- A23, Crosstalk
> >
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

[VOTE] The third HBase 2.2.6 release candidate (RC2) is available

[Guanghao Zhang]

Please vote on this release candidate (RC2) for Apache HBase 2.2.6.

The VOTE will remain open for at least 72 hours.

[ ] +1 Release this package as Apache HBase 2.2.6
[ ] -1 Do not release this package because ...

The tag to be voted on is 2.2.6RC2. The release files, including
signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/hbase/2.2.6RC2/

Maven artifacts are available in a staging repository at:
https://repository.apache.org/content/repositories/orgapachehbase-1407

Signatures used for HBase RCs can be found in this file:
https://dist.apache.org/repos/dist/release/hbase/KEYS

The list of bug fixes going into 2.2.6 can be found in included
CHANGES.md and RELEASENOTES.md available here:
https://dist.apache.org/repos/dist/dev/hbase/2.2.6RC2/CHANGES.md
https://dist.apache.org/repos/dist/dev/hbase/2.2.6RC2/RELEASENOTES.md

A detailed source and binary compatibility report for this release is
available at:
https://dist.apache.org/repos/dist/dev/hbase/2.2.6RC2/api_compare_2.2.6RC2_to_2.2.5.html

To learn more about Apache HBase, please see http://hbase.apache.org/

Thanks,
Guanghao Zhang

[VOTE] The second HBase 2.2.6 release candidate (RC1) is available

[Guanghao Zhang]

Please vote on this release candidate (RC) for Apache HBase 2.2.6.

The VOTE will remain open for at least 72 hours.

[ ] +1 Release this package as Apache HBase 2.2.6
[ ] -1 Do not release this package because ...

The tag to be voted on is 2.2.6RC1. The release files, including
signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/hbase/2.2.6RC1/

Maven artifacts are available in a staging repository at:
https://repository.apache.org/content/repositories/orgapachehbase-1406/

Signatures used for HBase RCs can be found in this file:
https://dist.apache.org/repos/dist/release/hbase/KEYS

The list of bug fixes going into 2.2.6 can be found in included
CHANGES.md and RELEASENOTES.md available here:
https://dist.apache.org/repos/dist/dev/hbase/2.2.6RC1/CHANGES.md
https://dist.apache.org/repos/dist/dev/hbase/2.2.6RC1/RELEASENOTES.md

A detailed source and binary compatibility report for this release is
available at:
https://dist.apache.org/repos/dist/dev/hbase/2.2.6RC1/api_compare_2.2.6RC1_to_2.2.5.html

To learn more about Apache HBase, please see http://hbase.apache.org/

Thanks,
Guanghao Zhang

Re: hbase 主备集群同时开启replication后WALs目录越积越大

[Guanghao Zhang]

需要多次jstack看看replicationSource线程卡在哪里，可能就是卡在某个循环里出不来了

zheng wang <18031...@qq.com> 于2020年7月22日周三 下午12:20写道：

> 2.0.x 都有这个问题，2.1.0不确定，但很可能也有。
>
>
>
>
> --原始邮件--
> 发件人:
>   "user-zh"
> <
> palomino...@gmail.com;
> 发送时间:2020年7月22日(星期三) 中午12:09
> 收件人:"user-zh"
> 主题:Re: hbase 主备集群同时开启replication后WALs目录越积越大
>
>
>
> 对，你看看卡在哪了到底
>
> sudo rm -rf /* 
>  机器已经重启过了，现在replication是没有延时的，昨天的异常场景
>  现在没办法重现，只是重启过机器后，replication已经没有延时了，WALs
> 还在持续增长，现在WALs的量从昨晚到现在已经增至670G，
>  还在持续增长，我们RegionServer是5台，RS的堆内存设置的是32G，按理说数据持续flush
>  ，replication又没有延时，WALS应该及时被清理才对。我们的MajorCompact是手动触发的，目前禁用了Major
> 
> 
> 
> 
> 
> 
>  --nbsp;原始邮件nbsp;--
>  发件人:
> 
> "user-zh"
> 
> <
>  binli...@gmail.comgt;;
>  发送时间:nbsp;2020年7月22日(星期三) 中午11:38
>  收件人:nbsp;"user-zh" 
>  主题:nbsp;Re: hbase 主备集群同时开启replication后WALs目录越积越大
> 
> 
> 
>  你把延迟严重的regionserver的jstack发出来看看replication主要耗时在那个阶段
> 
>  sudo rm -rf /*  
>  gt; 主备集群两边是通的，重启之前某一个RegionServer的replication队列一直积压，是卡在了一个WAls
>  gt; log上，这个数据包就一直卡着，不发。amp;nbsp;
>  gt; 重启之后，主备集群两边的replication延时都没有了，但是WALs 依旧持续增加amp;nbsp;
>  gt;
>  gt;
>  gt;
>  gt;
>  gt;
> --amp;nbsp;原始邮件amp;nbsp;--
>  gt; 发件人:
> 
> gt;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;
>  "user-zh"
> 
> gt;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;
>  <
>  gt; palomino...@gmail.comamp;gt;;
>  gt; 发送时间:amp;nbsp;2020年7月22日(星期三) 中午11:25
>  gt; 收件人:amp;nbsp;"user-zh" amp;gt;;
>  gt;
>  gt; 主题:amp;nbsp;Re: hbase 主备集群同时开启replication后WALs目录越积越大
>  gt;
>  gt;
>  gt;
>  gt; 你看看为啥有一个region server的replication延迟很大？
>  gt; 是写入有热点？还是对面集群不收？
>  gt;
>  gt; sudo rm -rf /*  上午10:59写道：
>  gt;
>  gt; amp;gt; hi:
>  gt; amp;gt; amp;amp;nbsp; amp;amp;nbsp;
> 我们线上集群的版本是cdh6.3.2-hbase2.1.0,
>  gt; amp;gt;
>  主备双集群，备集群作为容灾集群。两集群开启了replication，且互相replication。最近发现，主集群的WALs目录持续增加，
>  gt; amp;gt;
>  gt;
> 
> 十几天膨胀到十几个T，排查后发现某一个RegionServer节点的replication延迟很大，复制队列中积累了大量的数据，延迟卡在了十几天前的WALs
>  gt; amp;gt; log，我们通过重建peeer,重启集群，这部分积累的WALs才
>  gt; amp;gt;
>  gt; amp;gt;
>  gt;
> 
> 被移动到oldWALs目录中，然后被清除了。重启完集群，replication的延时没了，但现在依然发现，WALs持续在增加，但此时replication的状态是正常的。
>  gt; amp;gt;
>  gt; amp;gt;
>  gt; amp;gt;
>  gt; amp;gt;
>  gt; amp;gt;
>  gt; amp;gt;
>  gt;
> 
> 与之对应的是备集群的oldWALs目录持续膨胀，不会被清除，实在不知道这样的问题该如何去分析，我们开启replication的过程参考的是官网的步骤。replication的相关参数，用的是默认的。
>  gt; amp;gt; 还麻烦张老师帮忙看一下，该怎么去分析这个问题。
> 
> 
> 
>  --
>  *Best Regards,*
>  nbsp;lijin bin

Re: Re: 开启replication后，hbase读写性能下降问题

[Guanghao Zhang]

是A集群replication到B集群, A集群自身被Replication影响?

zheng wang <18031...@qq.com> 于2020年7月21日周二 下午1:46写道：

> replication确实是异步的，影响主要是在资源上，比如GC暂停、IO等待这些。
>
>
>
>
> --原始邮件--
> 发件人:
>
> "user-zh@hbase.apache.orgww112...@sina.com"
>;
> 发送时间:2020年7月21日(星期二) 中午1:37
> 收件人:"user-zh"
> 主题:回复：Re: 开启replication后，hbase读写性能下降问题
>
>
>
> 写入每秒3万，只有一个peer。 看了下磁盘的写入每秒在4-5M，读取在1M，这个写入量对ssd应该没什么压力吧。
> 我理解replication的过程应该是异步的，为什么会对读取和写入造成那么大的影响。
> 该怎么看瓶颈在哪？
>
> - 原始邮件 -
> 发件人：张铎(Duo Zhang)  收件人：user-zh  主题：Re: 开启replication后，hbase读写性能下降问题
> 日期：2020年07月21日 12点19分
>
> 写入的很猛吗？需要看看瓶颈在哪，我们之前遇到过peer很多的情况下，replication对HDFS的压力很大，因为每个peer都要读一遍WAL。
> 你这个是SSD的话，看起来是replication本身对region server的压力很大？
> zheng wang <18031...@qq.com 于2020年7月21日周二 下午12:05写道：
>  可以考虑对replication限速。
> 
> 
> 
> 
>  -- 原始邮件 --
>  发件人:
> 
>  "user-zh@hbase.apache.orgww112...@sina.com"
> 
>   gt;;
>  发送时间: 2020年7月21日(星期二) 中午11:51
>  收件人: "user-zh" 
>  主题: 开启replication后，hbase读写性能下降问题
> 
> 
> 
> 
>  在开启hbase的replication后，集群的slowGetCount和slowPutCount都会大幅增长。请问这种情况怎么解决。
>  开启复制之后，日志里边还有大量的slow sync日志。日志如下：
>  2020-07-21 11:08:11,314 INFO
>  org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL: Slow sync
> cost: 635
>  ms, current pipeline:
> 
> [DatanodeInfoWithStorage[x:9866,DS-6522f59f-3ca9-46a9-b25a-5d86cdbb656e,SSD],
> 
> DatanodeInfoWithStorage[x:9866,DS-8768f8b8-732f-4238-b51d-90ba83343353,SSD],
> 
> DatanodeInfoWithStorage[x:9866,DS-a5792087-6c58-4600-a239-9a1f61f1c6c0,SSD]]2020-07-21
>  11:08:11,315 INFO
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL:
>  Slow sync cost: 564 ms, current pipeline:
> 
> [DatanodeInfoWithStorage[x:9866,DS-6522f59f-3ca9-46a9-b25a-5d86cdbb656e,SSD],
> 
> DatanodeInfoWithStorage[x:9866,DS-8768f8b8-732f-4238-b51d-90ba83343353,SSD],
> 
> DatanodeInfoWithStorage[x:9866,DS-a5792087-6c58-4600-a239-9a1f61f1c6c0,SSD]]2020-07-21
>  11:08:11,315 INFO
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL:
>  Slow sync cost: 535 ms, current pipeline:
> 
> [DatanodeInfoWithStorage[x:9866,DS-6522f59f-3ca9-46a9-b25a-5d86cdbb656e,SSD],
> 
> DatanodeInfoWithStorage[x:9866,DS-8768f8b8-732f-4238-b51d-90ba83343353,SSD],
> 
> DatanodeInfoWithStorage[x:9866,DS-a5792087-6c58-4600-a239-9a1f61f1c6c0,SSD]]

[VOTE] The first HBase 2.2.6 release candidate (RC0) is available

[Guanghao Zhang]

Please vote on this release candidate (RC) for Apache HBase 2.2.6.

The VOTE will remain open for at least 72 hours.

[ ] +1 Release this package as Apache HBase 2.2.6
[ ] -1 Do not release this package because ...

The tag to be voted on is 2.2.6RC0. The release files, including
signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/hbase/2.2.6RC0/

Maven artifacts are available in a staging repository at:
https://repository.apache.org/content/repositories/orgapachehbase-1401

Signatures used for HBase RCs can be found in this file:
https://dist.apache.org/repos/dist/release/hbase/KEYS

The list of bug fixes going into 2.2.6 can be found in included
CHANGES.md and RELEASENOTES.md available here:
https://dist.apache.org/repos/dist/dev/hbase/2.2.6RC0/CHANGES.md
https://dist.apache.org/repos/dist/dev/hbase/2.2.6RC0/RELEASENOTES.md

A detailed source and binary compatibility report for this release is
available at:
https://dist.apache.org/repos/dist/dev/hbase/2.2.6RC0/api_compare_2.2.6RC0_to_2.2.5.html

To learn more about Apache HBase, please see http://hbase.apache.org/

Thanks,
Guanghao Zhang

Re: HBase资源隔离队列参数貌似不生效

[Guanghao Zhang]

是RegionServer CPU高, 还是Client测CPU高?

sudo rm -rf /*  于2020年6月3日周三 上午10:44写道：

> 我用的HBase版本是cdh6.3.2-hbase2.1.0
>
>
>
>
> --原始邮件--
> 发件人:"张铎(Duo Zhang)" 发送时间:2020年6月3日(星期三) 上午10:22
> 收件人:"user-zh"
> 主题:Re: HBase资源隔离队列参数貌似不生效
>
>
>
> 图好像看不到？
>
> sudo rm -rf /* 
>  配置了队列隔离参数，参数如下：
>  hbase.regionserver.handler.count 30
>  hbase.ipc.server.callqueue.handler.factor 1
>  hbase.ipc.server.callqueue.read.ratio 0.5
>  hbase.ipc.server.callqueue.scan.ratio 0.1
> 
>  这样我控制scan的线程为1，
>  此时，查看线程的分配情况
> 
> 
>  接着下运行测试代码，只有scan的操作。
>  from happybase import ConnectionPool
>  import time
> 
>  pool = ConnectionPool(size=1, host='ip', port=9090, timeout=2000)
>  for i in range(200):
>  start = time.time()
>  try:
>  with
> pool.connection(2000) as con:
> 
> table = con.table("table_name")
> 
> res = list(table.scan(filter="PrefixFilter('273810955|')",
> 
> 
> row_start='\x0f\x10R\xca\xdf\x96\xcb\xe2\xad7$\xad9khE\x19\xfd\xaa\x87\xa5\xdd\xf7\x85\x1c\x81ku
>  ^\x92k',
> 
> limit=3))
>  except Exception as e:
>  pass
>  end = time.time()
>  print 'timeout: %d' % (end - start)
> 
>  此时线程的运行情况
> 
>  此时我们的CPU占用，已经被跑满了
> 
> 为什么我配置了队列隔离，只运行scan操作，还会有请求占用get的处理线程，我们的目的是想要限制scan所使用的资源，但现在看来，好像没有生效
> 
> 

Re: 这个是触发了gc的问题吗

[Guanghao Zhang]

>>> 13193.635 / 100818.0
0.13086586720625284

平均GC STW时间130ms, 你配置的是-XX:MaxGCPauseMillis=100, 对于G1GC这个不是绝对保证, 所以还算正常

 于2020年5月27日周三 下午9:27写道：

> rs的配置如下：
>hbase版本：1.2.0
>jvm:1.8.0_172
>gc :  g1
>heap : -Xms120g -Xmx120g
>
> rs进程如下：
> /usr/java/latest/bin/java -Dproc_regionserver -XX:OnOutOfMemoryError=kill
> -9 %p -Djava.net.preferIPv4Stack=true -Xms16106127360 -Xmx16106127360
> -XX:+UseG1GC -Xms120g -Xmx120g -XX:+UnlockExperimentalVMOptions
> -XX:G1NewSizePercent=1 -XX:MaxGCPauseMillis=100 -verbose:gc
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> -Xloggc:/var/log/hbase/hbase-gc.log -XX:+UseGCLogFileRotation
> -XX:NumberOfGCLogFiles=4 -XX:GCLogFileSize=512M
> -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime
> -XX:InitiatingHeapOccupancyPercent=70 -Djute.maxbuffer=41943040
> -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/tmp/hbase2_hbase2-REGIONSERVER-f972621f41cb4c66f7520eb42f0bccca_pid189753.hprof
> -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh
> -Dhbase.log.dir=/var/log/hbase
> -Dhbase.log.file=hbase-cmf-hbase2-REGIONSERVER-hbase-common-10.bigdata.lf.hw.lan.log.out
> -Dhbase.home.dir=/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hbase
> -Dhbase.id.str= -Dhbase.root.logger=INFO,RFA
> -Djava.library.path=/opt/cloudera/parcels/GPLEXTRAS-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/lib/native:/opt/cloudera/parcels/GPLEXTRAS-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hbase/lib/native/Linux-amd64-64
> -Dhbase.security.logger=INFO,RFAS
> org.apache.hadoop.hbase.regionserver.HRegionServer start
>
> 通过火焰图查看，gc的占用了很大一部分。
>
> 通过jstat -gcutil看gc频次和gc耗时看不出什么问题。截取一部分日志：
> 0.00 100.00  71.43  54.50  98.70  - 100818 13193.635 00.000
> 13193.635
>   0.00 100.00  72.35  54.50  98.70  - 100818 13193.635 00.000
> 13193.635
>   0.00 100.00  73.27  54.50  98.70  - 100818 13193.635 00.000
> 13193.635
>   0.00 100.00  74.19  54.50  98.70  - 100818 13193.635 00.000
> 13193.635
>   0.00 100.00  75.12  54.50  98.70  - 100818 13193.635 00.000
> 13193.635
>   0.00 100.00  76.04  54.50  98.70  - 100818 13193.635 00.000
> 13193.635
>   0.00 100.00  76.96  54.50  98.70  - 100818 13193.635 00.000
> 13193.635
>   0.00 100.00  77.88  54.50  98.70  - 100818 13193.635 00.000
> 13193.635
>   0.00 100.00  78.80  54.50  98.70  - 100818 13193.635 00.000
> 13193.635
>   0.00 100.00  79.72  54.50  98.70  - 100818 13193.635 00.000
> 13193.635
>   0.00 100.00  81.11  54.50  98.70  - 100818 13193.635 00.000
> 13193.635
>   0.00 100.00  82.49  54.50  98.70  - 100818 13193.635 00.000
> 13193.635
>   0.00 100.00  83.87  54.50  98.70  - 100818 13193.635 00.000
> 13193.635
>   0.00 100.00  84.79  54.50  98.70  - 100818 13193.635 00.000
> 13193.635
>   0.00 100.00  85.71  54.50  98.70  - 100818 13193.635 00.000
> 13193.635
>   0.00 100.00  87.10  54.50  98.70  - 100818 13193.635 00.000
> 13193.635
>   0.00 100.00  88.48  54.50  98.70  - 100818 13193.635 00.000
> 13193.635
>   0.00 100.00  89.40  54.50  98.70  - 100818 13193.635 00.000
> 13193.635
>   0.00 100.00  90.78  54.50  98.70  - 100818 13193.635 00.000
> 13193.635
>   0.00 100.00  92.17  54.50  98.70  - 100818 13193.635 00.000
> 13193.635
>   0.00 100.00  93.09  54.50  98.70  - 100818 13193.635 00.000
> 13193.635
>   0.00 100.00  94.01  54.50  98.70  - 100818 13193.635 00.000
> 13193.635
>   0.00 100.00   1.56  55.13  98.70  - 100819 13193.718 00.000
> 13193.718
>   0.00 100.00   1.95  55.13  98.70  - 100819 13193.718 00.000
> 13193.718
>   0.00 100.00   3.12  55.13  98.70  - 100819 13193.718 00.000
> 13193.718
>   0.00 100.00   3.91  55.13  98.70  - 100819 13193.718 00.000
> 13193.718
>   0.00 100.00   5.08  55.13  98.70  - 100819 13193.718 00.000
> 13193.718
>   0.00 100.00   5.86  55.13  98.70  - 100819 13193.718 00.000
> 13193.718
>   0.00 100.00   7.03  55.13  98.70  - 100819 13193.718 00.000
> 13193.718
>   0.00 100.00   7.42  55.13  98.70  - 100819 13193.718 00.000
> 13193.718
>   0.00 100.00   8.20  55.13  98.70  - 100819 13193.718 00.000
> 13193.718
>   0.00 100.00   8.98  55.13  98.70  - 100819 13193.718 00.000
> 13193.718
>   0.00 100.00   9.77  55.13  98.70  - 100819 13193.718 00.000
> 13193.718
>   0.00 100.00  10.16  55.13  98.70  - 100819 13193.718 00.000
> 13193.718
>   0.00 100.00  11.33  55.13  98.70  - 100819 13193.718 00.000
> 13193.718
>   0.00 100.00  12.50  55.13  98.70  - 100819 13193.718 00.000
> 13193.718
>
>
>

[通知]Apache HBase 2.2.5版本现在可以下载了

[Guanghao Zhang]

HBase团队很高兴宣布HBase 2.2.5版本现在可用了

Apache HBase™是一种开源的，分布式的，多版本的非关系型数据库。 Apache HBase可以部署在低成本硬件上,
来支持数十亿行/数百万列的超级大表，同时保证低延迟的随机读写能力。要了解更多有关HBase详情，参见https://hbase.apache.org/。

要下载2.2.5，请访问我们的下载页面：
http://hbase.apache.org/downloads.html

HBase 2.2.5是HBase 2.2系列的最新版本，目标是继续为Apache大数据生态带来一款稳定可靠的数据库。对比2.2.4版本,
2.2.5版本总共包括85个Bug修复和改进, 详细的改进列表参考:
https://downloads.apache.org/hbase/2.2.5/CHANGES.md

注意：RemoteHTable和RemoteAdmin这两个接口在本次版本中有一些不兼容修改, 因为这两个接口一直是专门为测试使用的,
所以在2.2.5版本中将它们标记为私有类, 更多详细信息，请参见HBASE-24115。

有关验证ASF版本下载的说明，请参阅
https://www.apache.org/dyn/closer.cgi#verify

可以在下面链接找到项目成员签名密钥
https://www.apache.org/dist/hbase/KEYS

随时欢迎提出问题和意见：
d...@hbase.apache.org
u...@hbase.apache.org
user-zh@hbase.apache.org

感谢所有为该版本做出贡献的contributor, committer!

HBase开发团队

[ANNOUNCE] Apache HBase 2.2.5 is now available for download

[Guanghao Zhang]

The HBase team is happy to announce the immediate availability of HBase
2.2.5.

Apache HBase™ is an open-source, distributed, versioned, non-relational
database. Apache HBase gives you low latency random access to billions of
rows with
millions of columns atop non-specialized hardware. To learn more about
HBase,
see https://hbase.apache.org/.

To download 2.2.5, visit our download page:
http://hbase.apache.org/downloads.html

HBase 2.2.5 is the latest release of the HBase 2.2 line, continuing on the
theme of bringing a stable, reliable database to the Apache Big Data
ecosystem and beyond. 2.2.5 includes 85 bug fixes and improvement  done
since the 2.2.4.

The list of 85 changes made since 2.2.4 can be found here:
https://downloads.apache.org/hbase/2.2.5/CHANGES.md

NOTICE: There are some incompatibility changes for RemoteHTable and
RemoteAdmin interface. They are test only and marked to private now. See
HBASE-24115 for more details.

For instructions on verifying ASF release downloads, please see
https://www.apache.org/dyn/closer.cgi#verify

Project member signature keys can be found at
https://www.apache.org/dist/hbase/KEYS

Question, comments, and problems are always welcome at:
d...@hbase.apache.org
user@hbase.apache.org
user...@hbase.apache.org

Thanks to all who contributed and made this release possible.

Cheers,
The HBase Dev Team

[通知]欢迎Lijin Bin加入HBase PMC

[Guanghao Zhang]

我很高兴代表Apache HBase PMC宣布Lijin Bin已接受我们的邀请，成为Apache HBase项目的PMC成员。感谢Lijin
Bin为HBase项目做出的贡献，期待未来在HBase项目中承担更多责任。

欢迎Lijin Bin加入HBase PMC！

[ANNOUNCE] Please welcome Lijin Bin to the HBase PMC

[Guanghao Zhang]

On behalf of the Apache HBase PMC I am pleased to announce that Lijin Bin
has accepted our invitation to become a PMC member on the Apache HBase
project. We appreciate Lijin Bin stepping up to take more responsibility in
the HBase project.

Please join me in welcoming Lijin Bin to the HBase PMC!

Re: [ANNOUNCE] New HBase committer Wei-Chiu Chuang

[Guanghao Zhang]

Congratulations and welcome Wei-Chiu!

Wellington Chevreuil  于2020年5月14日周四
下午6:01写道：

> Congratulations, Wei-Chiu! Welcome!
>
> Em qui., 14 de mai. de 2020 às 10:12, Jan Hentschel <
> jan.hentsc...@ultratendency.com> escreveu:
>
> > Congratulations Wei-Chiu and welcome!
> >
> > From: Sean Busbey 
> > Reply-To: "d...@hbase.apache.org" 
> > Date: Wednesday, May 13, 2020 at 9:10 PM
> > To: dev , Hbase-User 
> > Subject: [ANNOUNCE] New HBase committer Wei-Chiu Chuang
> >
> > Folks,
> >
> > On behalf of the Apache HBase PMC I am pleased to announce that Wei-Chiu
> > Chuang has accepted the PMC's invitation to become a committer on the
> > project.
> >
> > We appreciate all of the great contributions Wei-Chiu has made to the
> > community thus far and we look forward to his continued involvement.
> >
> > Allow me to be the first to congratulate Wei-Chiu on his new role!
> >
> > thanks,
> > busbey
> >
> >
>

[ANNOUNCE] Apache HBase 2.2.4 is now available for download

[Guanghao Zhang]

The HBase team is happy to announce the immediate availability of HBase
2.2.4.

Apache HBase™ is an open-source, distributed, versioned, non-relational
database. Apache HBase gives you low latency random access to billions of
rows with millions of columns atop non-specialized hardware. To learn more
about HBase, see https://hbase.apache.org/.

To download 2.2.4, visit our download page:
http://hbase.apache.org/downloads.html

HBase 2.2.4 is the latest release of the HBase 2.2 line, continuing on the
theme of bringing a stable, reliable database to the Apache Big Data
ecosystem and beyond. 2.2.4 includes 42 bug fixes and improvement  done
since the 2.2.3.

The list of 42 changes made since 2.2.4 can be found here:
https://downloads.apache.org/hbase/2.2.4/CHANGES.md

For instructions on verifying ASF release downloads, please see
https://www.apache.org/dyn/closer.cgi#verify

Project member signature keys can be found at
https://www.apache.org/dist/hbase/KEYS

Question, comments, and problems are always welcome at: d...@hbase.apache.org

Thanks to all who contributed and made this release possible.

Cheers,
The HBase Dev Team

Re: [VOTE] The first HBase 2.2.4 release candidate (RC0) is available

[Guanghao Zhang]

+1 from me. With 3 binding +1s and 1 non-binding +1, the vote passes. Let
me push out the release.

Jan Hentschel  于2020年3月13日周五 下午10:22写道：

> +1 (binding)
>
> * Signature: ok
> * Checksum : ok
> * Rat check (1.8.0_202-ea): ok
>  - mvn clean apache-rat:check
> * Built from source (1.8.0_202-ea): ok
>  - mvn clean install -DskipTests
> * Unit tests pass (1.8.0_202-ea): ok
>  - mvn package -P runSmallTests
>
> From: Guanghao Zhang 
> Reply-To: "d...@hbase.apache.org" 
> Date: Wednesday, March 11, 2020 at 8:03 AM
> To: HBase Dev List , Hbase-User <
> user@hbase.apache.org>
> Subject: [VOTE] The first HBase 2.2.4 release candidate (RC0) is available
>
> Please vote on this release candidate (RC) for Apache HBase 2.2.4.
>
> The VOTE will remain open for at least 72 hours.
> [ ] +1 Release this package as Apache HBase 2.2.4
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 2.2.4RC0. The release files, including
> signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/hbase/2.2.4RC0/
>
> Maven artifacts are available in a staging repository at:
> https://repository.apache.org/content/repositories/orgapachehbase-1385/
>
> Signatures used for HBase RCs can be found in this file:
> https://dist.apache.org/repos/dist/release/hbase/KEYS
>
> The list of bug fixes going into 2.2.4 can be found in included
> CHANGES.md and RELEASENOTES.md available here:
> https://dist.apache.org/repos/dist/dev/hbase/2.2.4RC0/CHANGES.md
> https://dist.apache.org/repos/dist/dev/hbase/2.2.4RC0/RELEASENOTES.md
>
> A detailed source and binary compatibility report for this release is
> available at:
>
> https://dist.apache.org/repos/dist/dev/hbase/2.2.4RC0/api_compare_2.2.4RC0_to_2.2.3.html
>
> To learn more about Apache HBase, please see http://hbase.apache.org/
>
>
> Thanks,
> Guanghao Zhang
>
>

[VOTE] The first HBase 2.2.4 release candidate (RC0) is available

[Guanghao Zhang]

Please vote on this release candidate (RC) for Apache HBase 2.2.4.

The VOTE will remain open for at least 72 hours.
[ ] +1 Release this package as Apache HBase 2.2.4
[ ] -1 Do not release this package because ...

The tag to be voted on is 2.2.4RC0. The release files, including
signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/hbase/2.2.4RC0/

Maven artifacts are available in a staging repository at:
https://repository.apache.org/content/repositories/orgapachehbase-1385/

Signatures used for HBase RCs can be found in this file:
https://dist.apache.org/repos/dist/release/hbase/KEYS

The list of bug fixes going into 2.2.4 can be found in included
CHANGES.md and RELEASENOTES.md available here:
https://dist.apache.org/repos/dist/dev/hbase/2.2.4RC0/CHANGES.md
https://dist.apache.org/repos/dist/dev/hbase/2.2.4RC0/RELEASENOTES.md

A detailed source and binary compatibility report for this release is
available at:
https://dist.apache.org/repos/dist/dev/hbase/2.2.4RC0/api_compare_2.2.4RC0_to_2.2.3.html

To learn more about Apache HBase, please see http://hbase.apache.org/


Thanks,
Guanghao Zhang

Re: 关于HBase-23286 Improve MTTR: Split WAL to HFile的一些想法

[Guanghao Zhang]

这个慢也有可能是
hbase.split.writer.creation.bounded这个feature造成的，这个是为了解决HDFS同时写小文件过多可能导致卡住的问题，但是性能数据还需要确认下。Split
WAL to HFile这个功能，是默认开启了bounded这个feature的

张铎(Duo Zhang)  于2020年3月2日周一 下午4:18写道：

> 我理解，kill -9之后zk是不会马上就知道rs挂了的吧，本来也需要等一段时间？所以那个40s，是从kill
> -9开始算起，还是从master真正开始split开始算起？
>
> mintao  于2020年2月28日周五 下午5:06写道：
>
> > 1.测试表只有1个cf；
> > 2. hbase.split.writer.creation.bounded没有配置，但是我配置了
> > （1）hbase.regionserver.hlog.splitlog.writer.threads=20
> > （2）hbase.regionserver.wal.maxsplitters=10
> >
> >
> > | |
> > mintao
> > |
> > |
> > mintaoisj...@163.com
> > |
> > 签名由网易邮箱大师定制
> > 在2020年2月28日 16:53，Guanghao Zhang 写道：
> > 表有几个CF呢? split log生成的文件个数等于 WAL个数 * region个数, 而split生成HFile的时候是WAL个数 *
> > region个数 * CF个数的, 在CF多的时候写出去的文件会比之前多.
> > 另外还有一个问题是, hbase.split.writer.creation.bounded这个是否有开启?
> > split生成HFile是默认是用了bounded这个feature的
> >
> > mintao  于2020年2月28日周五 下午4:46写道：
> >
> >
> >
> >
> 我大概测试了十来次，在wal的数据量为12GB左右，100个wal文件情况下，没有开启writeToHFile时，split平均需要40s，assign平均需要30s，开启writeToHFile之后，split平均需要54s，assign需要4s。
> >
> >
> >
> >
> > | |
> > mintao
> > |
> > |
> > mintaoisj...@163.com
> > |
> > 签名由网易邮箱大师定制
> > 在2020年2月28日 16:23，Guanghao Zhang 写道：
> > 1. 可以贴下具体数据看看?
> > 2. 目前还没有修复, 欢迎提PR
> >
> > mintao  于2020年2月28日周五 下午4:19写道：
> >
> > 大家好：
> > 在HBase社区上看到HBase-23286这个jira，这里我有一些关于HBase-23286 Improve MTTR: Split WAL
> > to HFile的想法：
> > （1）我自己拉了代码测试了一下，发现开启writeToHFile之后整体恢复时间是有所缩短，特别是region
> > assgin消耗的时间，但是split阶段花费的时间还是有所增加了。
> > 具体测试的环境和测试流程：
> > 测试环境是两个节点，每个节点2个regionserver，集群内总共300个region，故障RS上有77个region，
> > 100个wal，每个wal大概120MB，测试过程是通过kill -9 故障RS宕掉之后，通过观察master日志来确定恢复服务的时间。
> > 测试结果：
> > 测试环境是两个节点，每个节点2个regionserver，集群内总共300个region，故障RS上有77个region，
> > 100个wal，每个wal大概120MB，测试过程是通过kill -9 故障RS宕掉之后，通过观察master日志来确定恢复服务的时间；
> > 是不是我测试的过程中遗漏了什么步骤？跟jira上的测试结果有一些差异。
> > （2）这个功能在社区中反映是存在一些问题的，比如说存在数据丢失（
> > https://issues.apache.org/jira/browse/HBASE-23741
> > ），这个bug是否已经修复了，我这边已经定位到问题，应该是跟sequenceId有关，我这边本地已经复现并修复了，是否可以将该修复提交到社区？
> >
> >
> >
> >
> > | |
> > mintao
> > |
> > |
> > mintaoisj...@163.com
> > |
> > 签名由网易邮箱大师定制
> >
> >
>

Re: 关于HBase-23286 Improve MTTR: Split WAL to HFile的一些想法

[Guanghao Zhang]

表有几个CF呢? split log生成的文件个数等于 WAL个数 * region个数, 而split生成HFile的时候是WAL个数 *
region个数 * CF个数的, 在CF多的时候写出去的文件会比之前多.
另外还有一个问题是, hbase.split.writer.creation.bounded这个是否有开启?
split生成HFile是默认是用了bounded这个feature的

mintao  于2020年2月28日周五 下午4:46写道：

>
> 我大概测试了十来次，在wal的数据量为12GB左右，100个wal文件情况下，没有开启writeToHFile时，split平均需要40s，assign平均需要30s，开启writeToHFile之后，split平均需要54s，assign需要4s。
>
>
>
>
> | |
> mintao
> |
> |
> mintaoisj...@163.com
> |
> 签名由网易邮箱大师定制
> 在2020年2月28日 16:23，Guanghao Zhang 写道：
> 1. 可以贴下具体数据看看?
> 2. 目前还没有修复, 欢迎提PR
>
> mintao  于2020年2月28日周五 下午4:19写道：
>
> 大家好：
> 在HBase社区上看到HBase-23286这个jira，这里我有一些关于HBase-23286 Improve MTTR: Split WAL
> to HFile的想法：
> （1）我自己拉了代码测试了一下，发现开启writeToHFile之后整体恢复时间是有所缩短，特别是region
> assgin消耗的时间，但是split阶段花费的时间还是有所增加了。
> 具体测试的环境和测试流程：
> 测试环境是两个节点，每个节点2个regionserver，集群内总共300个region，故障RS上有77个region，
> 100个wal，每个wal大概120MB，测试过程是通过kill -9 故障RS宕掉之后，通过观察master日志来确定恢复服务的时间。
> 测试结果：
> 测试环境是两个节点，每个节点2个regionserver，集群内总共300个region，故障RS上有77个region，
> 100个wal，每个wal大概120MB，测试过程是通过kill -9 故障RS宕掉之后，通过观察master日志来确定恢复服务的时间；
> 是不是我测试的过程中遗漏了什么步骤？跟jira上的测试结果有一些差异。
> （2）这个功能在社区中反映是存在一些问题的，比如说存在数据丢失（
> https://issues.apache.org/jira/browse/HBASE-23741
> ），这个bug是否已经修复了，我这边已经定位到问题，应该是跟sequenceId有关，我这边本地已经复现并修复了，是否可以将该修复提交到社区？
>
>
>
>
> | |
> mintao
> |
> |
> mintaoisj...@163.com
> |
> 签名由网易邮箱大师定制
>

Re: 关于HBase-23286 Improve MTTR: Split WAL to HFile的一些想法

[Guanghao Zhang]

1. 可以贴下具体数据看看?
2. 目前还没有修复, 欢迎提PR

mintao  于2020年2月28日周五 下午4:19写道：

> 大家好：
> 在HBase社区上看到HBase-23286这个jira，这里我有一些关于HBase-23286 Improve MTTR: Split WAL
> to HFile的想法：
> （1）我自己拉了代码测试了一下，发现开启writeToHFile之后整体恢复时间是有所缩短，特别是region
> assgin消耗的时间，但是split阶段花费的时间还是有所增加了。
> 具体测试的环境和测试流程：
> 测试环境是两个节点，每个节点2个regionserver，集群内总共300个region，故障RS上有77个region，
> 100个wal，每个wal大概120MB，测试过程是通过kill -9 故障RS宕掉之后，通过观察master日志来确定恢复服务的时间。
> 测试结果：
> 测试环境是两个节点，每个节点2个regionserver，集群内总共300个region，故障RS上有77个region，
> 100个wal，每个wal大概120MB，测试过程是通过kill -9 故障RS宕掉之后，通过观察master日志来确定恢复服务的时间；
> 是不是我测试的过程中遗漏了什么步骤？跟jira上的测试结果有一些差异。
> （2）这个功能在社区中反映是存在一些问题的，比如说存在数据丢失（
> https://issues.apache.org/jira/browse/HBASE-23741
> ），这个bug是否已经修复了，我这边已经定位到问题，应该是跟sequenceId有关，我这边本地已经复现并修复了，是否可以将该修复提交到社区？
>
>
>
>
> | |
> mintao
> |
> |
> mintaoisj...@163.com
> |
> 签名由网易邮箱大师定制

[ANNOUNCE] Apache HBase 2.2.3 is now available for download

[Guanghao Zhang]

The HBase team is happy to announce the immediate availability of HBase
2.2.3.

Apache HBase™ is an open-source, distributed, versioned, non-relational
database. Apache HBase gives you low latency random access to billions of
rows with
millions of columns atop non-specialized hardware. To learn more about
HBase,
see https://hbase.apache.org/.

To download 2.2.3, visit our download page:
http://hbase.apache.org/downloads.html

HBase 2.2.3 is the latest release of the HBase 2.2 line, continuing on the
theme of bringing a stable, reliable database to the Apache Big Data
ecosystem and beyond. 2.2.3 includes 115 bug fixes and improvement  done
since the 2.2.2.

The list of 115 changes made since 2.2.2 can be found here:
https://apache.org/dist/hbase/2.2.3/CHANGES.md

For instructions on verifying ASF release downloads, please see
https://www.apache.org/dyn/closer.cgi#verify

Project member signature keys can be found at
https://www.apache.org/dist/hbase/KEYS

Question, comments, and problems are always welcome at: d...@hbase.apache.org

Thanks to all who contributed and made this release possible.

Cheers,
The HBase Dev Team

[VOTE] The first HBase 2.2.3 release candidate (RC1) is available

[Guanghao Zhang]

Please vote on this release candidate (RC) for Apache HBase 2.2.3.

The VOTE will remain open for at least 72 hours.

[ ] +1 Release this package as Apache HBase 2.2.3
[ ] -1 Do not release this package because ...

The tag to be voted on is 2.2.3RC1. The release files, including
signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/hbase/2.2.3RC1/

Maven artifacts are available in a staging repository at:
https://repository.apache.org/content/repositories/orgapachehbase-1380/

Signatures used for HBase RCs can be found in this file:
https://dist.apache.org/repos/dist/release/hbase/KEYS

The list of bug fixes going into 2.2.3 can be found in included
CHANGES.md and RELEASENOTES.md available here:
https://dist.apache.org/repos/dist/dev/hbase/2.2.3RC1/CHANGES.md
https://dist.apache.org/repos/dist/dev/hbase/2.2.3RC1/RELEASENOTES.md

A detailed source and binary compatibility report for this release is
available at:
https://dist.apache.org/repos/dist/dev/hbase/2.2.3RC1/api_compare_2.2.3RC1_to_2.2.2.html

To learn more about Apache HBase, please see http://hbase.apache.org/

Thanks,
Guanghao Zhang

Re: NOTICE: hbase-2.2.1, 2.1.6 and 2.0.6 have a bug may cause data loss

[Guanghao Zhang]

>
> Either we call merge manually, or the region normalizer is enabled?

Got it. I found this because I use hbck2 to fixMeta overlap many times.
fixMeta overlap will call merge manually.

张铎(Duo Zhang)  于2019年9月19日周四 上午11:04写道：

> I do not think we will merge regions automatically? Either we call merge
> manually, or the region normalizer is enabled? And the way to produce the
> bug is a bit complicated, need multiple round of mergings.
>
> So agree that we should roll new releases soon, but the actual damage
> should be fine.
>
> Guanghao Zhang  于2019年9月19日周四 上午8:55写道：
>
> > Only 2.2.1, 2.1.6 and 2.0.6 are effected. Earlier releases are fine.
> Please
> > avoid downloading these releases. If you already running these versions
> in
> > your cluster, please use splitormerge_switch to disable region merge
> > now. See HBASE-23044 for more details.
> >
> > Need to rolling new release for branch-2.1 and branch-2.2 line. Ping
> @stack
> > for branch-2.0, do we need a more release 2.0.7?
> >
> > Thanks
> > Guanghao Zhang
> >
>

NOTICE: hbase-2.2.1, 2.1.6 and 2.0.6 have a bug may cause data loss

[Guanghao Zhang]

Only 2.2.1, 2.1.6 and 2.0.6 are effected. Earlier releases are fine. Please
avoid downloading these releases. If you already running these versions in
your cluster, please use splitormerge_switch to disable region merge
now. See HBASE-23044 for more details.

Need to rolling new release for branch-2.1 and branch-2.2 line. Ping @stack
for branch-2.0, do we need a more release 2.0.7?

Thanks
Guanghao Zhang

[ANNOUNCE] Apache HBase 2.2.1 is now available for download

[Guanghao Zhang]

The HBase team is happy to announce the immediate availability of HBase
2.2.1.

Apache HBase™ is an open-source, distributed, versioned, non-relational
database. Apache HBase gives you low latency random access to billions of
rows with
millions of columns atop non-specialized hardware. To learn more about
HBase, see https://hbase.apache.org/.

To download 2.2.1, visit our download page:
http://hbase.apache.org/downloads.html

HBase 2.2.1 is the latest release of the HBase 2.2 line, continuing on the
theme of bringing a stable, reliable database to the Apache Big Data
ecosystem and beyond. 2.2.1 includes 161 bug and improvement fixes done
since the 2.2.0. Most hbck2 abilities introduced by HBASE-21745 was
released in 2.2.1 (HBASE-21745 addressed gaps between hbck1 and hbck2). And
There are several critical fixes:
-HBASE-22867 The ForkJoinPool in CleanerChore will spawn thousands of
threads in our cluster with thousands table
-HBASE-22169 Open region failed cause memory leak
-Fixes around WAL, which may cause WAL corruption or hang the region
server. Please see HBASE-22539, HBASE-22681, HBASE-22684 for more details.

The list of 161 changes made since 2.2.0 can be found here:
https://apache.org/dist/hbase/2.2.1/CHANGES.md

For instructions on verifying ASF release downloads, please see
https://www.apache.org/dyn/closer.cgi#verify

Project member signature keys can be found at
https://www.apache.org/dist/hbase/KEYS

Question, comments, and problems are always welcome at: d...@hbase.apache.org

Thanks to all who contributed and made this release possible.

Cheers,
The HBase Dev Team

Re: [VOTE] The third HBase 2.2.1 release candidate (RC2) is available

[Guanghao Zhang]

With 4 binding +1s, the vote passes.

Let me push out the release and send a notice.

Thanks all for verifying the release candidate.

Stack  于2019年9月14日周六 上午6:46写道：

> +1
> IMO, CHANGES and RELEASENOTES should accumulate all changes, not just show
> what is in current release.
> Compat report looks good.
> Built from src.
> Ran a loading and verified it there across restart.
> Checked sigs and hash on src tgz (Duo, sign Guanghao's key and vice-versa
> -- key is not 'a trusted signature!').
> S
>
>
>
> On Tue, Sep 10, 2019 at 4:06 AM Guanghao Zhang  wrote:
>
> > Please vote on this release candidate (RC2) for Apache HBase 2.2.1.
> >
> > As this RC only changed the tar ball revison, this VOTE will remain open
> > for 24 hours.
> >
> > [ ] +1 Release this package as Apache HBase 2.2.1
> > [ ] -1 Do not release this package because ...
> >
> > The tag to be voted on is 2.2.1RC2. The release files, including
> > signatures, digests, etc. can be found at:
> > https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC2/
> >
> > Maven artifacts are available in a staging repository at:
> > https://repository.apache.org/content/repositories/orgapachehbase-1328/
> >
> > Signatures used for HBase RCs can be found in this file:
> > https://dist.apache.org/repos/dist/release/hbase/KEYS
> >
> > The list of bug fixes going into 2.2.1 can be found in included
> > CHANGES.md and RELEASENOTES.md available here:
> > https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC2/CHANGES.md
> > https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC2/RELEASENOTES.md
> >
> > A detailed source and binary compatibility report for this release is
> > available at:
> >
> >
> https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC2/api_compare_2.2.1RC2_to_2.2.0.html
> >
> > To learn more about Apache HBase, please see http://hbase.apache.org/
> >
> > Thanks,
> > Guanghao Zhang
> >
>

Re: [VOTE] The third HBase 2.2.1 release candidate (RC2) is available

[Guanghao Zhang]

The UI problem HBASE-23005 is not a release blocker. Will fix it in next
release.

Guanghao Zhang  于2019年9月10日周二 下午7:07写道：

> +1 from me.
>
> Guanghao Zhang  于2019年9月10日周二 下午7:06写道：
>
>> Please vote on this release candidate (RC2) for Apache HBase 2.2.1.
>>
>> As this RC only changed the tar ball revison, this VOTE will remain open
>> for 24 hours.
>>
>> [ ] +1 Release this package as Apache HBase 2.2.1
>> [ ] -1 Do not release this package because ...
>>
>> The tag to be voted on is 2.2.1RC2. The release files, including
>> signatures, digests, etc. can be found at:
>> https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC2/
>>
>> Maven artifacts are available in a staging repository at:
>> https://repository.apache.org/content/repositories/orgapachehbase-1328/
>>
>> Signatures used for HBase RCs can be found in this file:
>> https://dist.apache.org/repos/dist/release/hbase/KEYS
>>
>> The list of bug fixes going into 2.2.1 can be found in included
>> CHANGES.md and RELEASENOTES.md available here:
>> https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC2/CHANGES.md
>> https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC2/RELEASENOTES.md
>>
>> A detailed source and binary compatibility report for this release is
>> available at:
>>
>> https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC2/api_compare_2.2.1RC2_to_2.2.0.html
>>
>> To learn more about Apache HBase, please see http://hbase.apache.org/
>>
>> Thanks,
>> Guanghao Zhang
>>
>

Re: [VOTE] The third HBase 2.2.1 release candidate (RC2) is available

[Guanghao Zhang]

+1 from me.

Guanghao Zhang  于2019年9月10日周二 下午7:06写道：

> Please vote on this release candidate (RC2) for Apache HBase 2.2.1.
>
> As this RC only changed the tar ball revison, this VOTE will remain open
> for 24 hours.
>
> [ ] +1 Release this package as Apache HBase 2.2.1
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 2.2.1RC2. The release files, including
> signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC2/
>
> Maven artifacts are available in a staging repository at:
> https://repository.apache.org/content/repositories/orgapachehbase-1328/
>
> Signatures used for HBase RCs can be found in this file:
> https://dist.apache.org/repos/dist/release/hbase/KEYS
>
> The list of bug fixes going into 2.2.1 can be found in included
> CHANGES.md and RELEASENOTES.md available here:
> https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC2/CHANGES.md
> https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC2/RELEASENOTES.md
>
> A detailed source and binary compatibility report for this release is
> available at:
>
> https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC2/api_compare_2.2.1RC2_to_2.2.0.html
>
> To learn more about Apache HBase, please see http://hbase.apache.org/
>
> Thanks,
> Guanghao Zhang
>

[VOTE] The third HBase 2.2.1 release candidate (RC2) is available

[Guanghao Zhang]

Please vote on this release candidate (RC2) for Apache HBase 2.2.1.

As this RC only changed the tar ball revison, this VOTE will remain open
for 24 hours.

[ ] +1 Release this package as Apache HBase 2.2.1
[ ] -1 Do not release this package because ...

The tag to be voted on is 2.2.1RC2. The release files, including
signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC2/

Maven artifacts are available in a staging repository at:
https://repository.apache.org/content/repositories/orgapachehbase-1328/

Signatures used for HBase RCs can be found in this file:
https://dist.apache.org/repos/dist/release/hbase/KEYS

The list of bug fixes going into 2.2.1 can be found in included
CHANGES.md and RELEASENOTES.md available here:
https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC2/CHANGES.md
https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC2/RELEASENOTES.md

A detailed source and binary compatibility report for this release is
available at:
https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC2/api_compare_2.2.1RC2_to_2.2.0.html

To learn more about Apache HBase, please see http://hbase.apache.org/

Thanks,
Guanghao Zhang

Re: [VOTE] The second HBase 2.2.1 release candidate (RC1) is available

[Guanghao Zhang]

+1 from me.

hbase-2.2.1-bin.tar.gz (openjdk 1.8.0_202)
- Verified sha512sum: ok
- Start HBase in standalone mode: ok
- Verified with shell, create/disable/enable/drop/get/put/scan/delete: ok
- Checked master/regionserver/table/region Web UI: ok

hbase-2.2.1-src.tar.gz (openjdk 1.8.0_202)
- Verified sha512sum: ok
- Build tarball: ok
- Start HBase in standalone mode: ok
- Verified with shell, create/disable/enable/drop/get/put/scan/delete: ok
- Checked master/regionserver/table/region Web UI: ok

Guanghao Zhang  于2019年9月4日周三 下午5:52写道：

> Please vote on this release candidate (RC1) for Apache HBase 2.2.1.
>
> The VOTE will remain open for at least 72 hours.
>
> [ ] +1 Release this package as Apache HBase 2.2.1
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 2.2.1RC1. The release files, including
> signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC1/
>
> Maven artifacts are available in a staging repository at:
> https://repository.apache.org/content/repositories/orgapachehbase-1332/
>
> Signatures used for HBase RCs can be found in this file:
> https://dist.apache.org/repos/dist/release/hbase/KEYS
>
> The list of bug fixes going into 2.2.1 can be found in included
> CHANGES.md and RELEASENOTES.md available here:
> https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC1/CHANGES.md
> https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC1/RELEASENOTES.md
>
> A detailed source and binary compatibility report for this release is
> available at:
>
> https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC1/api_compare_2.2.1RC1_to_2.2.0.html
>
> To learn more about Apache HBase, please see http://hbase.apache.org/
>
> Thanks,
> Guanghao Zhang
>

[VOTE] The second HBase 2.2.1 release candidate (RC1) is available

[Guanghao Zhang]

Please vote on this release candidate (RC1) for Apache HBase 2.2.1.

The VOTE will remain open for at least 72 hours.

[ ] +1 Release this package as Apache HBase 2.2.1
[ ] -1 Do not release this package because ...

The tag to be voted on is 2.2.1RC1. The release files, including
signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC1/

Maven artifacts are available in a staging repository at:
https://repository.apache.org/content/repositories/orgapachehbase-1332/

Signatures used for HBase RCs can be found in this file:
https://dist.apache.org/repos/dist/release/hbase/KEYS

The list of bug fixes going into 2.2.1 can be found in included
CHANGES.md and RELEASENOTES.md available here:
https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC1/CHANGES.md
https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC1/RELEASENOTES.md

A detailed source and binary compatibility report for this release is
available at:
https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC1/api_compare_2.2.1RC1_to_2.2.0.html

To learn more about Apache HBase, please see http://hbase.apache.org/

Thanks,
Guanghao Zhang

Re: [VOTE] The first HBase 2.2.1 release candidate (RC0) is available

[Guanghao Zhang]

File HBASE-22857. It just the unit test problem, there are duplicate code...

Stack  于2019年8月15日周四 上午11:34写道：

> See
>
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.2/lastSuccessfulBuild/artifact/dashboard.html
>
> See the second paragraph, the tests that run on gce. Looks like they fail
> 100%.
>
> S
>
> On Wed, Aug 14, 2019 at 7:35 PM 张铎(Duo Zhang) 
> wrote:
>
> > Nightly will exclude the flaky tests, IIRC.
> >
> > Guanghao Zhang  于2019年8月15日周四 上午10:22写道：
> >
> > > Let me take a try. But the nightly job not failed by these unit tests.
> > See
> > > https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/
> > >
> > > 张铎(Duo Zhang)  于2019年8月15日周四 上午9:59写道：
> > >
> > > >
> > > >
> > >
> >
> https://builds.apache.org/job/HBASE-Find-Flaky-Tests/job/branch-2.2/lastSuccessfulBuild/artifact/dashboard.html
> > > >
> > > >
> > > > Seems TestHRegion and TestHRegionWithInMemoryFlush are always
> failing?
> > > >
> > > > Guanghao Zhang  于2019年8月14日周三 下午9:28写道：
> > > >
> > > > > Please vote on this release candidate (RC) for Apache HBase 2.2.1.
> > > > >
> > > > > The VOTE will remain open for at least 72 hours.
> > > > >
> > > > > [ ] +1 Release this package as Apache HBase 2.2.1
> > > > > [ ] -1 Do not release this package because ...
> > > > >
> > > > > The tag to be voted on is 2.2.1RC0. The release files, including
> > > > > signatures, digests, etc. can be found at:
> > > > > https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC0/
> > > > >
> > > > > Maven artifacts are available in a staging repository at:
> > > > >
> > >
> https://repository.apache.org/content/repositories/orgapachehbase-1328/
> > > > >
> > > > > Signatures used for HBase RCs can be found in this file:
> > > > > https://dist.apache.org/repos/dist/release/hbase/KEYS
> > > > >
> > > > > The list of bug fixes going into 2.2.1 can be found in included
> > > > > CHANGES.md and RELEASENOTES.md available here:
> > > > > https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC0/CHANGES.md
> > > > >
> > https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC0/RELEASENOTES.md
> > > > >
> > > > > A detailed source and binary compatibility report for this release
> is
> > > > > available at:
> > > > >
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC0/api_compare_2.2.1RC0_to_2.2.0.html
> > > > >
> > > > > To learn more about Apache HBase, please see
> > http://hbase.apache.org/
> > > > >
> > > > > Thanks,
> > > > > Guanghao Zhang
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] The first HBase 2.2.1 release candidate (RC0) is available

[Guanghao Zhang]

Let me take a try. But the nightly job not failed by these unit tests. See
https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/

张铎(Duo Zhang)  于2019年8月15日周四 上午9:59写道：

>
> https://builds.apache.org/job/HBASE-Find-Flaky-Tests/job/branch-2.2/lastSuccessfulBuild/artifact/dashboard.html
>
>
> Seems TestHRegion and TestHRegionWithInMemoryFlush are always failing?
>
> Guanghao Zhang  于2019年8月14日周三 下午9:28写道：
>
> > Please vote on this release candidate (RC) for Apache HBase 2.2.1.
> >
> > The VOTE will remain open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache HBase 2.2.1
> > [ ] -1 Do not release this package because ...
> >
> > The tag to be voted on is 2.2.1RC0. The release files, including
> > signatures, digests, etc. can be found at:
> > https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC0/
> >
> > Maven artifacts are available in a staging repository at:
> > https://repository.apache.org/content/repositories/orgapachehbase-1328/
> >
> > Signatures used for HBase RCs can be found in this file:
> > https://dist.apache.org/repos/dist/release/hbase/KEYS
> >
> > The list of bug fixes going into 2.2.1 can be found in included
> > CHANGES.md and RELEASENOTES.md available here:
> > https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC0/CHANGES.md
> > https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC0/RELEASENOTES.md
> >
> > A detailed source and binary compatibility report for this release is
> > available at:
> >
> >
> https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC0/api_compare_2.2.1RC0_to_2.2.0.html
> >
> > To learn more about Apache HBase, please see http://hbase.apache.org/
> >
> > Thanks,
> > Guanghao Zhang
> >
>

[VOTE] The first HBase 2.2.1 release candidate (RC0) is available

[Guanghao Zhang]

Please vote on this release candidate (RC) for Apache HBase 2.2.1.

The VOTE will remain open for at least 72 hours.

[ ] +1 Release this package as Apache HBase 2.2.1
[ ] -1 Do not release this package because ...

The tag to be voted on is 2.2.1RC0. The release files, including
signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC0/

Maven artifacts are available in a staging repository at:
https://repository.apache.org/content/repositories/orgapachehbase-1328/

Signatures used for HBase RCs can be found in this file:
https://dist.apache.org/repos/dist/release/hbase/KEYS

The list of bug fixes going into 2.2.1 can be found in included
CHANGES.md and RELEASENOTES.md available here:
https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC0/CHANGES.md
https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC0/RELEASENOTES.md

A detailed source and binary compatibility report for this release is
available at:
https://dist.apache.org/repos/dist/dev/hbase/2.2.1RC0/api_compare_2.2.1RC0_to_2.2.0.html

To learn more about Apache HBase, please see http://hbase.apache.org/

Thanks,
Guanghao Zhang

Re: [ANNOUNCE] new HBase committer Sakthi

[Guanghao Zhang]

Congratulations!

Allan Yang  于2019年8月1日周四 下午8:46写道：

> Congratulations Sakthi !
>
> Best Regards
> Allan Yang
>
>
> Nihal Jain  于2019年8月1日周四 下午8:40写道：
>
> > Congrats Sakthi. More power to you!
> >
> > On Thu, 1 Aug, 2019, 4:42 PM ramkrishna vasudevan, <
> > ramkrishna.s.vasude...@gmail.com> wrote:
> >
> > > Congratulations Sakthi !!!
> > >
> > > On Thu, Aug 1, 2019 at 3:34 PM 张铎(Duo Zhang) 
> > > wrote:
> > >
> > > > Congratulations!
> > > >
> > > > Pankaj kr  于2019年8月1日周四 下午5:56写道：
> > > >
> > > > > Congratulation Sakthi..!!
> > > > >
> > > > > Regards,
> > > > > Pankaj
> > > > >
> > > > > -Original Message-
> > > > > From: Sean Busbey [mailto:bus...@apache.org]
> > > > > Sent: 01 August 2019 05:35
> > > > > To: user@hbase.apache.org; dev 
> > > > > Subject: [ANNOUNCE] new HBase committer Sakthi
> > > > >
> > > > > On behalf of the HBase PMC, I'm pleased to announce that Sakthi has
> > > > > accepted our invitation to become an HBase committer.
> > > > >
> > > > > We'd like to thank Sakthi for all of his diligent contributions to
> > the
> > > > > project thus far. We look forward to his continued participation in
> > our
> > > > > community.
> > > > >
> > > > > Congrats and welcome Sakthi!
> > > > >
> > > >
> > >
> >
>

Re: [Announce] 张铎 (Duo Zhang) is Apache HBase PMC chair

[Guanghao Zhang]

Congratulations!

Duo Zhang  于2019年7月19日周五 上午9:33写道：

> Thanks Misty for the great job you have done these years.
>
> And thanks all for trusting me. Will try my best.
>
> Jan Hentschel  于2019年7月19日周五 上午3:55写道：
>
> > Congrats Duo!
> >
> > From: Andrew Purtell 
> > Reply-To: "user@hbase.apache.org" 
> > Date: Thursday, July 18, 2019 at 7:52 PM
> > To: Hbase-User 
> > Cc: HBase Dev List , Duo Zhang <
> zhang...@apache.org>,
> > "priv...@hbase.apache.org" 
> > Subject: Re: [Announce] 张铎 (Duo Zhang) is Apache HBase PMC chair
> >
> > Congratulations Duo!
> > Thank you for taking on the role of Chair.
> >
> > On Thu, Jul 18, 2019 at 10:46 AM Misty Linville   > mi...@apache.org>> wrote:
> >
> > Each Apache project has a project management committee (PMC) that
> oversees
> > governance of the project, votes on new committers and PMC members, and
> > ensures that the software we produce adheres to the standards of the
> > Foundation. One of the roles on the PMC is the PMC chair. The PMC chair
> > represents the project as a Vice President of the Foundation and
> > communicates to the board about the project's health, once per quarter
> and
> > at other times as needed.
> >
> > It's been my honor to serve as your PMC chair since 2017, when I took
> over
> > from Andrew Purtell. I've decided to step back from my volunteer ASF
> > activities to leave room in my life for other things. The HBase PMC
> > nominated Duo for this role, and Duo has kindly agreed! The board passed
> > this resolution in its meeting yesterday[1] and it is already
> official[2].
> > Congratulations, Duo, and thank you for continuing to honor the project
> > with your dedication.
> >
> > Misty
> >
> > [1] The minutes have not yet posted at the time of this email, but will
> be
> > available at http://www.apache.org/foundation/records/minutes/2019/.
> > [2] https://www.apache.org/foundation/#who-runs-the-asf
> >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> >- A23, Crosstalk
> >
> >
>

[ANNOUNCE] Apache HBase 2.2.0 is now available for download

[Guanghao Zhang]

The HBase team is happy to announce the immediate availability of HBase
2.2.0.

Apache HBase™ is an open-source, distributed, versioned, non-relational
database. Apache HBase gives you low latency random access to billions of
rows with
millions of columns atop non-specialized hardware. To learn more about
HBase,
see https://hbase.apache.org/.

To download 2.2.0, visit our download page:
http://hbase.apache.org/downloads.html

HBase 2.2.0 is the third minor release in the HBase 2.x line, which aims to
improve the stability and reliability of HBase. This release includes roughly
660+ resolved issues since 2.1.0.

Notable new features include:
- HBASE-20881 introduced a new TransitRegionStateProcedure to replace the
old AssignProcedure/UnassignProcedure/MoveRegionProcedure. This is an
incompatible change and different implemenation with 2.0 and 2.1. Please
see the release note of HBASE-21970 about how to upgrade from 2.0 or 2.1 to
2.2+.
- HBASE-22399 The default hadoop-two.version has been changed to 2.8.5, and
all hadoop versions before 2.8.2(exclude) will not be supported any more.
- HBASE-20610 introduced a new way to do WAL splitting coordination by
procedure v2 framework.

For instructions on verifying ASF release downloads, please see
https://www.apache.org/dyn/closer.cgi#verify

Project member signature keys can be found at

https://www.apache.org/dist/hbase/KEYS

The list of 660+ changes made since 2.1.0 can be found here:
https://apache.org/dist/hbase/2.2.0/CHANGES.md

The compatibility report vs 2.1.5:
https://apache.org/dist/hbase/2.2.0/api_compare_2.2.0RC6_to_2.1.5.html

Question, comments, and problems are always welcome at: d...@hbase.apache.org

Thanks to all who contributed and made this release possible.

Cheers,
The HBase Dev Team

[ANNOUNCE] New HBase committer Yi Mei

[Guanghao Zhang]

On behalf of the Apache HBase PMC, I am pleased to announce that Yi Mei has
accepted the PMC's invitation to become a committer on the project. We
appreciate all of Yi Mei's generous contributions thus far and look forward
to Yi Mei's continued involvement.

Congratulations and welcome, Yi Mei!

Re: [ANNOUNCE] Please welcome Jan Hentschel to the Apache HBase PMC

[Guanghao Zhang]

Congratulations

Jan Hentschel  于2019年5月13日周一 上午3:19写道：

> Thanks everybody. It’s an honor and I’ll try to do my best to help the
> project and the community.
>
> From: Balazs Meszaros 
> Reply-To: "d...@hbase.apache.org" 
> Date: Thursday, May 9, 2019 at 1:19 PM
> To: "d...@hbase.apache.org" 
> Cc: "user@hbase.apache.org" 
> Subject: Re: [ANNOUNCE] Please welcome Jan Hentschel to the Apache HBase
> PMC
>
> Congratulations Jan!
>
> On Thu, May 9, 2019 at 12:07 PM Lars Francke  > wrote:
>
> Congratulations Jan and especially thank you for your work on the
> deprecations
>
> On Wed, May 8, 2019 at 11:37 PM Sean Busbey  bus...@apache.org>> wrote:
>
> > On behalf of the Apache HBase PMC I am pleased to announce that Jan
> > Hentschel has accepted our invitation to become a PMC member on the
> > HBase project. We appreciate Jan stepping up to take more
> > responsibility in the HBase project.
> >
> > Please join me in welcoming Jan to the HBase PMC!
> >
> >
> >
> > As a reminder, if anyone would like to nominate another person as a
> > committer or PMC member, even if you are not currently a committer or
> > PMC member, you can always drop a note to priv...@hbase.apache.org
>  to
> > let us know.
> >
> > -busbey
> >
>
>
>

[VOTE] The third HBase 2.2.0 release candidate (RC3) is available

[Guanghao Zhang]

Please vote on this release candidate (RC) for Apache HBase 2.2.0.
This is the first release of the branch-2.2 line.

The VOTE will remain open for at least 72 hours.

[ ] +1 Release this package as Apache HBase 2.2.0
[ ] -1 Do not release this package because ...

The tag to be voted on is 2.2.0RC3. The release files, including
signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/hbase/2.2.0RC3/

Maven artifacts are available in a staging repository at:
https://repository.apache.org/content/repositories/orgapachehbase-1308

Signatures used for HBase RCs can be found in this file:
https://dist.apache.org/repos/dist/release/hbase/KEYS

The list of bug fixes going into 2.2.0 can be found in included
CHANGES.md and RELEASENOTES.md available here:
https://dist.apache.org/repos/dist/dev/hbase/2.2.0RC3/CHANGES.md
https://dist.apache.org/repos/dist/dev/hbase/2.2.0RC3/RELEASENOTES.md

A detailed source and binary compatibility report for this release is
available at
https://dist.apache.org/repos/dist/dev/hbase/2.2.0RC3/api_compare_2.2.0RC3_to_2.1.4.html

To learn more about Apache HBase, please see http://hbase.apache.org/

Thanks,
Guanghao Zhang

Re: [ANNOUNCE] Please welcome Peter Somogyi to the HBase PMC

[Guanghao Zhang]

Congratulations!

Yu Li  于2019年1月22日周二 上午10:48写道：

> Congratulations, Peter!
>
> Best Regards,
> Yu
>
>
> On Tue, 22 Jan 2019 at 10:38, Guangxu Cheng 
> wrote:
>
> > Congratulations Peter!
> >
> > -
> > Best Regards
> > Guangxu Cheng
> >
> > Allan Yang  于2019年1月22日周二 上午10:15写道：
> >
> > > Congratulations Peter!
> > > Best Regards
> > > Allan Yang
> > >
> > >
> > > Pankaj kr  于2019年1月22日周二 上午9:49写道：
> > >
> > > >
> > > > Congratulations Peter...!!!
> > > >
> > > > Regards,
> > > > Pankaj
> > > >
> > > > --
> > > > Pankaj Kumar
> > > > M: +91-9535197664(India Contact Number)
> > > > E: pankaj...@huawei.com
> > > > 2012实验室-班加罗尔研究所IT BU分部
> > > > 2012 Laboratories-IT BU Branch Dept.HTIPL
> > > > From:Duo Zhang 
> > > > To:HBase Dev List ;hbase-user <
> > > user@hbase.apache.org
> > > > >
> > > > Date:2019-01-22 07:06:43
> > > > Subject:[ANNOUNCE] Please welcome Peter Somogyi to the HBase PMC
> > > >
> > > > On behalf of the Apache HBase PMC I am pleased to announce that Peter
> > > > Somogyi
> > > > has accepted our invitation to become a PMC member on the Apache
> HBase
> > > > project.
> > > > We appreciate Peter stepping up to take more responsibility in the
> > HBase
> > > > project.
> > > >
> > > > Please join me in welcoming Peter to the HBase PMC!
> > > >
> > >
> >
>

Re: [DISCUSS] EOL branch-1.3

[Guanghao Zhang]

+1. But branch-1.2 is not EOL now?

张铎(Duo Zhang)  于2018年12月8日周六 上午9:28写道：

> +1.
>
> Andrew Purtell  于2018年12月8日周六 上午5:45写道：
>
> > I'm good with doing one more 1.3 release. It would be my pleasure to
> offer
> > that service to the community. I like RM-ing.
> >
> >
> > On Fri, Dec 7, 2018 at 12:29 PM Stack  wrote:
> >
> > > +1
> > >
> > > (Pity you have to make a release to EOL it).
> > >
> > > S
> > >
> > > On Fri, Dec 7, 2018 at 11:25 AM Andrew Purtell 
> > > wrote:
> > >
> > > > We haven't had a release from branch-1.3 for a long time and do not
> > > appear
> > > > to have an active RM for it. Unless a RM for 1.3 steps forward and
> > > promises
> > > > to make a release in the very near future, I propose we make one more
> > > > release of 1.3, from the head of branch-1.3, and then retire the
> > branch.
> > > If
> > > > this is acceptable I can RM the final 1.3 release.
> > > >
> > > > --
> > > > Best regards,
> > > > Andrew
> > > >
> > > > Words like orphans lost among the crosstalk, meaning torn from
> truth's
> > > > decrepit hands
> > > >- A23, Crosstalk
> > > >
> > >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> >- A23, Crosstalk
> >
>

Re: [ANNOUNCE] Asynchronous client is available in HBase 2.0

[Guanghao Zhang]

@stack, sorry for reply late, sir. As HBASE-16388
<https://issues.apache.org/jira/browse/HBASE-16388> said, the client
threads may be blocked by only one slow region server. But a better
solution for this problem is to use async hbase client. And there are many
reasons to make a region server slow, slow read from hdfs, gc stw, etc. So
this should be a advantage of the async client. PS: my colleague Zheng Hu
talked about this problem on today's HBaseCon Asia :-). About the
performance of async client, there are two important metrics: latency and
QPS. Our PE tool can be used to test latency and it will be addressed by
HBASE-17994 <https://issues.apache.org/jira/browse/HBASE-17994>. And YCSB
maybe better to test QPS. I will publish the test result when we get more
test data. Thanks.


2017-07-22 0:35 GMT+08:00 Andrew Purtell <apurt...@apache.org>:

> This is exciting!
> Thanks so much for this hard work.
>
> On Fri, Jul 21, 2017 at 2:21 AM, Phil Yang <ud1...@gmail.com> wrote:
>
> > I filed an issue HBASE-18334
> > <https://issues.apache.org/jira/browse/HBASE-18334> to remove the old
> sync
> > implementation. I think we can do it in 3.0 and keep sync logic in
> > branch-2? Of course we can also remove it in 2.1 or higher 2.x branch but
> > if we remove it since 2.0 it may be a little radical  :)
> >
> > Thanks,
> > Phil
> >
> >
> > 2017-07-21 6:15 GMT+08:00 Stack <st...@duboce.net>:
> >
> > > Very sweet Guanghao. Thanks for all the hard work (you, Duo and all who
> > > contributed to the async work).
> > >
> > > When you think we should move the sync client to be on top of the async
> > > chassis?
> > >
> > > Thanks,
> > > St.Ack
> > >
> > > P.S. It looks like you updated its status in this doc,
> > > https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> > > ktczrlKHK8N4SZzs/edit#,
> > > thanks.
> > >
> > > On Thu, Jul 20, 2017 at 4:51 AM, Guanghao Zhang <zghao...@gmail.com>
> > > wrote:
> > >
> > > > Dear all,
> > > >
> > > > Asynchronous client aims to provide the ability to access HBase
> > > > asynchronously. You can obtain an AsyncConnection from
> > ConnectionFactory,
> > > > and then get an asynchronous table instance (for DML operations) or
> an
> > > > asynchronous admin instance (for DDL operations) from it to access
> > HBase.
> > > > For the asynchronous table or admin, most methods have the same
> meaning
> > > > with the old Table or Admin interface, expect that the return value
> is
> > > > wrapped with a CompletableFuture (java 8 or higher is required)
> > usually.
> > > > You can get more details from the documentation[1][2].
> > > >
> > > > There are two examples about the asynchronous client.
> > > AsyncClientExample[3]
> > > > is a simple example to show you how to use AsyncTable.
> > > HttpProxyExample[4]
> > > > is an example for advance user to show you how to use RawAsyncTable
> to
> > > > write a fully asynchronous HTTP proxy server. Welcome to try the
> > > > asynchronous client in your project.
> > > >
> > > > We tested the performance of asynchronous client by
> > > > the PerformanceEvaluation tool. And it showed that async client has
> > > almost
> > > > same latency with sync client[5]. Notice: when we did performance
> test
> > > for
> > > > asynchronous client. We found some performance issue when use jdk
> > > 1.8_111.
> > > > And it was fixed when use jdk 1.8_131 [6]. So suggest use a bigger
> > > version
> > > > than jdk8_131 when you use the asynchronous client.
> > > >
> > > > The major umbrella issues include:
> > > > HBASE-16432 Revisit the asynchronous ipc implementation[7]
> > > > HBASE-16833 Implement asynchronous hbase client based on
> HBASE-15921[8]
> > > > HBASE-17359 Implement async admin[9]
> > > >
> > > > There is also an issue HBASE-17856 to track the further improvements
> > for
> > > > asynchronous hbase client[10].
> > > >
> > > > Finally, thanks Duo Zhang and Zheng Hu who worked with me together to
> > > > implement the asynchronous client.
> > > >
> > > > Best Regards,
> > > > Guanghao
> > > >
> > > > [1] https://hbase.apache.org/book.html#async.client
> > > > [2] https://hbase.apache.org/book.htm

Re: [ANNOUNCE] New HBase committer Mike Drob

[Guanghao Zhang]

Congratulations！

2017-08-04 5:34 GMT+08:00 Ted Yu :

> Congratulations, Mike.
>
> (Just got access to gmail).
>
> On Tue, Aug 1, 2017 at 8:38 AM, Josh Elser  wrote:
>
> > On behalf of the Apache HBase PMC, I'm pleased to announce that Mike Drob
> > has accepted the PMC's invitation to become a committer.
> >
> > Mike has been doing some great things lately in the project and this is a
> > simple way that we can express our thanks. As my boss likes to tell me:
> the
> > reward for a job well-done is more work to do! We're all looking forward
> to
> > your continued involvement :)
> >
> > Please join me in congratulating Mike!
> >
> > - Josh
> >
>

[ANNOUNCE] Asynchronous client is available in HBase 2.0

[Guanghao Zhang]

Dear all,

Asynchronous client aims to provide the ability to access HBase
asynchronously. You can obtain an AsyncConnection from ConnectionFactory,
and then get an asynchronous table instance (for DML operations) or an
asynchronous admin instance (for DDL operations) from it to access HBase.
For the asynchronous table or admin, most methods have the same meaning
with the old Table or Admin interface, expect that the return value is
wrapped with a CompletableFuture (java 8 or higher is required) usually.
You can get more details from the documentation[1][2].

There are two examples about the asynchronous client. AsyncClientExample[3]
is a simple example to show you how to use AsyncTable. HttpProxyExample[4]
is an example for advance user to show you how to use RawAsyncTable to
write a fully asynchronous HTTP proxy server. Welcome to try the
asynchronous client in your project.

We tested the performance of asynchronous client by
the PerformanceEvaluation tool. And it showed that async client has almost
same latency with sync client[5]. Notice: when we did performance test for
asynchronous client. We found some performance issue when use jdk 1.8_111.
And it was fixed when use jdk 1.8_131 [6]. So suggest use a bigger version
than jdk8_131 when you use the asynchronous client.

The major umbrella issues include:
HBASE-16432 Revisit the asynchronous ipc implementation[7]
HBASE-16833 Implement asynchronous hbase client based on HBASE-15921[8]
HBASE-17359 Implement async admin[9]

There is also an issue HBASE-17856 to track the further improvements for
asynchronous hbase client[10].

Finally, thanks Duo Zhang and Zheng Hu who worked with me together to
implement the asynchronous client.

Best Regards,
Guanghao

[1] https://hbase.apache.org/book.html#async.client
[2] https://hbase.apache.org/book.html#async.admin
[3]
https://github.com/apache/hbase/blob/master/hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/AsyncClientExample.java
[4]
https://github.com/apache/hbase/blob/master/hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java
[5] https://www.slideshare.net/HBaseCon/hbase-at-xiaomi
[6]
http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/java/util/concurrent/CompletableFuture.java
[7] https://issues.apache.org/jira/browse/HBASE-16432
[8] https://issues.apache.org/jira/browse/HBASE-16833
[9] https://issues.apache.org/jira/browse/HBASE-17359
[10] https://issues.apache.org/jira/browse/HBASE-17856

Re: Re: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC

[Guanghao Zhang]

Congratulations!

2017-07-04 16:21 GMT+08:00 Ted Yu :

> Congratulations, Chunhui.
>  Original message From: Allan Yang 
> Date: 7/3/17  11:00 PM  (GMT-08:00) To: user@hbase.apache.org Subject:
> Re: Re: [ANNOUNCE] Chunhui Shen joins the Apache HBase PMC
> Congratulations!
>
> Best Regards
> Allan Yang
>
> Best Regards
> Allan Yang
>
> 2017-07-04 13:43 GMT+08:00 Chunhui Shen :
>
> > Thanks all !
> > Will do my best for better HBase community.
> >
> >
> >
> >
> > Best Regards
> > Chunhui
> > 在 2017-07-04 13:31:27，"ramkrishna vasudevan" <
> > ramkrishna.s.vasude...@gmail.com> 写道：
> > >Congratulations !!
> > >
> > >On Tue, Jul 4, 2017 at 11:00 AM, 张铎(Duo Zhang) 
> > >wrote:
> > >
> > >> Congratulations!
> > >>
> > >> Yu Li 于2017年7月4日 周二13:24写道：
> > >>
> > >> > On behalf of the Apache HBase PMC I am pleased to announce that
> > Chunhui
> > >> > Shen
> > >> > has accepted our invitation to become a PMC member on the Apache
> > >> > HBase project. He has been an active contributor to HBase for past
> > many
> > >> > years. Looking forward for many more contributions from him.
> > >> >
> > >> > Please join me in welcoming Chunhui to the HBase PMC!
> > >> >
> > >> > Best Regards,
> > >> > Yu
> > >> >
> > >>
> >
>

Re: [ANNOUNCE] New Apache HBase committer Ashu Pachauri

[Guanghao Zhang]

Congratulations and welcome!

2017-06-17 7:27 GMT+08:00 Gary Helmling :

> On behalf of the Apache HBase PMC, I am pleased to announce that Ashu
> Pachauri has accepted the PMC's invitation to become a committer on the
> project.  We appreciate all of Ashu's generous contributions thus far and
> look forward to his continued involvement.
>
> Congratulations and welcome, Ashu!
>

Re: ANNOUNCE: Yu Li joins the Apache HBase PMC

[Guanghao Zhang]

Congratulations!

2017-04-16 21:36 GMT+08:00 Yu Li :

> Thanks all! My honor and will do my best.
>
> Best Regards,
> Yu
>
> On 16 April 2017 at 14:20, 张铎(Duo Zhang)  wrote:
>
> > Congratulations!
> >
> > 2017-04-16 11:24 GMT+08:00 Mikhail Antonov :
> >
> >> Congratulations Yu!
> >>
> >> -Mikhail
> >>
> >> On Sat, Apr 15, 2017 at 12:44 PM, Nick Dimiduk 
> >> wrote:
> >>
> >> > Congratulations Yu and thanks a lot! Keep up the good work!
> >> >
> >> > On Fri, Apr 14, 2017 at 7:22 AM Anoop John 
> >> wrote:
> >> >
> >> > > On behalf of the Apache HBase PMC I"m pleased to announce that Yu Li
> >> > > has accepted our invitation to become a PMC member on the Apache
> HBase
> >> > > project. He has been an active contributor to HBase for past many
> >> > > years. Looking forward for
> >> > > many more contributions from him.
> >> > >
> >> > > Welcome to the PMC, Yu Li...
> >> > >
> >> > >
> >> > > -Anoop-
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Thanks,
> >> Michael Antonov
> >>
> >
> >
>

Re: [ANNOUNCE] - Welcome our new HBase committer Anastasia Braginsky

[Guanghao Zhang]

Congratulations and welcome!

2017-03-28 8:05 GMT+08:00 Jerry He :

> Congrats and welcome!
>
> Jerry
>

Re: [ANNOUNCE] New HBase committer Guanghao Zhang

[Guanghao Zhang]

Thanks all. Looking forward to work with you guys and keep contributing for
HBase. Thanks.

2016-12-20 16:48 GMT+08:00 Yu Li <car...@gmail.com>:

> Congratulations and welcome Guanghao!
>
> Best Regards,
> Yu
>
> On 20 December 2016 at 12:59, 宾莉金 or binlijin <binli...@gmail.com> wrote:
>
> > Congratulations and welcome!
> >
> > 2016-12-20 12:54 GMT+08:00 Nick Dimiduk <ndimi...@gmail.com>:
> >
> > > Congratulations Guanghao and thank you for all your contributions!
> > >
> > > On Mon, Dec 19, 2016 at 5:37 PM Duo Zhang <zhang...@apache.org> wrote:
> > >
> > > > On behalf of the Apache HBase PMC, I am pleased to announce that
> > Guanghao
> > > >
> > > > Zhang has accepted the PMC's invitation to become a committer on the
> > > >
> > > > project. We appreciate all of Guanghao's generous contributions thus
> > far
> > > >
> > > > and look forward to his continued involvement.
> > > >
> > > >
> > > >
> > > > Congratulations and welcome, Guanghao!
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > *Best Regards,*
> >  lijin bin
> >
>

Re: How to limit a single row being filled with millions of columus?

[Guanghao Zhang]

Now the scan context has size limit. And scan can break between cells. This
should help for this. What is the version of your cluster?

2016-12-06 13:35 GMT+08:00 聪聪 <175998...@qq.com>:

> I am glad to receive your reply！How can I find a big row quickly? If not
> ,when we proceed major compact,the regionserver dose not work all the time .
>
>
>
>
>
>
>
> ------ 原始邮件 --
> 发件人: "Guanghao Zhang";<zghao...@gmail.com>;
> 发送时间: 2016年12月6日(星期二) 中午12:13
> 收件人: "user"<user@hbase.apache.org>;
>
> 主题: Re: How to limit a single row being filled with millions of columus?
>
>
>
> There are a config hbase.table.max.rowsize but only used for user get/scan.
> It will throw RowTooBigException when you scan a big row with millions of
> columns. But it doesn't work for compact. If i am not wrong, there are not
> a way to prevent a single row being filled with millions of columns.
>
> 2016-12-06 11:52 GMT+08:00 聪聪 <175998...@qq.com>:
>
> > Recently, I have a problem that confused me a long time. The problem is
> > that as we all know in HBase,there are millions of columns in a single
> row.
> > Full gc will happen when region proceeds major compact and it results in
> > regionserver and hbase not working. Is there any good way to prevent a
> > single row being put(wrote) columns overmuch?
> > Hope to you soon!
>

Re: How to limit a single row being filled with millions of columus?

[Guanghao Zhang]

There are a config hbase.table.max.rowsize but only used for user get/scan.
It will throw RowTooBigException when you scan a big row with millions of
columns. But it doesn't work for compact. If i am not wrong, there are not
a way to prevent a single row being filled with millions of columns.

2016-12-06 11:52 GMT+08:00 聪聪 <175998...@qq.com>:

> Recently, I have a problem that confused me a long time. The problem is
> that as we all know in HBase,there are millions of columns in a single row.
> Full gc will happen when region proceeds major compact and it results in
> regionserver and hbase not working. Is there any good way to prevent a
> single row being put(wrote) columns overmuch?
> Hope to you soon!

Re: HBase Replication - Specific Tables To Different Clusters

[Guanghao Zhang]

HBase replication supported both of them. You can use add_peer or
set_peer_tableCFs to config specific tables for specific peers.

Re: Hbase throttling issues

[Guanghao Zhang]

Did you set any other quotas? The exception
ThrottlingException.throwReadSizeExceeded
shows that the throttle type is READ_SIZE, but your quota's throttle type
is REQUEST_SIZE. You can use list_quota shell cmd to check whether there
are READ_SIZE quota.

2016-09-22 12:06 GMT+08:00 Sumit Nigam <sumit_o...@yahoo.com.invalid>:

> No, did not try request number. I want to use size as my throttling
> factor.
>
> Thanks again!
>
>   From: Guanghao Zhang <zghao...@gmail.com>
>  To: user@hbase.apache.org; Sumit Nigam <sumit_o...@yahoo.com>
>  Sent: Thursday, September 22, 2016 9:28 AM
>  Subject: Re: Hbase throttling issues
>
> Did you try to use REQUEST_NUMBER as throttle type?
>
> 2016-09-22 11:33 GMT+08:00 Sumit Nigam <sumit_o...@yahoo.com.invalid>:
>
> > Hi Guanghao,
> >
> > This throttling error shows up as soon as I start hbase. So, ideally
> there
> > shouldn't be too many prior operations in play here. Plus, the error
> shows
> > up even when my table has hardly any data (possibly in kbs) and I have
> set
> > throttling limit to be ~10GB.
> >
> > Thanks,Sumit
> >
> >  From: Guanghao Zhang <zghao...@gmail.com>
> >  To: user@hbase.apache.org; Sumit Nigam <sumit_o...@yahoo.com>
> >  Sent: Thursday, September 22, 2016 7:37 AM
> >  Subject: Re: Hbase throttling issues
> >
> > All scan operations were throttled? Now it use the avg size of all
> previous
> > operations to check quota. Maybe the previous scan operation read too
> much
> > data.
> >
> > 2016-09-22 1:18 GMT+08:00 Sumit Nigam <sumit_o...@yahoo.com.invalid>:
> >
> > > Hi Hbase experts,
> > >
> > > Our application is unable to scan or read from hbase tables, when
> > > throttling is set. We are getting ThrottlingException every time. The
> > error
> > > is seen more frequently when the number of hbase pre splits is
> > > increased. The size tables for which this error is showing is empty (
> > > during some runs it was very low in the order of few kbs). Have tried
> > both
> > > rate limiters already - average and fixed. Can't understand why read
> rate
> > > limit exceeds when there is hardly any data in hbase. Anyone has faced
> > this
> > > issue before?
> > >
> > > Setup Details:
> > >
> > > Hbase version : 1.1.2
> > > Number of region servers :4
> > > Number of regions : 116
> > > HeapMemory for Region Server : 2GB
> > >
> > > Quotas set :
> > > TABLE => ns1:table1 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE,
> > LIMIT
> > > => 10G/sec, SCOPE => MACHINE
> > > TABLE => ns2:table2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE,
> > LIMIT
> > > => 10G/sec, SCOPE => MACHINE
> > >
> > > Following is the error we faced. Pasting debug logs of regions server:
> > >
> > > 2016-09-17 22:35:40,674 DEBUG [B.defaultRpcServer.handler=
> > 55,queue=1,port=58526]
> > > quotas.RegionServerQuotaManager: Throttling exception for user=root
> > > table=ns1:table1 numWrites=0 numReads=0 numScans=1: read size limit
> > > exceeded - wait 0.00sec
> > >
> > > 2016-09-17 22:35:40,676 DEBUG [B.defaultRpcServer.handler=
> > 55,queue=1,port=58526]
> > > ipc.RpcServer: B.defaultRpcServer.handler=55,queue=1,port=58526:
> callId:
> > > 52 service: ClientService methodName: Scan size: 28 connection:
> > > 10.65.141.170:42806
> > >
> > > org.apache.hadoop.hbase.quotas.ThrottlingException: read size limit
> > > exceeded - wait 0.00sec
> > >
> > > at org.apache.hadoop.hbase.quotas.ThrottlingException.
> > > throwThrottlingException(ThrottlingException.java:107)
> > >
> > > at org.apache.hadoop.hbase.quotas.ThrottlingException.
> > > throwReadSizeExceeded(ThrottlingException.java:101)
> > >
> > > at org.apache.hadoop.hbase.quotas.TimeBasedLimiter.
> > > checkQuota(TimeBasedLimiter.java:139)
> > >
> > > at org.apache.hadoop.hbase.quotas.DefaultOperationQuota.checkQuota(
> > > DefaultOperationQuota.java:59)
> > >
> > > at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota(
> > > RegionServerQuotaManager.java:180)
> > >
> > > at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota(
> > > RegionServerQuotaManager.java:125)
> > >
> > > at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> > > scan(RSRpcServices.java:2265)
> > >
> > > at org.apache.hadoop.hbase.protobuf.generated.
> > > ClientProtos$ClientService$2.callBlockingMethod(
> ClientProtos.java:32205)
> > >
> > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> > >
> > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> > >
> > > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
> > > RpcExecutor.java:130)
> > >
> > > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> > >
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > >
> > > Thanks,
> > >
> > > Sumit
> >
> >
> >
> >
>
>
>
>

Re: Hbase throttling issues

[Guanghao Zhang]

Did you try to use REQUEST_NUMBER as throttle type?

2016-09-22 11:33 GMT+08:00 Sumit Nigam <sumit_o...@yahoo.com.invalid>:

> Hi Guanghao,
>
> This throttling error shows up as soon as I start hbase. So, ideally there
> shouldn't be too many prior operations in play here. Plus, the error shows
> up even when my table has hardly any data (possibly in kbs) and I have set
> throttling limit to be ~10GB.
>
> Thanks,Sumit
>
>   From: Guanghao Zhang <zghao...@gmail.com>
>  To: user@hbase.apache.org; Sumit Nigam <sumit_o...@yahoo.com>
>  Sent: Thursday, September 22, 2016 7:37 AM
>  Subject: Re: Hbase throttling issues
>
> All scan operations were throttled? Now it use the avg size of all previous
> operations to check quota. Maybe the previous scan operation read too much
> data.
>
> 2016-09-22 1:18 GMT+08:00 Sumit Nigam <sumit_o...@yahoo.com.invalid>:
>
> > Hi Hbase experts,
> >
> > Our application is unable to scan or read from hbase tables, when
> > throttling is set. We are getting ThrottlingException every time. The
> error
> > is seen more frequently when the number of hbase pre splits is
> > increased. The size tables for which this error is showing is empty (
> > during some runs it was very low in the order of few kbs). Have tried
> both
> > rate limiters already - average and fixed. Can't understand why read rate
> > limit exceeds when there is hardly any data in hbase. Anyone has faced
> this
> > issue before?
> >
> > Setup Details:
> >
> > Hbase version : 1.1.2
> > Number of region servers :4
> > Number of regions : 116
> > HeapMemory for Region Server : 2GB
> >
> > Quotas set :
> > TABLE => ns1:table1 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE,
> LIMIT
> > => 10G/sec, SCOPE => MACHINE
> > TABLE => ns2:table2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE,
> LIMIT
> > => 10G/sec, SCOPE => MACHINE
> >
> > Following is the error we faced. Pasting debug logs of regions server:
> >
> > 2016-09-17 22:35:40,674 DEBUG [B.defaultRpcServer.handler=
> 55,queue=1,port=58526]
> > quotas.RegionServerQuotaManager: Throttling exception for user=root
> > table=ns1:table1 numWrites=0 numReads=0 numScans=1: read size limit
> > exceeded - wait 0.00sec
> >
> > 2016-09-17 22:35:40,676 DEBUG [B.defaultRpcServer.handler=
> 55,queue=1,port=58526]
> > ipc.RpcServer: B.defaultRpcServer.handler=55,queue=1,port=58526: callId:
> > 52 service: ClientService methodName: Scan size: 28 connection:
> > 10.65.141.170:42806
> >
> > org.apache.hadoop.hbase.quotas.ThrottlingException: read size limit
> > exceeded - wait 0.00sec
> >
> > at org.apache.hadoop.hbase.quotas.ThrottlingException.
> > throwThrottlingException(ThrottlingException.java:107)
> >
> > at org.apache.hadoop.hbase.quotas.ThrottlingException.
> > throwReadSizeExceeded(ThrottlingException.java:101)
> >
> > at org.apache.hadoop.hbase.quotas.TimeBasedLimiter.
> > checkQuota(TimeBasedLimiter.java:139)
> >
> > at org.apache.hadoop.hbase.quotas.DefaultOperationQuota.checkQuota(
> > DefaultOperationQuota.java:59)
> >
> > at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota(
> > RegionServerQuotaManager.java:180)
> >
> > at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota(
> > RegionServerQuotaManager.java:125)
> >
> > at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> > scan(RSRpcServices.java:2265)
> >
> > at org.apache.hadoop.hbase.protobuf.generated.
> > ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
> >
> > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> >
> > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> >
> > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
> > RpcExecutor.java:130)
> >
> > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> >
> > at java.lang.Thread.run(Thread.java:745)
> >
> >
> > Thanks,
> >
> > Sumit
>
>
>
>

Re: Hbase throttling issues

[Guanghao Zhang]

All scan operations were throttled? Now it use the avg size of all previous
operations to check quota. Maybe the previous scan operation read too much
data.

2016-09-22 1:18 GMT+08:00 Sumit Nigam :

> Hi Hbase experts,
>
> Our application is unable to scan or read from hbase tables, when
> throttling is set. We are getting ThrottlingException every time. The error
> is seen more frequently when the number of hbase pre splits is
> increased. The size tables for which this error is showing is empty (
> during some runs it was very low in the order of few kbs). Have tried both
> rate limiters already - average and fixed. Can't understand why read rate
> limit exceeds when there is hardly any data in hbase. Anyone has faced this
> issue before?
>
> Setup Details:
>
> Hbase version : 1.1.2
> Number of region servers :4
> Number of regions : 116
> HeapMemory for Region Server : 2GB
>
> Quotas set :
> TABLE => ns1:table1 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT
> => 10G/sec, SCOPE => MACHINE
> TABLE => ns2:table2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT
> => 10G/sec, SCOPE => MACHINE
>
> Following is the error we faced. Pasting debug logs of regions server:
>
> 2016-09-17 22:35:40,674 DEBUG 
> [B.defaultRpcServer.handler=55,queue=1,port=58526]
> quotas.RegionServerQuotaManager: Throttling exception for user=root
> table=ns1:table1 numWrites=0 numReads=0 numScans=1: read size limit
> exceeded - wait 0.00sec
>
> 2016-09-17 22:35:40,676 DEBUG 
> [B.defaultRpcServer.handler=55,queue=1,port=58526]
> ipc.RpcServer: B.defaultRpcServer.handler=55,queue=1,port=58526: callId:
> 52 service: ClientService methodName: Scan size: 28 connection:
> 10.65.141.170:42806
>
> org.apache.hadoop.hbase.quotas.ThrottlingException: read size limit
> exceeded - wait 0.00sec
>
> at org.apache.hadoop.hbase.quotas.ThrottlingException.
> throwThrottlingException(ThrottlingException.java:107)
>
> at org.apache.hadoop.hbase.quotas.ThrottlingException.
> throwReadSizeExceeded(ThrottlingException.java:101)
>
> at org.apache.hadoop.hbase.quotas.TimeBasedLimiter.
> checkQuota(TimeBasedLimiter.java:139)
>
> at org.apache.hadoop.hbase.quotas.DefaultOperationQuota.checkQuota(
> DefaultOperationQuota.java:59)
>
> at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota(
> RegionServerQuotaManager.java:180)
>
> at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota(
> RegionServerQuotaManager.java:125)
>
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> scan(RSRpcServices.java:2265)
>
> at org.apache.hadoop.hbase.protobuf.generated.
> ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
>
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
>
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
>
> at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
> RpcExecutor.java:130)
>
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
> Thanks,
>
> Sumit

Re: Major compaction cannot remove deleted rows until the region is split. Strange!

[Guanghao Zhang]

We add some log in our production cluster.
2016-06-06,21:37:28,443 INFO org.apache.hadoop.hbase.regionserver.HRegion:
scannerReadPoints size is 0 and smallest read point is 4037995
2016-06-06,21:37:51,429 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Failed
openScannerjava.io.IOException: Could not seek StoreFileScanner
2016-06-06,21:38:02,103 INFO org.apache.hadoop.hbase.regionserver.HRegion:
scannerReadPoints size is 1 and smallest read point is 4038037
2016-06-13,17:17:10,516 INFO org.apache.hadoop.hbase.regionserver.HRegion:
scannerReadPoints size is 1 and smallest read point is 4038037

When new RegionScanner, it will add a scanner read point in
scannerReadPoints. But if we got a exception after add read point, the read
point will keep in regions server and the delete after this mvcc number
will never be compacted.
Create a issue about this. https://issues.apache.org/jira/browse/HBASE-16012

2016-06-10 3:08 GMT+08:00 Stack :

> Tian-ying:
>
> Can you try the suggestion below to see if it helps?
>
> Do you see the lease expired logs?
>
> St.Ack
>
>
> On Thu, Jun 2, 2016 at 7:03 PM, hao zhang  wrote:
>
> > Hi, Stack
> >
> >   We found this in our production cluster, too.  I take a look about the
> > code and found one case which will make the major compact not work.
> >
> >   1. put some rows
> >   2. scan
> >   3. delete
> >   4. scanner didn't close rightly, it will keep the  the read point in
> > region's scannerReadPoints.
> >   5. major compact. It can't work because we have a scanner which have
> > small read point than the delete.
> >
> >   But if move the region to new rs, the scannerReadPoints will update to
> > the biggest memstoreTs which form sotrefile. So major compact will work.
> >   I thought the try...catch module in Scanner.leaseExpired() method
> needs a
> > finally module to close the region scanner rightly.
> >
> > public void leaseExpired() {
> >   RegionScannerHolder rsh = scanners.remove(this.scannerName);
> >   if (rsh != null) {
> > RegionScanner s = rsh.s;
> > LOG.info("Scanner " + this.scannerName + " lease expired on
> region
> > "
> >   + s.getRegionInfo().getRegionNameAsString());
> > try {
> >   Region region =
> > regionServer.getRegion(s.getRegionInfo().getRegionName());
> >   if (region != null && region.getCoprocessorHost() != null) {
> > region.getCoprocessorHost().preScannerClose(s);
> >   }
> >   s.close();
> >   if (region != null && region.getCoprocessorHost() != null) {
> > region.getCoprocessorHost().postScannerClose(s);
> >   }
> > } catch (IOException e) {
> >   LOG.error("Closing scanner for "
> > + s.getRegionInfo().getRegionNameAsString(), e);
> > }
> >   } else {
> > LOG.warn("Scanner " + this.scannerName + " lease expired, but no
> > related" +
> >   " scanner found, hence no chance to close that related
> > scanner!");
> >   }
> > }
> >
> >
> > 2016-06-02 2:50 GMT+08:00 Stack :
> >
> > > On Wed, Jun 1, 2016 at 10:56 AM, Tianying Chang 
> > wrote:
> > >
> > > > Hi, Stack
> > > >
> > > > After moving the region and issue a major compact on that region, its
> > > size
> > > > shrink from 99G down to 24G. So it looks like the region is in a bad
> > > state
> > > > that cannot recover, close/open it fixed the issue. And from the
> region
> > > > size metric graph, we can see major compaction stop working  since
> > March
> > > > 31, so some bug that caused region enter into bad state...
> > Unfortunately,
> > > > we don't have DEBUG enabled and that is the last region that has the
> > > issue,
> > > > it is hard to figure out what is the bug that caused the bad state...
> > > >
> > > >
> > > Interesting. So moving it to another RS make it major-compactable? That
> > > would seem to indicate some state kept in the RS memory is preventing
> the
> > > major compaction running. Is moving the region a workaround for you
> until
> > > we figure what it is Tian-Ying?
> > >
> > > St.
> > >
> > >
> > >
> > > > Thanks
> > > > Tian-Ying
> > > >
> > > > On Tue, May 31, 2016 at 3:43 PM, Tianying Chang 
> > > wrote:
> > > >
> > > > > Hi, Stack
> > > > >
> > > > > Based on the log, the major compaction was run, and it took 5+
> hours.
> > > > And
> > > > > I also manually run major_compact from hbase shell explicitly to
> > > verify.
> > > > >
> > > > > I just moved the region to a different RS and issued a
> major_compact
> > on
> > > > > that region again, let me see if the major compaction can succeed
> and
> > > > will
> > > > > report back.
> > > > >
> > > > > Thanks
> > > > > Tian-Ying
> > > > >
> > > > > On Sun, May 29, 2016 at 4:35 PM, Stack  wrote:
> > > > >
> > > > >> On Fri, May 27, 2016 at 3:17 PM, Tianying Chang <
> tych...@gmail.com>
> > > > >> wrote:
> > > >