Re: [DISCUSS] Removing problematic terms from our project

[Xu Cang]

Strongly agree with what Nick said here:

 " From my perspective, we gain nothing as a project or as a community be
willfully retaining use of language that is well understood to be
problematic or hurtful, On the contrary, we have much to gain by
encouraging
contributions from as many people as possible."

+1 to Andrew's proposal.

It might be good to have a source of truth web page or README file for
developers and users to refer to regarding all naming transitions. It's
going to help both developers changing the code and users looking for some
answers online that use old namings.

Xu

On Wed, Jun 24, 2020 at 2:21 PM Nick Dimiduk  wrote:

> On Tue, Jun 23, 2020 at 13:11 Sean Busbey  wrote:
>
> > I would like to make sure I am emphatically clear that "master" by itself
> > is not okay if the context is the same as what would normally be a
> > master/slave context. Furthermore our use of master is clearly such a
> > context.
>
>
> I agree: to me “Master”, as in “HMaster” caries with it the master/slave
> baggage. As an alternative, I prefer the term “coordinator” over “leader”.
> Thus we would have daemons called “coordinator” and “region server”.
>
> To me, “master” as in “master branch” does not carry the same baggage, but
> I’m also in favor changing the name of our default branch to a word that is
> less conflicted. I see nothing that we gain as a community by continuing to
> use this word.
>
> It seems to me we have, broadly speaking, consensus around making *some*
> > changes. I haven't seen a strong push for "break everything in the name
> of
> > expediency" (I would personally be fine with this). So barring additional
> > discussion that favors breaking changes, current approaches should
> comport
> > with our existing project compatibility goals.
> >
> > Maybe we could stop talking about what-ifs and look at actual practical
> > examples? If anyone is currently up for doing the work of a PR we can
> look
> > at for one of these?
> >
> > If folks would prefer we e.g. just say "we should break whatever we need
> to
> > in 3.0.0 to make this happen" then it would be good to speak up.
> Otherwise
> > likely we would be done with needed changes circa hbase 4, probably late
> > 2021 or 2022.
> >
> >
> > On Tue, Jun 23, 2020, 03:03 zheng wang <18031...@qq.com> wrote:
> >
> > > IMO, master is ok if not used with slave together.
> > >
> > >
> > > -1/+1/+1/+1
> > >
> > >
> > > --原始邮件--
> > > 发件人:"Andrew Purtell" > > 发送时间:2020年6月23日(星期二) 凌晨5:24
> > > 收件人:"Hbase-User" > > 抄送:"dev" > > 主题:Re: [DISCUSS] Removing problematic terms from our project
> > >
> > >
> > >
> > > In observing something like voting happening on this thread to express
> > > alignment or not, it might be helpful to first, come up with a list of
> > > terms to change (if any), and then propose replacements, individually.
> So
> > > far we might break this apart into four proposals:
> > >
> > > 1. Replace "master"/"hmaster" with ??? ("coordinator" is one option),
> > this
> > > one has by far the most significant impact and both opinion and
> > > interpretation on this one is mixed.
> > >
> > > 2. Replace "slave" with "follower", seems to impact the cross cluster
> > > replication subsystem only.
> > >
> > > 3. Replace "black list" with "deny list".
> > >
> > > 4. Replace "white list" with "accept list".
> > >
> > > Perhaps if you are inclined to respond with a +1/-1/+0/-0, it would be
> > > useful to give such an indication for each line item above. Or, offer
> > > alternative proposals. Or, if you have a singular opinion, that's fine
> > too.
> > >
> > >
> > >
> > > On Mon, Jun 22, 2020 at 2:09 PM Geoffrey Jacoby  
> > > wrote:
> > >
> > >  For most of the proposals (slave - worker, blacklist -
> > > denylist,
> > >  whitelist- allowlist), I'm +1 (nonbinding). Denylist and
> > > acceptlist even
> > >  have the advantage of being clearer than the terms they're
> > replacing.
> > > 
> > >  However, I'm not convinced about changing "master" to
> "coordinator",
> > > or
> > >  something similar. Unlike "slave", which is negative in any
> context,
> > >  "master" has many definitions, including some common ones which do
> > not
> > >  appear problematic. See
> > > https://www.merriam-webster.com/dictionary/master
> > >  ; for
> > >  examples. In particular, the progression of an artisan was from
> > >  "apprentice" to "journeyman" to "master". A master smith,
> carpenter,
> > > or
> > >  artist would run a shop managing lots of workers and apprentices
> who
> > > would
> > >  hope to become masters of their own someday. So "master" and
> > "worker"
> > > can
> > >  still go together.
> > > 
> > >  Since it's the least problematic term, and by far the hardest term
> > to
> > >  change (both within HBase and with effects on downstream projects
> > > such as
> > >  Ambari), I'm -0 (nonbinding) on changing "master".
> > > 
> > >  Geoffrey
> > > 
> > >  On Mon, Jun 

[jira] [Created] (HBASE-24028) MapReduce on snapshot restores and opens all regions in each mapper

[Xu Cang (Jira)]

Xu Cang created HBASE-24028:
---

 Summary: MapReduce on snapshot restores and opens all regions in 
each mapper
 Key: HBASE-24028
 URL: https://issues.apache.org/jira/browse/HBASE-24028
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.6.0, 2.3.0
Reporter: Xu Cang


Given this scenario: one MR job scans a table (with many regions). I will use 
'RestoreSnapshotHelper' to restore snapshot for all regions in each mapper. 

In the code 
[https://github.com/apache/hbase/blob/branch-2.0/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/RestoreSnapshotHelper.java#L183]

Seems there is no way to only restore relevant regions from snapshot to region.

This leads to extreme slowness and waste of resource. 

Please correct me if I am wrong or miss anything. thanks.

 

One quick example I san show as below, in my test, there are 2 regions in a 
testing table. and each mapper opens and iterates 2 regions. 

2020-03-19 18:58:15,225 INFO [main] mapred.MapTask - Map output collector class 
= org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2020-03-19 18:58:15,285 INFO [main] snapshot.RestoreSnapshotHelper - region to 
add: *d7f85b4a9d3fa22a5e7b88bda39f6d50*
2020-03-19 18:58:15,285 INFO [main] snapshot.RestoreSnapshotHelper - region to 
add: *69dd3fdba3698f827f8883ed911161ef*
2020-03-19 18:58:15,286 INFO [main] snapshot.RestoreSnapshotHelper - clone 
region=d7f85b4a9d3fa22a5e7b88bda39f6d50 as d7f85b4a9d3fa22a5e7b88bda39f6d50

 

So if I misunderstood anything, can anyone point to me where in this class, can 
distinguish which region to go through for different mappers? 

 

btw the original implementation for MR on Snapshot is here, there weren't too 
many big changes after that HBASE-8369 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] First Release Candidate for HBase 1.6.0 is available for download

[Xu Cang]

+1 (Non-binding)

* Signature: pass
* Checksum : pass
* Rat check (1.8.0_152-b16): pass
 - mvn clean apache-rat:check
* Built from source (1.8.0_152-b16): pass
 - mvn clean install -DskipTests
* Unit tests pass (1.8.0_152-b16): pass
 * Tested basic WebUI: pass


Xu

On Fri, Feb 28, 2020 at 1:23 AM Peter Somogyi  wrote:
>
> +1 (binding)
>
> Signature, checksum: ok
> Rat check: ok
> Build from source: ok
> Unit tests: ok
> LTT 1M rows: ok
> Basic shell commands: ok
>
> On Fri, Feb 28, 2020 at 12:15 AM Jan Hentschel <
> jan.hentsc...@ultratendency.com> wrote:
>
> > +1 (binding)
> >
> > * Signature: ok
> > * Checksum : ok
> > * Rat check (1.8.0_202-ea): ok
> >  - mvn clean apache-rat:check
> > * Built from source (1.8.0_202-ea): ok
> >  - mvn clean install -DskipTests
> > * Unit tests pass (1.8.0_202-ea): ok
> >  - mvn package -P runSmallTests
> >
> > From: Andrew Purtell 
> > Reply-To: "dev@hbase.apache.org" 
> > Date: Monday, February 24, 2020 at 6:05 PM
> > To: dev 
> > Subject: [VOTE] First Release Candidate for HBase 1.6.0 is available for
> > download
> >
> > Please vote on this release candidate (RC0) for Apache HBase 1.6.0.
> >
> > The VOTE will remain open for at least 72 hours.
> >
> > The tag to be voted on is '1.6.0RC0' (5ec5a5b115), available for download
> > at
> > https://dist.apache.org/repos/dist/dev/hbase/1.6.0RC0/
> >
> > Maven artifacts are available in a staging repository at
> > https://repository.apache.org/content/repositories/orgapachehbase-1383
> >
> > A list of the 73 issues resolved in this release can be found at
> > https://s.apache.org/b7eml
> >
> > A list of exclude patters for the known flaky unit tests can be found at:
> >
> > https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-1/
> >
> > A detailed source and binary compatibility report for this release is
> > available at
> >
> > https://dist.apache.org/repos/dist/dev/hbase/1.6.0RC0/api_compare_1.5.0_to_1.6.0RC0.html
> > .
> >
> > There are no binary or source compatibility issues reported.
> >
> > Prior to making this announcement I made the following preflight checks:
> >
> > RAT check passes (7u80)
> > Unit test suite passes (7u80, 8u232)
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> >

Re: [VOTE] First Release Candidate for HBase 1.4.13 is available for download

[Xu Cang]

+1 (non-binding)

* Signature: pass
* Checksum : pass
* Rat check (1.8.0_152-b16): pass
 - mvn clean apache-rat:check
* Built from source (1.8.0_152-b16): pass
 - mvn clean install -DskipTests
* Unit tests pass (1.8.0_152-b16): pass
 - mvn package -P runSmallTests
 * Tested basic WebUI: pass


Xu

On Thu, Feb 27, 2020 at 2:07 PM Jan Hentschel
 wrote:
>
> +1 (binding)
>
> * Signature: ok
> * Checksum : ok
> * Rat check (1.8.0_202-ea): ok
>  - mvn clean apache-rat:check
> * Built from source (1.8.0_202-ea): ok
>  - mvn clean install -DskipTests
> * Unit tests pass (1.8.0_202-ea): ok
>  - mvn package -P runSmallTests
>
> Also checked the compatibility report and the release notes.
>
> From: Sakthi 
> Reply-To: "dev@hbase.apache.org" 
> Date: Monday, February 24, 2020 at 12:20 PM
> To: HBase Dev List 
> Subject: [VOTE] First Release Candidate for HBase 1.4.13 is available for 
> download
>
> Please vote on this release candidate (RC0) for Apache HBase 1.4.13.
>
> The VOTE will remain open for at least 72 hours.
>
> [ ] +1 Release this package as Apache HBase 1.4.13
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is '1.4.13RC0' (38bf65a
> 
> ):
> https://github.com/apache/hbase/tree/1.4.13RC0
>
> It's available for download at:
> https://dist.apache.org/repos/dist/dev/hbase/1.4.13RC0/
>
> Maven artifacts are available in a staging repository at:
> https://repository.apache.org/content/repositories/orgapachehbase-1384/
>
> Artifacts are signed with my key (851528A6) published in our KEYS file at:
> https://dist.apache.org/repos/dist/release/hbase/KEYS
>
> A list of the 13 issues resolved in this release can be found at:
> https://s.apache.org/s8i5w
>
> A list of exclude patters for the known flakey tests can be found at:
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-1.4/
>
> A detailed source and binary compatibility report for this release is
> available at:
> https://dist.apache.org/repos/dist/dev/hbase/1.4.13RC0/compat-check-report.html
>
>
> *PREFLIGHT CHECKS DONE:*
> Java Version - Zulu 1.7.0_232-b6
>
> Fully-Distributed Mode (3 Nodes)
>
> Hadoop Version - 2.7.7
>
>
>
>-
>
>Built bin tarball tarball from Source
>-
>
>Checksums & Signatures - OK
>-
>
>RAT check - OK
>-
>
>ITBLL (8M) rows (& MR Tasks using Resource Manager [Verified]) - OK
>-
>
>LTT (5M) rows - OK
>-
>
>Shell [CRUD Operations] - OK
>-
>
>Web UI, Logs - OK
>-
>
>REST Server - OK
>-
>
>   CRUD Operations, Web UI, Logs - OK
>   -
>
>Thrift Server - OK
>-
>
>   CRUD Operations, Web UI, Logs - OK
>   -
>
>Compatibility Report - OK
>-
>
>HDFS directory structure - OK
>
> Java Version - Corretto 1.8.0_222-b10
> Standalone mode
>
>- Built bin tarball from Source
>-
>
>Shell [CRUD Operations] - OK
>-
>
>Web UI, Logs - OK
>
> To learn more about Apache HBase, please see http://hbase.apache.org/
>
> Thanks,
> Sakthi
>

Re: [ANNOUNCE] New HBase committer Ankit Singhal

[Xu Cang]

Congratulations Ankit, welcome!

Xu

On Tue, Nov 12, 2019 at 10:36 AM Andrew Purtell  wrote:

> Congratulations and welcome Ankit
>
> On Tue, Nov 12, 2019 at 8:39 AM Josh Elser  wrote:
>
> > On behalf of the Apache HBase PMC, I'm pleased to announce that Ankit
> > Singhal has accepted our invitation to become an HBase committer.
> >
> > Thanks for all of your contributions to the HBase project and we look
> > forward to your continued growth and participation.
> >
> > Congratulations!
> >
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

Jenkins Precommit job issue

[Xu Cang]

Hi All,
Does anyone have some insights about Jenkins Precommit job issue here:
https://builds.apache.org/job/PreCommit-HBASE-Build/992/console ?
For last several runs, they all failed here after some gpg operations.


*10:31:20* curl: (22) The requested URL returned error: 404 Not Found


Thanks,

Xu

[jira] [Created] (HBASE-23143) Region Server Crash due to 2 cells out of order ( between 2 DELETEs)

[Xu Cang (Jira)]

Xu Cang created HBASE-23143:
---

 Summary: Region Server Crash due to 2 cells out of order ( between 
2 DELETEs)
 Key: HBASE-23143
 URL: https://issues.apache.org/jira/browse/HBASE-23143
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.3.2
Reporter: Xu Cang


Region Server Crash due to 2 cells out of order ( between 2 DELETEs)

 

Caused by: java.io.IOException: Added a key not lexically larger than previous.
 Current cell = 
00D7F00xxQ10D52v8UY6yV0057F00bPaGT\x00057F00bPaG/0:TABLE1_ID/*1570095189597*/DeleteColumn/vlen=0/seqid=*2128373*,
 
 lastCell = 
00D7F00xxQ10D52v8UY6yV0057F00bPaGT\x00057F00bPaG/0:TABLE1_ID/*1570095165147*/DeleteColumn/vlen=0/seqid=*2128378*

 

 

I am aware https://issues.apache.org/jira/browse/HBASE-22862

but it's slightly different, this issue is not caused by One Delete and One Put.

This issue I am seeing is caused by 2 Deletes

 

Has anyone seen this issue? 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] The second HBase Operator Tools 1.0.0 release candidate (RC1) is available

[Xu Cang]

(non-binding) +1
Built it from src.
Ran unit tests.
Checksum is ok.


Xu

On Fri, Sep 20, 2019 at 6:49 PM Guanghao Zhang  wrote:

> +1
>
> Built from source: ok (openjdk 1.8.0_202)
> Signatures and checksum: ok
> CHANGES and RELEASENOTES: ok
>
> Stack  于2019年9月21日周六 上午8:32写道：
>
> > +1
> >
> > Signatures and hash are good.
> > CHANGES and RELEASENOTES look complete.
> > Built from src and ran a few commands.
> >
> > S
> >
> >
> >
> > On Fri, Sep 20, 2019 at 8:34 AM Peter Somogyi 
> wrote:
> >
> > > Please vote on this Apache HBase Operator Tools Release Candidate (RC),
> > > 1.0.0.
> > >
> > > The VOTE will remain open for at least 72 hours.
> > >
> > > [ ] +1 Release this package as Apache HBase Operator Tools 1.0.0
> > > [ ] -1 Do not release this package because ...
> > >
> > > The tag to be voted on is 1.0.0RC1:
> > >
> > >  https://github.com/apache/hbase-operator-tools/tree/1.0.0RC1
> > >
> > > The release files, including signatures, digests, as well as CHANGES.md
> > > and RELEASENOTES.md included in this RC can be found at:
> > >
> > >  https://dist.apache.org/repos/dist/dev/hbase/1.0.0RC1/
> > >
> > > Maven artifacts are available in a staging repository at:
> > >
> > >
> https://repository.apache.org/content/repositories/orgapachehbase-1349/
> > >
> > > Artifacts were signed with the psomo...@apache.org key which can be
> > found
> > > in:
> > >
> > >  https://dist.apache.org/repos/dist/release/hbase/KEYS
> > >
> > > To learn more about Apache HBase Operator Tools, please see
> > > http://hbase.apache.org/
> > >
> > > Thanks,
> > > Your HBase Release Manager
> > >
> >
>

Re: [VOTE] The first HBase Operator Tools 1.0.0 release candidate (RC0) is available

[Xu Cang]

non-binding +1

Build source code. Ran unit tests. Run some commands against cluster.

Xu

On Mon, Sep 16, 2019 at 8:33 PM Stack  wrote:

> +1
>
> Checked sigs and hashes.
> Built src. Ran a few commands against a cluster. Seemed to work fine.
>
> S
>
> On Mon, Sep 16, 2019 at 11:31 AM Peter Somogyi 
> wrote:
>
> > Please vote on this Apache HBase Operator Tools Release Candidate (RC),
> > 1.0.0.
> >
> > The VOTE will remain open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache HBase Operator Tools 1.0.0
> > [ ] -1 Do not release this package because ...
> >
> > The tag to be voted on is 1.0.0RC0:
> >
> >  https://github.com/apache/hbase-operator-tools/tree/1.0.0RC0
> >
> > The release files, including signatures, digests, as well as CHANGES.md
> > and RELEASENOTES.md included in this RC can be found at:
> >
> >  https://dist.apache.org/repos/dist/dev/hbase/1.0.0RC0/
> >
> > Maven artifacts are available in a staging repository at:
> >
> >  https://repository.apache.org/content/repositories/orgapachehbase-1348/
> >
> > Artifacts were signed with the psomo...@apache.org key which can be
> found
> > in:
> >
> >  https://dist.apache.org/repos/dist/release/hbase/KEYS
> >
> > To learn more about Apache HBase Operator Tools, please see
> > http://hbase.apache.org/
> >
> > Thanks,
> > Your HBase Release Manager
> >
>

[jira] [Resolved] (HBASE-22804) Provide an API to get list of successful regions and total expected regions in Canary

[Xu Cang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-22804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Cang resolved HBASE-22804.
-
Fix Version/s: 1.4.12
   2.2.2
   2.1.7
   1.3.6
   2.3.1
   3.0.0
   Resolution: Fixed

> Provide an API to get list of successful regions and total expected regions 
> in Canary
> -
>
> Key: HBASE-22804
> URL: https://issues.apache.org/jira/browse/HBASE-22804
> Project: HBase
>  Issue Type: Improvement
>  Components: canary
>Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0, 2.1.5, 2.2.1
>Reporter: Caroline
>Assignee: Caroline
>Priority: Minor
>  Labels: Canary
> Fix For: 3.0.0, 1.5.0, 2.3.1, 1.3.6, 2.1.7, 2.2.2, 1.4.12
>
> Attachments: HBASE-22804.branch-1.001.patch, 
> HBASE-22804.branch-1.002.patch, HBASE-22804.branch-1.003.patch, 
> HBASE-22804.branch-1.004.patch, HBASE-22804.branch-1.005.patch, 
> HBASE-22804.branch-1.006.patch, HBASE-22804.branch-1.007.patch, 
> HBASE-22804.branch-1.008.patch, HBASE-22804.branch-1.009.patch, 
> HBASE-22804.branch-1.009.patch, HBASE-22804.branch-1.010.patch, 
> HBASE-22804.branch-2.001.patch, HBASE-22804.branch-2.002.patch, 
> HBASE-22804.branch-2.003.patch, HBASE-22804.branch-2.004.patch, 
> HBASE-22804.branch-2.005.patch, HBASE-22804.branch-2.006.patch, 
> HBASE-22804.master.001.patch, HBASE-22804.master.002.patch, 
> HBASE-22804.master.003.patch, HBASE-22804.master.004.patch, 
> HBASE-22804.master.005.patch, HBASE-22804.master.006.patch
>
>
> At present HBase Canary tool only prints the successes as part of logs. 
> Providing an API to get the list of successes, as well as total number of 
> expected regions, will make it easier to get a more accurate availability 
> estimate.
>   



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: [NOTICE] reminder, do not force push except to feature branches

[Xu Cang]

ACK.

Thanks!

On Thu, Sep 12, 2019 at 2:24 PM Sean Busbey  wrote:

> Hi folks!
>
> Friendly reminder, please do not force push to the project repositories.
>
> The one exception to this is feature branches, which should be named
> for a jira key (e.g. HBASE-23000)
>
> I've filed INFRA-19018 to get our release line branches protected from
> force pushes similar to how we do for master and release tags, but
> that system of protection has fallen down before so please try to stay
> in the habit of not trying.
>
> If you mistakenly commit something to a branch and want to undo it
> please just push revert commits.
>
> -busbey
>

Re: HBase Hadoop-QA refused to process patch due to Yetus "--skip-errorprone"

[Xu Cang]

Thanks Peter,
Should we port HBASE-22981's patch to branch-1 and possibly branch-2 too?

Xu

On Wed, Sep 11, 2019 at 3:07 PM Peter Somogyi  wrote:

> Hi,
>
> The problem is the same that was fixed in HBASE-22981 for the nightly jobs.
> Yetus 0.11.0 fails the build in case there are unprocessed parameters. We
> need to handle --skip-errorprone flag properly on branch-1s or
> add --ignore-unknown-options=true to the Jenkinsfile in use.
>
> Peter
>
> On Wed, Sep 11, 2019 at 6:53 PM Xu Cang 
> wrote:
>
> > Hi,
> > There is a JIRA my coworker is working on that hit this issue: HBase
> > Hadoop-QA refused to process patch due to Yetus "--skip-errorprone":
> >
> >
> >
> https://issues.apache.org/jira/browse/HBASE-22804?focusedCommentId=16926326=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16926326
> >
> > We guess it's due to many pre-check failures.(?)
> > Could someone please shed some lights on how to overcome this? Thank you.
> >
> > cc. Caroline Zhou
> >
> > Xu
> >
>

HBase Hadoop-QA refused to process patch due to Yetus "--skip-errorprone"

[Xu Cang]

Hi,
There is a JIRA my coworker is working on that hit this issue: HBase
Hadoop-QA refused to process patch due to Yetus "--skip-errorprone":

https://issues.apache.org/jira/browse/HBASE-22804?focusedCommentId=16926326=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16926326

We guess it's due to many pre-check failures.(?)
Could someone please shed some lights on how to overcome this? Thank you.

cc. Caroline Zhou

Xu

Can anyone add 'sandeep.pal' to HBASE contributor on JIRA

[Xu Cang]

Hi,
Can anyone add my coworker 'sandeep.pal'  (this is his apache id) to HBASE
contributor list on JIRA in order for him to be able to assign JIRAs to
himself?
Thank you in advance.

Best,
Xu

Re: [ANNOUNCE] new HBase committer Sakthi

[Xu Cang]

Congratulations and welcome Sakthi!

On Mon, Aug 5, 2019 at 1:11 PM Tak-Lon (Stephen) Wu 
wrote:

> Congratulations Sakthi ;) !
>
> -Stephen
>
> On Sun, Aug 4, 2019 at 7:44 AM Anoop John  wrote:
>
> > Congratulations Sakthi 
> >
> > -Anoop-
> >
> > On Sat, Aug 3, 2019 at 2:11 AM Stack  wrote:
> >
> > > Hurray!
> > > S
> > >
> > > On Wed, Jul 31, 2019 at 5:04 PM Sean Busbey  wrote:
> > >
> > > > On behalf of the HBase PMC, I'm pleased to announce that Sakthi has
> > > > accepted our invitation to become an HBase committer.
> > > >
> > > > We'd like to thank Sakthi for all of his diligent contributions to
> the
> > > > project thus far. We look forward to his continued participation in
> our
> > > > community.
> > > >
> > > > Congrats and welcome Sakthi!
> > > >
> > >
> >
>

Re: [ANNOUNCE] New HBase committer Tak-Lon (Stephen) Wu

[Xu Cang]

Congratulations and welcome Tak-Lon!

On Mon, Aug 5, 2019 at 1:08 PM Zach York 
wrote:

> Congratulations and welcome Stephen!
>
> On Mon, Aug 5, 2019 at 1:00 PM Andrew Purtell  wrote:
>
> > Congratulations and welcome, Stephen.
> >
> > On Mon, Aug 5, 2019 at 11:53 AM Sean Busbey  wrote:
> >
> > > On behalf of the Apache HBase PMC I am super pleased to announce that
> > > Tak-Lon (Stephen) Wu has accepted the PMC's invitation to become a
> > > commiter on the project.
> > >
> > > Thanks so much for the work you've been contributing. We look forward
> > > to your continued involvement.
> > >
> > > Congratulations and welcome!
> > >
> > > -busbey
> > >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> >- A23, Crosstalk
> >
>

Re: [ANNOUNCE] Please welcome Zheng Hu to the HBase PMC

[Xu Cang]

Congratulations, Zheng Hu!

On Mon, Aug 5, 2019 at 6:07 AM Guangxu Cheng  wrote:

> Congratulations, Zheng!
>
> Allan Yang  于2019年8月5日周一 下午5:13写道：
>
> > Congratulations, Hu！
> > Best Regards
> > Allan Yang
> >
> >
> > Peter Somogyi  于2019年8月5日周一 下午4:47写道：
> >
> > > Congratulations!
> > >
> > > On Mon, Aug 5, 2019 at 8:57 AM Pankaj kr  wrote:
> > >
> > > > Congratulations Zheng..!!
> > > >
> > > > Regards,
> > > > Pankaj
> > > >
> > > > -Original Message-
> > > > From: Duo Zhang [mailto:zhang...@apache.org]
> > > > Sent: 05 August 2019 07:38
> > > > To: HBase Dev List ; hbase-user <
> > > > u...@hbase.apache.org>
> > > > Subject: [ANNOUNCE] Please welcome Zheng Hu to the HBase PMC
> > > >
> > > > On behalf of the Apache HBase PMC I am pleased to announce that Zheng
> > Hu
> > > > has accepted our invitation to become a PMC member on the Apache
> HBase
> > > > project. We appreciate Zheng Hu stepping up to take more
> responsibility
> > > in
> > > > the HBase project.
> > > >
> > > > Please join me in welcoming Zheng Hu to the HBase PMC!
> > > >
> > >
> >
>

[jira] [Created] (HBASE-22775) Enhance logging for peer related operations

[Xu Cang (JIRA)]

Xu Cang created HBASE-22775:
---

 Summary: Enhance logging for peer related operations
 Key: HBASE-22775
 URL: https://issues.apache.org/jira/browse/HBASE-22775
 Project: HBase
  Issue Type: Improvement
Reporter: Xu Cang


Now we don't have good logging regarding peer operations, for example addPeer 
does not log itself:

[https://github.com/apache/hbase/blob/master/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationPeerStorage.java#L102]

This Jira is aiming to enhancing this area



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [Announce] 张铎 (Duo Zhang) is Apache HBase PMC chair

[Xu Cang]

Thank you Misty!
Congratulations Duo, thanks for taking extra work!

On Fri, Jul 19, 2019 at 11:23 AM Zach York 
wrote:

> Congratulations Duo! Thanks for offering to take on the additional work!
>
> On Fri, Jul 19, 2019 at 10:34 AM Stack  wrote:
>
> > Thank you Misty for your years of service (FYI, for non-PMCers, the
> reports
> > Misty wrote to the Apache Board on our behalf were repeatedly called out
> > for their quality and thoughtfulness).
> >
> > Duo Zhang, thank you for taking on the mantle.
> >
> > S
> >
> > On Thu, Jul 18, 2019 at 10:46 AM Misty Linville 
> wrote:
> >
> > > Each Apache project has a project management committee (PMC) that
> > oversees
> > > governance of the project, votes on new committers and PMC members, and
> > > ensures that the software we produce adheres to the standards of the
> > > Foundation. One of the roles on the PMC is the PMC chair. The PMC chair
> > > represents the project as a Vice President of the Foundation and
> > > communicates to the board about the project's health, once per quarter
> > and
> > > at other times as needed.
> > >
> > > It's been my honor to serve as your PMC chair since 2017, when I took
> > over
> > > from Andrew Purtell. I've decided to step back from my volunteer ASF
> > > activities to leave room in my life for other things. The HBase PMC
> > > nominated Duo for this role, and Duo has kindly agreed! The board
> passed
> > > this resolution in its meeting yesterday[1] and it is already
> > official[2].
> > > Congratulations, Duo, and thank you for continuing to honor the project
> > > with your dedication.
> > >
> > > Misty
> > >
> > > [1] The minutes have not yet posted at the time of this email, but will
> > be
> > > available at http://www.apache.org/foundation/records/minutes/2019/.
> > > [2] https://www.apache.org/foundation/#who-runs-the-asf
> > >
> >
>

Re: [VOTE] The first HBase 1.3.5 release candidate (RC0) is available

[Xu Cang]

+1 (non-binding)
Tested with hbase vote script.
* Signature: ok
* Checksum : ok
* Rat check (1.8.0_152): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_152): ok
 - mvn clean install -DskipTests
* Unit tests pass (1.8.0_152): ok
 - mvn test -P runAllTests

Also run LTT with 1M rows. Pass
Tested basic UI.

On Sat, Jun 8, 2019 at 2:46 PM Andrew Purtell 
wrote:

> Thanks for the report Jan. If you still have some of the surefire output
> files for the failed tests please consider filing a JIRA and attaching them
> to it.
>
> > On Jun 8, 2019, at 2:23 PM, Jan Hentschel <
> jan.hentsc...@ultratendency.com> wrote:
> >
> > +1 (binding)
> >
> > Tested via the vote script.
> >
> > * Signature: ok
> > * Checksum : ok
> > * Rat check (1.8.0_202-ea): ok
> > - mvn clean apache-rat:check
> > * Built from source (1.8.0_202-ea): ok
> > - mvn clean install -DskipTests
> > * Unit tests pass (1.8.0_202-ea): failed
> > - mvn test -P runAllTests
> >
> > Had some problems with the tests (especially TestCanaryTool). Running
> them in isolation works. Also, played around with the shell and all basic
> commands work.
> >
> > From: Artem Ervits 
> > Reply-To: "dev@hbase.apache.org" 
> > Date: Friday, June 7, 2019 at 3:44 PM
> > To: dev 
> > Subject: Re: [VOTE] The first HBase 1.3.5 release candidate (RC0) is
> available
> >
> > +1 (non-binding)
> >
> > -Tested with "hbase-vote.sh -s
> > https://dist.apache.org/repos/dist/dev/hbase/1.3.5RC0/
> >
> >* Signature: ok
> >* Checksum : ok
> >* Rat check (1.8.0_172): ok
> > - mvn clean apache-rat:check
> >* Built from source (1.8.0_172): ok
> > - mvn clean install -DskipTests
> >* Unit tests pass (1.8.0_172): ok
> > - mvn test -P runSmallTests
> >
> > I switched to small tests as runAllTests takes a long time on my machine
> > Maybe good to add https://issues.apache.org/jira/browse/HBASE-20091, if
> of
> > value, I can reopen and see if it applies, we can handle it in 1.3.6
> > installed on local filesystem: OK
> > hbase shell: create, put, get, alter, disable, drop: OK
> > logs: OK
> > Phoenix 4.14.2-HBase-1.3: loaded some examples, accessible from phoenix
> and
> > hbase: OK
> >
> >
> > On Thu, Jun 6, 2019 at 11:56 AM Andrew Purtell  >
> > wrote:
> >
> > The dist URL is https://dist.apache.org/repos/dist/dev/hbase/1.3.5RC0/
> >
> >
> >> On Jun 5, 2019, at 5:06 PM, Andrew Purtell  apurt...@apache.org>> wrote:
> >>
> >> The first HBase 1.3.5 release candidate (RC0) is available for download
> > at
> >> https://dist.apache.org/repos/dist/dev/hbase/hbase-1.3.5RC0/ and Maven
> >> artifacts are available in the temporary repository
> > https://repository.apache.org/content/repositories/orgapachehbase-1322/
> .
> >>
> >> The git tag corresponding to the candidate is '1.3.5RC0' (b59afe7b1d).
> >>
> >> A detailed source and binary compatibility report for this release is
> >> available at
> >
> https://dist.apache.org/repos/dist/dev/hbase/hbase-1.3.5RC0/compat-check-report.html
> > . Please review and raise any concerns if you have them.
> >>
> >> A list of the 29 issues resolved in this release can be found at
> > https://s.apache.org/WLIm
> >>
> >> Please try out the candidate and vote +1/0/-1.
> >>
> >> Prior to making this announcement I made the following checks:
> >>
> >>RAT check passes (7u80)
> >>Unit test suite passes (8u172)
> >>LTT load 1M rows (8u172)
> >>
> >>
> >> --
> >> Best regards,
> >> Andrew
> >>
> >
> >
>

Re: [VOTE] The second HBase 1.4.10 release candidate (RC1) is available

[Xu Cang]

+1 (non-binding)
Tested with hbase-vote.sh
~/Downloads/hbase-1.4.10
* Signature: ok
* Checksum : ok
* Rat check (1.8.0_152): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_152): ok
 - mvn clean install -DskipTests
* Unit tests pass (1.8.0_152): ok
 - mvn test -P runAllTests
Run LTT with 1M rows. Pass
Tested basic UI.  Pass

On Fri, Jun 7, 2019 at 5:44 AM Peter Somogyi  wrote:

> +1
>
> Checked:
> - Signature, checksum
> - Build from source
> - Rat check
> - Unit tests
> - Basic shell commands
> - LTT 1M rows
>
> On Fri, Jun 7, 2019 at 2:09 PM Artem Ervits  wrote:
>
> > +1 (non-binding)
> >
> > -Tested with "hbase-vote.sh -s
> > https://dist.apache.org/repos/dist/dev/hbase/1.4.10RC1/; --output-dir
> > /tmp/vote-1.4.10RC1
> > * Signature: ok
> > * Checksum : ok
> > * Rat check (1.8.0_172): ok
> >  - mvn clean apache-rat:check
> > * Built from source (1.8.0_172): ok
> >  - mvn clean install -DskipTests
> > * Unit tests pass (1.8.0_172): failed
> >  - mvn test -P runAllTests
> >
> > on a separate vote my machine shut down and I was not able to preserve
> the
> > test results prior to hbase-thrift
> > installed source release on local filesystem (upgraded previously
> installed
> > 1.4.9 instance, all tables there): ok
> > hbase shell: ok
> > logs: ok
> > Java write 1M: ok
> > pointed to an existing Phoenix 4.14.2, tables are accessible from both
> > HBase and Phoenix: ok
> >
> >
> >
> > On Fri, Jun 7, 2019 at 7:16 AM Wellington Chevreuil <
> > wellington.chevre...@gmail.com> wrote:
> >
> > > +1
> > >
> > > -Tested with "hbase-vote.sh -s
> > > https://dist.apache.org/repos/dist/dev/hbase/1.4.10RC1/;
> > > * Signature: ok
> > > * Checksum : ok
> > > * Rat check (1.8.0_212): ok
> > >  - mvn clean apache-rat:check
> > > * Built from source (1.8.0_212): ok
> > >  - mvn clean install -DskipTests
> > > * Unit tests pass (1.8.0_212): ok
> > >  - mvn test -P runAllTests
> > > - Deployed binary built from source on local file system mode;
> > >* created table, put some rows, scan, delete, drops,
> > disable/enable,
> > > truncate worked fine;
> > >* UI: ok;
> > >* ran ltt for 100 rows;
> > >* hbase shell count matches ltt ingested rows;
> > >
> > >
> > > Em sex, 7 de jun de 2019 às 04:07, Tak-Lon (Stephen) Wu <
> > > taklo...@gmail.com>
> > > escreveu:
> > >
> > > > +1 (non-binding)
> > > >
> > > >   * Signature: ok
> > > >   * Checksum : ok
> > > >   * Rat check (1.8.0_212): ok
> > > >- mvn clean apache-rat:check
> > > >   * Built from source (1.8.0_212): ok
> > > >- mvn clean install -DskipTests
> > > >   * Unit tests pass (1.8.0_212): ok
> > > >- mvn test -P runAllTests
> > > >   * on Cluster tests with hadoop-2.8.5 (1.8.0_201) with 6
> > > m4.xlarge
> > > > nodes: ok
> > > >- Checked UI: ok
> > > >- basic operations
> > > > (create/put/get/scan/flush/list/disable/drop): ok
> > > >- Run read/write LTT tests with 1m keys passed: ok
> > > >  - hbase ltt -tn TestTable2 -write 10:16 -num_keys
> 100
> > > >  - hbase hbase ltt -tn TestTable2 -read 100 -num_keys
> > 100
> > > >- ITBLL 1m: ok
> > > >  - hbase
> > > > org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList -m
> > > serverKilling
> > > > loop 1 10 100 ${RANDOM} 1
> > > >
> > > >
> > > > Two tests below reran and passed the second time,
> TestRegionServerAbort
> > > is
> > > > not part of flaky tests
> > > > <
> > > >
> > >
> >
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-1.4/lastSuccessfulBuild/artifact/dashboard.html
> > > > >
> > > > ,
> > > > but passed the second run, so IMO it should not block voting +1 for
> > this
> > > > RC.
> > > >
> > > > -
> > >
> TestSnapshotFromMaster.testAsyncSnapshotWillNotBlockSnapshotHFileCleaner
> > > > - TestRegionServerAbort.testAbortFromRPC
> > > >
> > > > *### Check compatibility report
> > > > <
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hbase/1.4.10RC1/compat-check-report.html
> > > > >
> > > > *
> > > > - Verify those removed method and it looks good.
> > > >
> > > >
> > > > -Stephen
> > > >
> > > > On Thu, Jun 6, 2019 at 8:57 AM Andrew Purtell <
> > andrew.purt...@gmail.com>
> > > > wrote:
> > > >
> > > > > The dist URL is
> > > https://dist.apache.org/repos/dist/dev/hbase/1.4.10RC1/
> > > > >
> > > > > > On Jun 5, 2019, at 5:11 PM, Andrew Purtell 
> > > > wrote:
> > > > > >
> > > > > > The second HBase 1.4.10 release candidate (RC1) is available for
> > > > > download at
> > > > > > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.4.10RC1/
> and
> > > > Maven
> > > > > > artifacts are available in the temporary repository
> > > > >
> > 

Re: [ANNOUNCE] New Committer: Wellington Chevreuil

[Xu Cang]

Congrats Wellington and welcome!


On Fri, Jun 7, 2019 at 1:21 PM Sakthi  wrote:

> Hurray! Congrats Wellington. Well deserved one!
>
> Sakthi
>
> On Fri, Jun 7, 2019 at 12:58 PM Wei-Chiu Chuang
>  wrote:
>
> > Yay!! Congrats for the achievement!
> >
> > On Fri, Jun 7, 2019 at 12:56 PM Andrew Purtell  >
> > wrote:
> >
> > > Congratulations and welcome, Wellington.
> > >
> > > > On Jun 7, 2019, at 3:11 AM, Peter Somogyi 
> wrote:
> > > >
> > > > On behalf of the HBase PMC, I'm pleased to announce that Wellington
> > > > Chevreuil has accepted our invitation to become an HBase committer.
> > > >
> > > > Thanks for all your hard work Wellington; we look forward to more
> > > > contributions!
> > > >
> > > > Please join me in extending congratulations to Wellington!
> > >
> >
>

Re: [ANNOUNCE] New HBase committer Yi Mei

[Xu Cang]

Congratulations, Yi Mei!



On Fri, May 24, 2019 at 1:44 PM Esteban Gutierrez
 wrote:

> Congratulations, Yi Mei!
>
> --
> Cloudera, Inc.
>
>
>
> On Fri, May 24, 2019 at 2:12 AM Guanghao Zhang  wrote:
>
> > On behalf of the Apache HBase PMC, I am pleased to announce that Yi Mei
> has
> > accepted the PMC's invitation to become a committer on the project. We
> > appreciate all of Yi Mei's generous contributions thus far and look
> forward
> > to Yi Mei's continued involvement.
> >
> > Congratulations and welcome, Yi Mei!
> >
>

Does HBase persist history of replication peer operations?

[Xu Cang]

Hi All,
I am trying to find some history regarding permanent replication peer
change history. After reading the code I see there is only
ZKReplicationPeerStorage implementation to interface
ReplicationPeerStorage.  And ZK does not store historical data regarding
peer ops. (Except logging) is there any other place I can find this info,
such as 'add_peer' , 'remove_peer'?

Thank you!
Xu

[jira] [Created] (HBASE-22391) Fix flaky tests from TestFromClientSide

[Xu Cang (JIRA)]

Xu Cang created HBASE-22391:
---

 Summary: Fix flaky tests from TestFromClientSide
 Key: HBASE-22391
 URL: https://issues.apache.org/jira/browse/HBASE-22391
 Project: HBase
  Issue Type: New Feature
  Components: test
Affects Versions: 2.0.5, 3.0.0, 1.5.1
Reporter: Xu Cang


tests in TestFromClientSide.java in general are flaky due to the reason that 
after createTable, they did not wait for table to be ready before adding data 
into table.

Found this issue when working on HBASE-22274

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HBASE-22215) Backport MultiRowRangeFilter does not work with reverse scans

[Xu Cang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-22215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Cang resolved HBASE-22215.
-
Resolution: Fixed

> Backport MultiRowRangeFilter does not work with reverse scans
> -
>
> Key: HBASE-22215
> URL: https://issues.apache.org/jira/browse/HBASE-22215
> Project: HBase
>  Issue Type: Sub-task
>  Components: Filters
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 1.5.0, 1.4.10
>
> Attachments: HBASE-22215.001.branch-1.patch, HBASE-22215.001.patch
>
>
> See parent. Modify and apply to 1.x lines.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (HBASE-22215) Backport MultiRowRangeFilter does not work with reverse scans

[Xu Cang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-22215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Cang reopened HBASE-22215:
-

> Backport MultiRowRangeFilter does not work with reverse scans
> -
>
> Key: HBASE-22215
> URL: https://issues.apache.org/jira/browse/HBASE-22215
> Project: HBase
>  Issue Type: Sub-task
>  Components: Filters
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 1.5.0, 1.4.10
>
> Attachments: HBASE-22215.001.branch-1.patch, HBASE-22215.001.patch
>
>
> See parent. Modify and apply to 1.x lines.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-22274) Cell size limit check on append should consider cell's previous size.

[Xu Cang (JIRA)]

Xu Cang created HBASE-22274:
---

 Summary: Cell size limit check on append should consider cell's 
previous size.
 Key: HBASE-22274
 URL: https://issues.apache.org/jira/browse/HBASE-22274
 Project: HBase
  Issue Type: New Feature
Reporter: Xu Cang


Now we have cell size limit check based on this parameter 
*hbase.server.keyvalue.maxsize* 

One case was missing: appending to a cell only take append op's cell size into 
account against this limit check. we should check against the potential final 
cell size after the append.'

It's easy to reproduce this :

 

Apply this diff

 
{code:java}
diff --git 
a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 
b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 index 5a285ef6ba..8633177ebe 100644 --- 
a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 +++ 
b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 @@ -6455,7 +6455,7 @@ public class TestFromClientSide { // expected } try { - 
t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[10 * 1024])); + 
t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[2 * 1024])); 
fail("Oversize cell failed to trigger exception"); } catch (IOException e) { // 
expected{code}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: The fourth HBase 1.5.0 release candidate (RC3) is available

[Xu Cang]

I just saw this email, Andrew. Should I re-open HBASE-21959? And revert it
before we understand/fix why it caused the test failure?
Regarding the failing test, do you mean this one "TestBlocksRead"?
Thanks,

Xu

On Tue, Apr 16, 2019 at 9:47 PM Andrew Purtell 
wrote:

> I've bisected twice and it lands on this commit:
>
> commit 6bc46bb10920c1c335b784b01d2a326db1a3d587 (HEAD, refs/bisect/bad)
> HBASE-21959 CompactionTool should close the store it uses for
> compacting files, in order to properly archive compacted files.
>
>  
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
> |   2 ++
>
>  
> hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionTool.java
> | 100
>
> At first glance it's hard to see how this change is relevant, but it does
> introduce a new unit test.
>
>
> On Tue, Apr 16, 2019 at 7:48 PM Andrew Purtell 
> wrote:
>
> > I’ve been able to reproduce it sometimes too and am bisecting. It may be
> > an interaction between test cases, not a failure per se, but does seem
> have
> > a recent cause, as you pointed out. I’ll be looking at it.
> >
> > Thank you for your kind consideration and for revoking your veto.
> >
> > A coprocessor API fix was just committed to branch-1 so I want to roll a
> > new RC soon to include it. There is also an issue open to improve the
> > behavior of the UI when the profiler link is clicked but system support
> is
> > not available.
> >
> > > On Apr 16, 2019, at 7:40 PM, Yu Li  wrote:
> > >
> > > After more investigation, the ConnectionRefused exception could be
> > > reproduced with "mvn -Dtest= test" after a complete run of
> all
> > > cases through "mvn -PrunAllTests clean test", but cannot by a clean
> > > standalone run (with "mvn *clean* test"). So now I'm more convinced
> it's
> > > some kind of environment chaos caused by parallel execution of test
> > cases,
> > > and not a blocker issue.
> > >
> > > @Andrew It seems to me that kerby jar is not included in our binary
> > > package, so I'm not sure whether a new RC is required by HBASE-22219.
> > > Anyway I'd like to revoke my -1 vote now. Thanks.
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > >> On Tue, 16 Apr 2019 at 10:19, Yu Li  wrote:
> > >>
> > >> Sorry for the late response due to job priority.
> > >>
> > >> This ConnectionRefused issue cannot be reproduced on my laptop (MacOS
> > >> 10.14.4) but could on the linux env. And I've checked and confirmed it
> > >> could pass with 1.4.7/1.4.9 source package but stably failed with
> 1.5.0,
> > >> performing a git bisect now, will report back later.
> > >>
> > >> Best Regards,
> > >> Yu
> > >>
> > >>
> > >> On Sat, 13 Apr 2019 at 00:38, Andrew Purtell <
> andrew.purt...@gmail.com>
> > >> wrote:
> > >>
> > >>> I also see the occasional ConnectionRefused errors. They don’t
> > reproduce
> > >>> if you run the test standalone. I also only see them on a Linux dev
> > host.
> > >>> That may be enough to find by bisect the commit that introduced this
> > >>> behavior. Working on it. There is a JIRA filed for this one. Search
> for
> > >>> “TestBlocksRead” and label “branch-1”.
> > >>>
> > >>> Thanks for the investigations.
> > >>>
> >  On Apr 12, 2019, at 6:36 AM, Yu Li  wrote:
> > 
> >  Quick updates:
> > 
> >  W/ patch of HBASE-22219 or say upgrading kerby version to 1.0.1, the
> >  failures listed above in the 1st part of hbase-server disappeared.
> > 
> >  However, in the 2nd part of hbase-server UT there're still many
> >  ConnectionRefused exceptions (17 errors in total) as shown below,
> > which
> >  could be reproduced easily with -Dtest=xxx command on my
> environments,
> >  still checking the root cause.
> > 
> >  [INFO] Running org.apache.hadoop.hbase.regionserver.TestBlocksRead
> >  [ERROR] Tests run: 4, Failures: 0, Errors: 4, Skipped: 0, Time
> > elapsed:
> >  0.853 s <<< FAILURE! - in
> >  org.apache.hadoop.hbase.regionserver.TestBlocksRead
> >  [ERROR]
> > 
> > >>>
> >
> testBlocksStoredWhenCachingDisabled(org.apache.hadoop.hbase.regionserver.TestBlocksRead)
> >  Time elapsed: 0.17 s  <<< ERROR!
> >  java.net.ConnectException: Call From
> >  z05f06378.sqa.zth.tbsite.net/11.163.183.195 to localhost:35669
> failed
> > >>> on
> >  connection exception: java.net.ConnectException: Connection refused;
> > For
> >  more details see:
> >  http://wiki.apache.org/hadoop/ConnectionRefused
> >    at
> > 
> > >>>
> >
> org.apache.hadoop.hbase.regionserver.TestBlocksRead.initHRegion(TestBlocksRead.java:112)
> >    at
> > 
> > >>>
> >
> org.apache.hadoop.hbase.regionserver.TestBlocksRead.testBlocksStoredWhenCachingDisabled(TestBlocksRead.java:389)
> >  Caused by: java.net.ConnectException: Connection refused
> >    at
> > 
> > >>>
> >
> org.apache.hadoop.hbase.regionserver.TestBlocksRead.initHRegion(TestBlocksRead.java:112)
> >    at
> > 
> > >>>
> >

[jira] [Created] (HBASE-22217) HBase shell command proposal : "rit assign all"

[Xu Cang (JIRA)]

Xu Cang created HBASE-22217:
---

 Summary: HBase shell command proposal : "rit assign all" 
 Key: HBASE-22217
 URL: https://issues.apache.org/jira/browse/HBASE-22217
 Project: HBase
  Issue Type: New Feature
Reporter: Xu Cang


HBase shell command proposal : "rit assign all" 

 

Currently we have shell command "rit" to list all RITs.

It would be handy having a command "rit assign all" to assign all RITs.

This equals to getting the list of RITs from 'rit' command and running "assign 
" one by one.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-22216) "Waiting on master failover to complete" shows 30 to 40 time per millisecond

[Xu Cang (JIRA)]

Xu Cang created HBASE-22216:
---

 Summary: "Waiting on master failover to complete" shows 30 to 40 
time per millisecond 
 Key: HBASE-22216
 URL: https://issues.apache.org/jira/browse/HBASE-22216
 Project: HBase
  Issue Type: Bug
  Components: proc-v2
Affects Versions: 1.3.0
Reporter: Xu Cang


"Waiting on master failover to complete" shows 30 to 40 time per millisecond 
from one host when master is initializing. 

This message is too noisy. Need to fix this. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: precommit is producing some unnecessary pain; let's clean it up

[Xu Cang]

"I think we need to revert the recent error-prone related work"
-- Andrew, can you please give an example about this? Such as a JIRA that
has this kind of failure in pre-commit build.

" Checkstyle's
ImportOrder is one that always trips me up and no matter where I place the
imports continues to complain."

-- I had some struggles about this too,  though, after I installed
checkstyle plugin in intellij and used it before generating patches, it
became less painful. I don't have any objection removing the check at the
same time. Contributors should still try their best to organize imports
cleanly and orderly.


""



On Wed, Apr 3, 2019 at 12:02 PM Andrew Purtell  wrote:

> I have been contributing to this project for more than ten years and have
> noticed it is increasingly difficult to do so.
>
> For me the issues come down to precommit results. Precommit is a very
> useful tool, but *only if committers are attentive to fixing breaking
> changes immediately*. This has been an eternal problem.
>
> Also in fairness some problems I've thought are external to my patch have
> turned out to be indirect consequences. Here the issue is I'm not able to
> trust precommit so true positive results are still sometimes suspect.
>
> I think we need to revert the recent error-prone related work, this seems
> to be the cause of some of the false failures in precommit jobs I've looked
> at.
>
> In other cases we should adjust some static check settings. Checkstyle's
> ImportOrder is one that always trips me up and no matter where I place the
> imports continues to complain. I'm at a loss and it's really a trivial
> matter. Let's just turn it off.
>
> The transient issues we sometimes face with Apache build infra are possibly
> tolerable, I'm not referring to those.
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

[jira] [Resolved] (HBASE-21752) Backport getProcedures() to branch-1 from branch-2 in HMaster class

[Xu Cang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Cang resolved HBASE-21752.
-
Resolution: Won't Fix

> Backport getProcedures() to branch-1 from branch-2 in HMaster class
> ---
>
> Key: HBASE-21752
> URL: https://issues.apache.org/jira/browse/HBASE-21752
> Project: HBase
>  Issue Type: Improvement
> Environment: Backport getProcedures() to branch-1 from branch-2 in 
> HMaster class
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Fix For: 1.5.1
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HBASE-21846) Flaky Test: testMultiRowRangeWithFilterListOrOperatorWithBlkCnt

[Xu Cang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Cang resolved HBASE-21846.
-
  Resolution: Resolved
Release Note: test is not flaky anymore after the revert

> Flaky Test: testMultiRowRangeWithFilterListOrOperatorWithBlkCnt
> ---
>
> Key: HBASE-21846
> URL: https://issues.apache.org/jira/browse/HBASE-21846
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.5.0
>    Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Trivial
>
> Flaky test:
> [ERROR]   
> TestFilterListOrOperatorWithBlkCnt.testMultiRowRangeWithFilterListOrOperatorWithBlkCnt:127
>  expected:<4> but was:<5>
> Added some debugging logs and test result as below:
> 1028 2019-02-05 01:14:13,525 INFO  [main] 
> filter.TestFilterListOrOperatorWithBlkCnt(118): 0. blocksStart: 0
> 1029 2019-02-05 01:14:13,572 INFO  [main] 
> filter.TestFilterListOrOperatorWithBlkCnt(121): found 20 results
> 1030 2019-02-05 01:14:13,572 INFO  [main] 
> filter.TestFilterListOrOperatorWithBlkCnt(124): 1. Diff in number of blocks 3 
> blocksEnd is: 3 blocksStart: 0
> 1031 2019-02-05 01:14:13,573 INFO  [main] 
> filter.TestFilterListOrOperatorWithBlkCnt(129): 2. Diff in number of blocks 4 
> blocksEnd is: 4 blocksStart: 0
> 1032 2019-02-05 01:14:13,576 INFO  [main] 
> filter.TestFilterListOrOperatorWithBlkCnt(136): 3. Diff in number of blocks 5 
> blocksEnd is: 5 blocksStart: 0
> Basically,in my testing environment the scan with filterList read 3 blocks. 
> Latter 2 scans read 1 respectively. 
> According to this flaky tests 
> list:https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-1/lastSuccessfulBuild/artifact/dashboard.html
> This test is always failing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] what's our biggest source of friction for getting new contributors going?

[Xu Cang]

The motivation for contributing opensource work sometimes affected by
projects' popularity. And open source as a whole is hugely impacted by
public cloud companies since their profit-driven mindset. One of the
biggest public cloud provider even discourages employees from contributing
to open source by asking them filling documents, going through reviews and
signing agreements before you can do that under their employment. Past
Redis license change story also reveals some difficulties that open source
community is facing. https://news.ycombinator.com/item?id=19221754

On Wed, Mar 20, 2019 at 12:03 PM Andrew Purtell  wrote:

> I don’t think it is realistic, unfortunately. If you remember our OWNERS
> initiative, which failed, the idea there was for various functional areas
> or components there would in effect be a mentor, someone you could
> at-mention for advice and review. This failed because if this isn’t your
> full time paid profession you can’t devote the necessary time. Life and day
> job intervene. Furthermore the available volunteer bandwidth of this
> community is in long term decline. The pendulum can shift back (viz
> zookeeper) but the trend as of today is similar to tends in the ASF
> incubator: seems like in the past there were more people around with more
> available time. There might be good intentions but little is more
> demoralizing than being ignored by someone expected to be a mentor, only
> because that mentor has severe time constraints due to day job or personal
> life.
>
>
> On Wed, Mar 20, 2019 at 10:48 AM Misty Linville  wrote:
>
> > Another open source project I’m on is exploring the idea of a mentoring
> > rotation where each mentor serves for a week and especially looks out for
> > and provides practical assistance and review to new contributors for that
> > week, on an office-hours type of basis. Many hands make light work. WDYT?
> > I’d say that this type of mentoring, if done well, would be another way
> to
> > put one on the road to recognition for leadership within the project.
> >
> > On Wed, Mar 20, 2019 at 9:45 AM Tak-Lon (Stephen) Wu  >
> > wrote:
> >
> > > +1 one for the mentoring if one existing member who knows the feature
> > plan
> > > and is willing to create tasks for pickup.
> > >
> > > my current barrier is to find a continuous topic/component and focus on
> > it,
> > > but I'm a bit too random with my approach.
> > >
> > > Currently, not sure this is the right way, I lookup JIRAs by filtering
> > tags
> > > (e.g. BUG) with priority and created dates to find
> > > newly reported issues and somehow didn't know which JIRA should be
> > resolved
> > > next release. meanwhile, I may miss
> > > older JIRA which potentially is critical (probably the longer it has no
> > one
> > > look at it, should it considered as less critical?).
> > >
> > > Thanks,
> > > Stephen
> > >
> > >
> > >
> > >
> > > On Wed, Mar 20, 2019 at 12:12 AM Artem Ervits 
> > > wrote:
> > >
> > > > Would be great to have mentors. Ted used to review my patches and
> > provide
> > > > feedback on timely basis, understand everyone is busy but would be
> nice
> > > to
> > > > have a point person (per feature, bug, branch, module, etc). I have
> > > cycles
> > > > to burn but currently it feels a bit overwhelming as community may
> feel
> > > > there's a certain level of familiarity with the process necessary.
> > > >
> > > > On Wed, Mar 20, 2019, 4:59 AM Misty Linville 
> wrote:
> > > >
> > > > > #1 this is difficult code. That’s probably the one we can’t
> surmount
> > > but
> > > > I
> > > > > wanted to put it out there.
> > > > >
> > > > > On Tue, Mar 19, 2019 at 5:03 PM Sean Busbey 
> > wrote:
> > > > >
> > > > > > I have my own opinions, obvs. But I'm curious what other folks
> > think
> > > > are
> > > > > > the biggest impediments to new contributors?
> > > > > >
> > > > >
> > > >
> > >
> >
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

[jira] [Reopened] (HBASE-22009) Improve RSGroupInfoManagerImpl#getDefaultServers()

[Xu Cang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-22009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Cang reopened HBASE-22009:
-

Re-opening for pending branch-1 fix.

> Improve RSGroupInfoManagerImpl#getDefaultServers()
> --
>
> Key: HBASE-22009
> URL: https://issues.apache.org/jira/browse/HBASE-22009
> Project: HBase
>  Issue Type: Improvement
>  Components: rsgroup
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 1.5.1, 2.2.1
>
> Attachments: HBASE-22009.master.000.patch, 
> call_stack_getDefaultServers.png
>
>
> {code:title=RSGroupInfoManagerImpl.java|borderStyle=solid}
> private SortedSet getDefaultServers() throws IOException {
>   SortedSet defaultServers = Sets.newTreeSet();
>   for (ServerName serverName : getOnlineRS()) {
> Address server = Address.fromParts(serverName.getHostname(), 
> serverName.getPort());
> boolean found = false;
> for (RSGroupInfo rsgi : listRSGroups()) {
>   if (!RSGroupInfo.DEFAULT_GROUP.equals(rsgi.getName()) && 
> rsgi.containsServer(server)) {
> found = true;
> break;
>   }
> }
> if (!found) {
>   defaultServers.add(server);
> }
>   }
>   return defaultServers;
> }
> {code}
> That is a logic of 2 nest loops. And for each server, listRSGroups() 
> allocates a new LinkedList and calls Map#values(), both of which are very 
> heavy operations.
> Maybe the inner loop could be moved out, that is
> # Build a list of servers of other groups than default group
> # Iterate each online servers and check if it is in the list above. If it is 
> not, then it belongs to default group.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-22067) Fix log line in StochasticLoadBalancer when balancer is an ill-fit for cluster size

[Xu Cang (JIRA)]

Xu Cang created HBASE-22067:
---

 Summary: Fix log line in StochasticLoadBalancer when balancer is 
an ill-fit for cluster size
 Key: HBASE-22067
 URL: https://issues.apache.org/jira/browse/HBASE-22067
 Project: HBase
  Issue Type: Bug
Reporter: Xu Cang


HBASE-21338 Added log lines regarding load balancer warnings. There is a bug in 
log that uses wrong parameter.
'maxRunningTime' is used , should be maxSteps.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Broken Blog Images

[Xu Cang]

 A really good article BTW.
Seems the article was posted by Mr. Stack and the error message from the
link is "ACL Denied"

Best,
Xu


On Wed, Mar 6, 2019 at 12:34 PM Sean Busbey  wrote:

> we could host them on the main project website if we still have the
> images somewhere.
>
> On Wed, Mar 6, 2019 at 2:28 PM Nick Dimiduk  wrote:
> >
> > Heya,
> >
> > There's some nice posts up on the Apache blog, such as [0]. Sadly, it
> looks
> > like the image refs are broken. Refs are pointing off to some 3rd party
> > content hosting site. Maybe they can be updated to local-to-the-blog
> > content?
> >
> > Thanks,
> > Nick
> >
> > [0]:
> https://blogs.apache.org/hbase/entry/hbase_zk_less_region_assignment
>

Re: The third HBase 1.5.0 release candidate (RC2) is available

[Xu Cang]

+1

RAT check passes
2 Jars' sha512 verified.
Unit test suite passes (8u191)
  Except this one:
https://issues.apache.org/jira/browse/HBASE-21952
  and this one:
TestSnapshotFromMaster.testAsyncSnapshotWillNotBlockSnapshotHFileCleaner ,
This is in our Flaky test list on Jenkins.
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-1/lastSuccessfulBuild/artifact/dashboard.html

Web UI basic functionalities verified.


Best,
Xu

On Thu, Feb 21, 2019 at 3:44 PM Andrew Purtell  wrote:

> The third HBase 1.5.0 release candidate (RC2) is available for download at
> https://dist.apache.org/repos/dist/dev/hbase/hbase-1.5.0RC2/ and Maven
> artifacts are available in the temporary repository
> https://repository.apache.org/content/repositories/orgapachehbase-1258/
>
> The git tag corresponding to the candidate is '1.5.0RC2' (b5c50b506c).
>
> A detailed source and binary compatibility report for this release is
> available for your review at
>
> https://dist.apache.org/repos/dist/dev/hbase/hbase-1.5.0RC2/compat-check-report.html
> .
>
> A list of the 94 issues resolved in this release can be found at
> https://s.apache.org/K4Wk . The 1.5.0 changelog is derived from the
> changelog of the last branch-1.4 release, 1.4.9.
>
> Please try out the candidate and vote +1/0/-1.
>
> The vote will be open for at least 72 hours. Unless objection I will try to
> close it Thursday February 28, 2019 if we have sufficient votes.
>
> Prior to making this announcement I made the following preflight checks:
>
> RAT check passes (7u80)
> Unit test suite passes (7u80, 8u181)*
> Opened the UI in a browser, poked around
> LTT load 100M rows with 100% verification and 20% updates (8u181)
> ITBLL 1B rows with slowDeterministic monkey (8u181)
> ITBLL 1B rows with serverKilling monkey (8u181)
>
> Some of this testing was done with recent 1.5.0-SNAPSHOT versions. During
> the month of February I plan to perform a number of additional tests,
> including performance regression checks. As more results become available I
> will post them to this thread.
>
> There are known flaky tests. See HBASE-21904 and HBASE-21905. These flaky
> tests do not represent serious test failures that would prevent a release
> in my opinion.
>
>
> --
> Best regards,
> Andrew
>

[jira] [Created] (HBASE-21952) Test Failure: TestClientOperationInterrupt.testInterrupt50Percent

[Xu Cang (JIRA)]

Xu Cang created HBASE-21952:
---

 Summary: Test Failure: 
TestClientOperationInterrupt.testInterrupt50Percent
 Key: HBASE-21952
 URL: https://issues.apache.org/jira/browse/HBASE-21952
 Project: HBase
  Issue Type: Improvement
Reporter: Xu Cang
 Fix For: 1.5.0


---
Test set: org.apache.hadoop.hbase.client.TestClientOperationInterrupt
---
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 51.861 s <<< 
FAILURE! - in org.apache.hadoop.hbase.client.TestClientOperationInterrupt
testInterrupt50Percent(org.apache.hadoop.hbase.client.TestClientOperationInterrupt)
  Time elapsed: 50.108 s  <<< FAILURE!
java.lang.AssertionError:  noEx: 53, badEx=0, noInt=0
at 
org.apache.hadoop.hbase.client.TestClientOperationInterrupt.testInterrupt50Percent(TestClientOperationInterrupt.java:149)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: The second HBase 1.5.0 release candidate (RC1) is available

[Xu Cang]

+1
RAT check passes (8u191)
Unit tests pass (8u191)
UI check pass
Sha512 check pass for 2 jars
 ITBLL 25 rows with slowDeterministic monkey (8u191)
 ITBLL 25 rows with serverKilling monkey (8u191)

Xu

On Fri, Feb 15, 2019 at 12:52 PM Andrew Purtell  wrote:

> The second HBase 1.5.0 release candidate (RC1) is available for download at
> https://dist.apache.org/repos/dist/dev/hbase/hbase-1.5.0RC1/ and Maven
> artifacts are available in the temporary repository
> https://repository.apache.org/content/repositories/orgapachehbase-1256/
>
> The git tag corresponding to the candidate is '1.5.0RC1' (afba39399a).
>
> A detailed source and binary compatibility report for this release is
> available for your review at
>
> https://dist.apache.org/repos/dist/dev/hbase/hbase-1.5.0RC1/compat-check-report.html
> .
>
> A list of the 89 issues resolved in this release can be found at
> https://s.apache.org/K4Wk . The 1.5.0 changelog is derived from the
> changelog of the last branch-1.4 release, 1.4.9.
>
> Please try out the candidate and vote +1/0/-1.
>
> The vote will be open for at least 72 hours. Unless objection I will try to
> close it Thursday February 28, 2019 if we have sufficient votes.
>
> Prior to making this announcement I made the following preflight checks:
>
> RAT check passes (7u80)
> Unit test suite passes (7u80, 8u181)*
> Opened the UI in a browser, poked around
> LTT load 100M rows with 100% verification and 20% updates (8u181)
> ITBLL 1B rows with slowDeterministic monkey (8u181)
> ITBLL 1B rows with serverKilling monkey (8u181)
>
> Some of this testing was done with recent 1.5.0-SNAPSHOT versions. During
> the month of February I plan to perform a number of additional tests,
> including performance regression checks. As more results become available I
> will post them to this thread.
>
> There are known flaky tests. See HBASE-21904 and HBASE-21905. These flaky
> tests do not represent serious test failures that would prevent a release
> in my opinion.
>
> --
> Best regards,
> Andrew
>

Re: [VOTE] First release candidate for HBase 2.1.3 is available for download

[Xu Cang]

+1
Verified 3 jar's sha512.
Did basic WebUI check.
Ran all unit tests, passed.


On Thu, Feb 7, 2019 at 4:03 AM 张铎(Duo Zhang)  wrote:

> The first release candidate for HBase 2.1.3 is available for download:
>
>   https://dist.apache.org/repos/dist/dev/hbase/hbase-2.1.3RC0
>
> Maven artifacts are also available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1253/
>
> Artifacts are signed with my key (9AD2AE49) published in our
> KEYS file at http://www.apache.org/dist/hbase/KEYS
>
> The RC corresponds to the signed tag 2.1.3RC0, which currently points to
> commit
>
>   169f2aea2a9f31831201d3add598e6d7fb23079b
>
> HBase 2.1.3 is the fourth maintenance release in the HBase 2.1 line,
> continuing on the theme of bringing a stable, reliable database to the
> Hadoop and NoSQL communities. It fixes CVE-2018-1320 by upgrading thrift
> dependency from 0.9.3 to 0.12.0, all hbase users who use thrift are highly
> recommended to upgrade.
>
> 2.1.3 includes ~55 bug and improvement fixes done since the 2.1.2. There is
> an imcompatible issue, HBASE-21684, where we change the superclass of
> StoppedRpcClientException from HBaseIOException to DoNotRetryIOException,
> should be low risk, and feel free to contact us if this breaks anything for
> you.
>
> The detailed source and binary compatibility report vs 2.1.2 has been
> published for your review, at:
>
>
>
> https://dist.apache.org/repos/dist/dev/hbase/hbase-2.1.3RC0/compatibility_report_2.1.2vs2.1.3RC0.html
>
> The report shows no incompatibilities.
>
> The full list of fixes included in this release is available in the
> CHANGES.md that ships as part of the release also available here:
>
>   https://dist.apache.org/repos/dist/dev/hbase/hbase-2.1.3RC0/CHANGES.md
>
> The RELEASENOTES.md are here:
>
>
> https://dist.apache.org/repos/dist/dev/hbase/hbase-2.1.3RC0/RELEASENOTES.md
>
> Please try out this candidate and vote +1/-1 on whether we should release
> these artifacts as HBase 2.1.3.
>
> The VOTE will remain open for at least 72 hours.
>
> Thanks
>

Re: [ANNOUNCE] New HBase committer Xu Cang

[Xu Cang]

Thank you all! Looking forward to contributing more and making more
meaningful impact on our community.

Xu

On Wed, Feb 6, 2019 at 1:13 AM Yuhao Bi  wrote:

> Congratulations!
>
> Pankaj kr  于2019年2月6日周三 下午4:40写道：
>
> > Congratulations Xu...!!
> >
> > Regards,
> > Pankaj
> >
> >
> > -Original Message-
> > From: Andrew Purtell [mailto:apurt...@apache.org]
> > Sent: 06 February 2019 04:19
> > To: dev ; Hbase-User 
> > Subject: [ANNOUNCE] New HBase committer Xu Cang
> >
> > On behalf of the Apache HBase PMC, I am pleased to announce that Xu Cang
> > has accepted the PMC's invitation to become a committer on the project.
> We
> > appreciate all of Xu's generous contributions thus far and look forward
> to
> > his continued involvement.
> >
> > Congratulations and welcome, Xu!
> >
> > --
> > Best regards,
> > Andrew
> >
>

[jira] [Resolved] (HBASE-21848) Fix tests in TestRegionLocationCaching

[Xu Cang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Cang resolved HBASE-21848.
-
Resolution: Fixed

> Fix tests in TestRegionLocationCaching 
> ---
>
> Key: HBASE-21848
> URL: https://issues.apache.org/jira/browse/HBASE-21848
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>    Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
>
> There are 4 flaky tests in TestRegionLocationCaching.
> They are in flaky tests list too: 
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-1/lastSuccessfulBuild/artifact/dashboard.html
>  
> [ERROR]   
> TestRegionLocationCaching.testCachingForHTableMultiPut:133->checkRegionLocationIsCached:148
>  Expected non-zero number of cached region locations. Actual: 0
> [ERROR]   
> TestRegionLocationCaching.testCachingForHTableMultiplexerMultiPut:95->checkRegionLocationIsCached:148
>  Expected non-zero number of cached region locations. Actual: 0
> [ERROR]   
> TestRegionLocationCaching.testCachingForHTableMultiplexerSinglePut:73->checkRegionLocationIsCached:148
>  Expected non-zero number of cached region locations. Actual: 0
> [ERROR]   
> TestRegionLocationCaching.testCachingForHTableSinglePut:116->checkRegionLocationIsCached:148
>  Expected non-zero number of cached region locations. Actual: 0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-21848) Fix tests in TestRegionLocationCaching

[Xu Cang (JIRA)]

Xu Cang created HBASE-21848:
---

 Summary: Fix tests in TestRegionLocationCaching 
 Key: HBASE-21848
 URL: https://issues.apache.org/jira/browse/HBASE-21848
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.3.0
Reporter: Xu Cang


There are 4 flaky tests in TestRegionLocationCaching.
They are in flaky tests list too: 
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-1/lastSuccessfulBuild/artifact/dashboard.html
 




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-21847) Fix test TestRegionServerMetrics#testRequestCount

[Xu Cang (JIRA)]

Xu Cang created HBASE-21847:
---

 Summary: Fix test  TestRegionServerMetrics#testRequestCount
 Key: HBASE-21847
 URL: https://issues.apache.org/jira/browse/HBASE-21847
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.3.0
Reporter: Xu Cang


This test is also in flaky test list:
[ERROR]   TestRegionServerMetrics.testRequestCount:137 Metrics Counters should 
be equal expected:<59> but was:<89>
The failutre is consistent.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-21846) Flaky Test: testMultiRowRangeWithFilterListOrOperatorWithBlkCnt

[Xu Cang (JIRA)]

Xu Cang created HBASE-21846:
---

 Summary: Flaky Test: 
testMultiRowRangeWithFilterListOrOperatorWithBlkCnt
 Key: HBASE-21846
 URL: https://issues.apache.org/jira/browse/HBASE-21846
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.5.0, 1.3.0
Reporter: Xu Cang


Flaky test:
[ERROR]   
TestFilterListOrOperatorWithBlkCnt.testMultiRowRangeWithFilterListOrOperatorWithBlkCnt:127
 expected:<4> but was:<5>


Added some debugging logs and test result as below:
1028 2019-02-05 01:14:13,525 INFO  [main] 
filter.TestFilterListOrOperatorWithBlkCnt(118): 0. blocksStart: 0
1029 2019-02-05 01:14:13,572 INFO  [main] 
filter.TestFilterListOrOperatorWithBlkCnt(121): found 20 results
1030 2019-02-05 01:14:13,572 INFO  [main] 
filter.TestFilterListOrOperatorWithBlkCnt(124): 1. Diff in number of blocks 3 
blocksEnd is: 3 blocksStart: 0
1031 2019-02-05 01:14:13,573 INFO  [main] 
filter.TestFilterListOrOperatorWithBlkCnt(129): 2. Diff in number of blocks 4 
blocksEnd is: 4 blocksStart: 0
1032 2019-02-05 01:14:13,576 INFO  [main] 
filter.TestFilterListOrOperatorWithBlkCnt(136): 3. Diff in number of blocks 5 
blocksEnd is: 5 blocksStart: 0

Basically,in my testing environment the scan with filterList read 3 blocks. 
Latter 2 scans read 1 respectively. 

According to this flaky tests 
list:https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-1/lastSuccessfulBuild/artifact/dashboard.html
This test is always failing.





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: The first HBase 1.5.0 release candidate (RC0) is available

[Xu Cang]

Thanks to Peter's link. I checked my failed tests, they are all in this
flaky tests list. (Going to create some JIRAs for flaky tests if there
aren't now)

+1 for this release now.

Best,
Xu


On Tue, Feb 5, 2019 at 12:36 AM Peter Somogyi  wrote:

> Just like Xu Cang I ran into similar test failures on Debian and many of
> these are on the flaky list for branch-1 with 100% flakyness.
>
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-1/lastSuccessfulBuild/artifact/dashboard.html
>
> These are the failed tests for me:
> regionserver.TestRegionServerMetrics
> client.TestRegionLocationCaching
> client.TestHCM
> client.TestClientOperationInterrupt
> util.TestHBaseFsck
> master.cleaner.TestSnapshotFromMaster
> master.TestMasterShutdown
> replication.TestReplicationDroppedTables
> filter.TestFilterListOrOperatorWithBlkCnt
> mapreduce.TestSecureLoadIncrementalHFilesSplitRecovery
> mapreduce.TestLoadIncrementalHFilesSplitRecovery
>
> On Tue, Feb 5, 2019 at 7:30 AM 张铎(Duo Zhang) 
> wrote:
>
> > I think HBASE-21727 should be partially reverted since it removed a
> public
> > method in HBaseConfiguration which is marked as IA.Public?
> >
> > la...@apache.org  于2019年2月5日周二 上午1:54写道：
> >
> > >  Based on my own testing I was going to vote +1.I built 1.5.0 from
> > source,
> > > and ran it with the tip of the Phoenix 4.x.
> > > I regularly load a lot of data, execute Phoenix queries, etc. Nothing
> > > undue, nothing undue in the logs either.
> > > I'll try to reproduce the test failures. Since Andy can't reproduce
> them
> > > there is something flaky, most likely it's the tests, but that's, of
> > > course, hard to say.
> > > -- Lars
> > > On Saturday, February 2, 2019, 4:03:53 PM PST, Andrew Purtell <
> > > andrew.purt...@gmail.com> wrote:
> > >
> > >  Thanks. As I am not able to produce those unit test results we will
> need
> > > your help to diagnose the issues. Please file JIRAs as needed, post the
> > > test output detail, etc. Thanks for trying the candidate out!
> > >
> > > The ITBLL results may be a tool usage problem. The numbers in the
> failure
> > > messages you posted are too round. I expect real failures to produce
> more
> > > irregular numbers. ITBLL can a bit hard to use. Contact me offline and
> > I’ll
> > > give you notes on how I ran the tests myself.
> > >
> > >
> > > > On Feb 2, 2019, at 3:45 PM, Xu Cang 
> > > wrote:
> > > >
> > > > 2 jars sha12 verification: pass.
> > > > Basic UI check: pass.
> > > > Unit test. Some failures in  hbase - server package. (see details
> > below,
> > > > not sure if these are flaky tests)
> > > > ITBLL tests with slowDeterministic and serverKilling monky. Both got
> > some
> > > > failures. (Not sure if this is my environment issue since I am using
> > *my
> > > > laptop* to conduct this testing)
> > > > Not voting for now since I have some doubts regarding my testing
> > result.
> > > > Will keep looking.
> > > >
> > > >
> > > >  - *Unit test failure: (failures are reproducable)*
> > > >
> > > > [INFO] Results:
> > > > [INFO]
> > > > [ERROR] Failures:
> > > > [ERROR]
> > > >
> > >
> >
> TestRegionLocationCaching.testCachingForHTableMultiPut:133->checkRegionLocationIsCached:148
> > > > Expected non-zero number of cached region locations. Actual: 0
> > > > [ERROR]
> > > >
> > >
> >
> TestRegionLocationCaching.testCachingForHTableMultiplexerMultiPut:95->checkRegionLocationIsCached:148
> > > > Expected non-zero number of cached region locations. Actual: 0
> > > > [ERROR]
> > > >
> > >
> >
> TestRegionLocationCaching.testCachingForHTableMultiplexerSinglePut:73->checkRegionLocationIsCached:148
> > > > Expected non-zero number of cached region locations. Actual: 0
> > > > [ERROR]
> > > >
> > >
> >
> TestRegionLocationCaching.testCachingForHTableSinglePut:116->checkRegionLocationIsCached:148
> > > > Expected non-zero number of cached region locations. Actual: 0
> > > > [ERROR]  TestReplicasClient.testHedgedRead:595 expected:<0> but
> was:<1>
> > > > [ERROR]
> > > >
> > >
> >
> TestFilterListOrOperatorWithBlkCnt.testMultiRowRangeWithFilterListOrOperatorWithBlkCnt:127
> > > &

Re: The first HBase 1.5.0 release candidate (RC0) is available

[Xu Cang]

2 jars sha12 verification: pass.
Basic UI check: pass.
Unit test. Some failures in  hbase - server package. (see details below,
not sure if these are flaky tests)
ITBLL tests with slowDeterministic and serverKilling monky. Both got some
failures. (Not sure if this is my environment issue since I am using *my
laptop* to conduct this testing)
Not voting for now since I have some doubts regarding my testing result.
Will keep looking.


   - *Unit test failure: (failures are reproducable)*

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
 
TestRegionLocationCaching.testCachingForHTableMultiPut:133->checkRegionLocationIsCached:148
Expected non-zero number of cached region locations. Actual: 0
[ERROR]
 
TestRegionLocationCaching.testCachingForHTableMultiplexerMultiPut:95->checkRegionLocationIsCached:148
Expected non-zero number of cached region locations. Actual: 0
[ERROR]
 
TestRegionLocationCaching.testCachingForHTableMultiplexerSinglePut:73->checkRegionLocationIsCached:148
Expected non-zero number of cached region locations. Actual: 0
[ERROR]
 
TestRegionLocationCaching.testCachingForHTableSinglePut:116->checkRegionLocationIsCached:148
Expected non-zero number of cached region locations. Actual: 0
[ERROR]   TestReplicasClient.testHedgedRead:595 expected:<0> but was:<1>
[ERROR]
 
TestFilterListOrOperatorWithBlkCnt.testMultiRowRangeWithFilterListOrOperatorWithBlkCnt:127
expected:<4> but was:<5>
[ERROR]   TestRegionServerMetrics.testRequestCount:137 Metrics Counters
should be equal expected:<59> but was:<89>
[INFO]
[ERROR] Tests run: 1870, Failures: 7, Errors: 0, Skipped: 17


   -
*ITBLL testing result: (failures are reproducable) *

 ./bin/hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList Loop
1 1 2500 /tmp/itbll 1 -m slowDeterministic

 2019-02-01 23:31:03,746 INFO  [main] mapreduce.Job:  map 100% reduce 100%
2019-02-01 23:31:03,746 INFO  [main] mapreduce.Job: Job
job_local221831554_0003 completed successfully
2019-02-01 23:31:03,758 INFO  [main] mapreduce.Job: 17518
Input split bytes=679
Combine input records=0
Combine output records=0
Reduce input groups=7500
Reduce shuffle bytes=517518
Reduce input records=15000
Reduce output records=0
Spilled Records=561948580
Shuffled Maps =3
Failed Shuffles=0
Merged Map outputs=3
GC time elapsed (ms)=1859
Total committed heap usage (bytes)=1846542336
HBase Counters
BYTES_IN_REMOTE_RESULTS=0
BYTES_IN_RESULTS=31125001574
MILLIS_BETWEEN_NEXTS=934178
NOT_SERVING_REGION_EXCEPTION=7
NUM_SCANNER_RESTARTS=0
NUM_SCAN_RESULTS_STALE=0
REGIONS_SCANNED=12
REMOTE_RPC_CALLS=0
REMOTE_RPC_RETRIES=0
ROWS_FILTERED=12
ROWS_SCANNED=7512
RPC_CALLS=14607
RPC_RETRIES=7
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Verify$Counts
REFERENCED=7500
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=108
2019-02-01 23:31:03,764 ERROR [main]
test.IntegrationTestBigLinkedList$Verify: *Expected referenced count does
not match with actual referenced count. expected referenced=2500
,actual=7500*


 ./bin/hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList Loop
1 4 100 /tmp/itbll 4 -m slowDeterministic
 2019-02-02 00:34:27,009 ERROR [main]
test.IntegrationTestBigLinkedList$Verify: Expected referenced count does
not match with actual referenced count. expected referenced=400
,actual=7900


On Fri, Feb 1, 2019 at 2:17 PM Andrew Purtell  wrote:

> The first HBase 1.5.0 release candidate (RC0) is available for download at
> https://dist.apache.org/repos/dist/dev/hbase/hbase-1.5.0RC0/ and Maven
> artifacts are available in the temporary repository
> https://repository.apache.org/content/repositories/orgapachehbase-1250/
>
> The git tag corresponding to the candidate is '1.5.0RC0' (ce6a6014da).
>
> A detailed source and binary compatibility report for this release is
> available for your review at
>
> https://dist.apache.org/repos/dist/dev/hbase/hbase-1.5.0RC0/compat-check-report.html
> . I do not believe there are any reported compatibility issues that are in
> violation of our compatibility policy for minor releases, but if you find
> something and feel differently, please file a JIRA.
>
> A list of the 88 issues resolved in this release can be found at
> https://s.apache.org/K4Wk . The 1.5.0 changelog is derived from the
> changelog of the last branch-1.4 release, 1.4.9.
>
> Please try out the candidate and vote +1/0/-1.
>
> The vote will be open for at least 72 hours. Unless objection I will try to
> close it Thursday February 28, 2019 if we have sufficient votes.
>
> Prior to making this announcement I made 

[jira] [Created] (HBASE-21752) Backport getProcedures() to branch-1 from branch-2 in HMaster class

[Xu Cang (JIRA)]

Xu Cang created HBASE-21752:
---

 Summary: Backport getProcedures() to branch-1 from branch-2 in 
HMaster class
 Key: HBASE-21752
 URL: https://issues.apache.org/jira/browse/HBASE-21752
 Project: HBase
  Issue Type: Improvement
 Environment: Backport getProcedures() to branch-1 from branch-2 in 
HMaster class
Reporter: Xu Cang






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: +How can I abort pending procedures?

[Xu Cang]

Hi Hossein,
If you are facing this issue for HBase branch-1. You could not use hbck2.
And be aware, some procedures are not abortable. The most practical
solution to your issue it to follow what Wellington mentioned above.
Removing the  "/hbase/MasterProcWALs" will remove all master procedures for
your cluster. I suggest you backing up this directory before removing. Then
you can failover active HMaster and you can retry creating tables you need.

Best,
Xu

On Mon, Dec 10, 2018 at 2:56 AM Wellington Chevreuil <
wellington.chevre...@gmail.com> wrote:

> Hi Hossein, for which hbase version are you facing this issue?
> Removing "/hbase/MasterProcWALs" would probably help sort the
> mentioned error, but there might be some risk of creating other
> inconsistencies, depending on which procedures are running. Does
> list_procedures command show any "running" procedure, or just list the
> finished ones?
> Em seg, 10 de dez de 2018 às 02:39, Sakthi Vel
>  escreveu:
> >
> > Hi Hossein,
> >
> > Aborting procedures can be dangerous (specially if the procedure is not
> > rolled back). AFAIK, you can use hbck2(apache/hbase-operator-tools) tool
> to
> > abort a procedure using the ('bypass')  option. I would like to quote the
> > official hbck2 doc here:
> >
> >  bypass [OPTIONS] ...
> >Options:
> > -o,--override   override if procedure is running/stuck
> > -r,--recursive  bypass parent and its children. SLOW! EXPENSIVE!
> > -w,--lockWait   milliseconds to wait on lock before giving up;
> > default=1
> >Pass one (or more) procedure 'pid's to skip to procedure finish.
> >Parent of bypassed procedure will also be skipped to the finish.
> >Entities will be left in an inconsistent state and will require
> >manual fixup. May need Master restart to clear locks still held.
> >Bypass fails if procedure has children. Add 'recursive' if all
> >you have is a parent pid to finish parent and children. This
> >is SLOW, and dangerous so use selectively. Does not always work.
> >
> > +Other members, please correct me if I am wrong.
> >
> > Sakthi
> >
> > On Sun, Dec 9, 2018 at 6:18 PM Hossein Zolfi 
> > wrote:
> >
> > > Hi,
> > > I run hbase performance tools, and thousands tables have been created.
> And
> > > our cluster is currently in inconsistent state (We dont know what is
> the
> > > cause but we try found it), at first I try to disable/drop created
> tables
> > > (1700 tables) but nothing done. list_procedure show 492 rows, and It's
> not
> > > possible to abort any of them. Then, I restart hmaster service, but
> now, I
> > > got infinite number of following exceptions:
> > >
> > > 2018-12-09 20:01:30,194 WARN
> [MASTER_SERVER_OPERATIONS-master-4:16000-0]
> > > master.AssignmentManager: Failed assignment of
> > > t53889,0007603345,1542715604227.4cc63591941dbe928663
> > > 88fbde075cac. to data-22-54,16020,1543392184445, waiting a little
> before
> > > trying on the same region server try=1 of 10
> > >
> org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> > >
> org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> > > Received OPEN for the region:t53889,000
> > > 7603345,1542715604227.4cc63591941dbe92866388fbde075cac. ,
> which
> > > we are already trying to CLOSE
> > > at
> > >
> > >
> org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:1604)
> > > at
> > >
> > >
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22239)
> > > at
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2196)
> > > at
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> > > at
> > >
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> > > at
> > > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> > > at java.lang.Thread.run(Thread.java:748)
> > >
> > > at
> sun.reflect.GeneratedConstructorAccessor10.newInstance(Unknown
> > > Source)
> > > at
> > >
> > >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > > at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> > > at
> > >
> > >
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> > > at
> > >
> > >
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> > > at
> > >
> > >
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:330)
> > > at
> > >
> > >
> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:772)
> > > at
> > >
> > >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2164)
> > > at
> > >
> > >
> 

[jira] [Created] (HBASE-21553) schedLock not released ni MasterProcedureScheduler

[Xu Cang (JIRA)]

Xu Cang created HBASE-21553:
---

 Summary: schedLock not released ni MasterProcedureScheduler
 Key: HBASE-21553
 URL: https://issues.apache.org/jira/browse/HBASE-21553
 Project: HBase
  Issue Type: Improvement
Reporter: Xu Cang


https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureScheduler.java#L749



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-21552) backport HBASE-16735(Procedure v2 - Fix yield while holding locks) to branch-1 .

[Xu Cang (JIRA)]

Xu Cang created HBASE-21552:
---

 Summary: backport  HBASE-16735(Procedure v2 - Fix yield while 
holding locks)  to branch-1 . 
 Key: HBASE-21552
 URL: https://issues.apache.org/jira/browse/HBASE-21552
 Project: HBase
  Issue Type: Improvement
  Components: proc-v2
Reporter: Xu Cang
Assignee: Xu Cang
 Attachments: Screen Shot 2018-12-05 at 4.34.05 PM.png

Please see screenshot for the stack trace. 
We met this issue in production: many createNamespaceProcedures cannot proceed.
After some debugging and JIRA digging, I think HBASE-16735 addressed this 
issue. It fixed the issue that WAITING procedure fails to be added back to the 
runQueue. 
But that change wasn't ported to branch-1. I am creating this JIRA for 
backporting it to branch-1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-21224) Handle compaction queue duplication

[Xu Cang (JIRA)]

Xu Cang created HBASE-21224:
---

 Summary: Handle compaction queue duplication
 Key: HBASE-21224
 URL: https://issues.apache.org/jira/browse/HBASE-21224
 Project: HBase
  Issue Type: Improvement
  Components: Compaction
Reporter: Xu Cang


Mentioned by [~allan163] that we may want to handle compaction queue 
duplication in this Jira https://issues.apache.org/jira/browse/HBASE-18451 

Creating this item for further assessment and discussion.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-21117) Backport HBASE-18350 (RSGroups are broken underAMv2) to branch-1 :

[Xu Cang (JIRA)]

Xu Cang created HBASE-21117:
---

 Summary: Backport HBASE-18350  (RSGroups are broken underAMv2)  to 
branch-1 :
 Key: HBASE-21117
 URL: https://issues.apache.org/jira/browse/HBASE-21117
 Project: HBase
  Issue Type: Bug
  Components: backport, rsgroup, shell
Affects Versions: 1.3.2
Reporter: Xu Cang
Assignee: Xu Cang


When working on HBASE-20666, I found out HBASE-18350 did not get ported to 
branch-1, which causes procedure to hang when #moveTables called sometimes. 

After looking into the 18350 patch, seems it's important since it fixes 4 
issues. This Jira is an attempt to backport it to branch-1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HBASE-21066) Improve isTableState() method to ensure caller gets correct info

[Xu Cang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Cang resolved HBASE-21066.
-
Resolution: Won't Fix

> Improve isTableState() method to ensure caller gets correct info
> 
>
> Key: HBASE-21066
> URL: https://issues.apache.org/jira/browse/HBASE-21066
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0, 1.3.0, 2.0.0
>    Reporter: Xu Cang
>Priority: Minor
> Attachments: HBASE-21066.master.001.patch, 
> HBASE-21066.master.002.patch
>
>
>  
> {code:java}
> public boolean isTableState(TableName tableName, TableState.State... states) {
>  try {
>  TableState tableState = getTableState(tableName);
>  return tableState.isInStates(states);
>  } catch (IOException e) {
>  LOG.error("Unable to get table " + tableName + " state", e);
>  // XXX: is it safe to just return false here?
>  return false;
>  }
>  }
>  
> {code}
>  
> When cannot get table state, returning false is not always safe or correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-21066) Improve isTableState() method to ensure caller gets correct info

[Xu Cang (JIRA)]

Xu Cang created HBASE-21066:
---

 Summary: Improve isTableState() method to ensure caller gets 
correct info
 Key: HBASE-21066
 URL: https://issues.apache.org/jira/browse/HBASE-21066
 Project: HBase
  Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Xu Cang


{{public boolean isTableState(TableName tableName, TableState.State... states) 
{}}
{{try {}}
{{TableState tableState = getTableState(tableName);}}
{{return tableState.isInStates(states);}}
{{} catch (IOException e) {}}
{{LOG.error("Unable to get table " + tableName + " state", e);}}
{{// XXX: is it safe to just return false here?}}
{{return false;}}
{{}}}
{{}}}

 

When cannot get table state, returning false is not always safe or correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (HBASE-20928) Rewrite calculation of midpoint in binarySearch functions to prevent overflow

[Xu Cang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Cang reopened HBASE-20928:
-

> Rewrite calculation of midpoint in binarySearch functions to prevent overflow
> -
>
> Key: HBASE-20928
> URL: https://issues.apache.org/jira/browse/HBASE-20928
> Project: HBase
>  Issue Type: Bug
>  Components: io
>Reporter: saurabh singh
>Assignee: saurabh singh
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HBASE-20928-addendum.patch, 
> HBASE-20928-fix-binarySearch-v5.patch, HBASE-20928-fix-binarySearch-v5.patch
>
>
> There are couple of issues in the function:
>  * {{>>>}} operator would mess the values if {{low}} + {{high}} end up being 
> negative. This shouldn't happen but I don't see anything to prevent this from 
> happening.
>  * The code fails around boundary values of {{low}} and {{high}}. This is a 
> well known binary search catch. 
> [https://ai.googleblog.com/2006/06/extra-extra-read-all-about-it-nearly.html]
>  
> Most of the code should already be covered by tests. I would have liked to 
> add a test that actually fails without the fix but given these are private 
> methods I am not sure on the best place to add the test. Suggestions?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20925) Canary test to expose results per table

[Xu Cang (JIRA)]

Xu Cang created HBASE-20925:
---

 Summary: Canary test to expose results per table
 Key: HBASE-20925
 URL: https://issues.apache.org/jira/browse/HBASE-20925
 Project: HBase
  Issue Type: Improvement
  Components: canary
Reporter: Xu Cang


Canary test to expose results per table.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20858) port HBASE-20695 to branch-1

[Xu Cang (JIRA)]

Xu Cang created HBASE-20858:
---

 Summary: port HBASE-20695 to branch-1
 Key: HBASE-20858
 URL: https://issues.apache.org/jira/browse/HBASE-20858
 Project: HBase
  Issue Type: Improvement
Reporter: Xu Cang


port HBASE-20695 to branch-1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20695) Implement table level RegionServer replication metrics

[Xu Cang (JIRA)]

Xu Cang created HBASE-20695:
---

 Summary: Implement table level RegionServer replication metrics 
 Key: HBASE-20695
 URL: https://issues.apache.org/jira/browse/HBASE-20695
 Project: HBase
  Issue Type: Improvement
  Components: metrics
Reporter: Xu Cang
Assignee: Xu Cang


Region server metrics now are mainly global metrics. It would be nice to have 
table level metrics such as table level source.AgeOfLastShippedOp to indicate 
operators which table's replication is lagging behind.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[PATCH] one minor fix for hardcoded hbase:meta

[Xu Cang]

Hi,

I saw a minor thing during working on meta table related stuff.
Not sure if I should open a Jira issue for this kind of minor change.
(Please let me know if so) See the patch in attachment.

I ran the unit test for this, looks good:

"[INFO] ---
[INFO]  T E S T S
[INFO] ---
[INFO] Running
org.apache.hadoop.hbase.coprocessor.TestCoprocessorServiceBackwardCompatibility
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
15.651 s - in
org.apache.hadoop.hbase.coprocessor.TestCoprocessorServiceBackwardCompatibility
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
"


Thanks,
Xu
diff --git 
a/hbase-endpoint/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorServiceBackwardCompatibility.java
 
b/hbase-endpoint/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorServiceBackwardCompatibility.java
index e7181bb..edb217f 100644
--- 
a/hbase-endpoint/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorServiceBackwardCompatibility.java
+++ 
b/hbase-endpoint/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorServiceBackwardCompatibility.java
@@ -118,7 +118,7 @@ public class TestCoprocessorServiceBackwardCompatibility {
 DummyResponse.getDefaultInstance());
 assertEquals(REGIONSERVER, DummyCoprocessorService.numRegionServer);
 
-
TEST_UTIL.getConnection().getTable(TableName.valueOf("hbase:meta")).batchCoprocessorService(
+
TEST_UTIL.getConnection().getTable(TableName.META_TABLE_NAME).batchCoprocessorService(
 DummyCoprocessorService.getDescriptor().findMethodByName("dummyCall"),
 DummyRequest.newBuilder().setValue(REGION).build(), Bytes.toBytes(""), 
Bytes.toBytes(""),
 DummyResponse.getDefaultInstance());

Re: [development] How to get Client ID or Client Name from Get object in coprocessor implementation

[Xu Cang]

Josh,
Thank you. The RPCServer.getRemoteIp() is what I was looking for.
Best,

On Thu, Apr 26, 2018 at 6:46 AM, Josh Elser <els...@apache.org> wrote:

> Xu,
>
> Can you be more specific about what you're asking for in a "clientID" or
> "clientName"?
>
> Or even better, look at the static methods on the class
> org.apache.hadoop.hbase.ipc.RpcServer. There are a number of getters
> which use ThreadLocal variables to provide context about the user who
> submitted the RPC with the current thread is processing.
>
>
> On 4/25/18 6:55 PM, Xu Cang wrote:
>
>> Hi,
>> I am trying to implement a coprocessor and one info I'd like to get is
>> clientID or clientName for each Get op. I haven't found a place I can
>> retrieve this. Is this possible?
>>
>> I am going to implement RegionObserver. The hook I refer to is this:
>>
>> void preGetOp(ObserverContext c, Get get,
>> List result)
>>
>>  From this preGetOp method, I have only observerContext and 'Get' objects.
>> Is client ID in any of these data structures? If not, are thee other ways
>> to get that info?
>>
>> Thanks,
>>
>> Xu
>>
>>

[development] How to get Client ID or Client Name from Get object in coprocessor implementation

[Xu Cang]

Hi,
I am trying to implement a coprocessor and one info I'd like to get is
clientID or clientName for each Get op. I haven't found a place I can
retrieve this. Is this possible?

I am going to implement RegionObserver. The hook I refer to is this:

void preGetOp(ObserverContext c, Get get,
List result)

>From this preGetOp method, I have only observerContext and 'Get' objects.
Is client ID in any of these data structures? If not, are thee other ways
to get that info?

Thanks,

Xu

Re: [jira] [Created] (HBASE-20233) [metrics] Ill-formatted numRegions metric in "Hadoop:service=HBase,name=RegionServer,sub=Regions" mbean

[Xu Cang]

Hi,
This is a small patch for this issue. (first time trying to use Apache
JIRA, it does not allow me to upload patch file. please let me know if this
is the wrong way to send patches.)

Best,
Xu

On Tue, Mar 20, 2018 at 1:05 PM, stack (JIRA)  wrote:

> stack created HBASE-20233:
> -
>
>  Summary: [metrics] Ill-formatted numRegions metric in
> "Hadoop:service=HBase,name=RegionServer,sub=Regions" mbean
>  Key: HBASE-20233
>  URL: https://issues.apache.org/jira/browse/HBASE-20233
>  Project: HBase
>   Issue Type: Bug
>   Components: metrics
>  Environment: Noticed this poking at metrics. The MBean
> "Hadoop:service=HBase,name=RegionServer,sub=Regions" carries per region
> metrics. They are all formatted like so:
>
> {code}
> Namespace_default_table_IntegrationTestBigLinkedList_
> metric_incrementTime_98th_percentile: 0,
> Namespace_default_table_IntegrationTestBigLinkedList_
> metric_incrementTime_99th_percentile: 0,
> Namespace_default_table_IntegrationTestBigLinkedList_
> metric_incrementTime_99.9th_percentile: 0,
> Namespace_default_table_IntegrationTestBigLinkedList_metric_appendTime_num_ops:
> 0,
> Namespace_default_table_IntegrationTestBigLinkedList_metric_appendTime_min:
> 0,
> Namespace_default_table_IntegrationTestBigLinkedList_metric_appendTime_max:
> 0,
> Namespace_default_table_IntegrationTestBigLinkedList_metric_appendTime_mean:
> 0,
> {code}
>
> In the middle of them all is a metric named...
> {code}
> numRegions: 15,
> {code}
>
> It has a different format and it is out of scope when compared to the
> other metrics in this mbean; it belongs better elsewhere, perhaps in the
> Server bean.
>
> I noticed it because it breaks the parse done by tcollector scraping our
> metrics from /jmx
>
> It was added long ago in HBASE-14166
> Reporter: stack
>
>
>
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v7.6.3#76005)
>

HFile.java code comment regarding block_size

[Xu Cang]

Hi,
I am reading HFile.java today and found out in the comment section,
it's a bit confusing to me.


 * Minimum block size. We recommend a setting of minimum block size between

 *__8KB to 1MB__ for general usage. Larger block size is preferred if files are

 * primarily for sequential access. However, it would lead to inefficient random

 * access (because there are more data to decompress). Smaller blocks are good

 * for random access, but require more memory to hold the block index, and may

 * be slower to create (because we must flush the compressor stream at the

 * conclusion of each data block, which leads to an FS I/O flush). Further, due

 * to the internal caching in Compression codec, the smallest possible block

 * size would be around __20KB-30KB__.

It mentioned block size should be 8k to 1M then it also says
__smallest__ block should be around 20k to 30k.
Should we fix this description? Sounds like the 20k is more accurate
regarding lower bound.
Thanks,

Xu