[jira] [Created] (HBASE-26311) Balancer gets stuck in cohosted replica distribution

2021-09-29 Thread Clara Xiong (Jira) 
Clara Xiong created HBASE-26311:
---

 Summary: Balancer gets stuck in cohosted replica distribution
 Key: HBASE-26311
 URL: https://issues.apache.org/jira/browse/HBASE-26311
 Project: HBase
  Issue Type: Bug
  Components: Balancer
Reporter: Clara Xiong


In production, we found a corner case where balancer cannot make progress when 
there is cohosted replica. This is reproed on master branch using test added in 
HBASE-26310. The cost function isn't provide proper evaluation so balancer 
could make progress.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26310) Repro balancer behavior during iterations

2021-09-29 Thread Clara Xiong (Jira) 
Clara Xiong created HBASE-26310:
---

 Summary: Repro balancer behavior during iterations
 Key: HBASE-26310
 URL: https://issues.apache.org/jira/browse/HBASE-26310
 Project: HBase
  Issue Type: Sub-task
Reporter: Clara Xiong


All existing tests expect balancer to complete in one run. This misses 
temporary imbalance or inefficiency during iterations on large clusters. 
Recently we found such issues for cohosted region distribution. To repro the 
problem and validate fixes, we need a test to simulate multiple iterations.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26309) Balancer tend to move regions to the server at the end of list

2021-09-29 Thread Clara Xiong (Jira) 
Clara Xiong created HBASE-26309:
---

 Summary: Balancer tend to move regions to the server at the end of 
list
 Key: HBASE-26309
 URL: https://issues.apache.org/jira/browse/HBASE-26309
 Project: HBase
  Issue Type: Bug
  Components: Balancer
Reporter: Clara Xiong


When we have a cohosted regions, cohosted replica distribution has priority to 
region count skew. So we won't see region count  getting balanced until the 
replicas distributed. During this time, master UI shows a drift of regions to 
the server at the end of list and causes significant overload. This shows a 
bias of random region pick and needs to be addressed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26308) Sum of multiplier of cost functions is not populated properly when we have a shortcut for trigger

2021-09-29 Thread Clara Xiong (Jira) 
Clara Xiong created HBASE-26308:
---

 Summary: Sum of multiplier of cost functions is not populated 
properly when we have a shortcut for trigger
 Key: HBASE-26308
 URL: https://issues.apache.org/jira/browse/HBASE-26308
 Project: HBase
  Issue Type: Bug
  Components: Balancer
Reporter: Clara Xiong


We have a couple of scenarios that we force balancing:
 * idle servers
 * co-hosted regions

The code path quit before populating the sum of multiplier of cost functions. 
This causes wrong value reported in logging. As below, the weighted average is 
not divide by total weight. This causes inconsistent log among iterations.
{quote}2021-09-24 21:46:57,881 INFO 
org.apache.hadoop.hbase.master.balancer.S*tocha*sticLoadBalancer: Running 
balancer because at least one server hosts replicas of the same region.

2021-09-24 21:46:57,881 INFO 
org.apache.hadoop.hbase.master.balancer.S*tocha*sticLoadBalancer: Start 
S*tocha*sticLoadBalancer.balancer, initial weighted average 
imbalance=6389.260497305375, functionCost=RegionCountSkewCostFunction : 
(multiplier=500.0, imbalance=0.06659036267913739); 
PrimaryRegionCountSkewCostFunction : (multiplier=500.0, 
imbalance=0.05296760285663541); MoveCostFunction : (multiplier=7.0, 
imbalance=0.0, balanced); ServerLocalityCostFunction : (multiplier=25.0, 
imbalance=0.46286750487559114); RackLocalityCostFunction : (multiplier=15.0, 
imbalance=0.2569525347374165); TableSkewCostFunction : (multiplier=500.0, 
imbalance=0.3760689783169534); RegionReplicaHostCostFunction : 
(multiplier=10.0, imbalance=0.0553889913899139); 
RegionReplicaRackCostFunction : (multiplier=1.0, 
imbalance=0.05854089790897909); ReadRequestCostFunction : (multiplier=5.0, 
imbalance=0.06969346106898068); WriteRequestCostFunction : (multiplier=5.0, 
imbalance=0.07834116112410174); MemStoreSizeCostFunction : (multiplier=5.0, 
imbalance=0.12533769793201735); StoreFileCostFunction : (multiplier=5.0, 
imbalance=0.06921401085082914);  computedMaxSteps=5577401600
{quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Removing problematic terms from our project

2021-09-29 Thread Andrew Purtell 
Just to be clear Bryan there is no issue with you asking questions or
reviving discussion. I was just trying to summarize for you because I do
believe we had a fairly clear outcome.

Because there has been no additional comment in a long time -- i.e. it's
now lazy consensus -- I felt I could be more definitive in the restatement
of the summary.

We can certainly reopen the discussion if I have mischaracterized the
consensus, or if there isn't actually a consensus, or if anyone has changed
their position in the intervening months.

Otherwise, we are just waiting for patch contribution...


On Wed, Sep 29, 2021 at 11:02 AM Bryan Beaudreault
 wrote:

> Sorry Andrew, I think I misinterpreted aspects of your last summary. It
> seemed like maybe there were still open questions and I was mostly just
> curious if something had been (or should be) captured anywhere else. Your
> new summary helps clarify the conclusion, thanks for providing it.
>
> On Wed, Sep 29, 2021 at 1:34 PM Andrew Purtell 
> wrote:
>
> > Bryan,
> >
> > Let me paraphrase the resolution of this discussion from the PMC
> > perspective: We are, broadly speaking, supportive of changes to improve
> > conscious language choices. Our project uses some words with known
> > controversial context. Unfortunately one word in particular, "master",
> does
> > not have a consensus that it is or isn't a valid term of art, and in any
> > case is deeply embedded in API and configuration contexts. Other terms,
> > like "slave", have consensus on removal. We would, generally speaking,
> > welcome for review any patches that change conscious language choices for
> > the better. The proposer of the patch can explain the context of the
> change
> > to help make the case it should be applied. The PMC would also provide
> > support, in the form of release management and voting, for necessary
> > deprecation-release-removal-release cycles where termonology changes
> impact
> > one or more of our compatibility guidelines.
> >
> > What has been missing since this thread closed with this conclusion?
> >
> > Actual patches.
> >
> > It's quite easy to advocate someone *else* make language changes.
> >
> >
> >
> > On Wed, Sep 29, 2021 at 5:26 AM Bryan Beaudreault
> >  wrote:
> >
> > > Sorry to revive a very old thread, but I just stumbled across this and
> > > don't see a clear resolution. I wonder if we should create a JIRA from
> > > Andrew's summary and treat that as an umbrella encompassing the
> original
> > 3
> > > JIRAs? I'm also cognizant of the fact that there are rumblings of doing
> > an
> > > initial 3.0 release, and I see above there was a proposal to deprecate
> > in 3
> > > and release in 4. I imagine we're slowly running out of time to make
> that
> > > change.
> > >
> > > If I missed a JIRA somewhere, maybe we can put a link here for
> posterity.
> > >
> > > On Fri, Jun 26, 2020 at 2:35 PM Andrew Purtell 
> > > wrote:
> > >
> > > > Circling back after more inputs, if we use this as a description of
> the
> > > > proposals:
> > > >
> > > > 1. Replace "master"/"hmaster" with ???, this one has by far the most
> > > > significant impact and both opinion and interpretation on this one is
> > > > mixed.
> > > >
> > > > 2. Replace "slave" with "follower", seems to impact the cross cluster
> > > > replication subsystem only.
> > > >
> > > > 3. Replace "black list" with "deny list".
> > > >
> > > > 4. Replace "white list" with "accept list".
> > > >
> > > > Then by my read of the responses we have consensus to do #2, #3, and
> > #4.
> > > > They were not controversial. JIRAs and patches will be welcome. Seems
> > > > pretty clear committers and PMC will approve and do what is needed to
> > > > complete any necessary deprecation cycle.
> > > >
> > > > Regarding #1, opinion is mixed. By my read I also think committers
> and
> > > PMC
> > > > will approve patches and do what is needed to complete any necessary
> > > > deprecation cycle for this one too. Enough PMC members expressed
> > support
> > > to
> > > > successfully vote on a release (although not if there were to be
> > opposing
> > > > votes). If a contributor were to open a JIRA and provide patches for
> > > this,
> > > > there would be more discussion. There is no consensus, yet, on what
> > > > replacement term is best. Personally, I can accept Zheng's recent
> > > > suggestion of "controller". I can see how syllable count matters.
> > > >
> > > > I don't mean this summary to close the conversation. It is only a
> > > > checkpoint.
> > > >
> > > > If anyone reading this has an opinion they do not wish to express
> > > > publically, you are welcome to write to priv...@hbase.apache.org to
> > > state
> > > > your opinion and the PMC will of course respectfully listen to it.
> > > >
> > > >
> > > >
> > > > On Thu, Jun 25, 2020 at 7:47 PM zheng wang <18031...@qq.com> wrote:
> > > >
> > > > > I like thecontroller.
> > > > >
> > > > >
> > > > > Coordinator is a bit long for me to write and speak.
> > > > > Manager

Re: [DISCUSS] Removing problematic terms from our project

2021-09-29 Thread Bryan Beaudreault 
Sorry Andrew, I think I misinterpreted aspects of your last summary. It
seemed like maybe there were still open questions and I was mostly just
curious if something had been (or should be) captured anywhere else. Your
new summary helps clarify the conclusion, thanks for providing it.

On Wed, Sep 29, 2021 at 1:34 PM Andrew Purtell  wrote:

> Bryan,
>
> Let me paraphrase the resolution of this discussion from the PMC
> perspective: We are, broadly speaking, supportive of changes to improve
> conscious language choices. Our project uses some words with known
> controversial context. Unfortunately one word in particular, "master", does
> not have a consensus that it is or isn't a valid term of art, and in any
> case is deeply embedded in API and configuration contexts. Other terms,
> like "slave", have consensus on removal. We would, generally speaking,
> welcome for review any patches that change conscious language choices for
> the better. The proposer of the patch can explain the context of the change
> to help make the case it should be applied. The PMC would also provide
> support, in the form of release management and voting, for necessary
> deprecation-release-removal-release cycles where termonology changes impact
> one or more of our compatibility guidelines.
>
> What has been missing since this thread closed with this conclusion?
>
> Actual patches.
>
> It's quite easy to advocate someone *else* make language changes.
>
>
>
> On Wed, Sep 29, 2021 at 5:26 AM Bryan Beaudreault
>  wrote:
>
> > Sorry to revive a very old thread, but I just stumbled across this and
> > don't see a clear resolution. I wonder if we should create a JIRA from
> > Andrew's summary and treat that as an umbrella encompassing the original
> 3
> > JIRAs? I'm also cognizant of the fact that there are rumblings of doing
> an
> > initial 3.0 release, and I see above there was a proposal to deprecate
> in 3
> > and release in 4. I imagine we're slowly running out of time to make that
> > change.
> >
> > If I missed a JIRA somewhere, maybe we can put a link here for posterity.
> >
> > On Fri, Jun 26, 2020 at 2:35 PM Andrew Purtell 
> > wrote:
> >
> > > Circling back after more inputs, if we use this as a description of the
> > > proposals:
> > >
> > > 1. Replace "master"/"hmaster" with ???, this one has by far the most
> > > significant impact and both opinion and interpretation on this one is
> > > mixed.
> > >
> > > 2. Replace "slave" with "follower", seems to impact the cross cluster
> > > replication subsystem only.
> > >
> > > 3. Replace "black list" with "deny list".
> > >
> > > 4. Replace "white list" with "accept list".
> > >
> > > Then by my read of the responses we have consensus to do #2, #3, and
> #4.
> > > They were not controversial. JIRAs and patches will be welcome. Seems
> > > pretty clear committers and PMC will approve and do what is needed to
> > > complete any necessary deprecation cycle.
> > >
> > > Regarding #1, opinion is mixed. By my read I also think committers and
> > PMC
> > > will approve patches and do what is needed to complete any necessary
> > > deprecation cycle for this one too. Enough PMC members expressed
> support
> > to
> > > successfully vote on a release (although not if there were to be
> opposing
> > > votes). If a contributor were to open a JIRA and provide patches for
> > this,
> > > there would be more discussion. There is no consensus, yet, on what
> > > replacement term is best. Personally, I can accept Zheng's recent
> > > suggestion of "controller". I can see how syllable count matters.
> > >
> > > I don't mean this summary to close the conversation. It is only a
> > > checkpoint.
> > >
> > > If anyone reading this has an opinion they do not wish to express
> > > publically, you are welcome to write to priv...@hbase.apache.org to
> > state
> > > your opinion and the PMC will of course respectfully listen to it.
> > >
> > >
> > >
> > > On Thu, Jun 25, 2020 at 7:47 PM zheng wang <18031...@qq.com> wrote:
> > >
> > > > I like thecontroller.
> > > >
> > > >
> > > > Coordinator is a bit long for me to write and speak.
> > > > Manager and Admin is used somewhere yet in HBase.
> > > >
> > > >
> > > >
> > > >
> > > > --原始邮件--
> > > > 发件人:"Andrew Purtell" > > > 发送时间:2020年6月26日(星期五) 上午9:08
> > > > 收件人:"Hbase-User" > > > 抄送:"dev" > > > 主题:Re: [DISCUSS] Removing problematic terms from our project
> > > >
> > > >
> > > >
> > > >  - AdminServer (as you already have AdminClient to talk to it).
> > > >
> > > > Oh... I like AdminServer. AdminServer (serving admin functions)
> and
> > > > RegionServer (serving region data).
> > > >
> > > > On Thu, Jun 25, 2020 at 4:46 PM Andrey Elenskiy
> > > >  > > >
> > > >   Is there a word that's not "master" and not "coordinator"
> > that
> > > > is clear
> > > >  and
> > > >  suitable for (diverse, polyglot) community?
> > > > 
> > > >  There are also:
> > > >  - captain (sounds pretty close to "master" without the negative
> >

Re: [DISCUSS] Removing problematic terms from our project

2021-09-29 Thread Andrew Purtell 
Bryan,

Let me paraphrase the resolution of this discussion from the PMC
perspective: We are, broadly speaking, supportive of changes to improve
conscious language choices. Our project uses some words with known
controversial context. Unfortunately one word in particular, "master", does
not have a consensus that it is or isn't a valid term of art, and in any
case is deeply embedded in API and configuration contexts. Other terms,
like "slave", have consensus on removal. We would, generally speaking,
welcome for review any patches that change conscious language choices for
the better. The proposer of the patch can explain the context of the change
to help make the case it should be applied. The PMC would also provide
support, in the form of release management and voting, for necessary
deprecation-release-removal-release cycles where termonology changes impact
one or more of our compatibility guidelines.

What has been missing since this thread closed with this conclusion?

Actual patches.

It's quite easy to advocate someone *else* make language changes.



On Wed, Sep 29, 2021 at 5:26 AM Bryan Beaudreault
 wrote:

> Sorry to revive a very old thread, but I just stumbled across this and
> don't see a clear resolution. I wonder if we should create a JIRA from
> Andrew's summary and treat that as an umbrella encompassing the original 3
> JIRAs? I'm also cognizant of the fact that there are rumblings of doing an
> initial 3.0 release, and I see above there was a proposal to deprecate in 3
> and release in 4. I imagine we're slowly running out of time to make that
> change.
>
> If I missed a JIRA somewhere, maybe we can put a link here for posterity.
>
> On Fri, Jun 26, 2020 at 2:35 PM Andrew Purtell 
> wrote:
>
> > Circling back after more inputs, if we use this as a description of the
> > proposals:
> >
> > 1. Replace "master"/"hmaster" with ???, this one has by far the most
> > significant impact and both opinion and interpretation on this one is
> > mixed.
> >
> > 2. Replace "slave" with "follower", seems to impact the cross cluster
> > replication subsystem only.
> >
> > 3. Replace "black list" with "deny list".
> >
> > 4. Replace "white list" with "accept list".
> >
> > Then by my read of the responses we have consensus to do #2, #3, and #4.
> > They were not controversial. JIRAs and patches will be welcome. Seems
> > pretty clear committers and PMC will approve and do what is needed to
> > complete any necessary deprecation cycle.
> >
> > Regarding #1, opinion is mixed. By my read I also think committers and
> PMC
> > will approve patches and do what is needed to complete any necessary
> > deprecation cycle for this one too. Enough PMC members expressed support
> to
> > successfully vote on a release (although not if there were to be opposing
> > votes). If a contributor were to open a JIRA and provide patches for
> this,
> > there would be more discussion. There is no consensus, yet, on what
> > replacement term is best. Personally, I can accept Zheng's recent
> > suggestion of "controller". I can see how syllable count matters.
> >
> > I don't mean this summary to close the conversation. It is only a
> > checkpoint.
> >
> > If anyone reading this has an opinion they do not wish to express
> > publically, you are welcome to write to priv...@hbase.apache.org to
> state
> > your opinion and the PMC will of course respectfully listen to it.
> >
> >
> >
> > On Thu, Jun 25, 2020 at 7:47 PM zheng wang <18031...@qq.com> wrote:
> >
> > > I like thecontroller.
> > >
> > >
> > > Coordinator is a bit long for me to write and speak.
> > > Manager and Admin is used somewhere yet in HBase.
> > >
> > >
> > >
> > >
> > > --原始邮件--
> > > 发件人:"Andrew Purtell" > > 发送时间:2020年6月26日(星期五) 上午9:08
> > > 收件人:"Hbase-User" > > 抄送:"dev" > > 主题:Re: [DISCUSS] Removing problematic terms from our project
> > >
> > >
> > >
> > >  - AdminServer (as you already have AdminClient to talk to it).
> > >
> > > Oh... I like AdminServer. AdminServer (serving admin functions) and
> > > RegionServer (serving region data).
> > >
> > > On Thu, Jun 25, 2020 at 4:46 PM Andrey Elenskiy
> > >  > >
> > >   Is there a word that's not "master" and not "coordinator"
> that
> > > is clear
> > >  and
> > >  suitable for (diverse, polyglot) community?
> > > 
> > >  There are also:
> > >  - captain (sounds pretty close to "master" without the negative
> side
> > > and it
> > >  should be relatable around the world)
> > >  - conductor (as in orchestra)
> > >  - controller (in kafka controller assigns partitions)
> > >  - RegionDriver (more relevant to what it's actually doing in hbase
> > and
> > >  borrowed from PlacementDrive of TiKV)
> > >  - AdminServer (as you already have AdminClient to talk to it).
> > > 
> > >  On Thu, Jun 25, 2020 at 3:49 PM Sean Busbey  
> > > wrote:
> > > 
> > >   How about "manager"?
> > >  
> > >   (It would help me if folks could explain what is lacking in
> > >  "coordinator".)
> > >  
> > >   On

Re: [DISCUSS] Contribution of a thrift2 python api

2021-09-29 Thread Sean Busbey 
yes, generally a PMC member associated with a specific release should
push the released artifacts to pypi. Someone will need to run down the
details of doing that, but there is precedent of other ASF projects
doing so.

On Tue, Sep 28, 2021 at 10:14 PM Yutong Xiao  wrote:
>
> Contributing to hbase-connectors is ok for me. But there is a question
> about how to release the package. Currently, I upload the package to pypi
> with my own pypi account. Not clear whether this should change if I
> contribute the client to hbase-connectors.
>
> Reid Chan  于2021年8月3日周二 下午10:27写道：
>
> > Brief summaries from this thread:
> >
> > I think you can push to hbase-connectors
> >  repo. Just create a new dir
> > hbase-thrift-python, and put your codes under it.
> > Your codes need to be reviewed by good Pythoners in the community.
> >
> > Some follow-up tasks include (but not limited to):
> >
> >- requirements.txt
> >- docs about how to release
> >- some client examples
> >- add some test codes (I'm not sure this part, whether there are similar
> >conventions in python world)
> >- other related information
> >
> >
> > About the *.thrfit files you mentioned, I can't come up a good idea by now.
> > Looks like we still need to create two separate PRs to update both hbase
> > and hbase-connectors repo.
> > However, as *.thrfit files seldom get updated, I feel it should be a ok.
> >
> >
> > --
> > Best Regards,
> > R.C
> >
> >
> > On Tue, Aug 3, 2021 at 10:42 AM Yutong Xiao  wrote:
> >
> > > Hello, any other thoughts about this?
> > >
> > > Yutong Xiao  于2021年7月21日周三 上午11:30写道：
> > >
> > > > Hi. I have removed the personal info in the licenses. For the point
> > 3.2,
> > > > thbase is dependent on the hbase.thrift file. I have involved the
> > > > hbase.thrift file, that thbase used, in the repo. In this case,  repo
> > > > separation will lead to a sync problem between the hbase.thrfit files
> > in
> > > > HBase repo and the connector repo. I am concerned this may make it hard
> > > for
> > > > maintenance. What do you think?
> > > >
> > > > 张铎(Duo Zhang)  于2021年7月17日周六 上午9:39写道：
> > > >
> > > >> One of the difficulties of moving hbase-thrift and hbase-rest out is
> > > >> because we make use of hbase-http in these two modules, at least for
> > > >> setting up the status servlet...
> > > >>
> > > >> Sean Busbey  于2021年7月16日周五 下午11:01写道：
> > > >>
> > > >> > maybe a good fit for the hbase-connectors repo? I know we've talked
> > a
> > > >> > few times about moving the thrift server out there. if we did both
> > > >> > then the compatibility question becomes just the standard
> > > >> > client/server compatibility provided the thrift server only uses our
> > > >> > public java client API.
> > > >> >
> > > >> > On Thu, Jul 15, 2021 at 10:21 PM Yutong Xiao 
> > > >> wrote:
> > > >> > >
> > > >> > > btw for point 2, if allowed I can do that.
> > > >> > > And for point 3.2 it is only a personal idea, the final decision
> > > >> should
> > > >> > be
> > > >> > > made by the community.
> > > >> > > Besides, many of my python user colleagues started using this
> > > library.
> > > >> > > I think many python users have the demand of a good HBase python
> > > >> client.
> > > >> > >
> > > >> > > Yutong Xiao  于2021年7月16日周五 上午11:07写道：
> > > >> > >
> > > >> > > > 1. The license is no problem.
> > > >> > > > 2. This should see if any committer or PMC has interests to do
> > > that.
> > > >> > > > 3. I can be responsible for those documents. About 3.2, as
> > thbase
> > > >> has
> > > >> > been
> > > >> > > > uploaded to Pypi, I think it would be better if it is a new,
> > > >> separate
> > > >> > repo.
> > > >> > > >
> > > >> > > > Wei-Chiu Chuang  于2021年7月6日周二 上午10:22写道：
> > > >> > > >
> > > >> > > >> Hi
> > > >> > > >> thanks for your interest in contributing the python api to the
> > > >> HBase
> > > >> > > >> project.
> > > >> > > >>
> > > >> > > >> I quickly check and it doesn't look like there's another active
> > > >> python
> > > >> > > >> HBase thrift client project at this point.
> > > >> > > >> I don't have a demand to use a python thrift hbase client
> > > library.
> > > >> If
> > > >> > > >> there
> > > >> > > >> are people who will benefit from this library, then it's a good
> > > >> idea
> > > >> > to
> > > >> > > >> make sure the library is well maintained, by having it become
> > > part
> > > >> of
> > > >> > the
> > > >> > > >> Apache HBase project and that more developers can contribute to
> > > it.
> > > >> > > >>
> > > >> > > >> As a hobbyist Python developer I can help review/commit the
> > > patch.
> > > >> > > >>
> > > >> > > >> My two cents:
> > > >> > > >> (1) license: the code is ASL 2.0 so it's compatible. The text
> > > >> > "Copyright
> > > >> > > >> 2021 Yutong Sean" would need to be removed.
> > > >> > > >> (2) Apache Infra does not manage PyPi. So we (the Apache HBase
> > > >> project
> > > >> > > >> committers/PMC) will have to do that.
>

[jira] [Resolved] (HBASE-26220) Use P2P communicate between region servers to sync the list for bootstrap node

2021-09-29 Thread Duo Zhang (Jira) 


 [ 
https://issues.apache.org/jira/browse/HBASE-26220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-26220.
---
Hadoop Flags: Reviewed
  Resolution: Fixed

Merged to master.

Thanks [~niuyulin] for reviewing.

> Use P2P communicate between region servers to sync the list for bootstrap node
> --
>
> Key: HBASE-26220
> URL: https://issues.apache.org/jira/browse/HBASE-26220
> Project: HBase
>  Issue Type: Sub-task
>  Components: master, regionserver
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> For returning bootstrap nodes to client, we do not need to always track 
> exactly the current live region server. But now, we just set a watcher on 
> zookeeper to watch all the live region servers, on every region server, for 
> large cluster, it will be a big pressure for zookeeper.
> So here I think, we could let region servers to communicate with each other 
> to sync the available bootstrap node, like a P2P network.
> Will put a simple design doc soon.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-26251) StochasticLoadBalancer metrics should update even if balancer doesn't run

2021-09-29 Thread Duo Zhang (Jira) 


 [ 
https://issues.apache.org/jira/browse/HBASE-26251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-26251.
---
Fix Version/s: 3.0.0-alpha-2
   2.5.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

Merged to master and branch-2.

Thanks [~GeorryHuang] for contributing and [~bbeaudreault] for reviewing!

> StochasticLoadBalancer metrics should update even if balancer doesn't run
> -
>
> Key: HBASE-26251
> URL: https://issues.apache.org/jira/browse/HBASE-26251
> Project: HBase
>  Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Zhuoyue Huang
>Priority: Minor
> Fix For: 2.5.0, 3.0.0-alpha-2
>
>
> Currently we only update StochasticLoadBalancerMetrics at the very end of a 
> balancer run, once a plan has been found. In fact, we update the metrics 
> based on that plan even if we don't end up executing the plan, which seems 
> incorrect. 
> Regardless of whether the balancer decides to run or not, cluster costs are 
> changing all the time. Since we don't update these metrics any other time, 
> operators miss out on important information about the balance of their 
> cluster over time. 
> I briefly looked into it and it would be relatively trivial to add another 
> call to updateStochasticCosts at the beginning of the balanceTable method, 
> before we determine if the cluster is in need of balancing. This would be an 
> improvement but would still miss cases where the balancer is disabled or 
> unable to run due to regions in transition, etc.
> It would be good if we could make it so updateStochasticCosts get called 
> periodically regardless of whether the balancer is enabled or can run.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Removing problematic terms from our project

2021-09-29 Thread Bryan Beaudreault 
Sorry to revive a very old thread, but I just stumbled across this and
don't see a clear resolution. I wonder if we should create a JIRA from
Andrew's summary and treat that as an umbrella encompassing the original 3
JIRAs? I'm also cognizant of the fact that there are rumblings of doing an
initial 3.0 release, and I see above there was a proposal to deprecate in 3
and release in 4. I imagine we're slowly running out of time to make that
change.

If I missed a JIRA somewhere, maybe we can put a link here for posterity.

On Fri, Jun 26, 2020 at 2:35 PM Andrew Purtell  wrote:

> Circling back after more inputs, if we use this as a description of the
> proposals:
>
> 1. Replace "master"/"hmaster" with ???, this one has by far the most
> significant impact and both opinion and interpretation on this one is
> mixed.
>
> 2. Replace "slave" with "follower", seems to impact the cross cluster
> replication subsystem only.
>
> 3. Replace "black list" with "deny list".
>
> 4. Replace "white list" with "accept list".
>
> Then by my read of the responses we have consensus to do #2, #3, and #4.
> They were not controversial. JIRAs and patches will be welcome. Seems
> pretty clear committers and PMC will approve and do what is needed to
> complete any necessary deprecation cycle.
>
> Regarding #1, opinion is mixed. By my read I also think committers and PMC
> will approve patches and do what is needed to complete any necessary
> deprecation cycle for this one too. Enough PMC members expressed support to
> successfully vote on a release (although not if there were to be opposing
> votes). If a contributor were to open a JIRA and provide patches for this,
> there would be more discussion. There is no consensus, yet, on what
> replacement term is best. Personally, I can accept Zheng's recent
> suggestion of "controller". I can see how syllable count matters.
>
> I don't mean this summary to close the conversation. It is only a
> checkpoint.
>
> If anyone reading this has an opinion they do not wish to express
> publically, you are welcome to write to priv...@hbase.apache.org to state
> your opinion and the PMC will of course respectfully listen to it.
>
>
>
> On Thu, Jun 25, 2020 at 7:47 PM zheng wang <18031...@qq.com> wrote:
>
> > I like thecontroller.
> >
> >
> > Coordinator is a bit long for me to write and speak.
> > Manager and Admin is used somewhere yet in HBase.
> >
> >
> >
> >
> > --原始邮件--
> > 发件人:"Andrew Purtell" > 发送时间:2020年6月26日(星期五) 上午9:08
> > 收件人:"Hbase-User" > 抄送:"dev" > 主题:Re: [DISCUSS] Removing problematic terms from our project
> >
> >
> >
> >  - AdminServer (as you already have AdminClient to talk to it).
> >
> > Oh... I like AdminServer. AdminServer (serving admin functions) and
> > RegionServer (serving region data).
> >
> > On Thu, Jun 25, 2020 at 4:46 PM Andrey Elenskiy
> >  >
> >   Is there a word that's not "master" and not "coordinator" that
> > is clear
> >  and
> >  suitable for (diverse, polyglot) community?
> > 
> >  There are also:
> >  - captain (sounds pretty close to "master" without the negative side
> > and it
> >  should be relatable around the world)
> >  - conductor (as in orchestra)
> >  - controller (in kafka controller assigns partitions)
> >  - RegionDriver (more relevant to what it's actually doing in hbase
> and
> >  borrowed from PlacementDrive of TiKV)
> >  - AdminServer (as you already have AdminClient to talk to it).
> > 
> >  On Thu, Jun 25, 2020 at 3:49 PM Sean Busbey  > wrote:
> > 
> >   How about "manager"?
> >  
> >   (It would help me if folks could explain what is lacking in
> >  "coordinator".)
> >  
> >   On Thu, Jun 25, 2020, 13:32 Nick Dimiduk  
> > wrote:
> >  
> >On Wed, Jun 24, 2020 at 10:14 PM 张铎(Duo Zhang) <
> > palomino...@gmail.com
> >wrote:
> >   
> > -0/+1/+1/+1
> >
> > I’m the one who asked whether ‘master’ is safe to use
> > without ‘slave’
> >   in
> > the private list.
> >
> > I’m still not convinced that it is really necessary
> > and I do not
> >  think
> > other words like ‘coordinator’ can fully describe the
> > role of HMaster
> >   in
> > HBase. HBase is more than 10 years old. In the
> context
> > of HBase, the
> >   word
> > ‘HMaster’ has its own meaning. Changing the name will
> > hurt our users
> >   and
> > make them confusing, especially for us non native
> > English speakers...
> >
> >   
> >Is there a word that's not "master" and not "coordinator"
> > that is clear
> >   and
> >suitable for (diverse, polyglot) community?
> >   
> >Stack  >
> >  +1/+1/+1/+1 where hbase3 adds the deprecation
> and
> > hbase4 follows
> >   hbase3
> >  soon after sounds good to me. I'm up for working
> > on this.
> >  S
> > 
> >  On Wed, Jun 24, 2020 at 2:26 PM Xu Cang <
> > xuc...@apache.org wrote:
> > 
> >   Strongly agree with what Nick said here:
> >  
> >   " From my perspective, we gain
> nothing
> > as a project

[jira] [Created] (HBASE-26307) Recovered ReplicationSource not terminated

2021-09-29 Thread zju_zsx (Jira) 
zju_zsx created HBASE-26307:
---

 Summary: Recovered ReplicationSource not terminated
 Key: HBASE-26307
 URL: https://issues.apache.org/jira/browse/HBASE-26307
 Project: HBase
  Issue Type: Bug
  Components: Replication
Affects Versions: 1.2.6
Reporter: zju_zsx


Recovered ReplicationSource not terminated after replicated all wal.

after all task done, it just remove source，but does not terminate it. so some 
resources(such as zk connection)  in replication endpoint leaks.
{code:java}
//代码占位符
if (replicationQueueInfo.isQueueRecovered()) {
  // use synchronize to make sure one last thread will clean the queue
  synchronized (workerThreads) {
Threads.sleep(100);// wait a short while for other worker thread to fully 
exit
boolean allOtherTaskDone = true;
for (ReplicationSourceWorkerThread worker : workerThreads.values()) {
  if (!worker.equals(this) && worker.isAlive()) {
allOtherTaskDone = false;
break;
  }
}
if (allOtherTaskDone) {
  manager.closeRecoveredQueue(this.source);
  LOG.info("Finished recovering queue " + peerClusterZnode
  + " with the following stats: " + getStats());
}
  }
}{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-26289) Hbase scan setMaxResultsPerColumnFamily not giving right results

2021-09-29 Thread Peter Somogyi (Jira) 


 [ 
https://issues.apache.org/jira/browse/HBASE-26289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Somogyi resolved HBASE-26289.
---
Fix Version/s: 2.4.7
   2.3.7
   3.0.0-alpha-2
   2.5.0
   Resolution: Fixed

Thanks [~richardantal]! Pushed to branch-2.3+.

> Hbase scan setMaxResultsPerColumnFamily not giving right results
> 
>
> Key: HBASE-26289
> URL: https://issues.apache.org/jira/browse/HBASE-26289
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 3.0.0-alpha-1, 2.1.9, 2.2.7, 2.3.6, 2.4.6
>Reporter: Richárd Antal
>Assignee: Richárd Antal
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.3.7, 2.4.7
>
>
> scan.setMaxResultsPerColumnFamily(1) is giving less number of row keys
> vs when it is not set in the scan, on a table that has multiple column 
> families and multiple qualifier or multiple versions for each cell 
> Docs for setMaxResultsPerColumnFamily:
> {code:java}
> /**
>  * Set the maximum number of values to return per row per Column Family
>  * @param limit the maximum number of values returned / row / CF
>  */{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26306) Backport "HBASE-26220 Use P2P communicate between region servers to sync the list for bootstrap node" to branch-2

2021-09-29 Thread Duo Zhang (Jira) 
Duo Zhang created HBASE-26306:
-

 Summary: Backport "HBASE-26220 Use P2P communicate between region 
servers to sync the list for bootstrap node" to branch-2
 Key: HBASE-26306
 URL: https://issues.apache.org/jira/browse/HBASE-26306
 Project: HBase
  Issue Type: Sub-task
Reporter: Duo Zhang






--
This message was sent by Atlassian Jira
(v8.3.4#803005)