[jira] [Comment Edited] (HBASE-18070) Enable memstore replication for meta replica
[
https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233790#comment-17233790
]
Michael Stack edited comment on HBASE-18070 at 11/18/20, 4:50 AM:
--
{quote}If you think you have something that can not be spoken out publicly then
I'm fine with a zoom call.
{quote}
Suggested it because I thought we'd agreed to go to face-to-face if a problem
communicating arose post Yu Li's help. That seems to be what is going on here
(and you put a -1 on top of it to boot).
On [https://github.com/apache/hbase/pull/2644], when I went there, I was late
to the game, trying to help clean up some crossed-wires around resolved Jira
but open PRs (or other way round). All conversations were 'resolved'. Your
'request changes' I presumed a vestige of a resolved conversation. Seemed like
minor stuff. No intentional dissing on my part. I can reopen if you want. Sorry.
Suggest you not get hung up on my representation. There is a design here with a
long-standing description of what the work here is about and you have helped
review the patches that comprise this work. Just use these instead.
was (Author: stack):
{quote}If you think you have something that can not be spoken out publicly then
I'm fine with a zoom call.
{quote}
Suggested it because I thought we'd agreed to go to face-to-face if a problem
communicating arose post Yu Li's help. That seems to be what is going on here
(and you put a -1 on top of it to boot).
On [https://github.com/apache/hbase/pull/2644], when I went there, I was late
to the game, trying to help clean up some crossed-wires around resolved Jira
but open PRs (or other way round). All conversations were 'resolved'. Your
'request changes' I presumed a vestige of a resolved conversation. Seemed like
minor stuff. No intentional dissing on my part. I can reopen if you want.
Suggest you not get hung up on my representation. There is a design here with a
long-standing description of what the work here is about and you have helped
review the patches that comprise this work. Just use these instead.
> Enable memstore replication for meta replica
>
>
> Key: HBASE-18070
> URL: https://issues.apache.org/jira/browse/HBASE-18070
> Project: HBase
> Issue Type: New Feature
>Reporter: Hua Xiang
>Assignee: Huaxiang Sun
>Priority: Critical
> Fix For: 3.0.0-alpha-1, HBASE-18070, HBASE-18070.branch-2, 2.5.0
>
>
> Based on the current doc, memstore replication is not enabled for meta
> replica. Memstore replication will be a good improvement for meta replica.
> Create jira to track this effort (feasibility, design, implementation, etc).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-18070) Enable memstore replication for meta replica
[
https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233167#comment-17233167
]
Huaxiang Sun edited comment on HBASE-18070 at 11/17/20, 12:25 AM:
--
{quote}So I think what we need to test here, is to write a special test which
will clearRegionLocationCache everytime before issuing the actual read/write
scan request, to simulate bad client behavior. The cluster without this feature
should be dead soon and the cluster with this feature should perform much
better. Of course if you put too many loads then no cluster could be alive, but
we only need to prove that, we can perform better.
{quote}
[~zhangduo], designing this test is complicated? As Region Server will push
back when it is under load and the client needs to account for how many errors
it run into with meta lookup. Right now, in the PE test, every meta lookup will
go through the meta region (by passing local table cache). We are planning to
increase the meta lookup load and get some testing data. This test is similar
to what you suggested (sorry, this is what you acked above).
We are working on it and hopefully will deliver some data in the following
couple days. At the same time, can we get +1 from you to merge into 2.4? Thanks.
was (Author: huaxiangsun):
{quote}So I think what we need to test here, is to write a special test which
will clearRegionLocationCache everytime before issuing the actual read/write
scan request, to simulate bad client behavior. The cluster without this feature
should be dead soon and the cluster with this feature should perform much
better. Of course if you put too many loads then no cluster could be alive, but
we only need to prove that, we can perform better.
{quote}
[~zhangduo], designing this test is complicated? As Region Server will push
back when it is under load and the client needs to account for how many errors
it run into with meta lookup. Right now, in the PE test, every meta lookup will
go through the meta region (by passing local table cache). We are planning to
increase the meta lookup load and get some testing data. This test is similar
to what you suggested (sorry, this is what you acked above).
We are working on it and hopefully will deliver some data in the following
couple data. At the same time, can we get +1 from you to merge into 2.4? Thanks.
> Enable memstore replication for meta replica
>
>
> Key: HBASE-18070
> URL: https://issues.apache.org/jira/browse/HBASE-18070
> Project: HBase
> Issue Type: New Feature
>Reporter: Hua Xiang
>Assignee: Huaxiang Sun
>Priority: Critical
> Fix For: 3.0.0-alpha-1, HBASE-18070, HBASE-18070.branch-2, 2.5.0
>
>
> Based on the current doc, memstore replication is not enabled for meta
> replica. Memstore replication will be a good improvement for meta replica.
> Create jira to track this effort (feasibility, design, implementation, etc).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-18070) Enable memstore replication for meta replica
[
https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233167#comment-17233167
]
Huaxiang Sun edited comment on HBASE-18070 at 11/17/20, 12:22 AM:
--
{quote}So I think what we need to test here, is to write a special test which
will clearRegionLocationCache everytime before issuing the actual read/write
scan request, to simulate bad client behavior. The cluster without this feature
should be dead soon and the cluster with this feature should perform much
better. Of course if you put too many loads then no cluster could be alive, but
we only need to prove that, we can perform better.
{quote}
[~zhangduo], designing this test is complicated? As Region Server will push
back when it is under load and the client needs to account for how many errors
it run into with meta lookup. Right now, in the PE test, every meta lookup will
go through the meta region (by passing local table cache). We are planning to
increase the meta lookup load and get some testing data. This test is similar
to what you suggested (sorry, this is what you acked above).
We are working on it and hopefully will deliver some data in the following
couple data. At the same time, can we get +1 from you to merge into 2.4? Thanks.
was (Author: huaxiangsun):
{quote}So I think what we need to test here, is to write a special test which
will clearRegionLocationCache everytime before issuing the actual read/write
scan request, to simulate bad client behavior. The cluster without this feature
should be dead soon and the cluster with this feature should perform much
better. Of course if you put too many loads then no cluster could be alive, but
we only need to prove that, we can perform better.
{quote}
[~zhangduo], designing this test is complicated? As Region Server will push
back when it is under load and the client needs to account for how many errors
it run into with meta lookup. Right now, in the PE test, every meta lookup will
go through the meta region (by passing local table cache). We are planning to
increase the meta lookup load and get some testing data. This test is similar
to what you suggested (sorry, this is what you acked above).
> Enable memstore replication for meta replica
>
>
> Key: HBASE-18070
> URL: https://issues.apache.org/jira/browse/HBASE-18070
> Project: HBase
> Issue Type: New Feature
>Reporter: Hua Xiang
>Assignee: Huaxiang Sun
>Priority: Critical
> Fix For: 3.0.0-alpha-1, HBASE-18070, HBASE-18070.branch-2, 2.5.0
>
>
> Based on the current doc, memstore replication is not enabled for meta
> replica. Memstore replication will be a good improvement for meta replica.
> Create jira to track this effort (feasibility, design, implementation, etc).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-18070) Enable memstore replication for meta replica
[
https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233167#comment-17233167
]
Huaxiang Sun edited comment on HBASE-18070 at 11/17/20, 12:17 AM:
--
{quote}So I think what we need to test here, is to write a special test which
will clearRegionLocationCache everytime before issuing the actual read/write
scan request, to simulate bad client behavior. The cluster without this feature
should be dead soon and the cluster with this feature should perform much
better. Of course if you put too many loads then no cluster could be alive, but
we only need to prove that, we can perform better.
{quote}
[~zhangduo], designing this test is complicated? As Region Server will push
back when it is under load and the client needs to account for how many errors
it run into with meta lookup. Right now, in the PE test, every meta lookup will
go through the meta region (by passing local table cache). We are planning to
increase the meta lookup load and get some testing data. This test is similar
to what you suggested (sorry, this is what you acked above).
was (Author: huaxiangsun):
{quote}So I think what we need to test here, is to write a special test which
will clearRegionLocationCache everytime before issuing the actual read/write
scan request, to simulate bad client behavior. The cluster without this feature
should be dead soon and the cluster with this feature should perform much
better. Of course if you put too many loads then no cluster could be alive, but
we only need to prove that, we can perform better.
{quote}
[~zhangduo], designing this test is complicated? As Region Server will push
back when it is under load and the client needs to account for how many errors
it run into with meta lookup. Right now, in the PE test, every meta lookup will
go through the meta region (by passing local table cache). We are planning to
increase the meta lookup load and get some testing data. This test is similar
to what you suggested, what do you think?
> Enable memstore replication for meta replica
>
>
> Key: HBASE-18070
> URL: https://issues.apache.org/jira/browse/HBASE-18070
> Project: HBase
> Issue Type: New Feature
>Reporter: Hua Xiang
>Assignee: Huaxiang Sun
>Priority: Critical
> Fix For: 3.0.0-alpha-1, HBASE-18070, HBASE-18070.branch-2, 2.5.0
>
>
> Based on the current doc, memstore replication is not enabled for meta
> replica. Memstore replication will be a good improvement for meta replica.
> Create jira to track this effort (feasibility, design, implementation, etc).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-18070) Enable memstore replication for meta replica
[
https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233167#comment-17233167
]
Huaxiang Sun edited comment on HBASE-18070 at 11/17/20, 12:16 AM:
--
{quote}So I think what we need to test here, is to write a special test which
will clearRegionLocationCache everytime before issuing the actual read/write
scan request, to simulate bad client behavior. The cluster without this feature
should be dead soon and the cluster with this feature should perform much
better. Of course if you put too many loads then no cluster could be alive, but
we only need to prove that, we can perform better.
{quote}
[~zhangduo], designing this test is complicated? As Region Server will push
back when it is under load and the client needs to account for how many errors
it run into with meta lookup. Right now, in the PE test, every meta lookup will
go through the meta region (by passing local table cache). We are planning to
increase the meta lookup load and get some testing data. This test is similar
to what you suggested, what do you think?
was (Author: huaxiangsun):
{quote}So I think what we need to test here, is to write a special test which
will clearRegionLocationCache everytime before issuing the actual read/write
scan request, to simulate bad client behavior. The cluster without this feature
should be dead soon and the cluster with this feature should perform much
better. Of course if you put too many loads then no cluster could be alive, but
we only need to prove that, we can perform better.
{quote}
[~zhangduo], designing this test is complicated? As Region Server will push
back when it is under load and the client needs to account for how many errors
it run into with meta lookup. Right now, in the PE test, every meta lookup will
go through the meta region (by passing local table cache). We are planning to
increase the meta lookup load and get some testing data. This test is similar
to what you described, what do you think?
> Enable memstore replication for meta replica
>
>
> Key: HBASE-18070
> URL: https://issues.apache.org/jira/browse/HBASE-18070
> Project: HBase
> Issue Type: New Feature
>Reporter: Hua Xiang
>Assignee: Huaxiang Sun
>Priority: Critical
> Fix For: 3.0.0-alpha-1, HBASE-18070, HBASE-18070.branch-2, 2.5.0
>
>
> Based on the current doc, memstore replication is not enabled for meta
> replica. Memstore replication will be a good improvement for meta replica.
> Create jira to track this effort (feasibility, design, implementation, etc).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-18070) Enable memstore replication for meta replica
[
https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232869#comment-17232869
]
Michael Stack edited comment on HBASE-18070 at 11/16/20, 4:20 PM:
--
{quote}I completely agree with [~zhangduo] on this point. It would be good to
choose between an "any replica" or "primary first" access pattern. In my
opinion we could have made that a follow up issue for improvement. I.e. this
feature is the main work; this alternate mode is a modest increment.
{quote}
I didn't disagree. I encouraged it.
{quote}I wish this discussion had taken place closer to when we agreed to hold
the 2.4 RC.
{quote}
Me too.
Only, there is no discussion here though, just a repeat of the failed
communication pattern (even same phrasings).
I've agreed to work on Duo's asks. I just wanted to do it after the merge so
this feature could make 2.4 and so I could shed the burden of keeping two
feature branches in sync.
was (Author: stack):
{quote}I completely agree with [~zhangduo] on this point. It would be good to
choose between an "any replica" or "primary first" access pattern. In my
opinion we could have made that a follow up issue for improvement. I.e. this
feature is the main work; this alternate mode is a modest increment.
{quote}
I didn't disagree. I encouraged it.
{quote}I wish this discussion had taken place closer to when we agreed to hold
the 2.4 RC.
{quote}
Me too.
Only, there is no discussion here though, just a repeat of failed the
communication pattern (even same phrasings).
I've agreed to work on Duo's asks. I just wanted to do it after the merge so
this feature could make 2.4 and so I could shed the burden of keeping two
feature branches in sync.
> Enable memstore replication for meta replica
>
>
> Key: HBASE-18070
> URL: https://issues.apache.org/jira/browse/HBASE-18070
> Project: HBase
> Issue Type: New Feature
>Reporter: Hua Xiang
>Assignee: Huaxiang Sun
>Priority: Critical
> Fix For: 3.0.0-alpha-1, HBASE-18070, HBASE-18070.branch-2, 2.5.0
>
>
> Based on the current doc, memstore replication is not enabled for meta
> replica. Memstore replication will be a good improvement for meta replica.
> Create jira to track this effort (feasibility, design, implementation, etc).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-18070) Enable memstore replication for meta replica
[
https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232550#comment-17232550
]
Michael Stack edited comment on HBASE-18070 at 11/16/20, 6:03 AM:
--
{quote}Where is the ‘HedgeRead’ word in my comments above? I was also talking
about the newly added load balance mode of meta location look up.
{quote}
I use 'hedged read' as a shorthand for what you describe as: "The general read
replica feature is not designed for performance either, it is designed for
availability. And for meta replica, I think what we need is to verify that, our
cluster could still be functional under heavy load on loaction lookup while the
cluster without this feature on will be dead."; i.e. a feature that has been in
the code base for years.
As I read it, you want a revisit of the original 'general read replica'
feature; proof it improves availability when the work here takes it as a basis,
building on "HBASE-10070 _HBase read high-availability using
timeline-consistent region replicas_" though this projet is about adding
liveness on existing hbase:meta read replica with a concern for load
distribution.
Regards tests for the 'general read replica', there are unit tests as you know
and it is enabled at my place of work apparently effective regards HA (as
described in an old hbasecon talk here
[https://www.youtube.com/watch?v=l6S-Vbs9WsU).]
I'll be happy to work on the test you suggest but do not think it should hold
up our committing this work. -1 if you disagree.
{quote}{quote}And I think here we could do much better than ‘HedgeRead’
solution.
{quote}{quote}
Sounds good. Lets make a follow-on issue?
was (Author: stack):
{quote}Where is the ‘HedgeRead’ word in my comments above? I was also talking
about the newly added load balance mode of meta location look up.
{quote}
I use 'hedged read' as a shorthand for what you describe as: "The general read
replica feature is not designed for performance either, it is designed for
availability. And for meta replica, I think what we need is to verify that, our
cluster could still be functional under heavy load on loaction lookup while the
cluster without this feature on will be dead."; i.e. a feature that has been in
the code base for years.
As I read it, you want a revisit of the original 'general read replica'
feature; proof it improves availability when the work here takes it as a basis,
building on "HBASE-10070 _HBase read high-availability using
timeline-consistent region replicas_" adding liveness on existing hbase:meta
read replica with a concern for load distribution.
Regards tests, there are unit tests as you know and it is enabled at my place
of work apparently effective regards HA (as described in an old hbasecon talk
here [https://www.youtube.com/watch?v=l6S-Vbs9WsU).]
I'll be happy to work on the test you suggest but do not think it should hold
up our committing this work. -1 if you disagree.
{quote}bq.And I think here we could do much better than ‘HedgeRead’ solution.
{quote}
Sounds good. Lets make a follow-on issue?
> Enable memstore replication for meta replica
>
>
> Key: HBASE-18070
> URL: https://issues.apache.org/jira/browse/HBASE-18070
> Project: HBase
> Issue Type: New Feature
>Reporter: Hua Xiang
>Assignee: Huaxiang Sun
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.4.0, HBASE-18070, HBASE-18070.branch-2
>
>
> Based on the current doc, memstore replication is not enabled for meta
> replica. Memstore replication will be a good improvement for meta replica.
> Create jira to track this effort (feasibility, design, implementation, etc).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-18070) Enable memstore replication for meta replica
[
https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232521#comment-17232521
]
Michael Stack edited comment on HBASE-18070 at 11/16/20, 4:49 AM:
--
{quote}This meta replica Load Balance mode should increase throughput as well.
The previous design is for High availability, so it is not for throughput. In
this new mode, load balance is introduced so read is offloaded mostly from
primary meta and distributed among multiple meta replica regions. In case of
the newly PE test, all location data is static, so if there are enough clients
does location lookup, it could exhaust the non-meta-replica region server, with
meta replica LB, it can support n(replica regon #) * read requests/second.
{quote}
[~huaxiangsun] yes... if enough load but we don't seem to be driving enough
load.. we seem bottlenecked on client, but improving perf is secondary?
{quote}With that said, the PE test should not increase primary read numbers at
all (there is no fall-back-to-primary path in the test), I will take a look
tomorrow to see why there are so many reads going through the primary meta
region, could be something wrong in the path.
{quote}
Yes, would be good to figure where the primary load is coming from. TODO.
[~zhangduo]
{quote}First you said you do not need to prove that the meta lookup request has
been distributed to meta replicas, as it has been done by the Meta Read
Replicas feature which has been done years ago.
{quote}
The 'hedged read' scenario has been in place forever, yes.
{quote}Then you said the feature is about distributing load of meta, which
directly objects your first argument?
{quote}
Distributing load is the 'LoadBalance' configuration, which is new and not the
'HedgedRead'.
{quote}Is it because my English is too bad? Could anyone help me to understand
better? Thanks a lot.
{quote}
Keep asking questions. I'll try and explain. I may not be doing a good job of
it. My impression is that you are want us to make reports around HA which would
be good to have but not the thrust of this issue. I think it would be good to
do but don't want it to be obstacle to commit. Thanks.
was (Author: stack):
{quote}This meta replica Load Balance mode should increase throughput as well.
The previous design is for High availability, so it is not for throughput. In
this new mode, load balance is introduced so read is offloaded mostly from
primary meta and distributed among multiple meta replica regions. In case of
the newly PE test, all location data is static, so if there are enough clients
does location lookup, it could exhaust the non-meta-replica region server, with
meta replica LB, it can support n(replica regon #) * read requests/second.
{quote}
[~huaxiangsun] yes... if enough load but we don't seem to be driving enough
load.. we seem bottlenecked on client, but improving perf is secondary?
{quote}With that said, the PE test should not increase primary read numbers at
all (there is no fall-back-to-primary path in the test), I will take a look
tomorrow to see why there are so many reads going through the primary meta
region, could be something wrong in the path.
{quote}
Yes, would be good to figure where the primary load is coming from. TODO.
[~zhangduo]
{quote}First you said you do not need to prove that the meta lookup request has
been distributed to meta replicas, as it has been done by the Meta Read
Replicas feature which has been done years ago.
{quote}
The 'hedged read' scenario has been in place forever, yes.
{quote}Then you said the feature is about distributing load of meta, which
directly objects your first argument?
{quote}
Distributing load is the 'LoadBalance' configuration, which is new and not the
'HedgedRead'.
{quote}Is it because my English is too bad? Could anyone help me to understand
better? Thanks a lot.
{quote}
Keep asking questions. I'll try and explain. My impression is that you are want
us to make reports around HA which would be good to have but not the thrust of
this issue. I think it would be good to do but don't want it to be obstacle to
commit. Thanks.
> Enable memstore replication for meta replica
>
>
> Key: HBASE-18070
> URL: https://issues.apache.org/jira/browse/HBASE-18070
> Project: HBase
> Issue Type: New Feature
>Reporter: Hua Xiang
>Assignee: Huaxiang Sun
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.4.0, HBASE-18070, HBASE-18070.branch-2
>
>
> Based on the current doc, memstore replication is not enabled for meta
> replica. Memstore replication will be a good improvement for meta replica.
> Create jira to track this effort (feasibility, design, implementation, etc).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-18070) Enable memstore replication for meta replica
[ https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230167#comment-17230167 ] Andrew Kyle Purtell edited comment on HBASE-18070 at 11/11/20, 7:01 PM: [~stack] [~huaxiangsun] Can you complete one round of testing by the end of the week? Sufficient to set in your mind the answer to the merge/don't merge question? Assuming a positive result, we can then do the merge, into 2.4. The RC process will allow more time for tests to be run, by more people. You asked for an extension to get this into 2.4 and I would like to honor that but time is running out to complete an RC by the end of this month. Having waited this long another couple of days is fine, but beyond that will be a miss and a wait for no result, which is not the outcome I would like to see here. was (Author: apurtell): [~stack] [~huaxiangsun] Can you complete one round of testing by the end of the week? The RC process will allow more time for tests to be run, by more people. You asked for an extension to get this into 2.4 and I would like to honor that but time is running out to complete an RC by the end of this month. Having waited this long another couple of days is fine, but beyond that will be a miss and a wait for no result, which is not the outcome I would like to see here. > Enable memstore replication for meta replica > > > Key: HBASE-18070 > URL: https://issues.apache.org/jira/browse/HBASE-18070 > Project: HBase > Issue Type: New Feature >Reporter: Hua Xiang >Assignee: Huaxiang Sun >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.4.0, HBASE-18070, HBASE-18070.branch-2 > > > Based on the current doc, memstore replication is not enabled for meta > replica. Memstore replication will be a good improvement for meta replica. > Create jira to track this effort (feasibility, design, implementation, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-18070) Enable memstore replication for meta replica
[ https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215613#comment-17215613 ] Huaxiang Sun edited comment on HBASE-18070 at 10/16/20, 7:45 PM: - [~apurtell] , We are working on the client side change at this moment. [~stack] has the source side changes committed to HBASE-18070 branch. HBASE-25125 changes is out for review, [~zhangduo]b and [~stack] had some comments. I agreed with Duo that client side change is more important to verify that load balancing mode for meta replica works. Trying to get the first cut for client side change out for review sometime next week, thanks. was (Author: huaxiangsun): [~apurtell] , We are working on the client side change at this moment. [~stack] has the source side changes committed to HBASE-18070 branch. HBASE-25125 changes is out for review, [~zhangduo]b and [~stack] made comments. I agreed with Duo that client side change is more important to verify that load balancing mode for meta replica works. Trying to get the first cut for client side change out for review sometime next week, thanks. > Enable memstore replication for meta replica > > > Key: HBASE-18070 > URL: https://issues.apache.org/jira/browse/HBASE-18070 > Project: HBase > Issue Type: New Feature >Reporter: Hua Xiang >Assignee: Huaxiang Sun >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Based on the current doc, memstore replication is not enabled for meta > replica. Memstore replication will be a good improvement for meta replica. > Create jira to track this effort (feasibility, design, implementation, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-18070) Enable memstore replication for meta replica
[ https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215597#comment-17215597 ] Andrew Kyle Purtell edited comment on HBASE-18070 at 10/16/20, 6:39 PM: [~zhangduo] I think "we" is colleagues of Stack (at Apple) and Huaxiang (?). I might also like to enable META replicas at my employer (Salesforce) someday. Its fine to mention there may be users of a feature. On the 2.4 release discussion thread on dev@ I think we (the HBase community) have agreed I'll serve as RM for 2.4.0 and I'm fine waiting the estimated week or so for this to land. Please let me know if there are objections to either the wait or the commit of this work. was (Author: apurtell): [~zhangduo] I think "we" is colleagues of Stack (at Apple) and Huaxiang (?). I might also like to enable META replicas at my employer (Salesforce) someday. Its fine to mention there may be users of a feature. On the 2.4 release discussion thread on dev@ I think we (the HBase community) have agreed I'll serve as RM for 2.4.0 and I'm fine waiting the estimated week or so for this to land. > Enable memstore replication for meta replica > > > Key: HBASE-18070 > URL: https://issues.apache.org/jira/browse/HBASE-18070 > Project: HBase > Issue Type: New Feature >Reporter: Hua Xiang >Assignee: Huaxiang Sun >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Based on the current doc, memstore replication is not enabled for meta > replica. Memstore replication will be a good improvement for meta replica. > Create jira to track this effort (feasibility, design, implementation, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-18070) Enable memstore replication for meta replica
[ https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215597#comment-17215597 ] Andrew Kyle Purtell edited comment on HBASE-18070 at 10/16/20, 6:39 PM: [~zhangduo] I think "we" is colleagues of Stack (at Apple) and Huaxiang (?). I might also like to enable META replicas at my employer (Salesforce) someday. Its fine to mention there may be users of a feature. On the 2.4 release discussion thread on dev@ I think we (the HBase community) have agreed I'll serve as RM for 2.4.0 and I'm fine waiting the estimated week or so for this to land. was (Author: apurtell): [~zhangduo] I think "we" is colleagues of Stack (at Apple) and Huaxiang (?). I might also like to enable META replicas at my employer (Salesforce) someday. Its fine to mention there may be users of a feature, that's why we develop them. On the thread I think we (the HBase community) have agreed I'll serve as RM for 2.4.0 and I'm fine waiting the estimated week or so for this to land. > Enable memstore replication for meta replica > > > Key: HBASE-18070 > URL: https://issues.apache.org/jira/browse/HBASE-18070 > Project: HBase > Issue Type: New Feature >Reporter: Hua Xiang >Assignee: Huaxiang Sun >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Based on the current doc, memstore replication is not enabled for meta > replica. Memstore replication will be a good improvement for meta replica. > Create jira to track this effort (feasibility, design, implementation, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-18070) Enable memstore replication for meta replica
[ https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215597#comment-17215597 ] Andrew Kyle Purtell edited comment on HBASE-18070 at 10/16/20, 6:35 PM: [~zhangduo] I think "we" is colleagues of Stack (at Apple) and Huaxiang (?). I might also like to enable META replicas at my employer (Salesforce) someday. Its fine to mention there may be users of a feature, that's why we develop them. On the thread I think we (the HBase community) have agreed I'll serve as RM for 2.4.0 and I'm fine waiting the estimated week or so for this to land. was (Author: apurtell): [~zhangduo] I think "we" is colleagues of Stack (at Apache) and Huaxiang (?). I might also like to enable META replicas at my employer (Salesforce) someday. Its fine to mention there may be users of a feature, that's why we develop them. On the thread I think we (the HBase community) have agreed I'll serve as RM for 2.4.0 and I'm fine waiting the estimated week or so for this to land. > Enable memstore replication for meta replica > > > Key: HBASE-18070 > URL: https://issues.apache.org/jira/browse/HBASE-18070 > Project: HBase > Issue Type: New Feature >Reporter: Hua Xiang >Assignee: Huaxiang Sun >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Based on the current doc, memstore replication is not enabled for meta > replica. Memstore replication will be a good improvement for meta replica. > Create jira to track this effort (feasibility, design, implementation, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-18070) Enable memstore replication for meta replica
[ https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177871#comment-17177871 ] Huaxiang Sun edited comment on HBASE-18070 at 8/14/20, 4:46 PM: Thanks [~zhangduo] for comments. Yeah, we actually plan to remove zookeeper dependency as it is not needed. Please see the in-progress doc for rough ideas. Will merge this into the doc [~stack] posted when it is complete and then open for review/comments. https://docs.google.com/document/d/1qOxr3svNe-Wv55dccgYFEUrUgHDDAp9KFTJab4i3I1s/edit?usp=sharing was (Author: huaxiangsun): Thanks [~zhangduo] for comments. Yeah, we actually plan to remove zookeeper dependency as it is not needed. Please see the in-progress [https://docs.google.com/document/d/1qOxr3svNe-Wv55dccgYFEUrUgHDDAp9KFTJab4i3I1s/edit#] for rough ideas. Will merge this into the doc [~stack] posted when it is complete and then open for review/comments. > Enable memstore replication for meta replica > > > Key: HBASE-18070 > URL: https://issues.apache.org/jira/browse/HBASE-18070 > Project: HBase > Issue Type: New Feature >Reporter: Hua Xiang >Assignee: Huaxiang Sun >Priority: Major > > Based on the current doc, memstore replication is not enabled for meta > replica. Memstore replication will be a good improvement for meta replica. > Create jira to track this effort (feasibility, design, implementation, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-18070) Enable memstore replication for meta replica
[ https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168331#comment-17168331 ] Huaxiang Sun edited comment on HBASE-18070 at 7/31/20, 2:02 AM: Yeah, we checked the phase 2 design of read replica, there will be no out of order for meta replica replication. It will only be used for region location for hbase client, not something else. For master's view of meta, it can always talk with primary meta replica (just like there is no replica for meta). Most of logic/foundation has been laid out by phase 2 of read replica, we are trying to put some glue logic for meta and some optimizations for this special case. was (Author: huaxiangsun): Yeah, we checked the phase 2 design of read replica, there will be no out of order for meta replica. Yeah, it will only be used for region location for hbase client, not something else. For master's view of meta, it can always talk with primary meta replica (just like there is no replica for meta). Most of logic/foundation has been laid out by phase 2 of read replica, we are trying to put some glue logic for meta and some optimizations for this special case. > Enable memstore replication for meta replica > > > Key: HBASE-18070 > URL: https://issues.apache.org/jira/browse/HBASE-18070 > Project: HBase > Issue Type: New Feature >Reporter: Hua Xiang >Assignee: Huaxiang Sun >Priority: Major > > Based on the current doc, memstore replication is not enabled for meta > replica. Memstore replication will be a good improvement for meta replica. > Create jira to track this effort (feasibility, design, implementation, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-18070) Enable memstore replication for meta replica
[ https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023904#comment-16023904 ] huaxiang sun edited comment on HBASE-18070 at 5/24/17 11:52 PM: Hi [~enis], I spent some time on this. As you suggested, the change is not that complicated. What I plan to do is that in RegionReplicaReplicationEndpoint#getWALEntryfilter(), replace the default SystemTableWALEntryFilter with a customer filter allowing meta WalEntry when memstore replication for meta table is enabled. Will prototype and test. was (Author: huaxiang): Hi [~enis], I spent some time on this. As you suggested, the change is not that complicated. What I plan to do is that in RegionReplicaReplicationEndpoint#getWALEntryfilter(), replace the default SystemTableWALEntryFilter with a customer filter allow meta WalEntry when memstore replication for meta table is enabled. Will prototype and test. > Enable memstore replication for meta replica > > > Key: HBASE-18070 > URL: https://issues.apache.org/jira/browse/HBASE-18070 > Project: HBase > Issue Type: New Feature >Reporter: huaxiang sun >Assignee: huaxiang sun > > Based on the current doc, memstore replication is not enabled for meta > replica. Memstore replication will be a good improvement for meta replica. > Create jira to track this effort (feasibility, design, implementation, etc). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
