[jira] [Commented] (HBASE-25761) POC: hbase:meta,,1 as ROOT
[ https://issues.apache.org/jira/browse/HBASE-25761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1737#comment-1737 ] Francis Christopher Liu commented on HBASE-25761: - [~stack] Sounds good I'll integrate it into the doc. Online meeting sounds good too. I think the meeting time in PST might need to be an hour later to accomodate the difference in hour that DST introduces. > POC: hbase:meta,,1 as ROOT > -- > > Key: HBASE-25761 > URL: https://issues.apache.org/jira/browse/HBASE-25761 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Assignee: Francis Christopher Liu >Priority: Major > > One of the proposals up in the split-meta design doc suggests a > sleight-of-hand where the current hard-coded hbase:meta,,1 Region is > leveraged to serve as first Region of a split hbase:meta but also does > double-duty as 'ROOT'. This suggestion was put aside as a complicating > recursion in chat but then Francis noticed on a re-read of the BigTable > paper, that this is how they describe they do 'ROOT': "The root tablet is > just the first tablet in the METADATA table, but is treated specially -- it > is never split..." > This issue is for playing around with this notion to see what the problems > are so can do a better description of this approach here, in the design: > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit?ts=606c120f#heading=h.ikbhxlcthjle -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25761) POC: hbase:meta,,1 as ROOT
[ https://issues.apache.org/jira/browse/HBASE-25761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367137#comment-17367137 ] Francis Christopher Liu commented on HBASE-25761: - I see, thanks for clarifying. I think I do understand, I just don't think it would go down the route of dying for reasons mention earlier: possibly enabling it by default in all or subset of tests, removing the compatibility switch and then there's adoption. In general I think the death of a feature is because of a mix of lack of support and adoption. Since we are guaranteeing rolling upgradeabbility from 2.x it is something that I think all implementations would run into, perhaps maybe a little more for single meta region than the others but arguably inconsequential from this perspective (eg the code for multiple meta is not exercised). Perhaps this is something we can discuss further in a video call sync-up with [~stack] and hopefully a bunch of other folks that are interested? I think we should have a sync-up discussion for split meta in general anyway. > POC: hbase:meta,,1 as ROOT > -- > > Key: HBASE-25761 > URL: https://issues.apache.org/jira/browse/HBASE-25761 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Assignee: Francis Christopher Liu >Priority: Major > > One of the proposals up in the split-meta design doc suggests a > sleight-of-hand where the current hard-coded hbase:meta,,1 Region is > leveraged to serve as first Region of a split hbase:meta but also does > double-duty as 'ROOT'. This suggestion was put aside as a complicating > recursion in chat but then Francis noticed on a re-read of the BigTable > paper, that this is how they describe they do 'ROOT': "The root tablet is > just the first tablet in the METADATA table, but is treated specially -- it > is never split..." > This issue is for playing around with this notion to see what the problems > are so can do a better description of this approach here, in the design: > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit?ts=606c120f#heading=h.ikbhxlcthjle -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25761) POC: hbase:meta,,1 as ROOT
[
https://issues.apache.org/jira/browse/HBASE-25761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17364723#comment-17364723
]
Francis Christopher Liu commented on HBASE-25761:
-
{quote}
The difference comparing to a general root table and master local region here
is that, even we do not split meta, the root table and master local region is
there, most of the code will be executed. For example, the tiered assignment
and initialization.
{quote}
Yes and I think more importantly applying a different perspective there's new
operational complexity exposed to the operator (and possibly the user).
{quote}
In general, for me, the first meta as root solution is very like a compromise,
you need to do lots of magics in the row key and comparator, just like what I
have done in the master local region implementation, as we will also store
procedure data in the same region. But at least, the master local region will
use different families to store the data, so when reading we do not need to add
special filters...
{quote}
I see. From my perspective the difference in logic between _first meta region_
and _root table_ implementation is fairly straightforward, the composite
comparator has fairly simple logic, route based on presence/absence of prefix
character. For data access the key just needs to be prepended with the prefix
character, I haven't thought of a case yet wherein we'd need a specialized
filter (the rough PoC doesn't have one). The prefixing approach is also a
common end-user design pattern to achieve spatial locality for keys. Although
yes it would be a bit cleaner if the data would be in a separate hbase:root
table but I felt there's a good chance the community might see that (IMHO
small) difference a reasonable tradeoff for its compelling backward
compatibility characteristics. Hence it felt reasonable enough to include in
the discussion and hear what people think.
{quote}
And about the reviewing the implementing, trust me, no matter root table or
master local region, there are no big differences on the implementing steps.
{quote}
I see. I think there seems to be a significant difference between _first meta
region_ and _root table_ such that the former would be smoother (eg includes
the incremental disruption the steps produce) to get in wether it is a a big
difference across all three proposed solutions that I have not looked into.
Since you don't see any difference and this currently isn't a deciding factor
we don't really need to delve further unless needed.
> POC: hbase:meta,,1 as ROOT
> --
>
> Key: HBASE-25761
> URL: https://issues.apache.org/jira/browse/HBASE-25761
> Project: HBase
> Issue Type: Sub-task
>Reporter: Michael Stack
>Assignee: Francis Christopher Liu
>Priority: Major
>
> One of the proposals up in the split-meta design doc suggests a
> sleight-of-hand where the current hard-coded hbase:meta,,1 Region is
> leveraged to serve as first Region of a split hbase:meta but also does
> double-duty as 'ROOT'. This suggestion was put aside as a complicating
> recursion in chat but then Francis noticed on a re-read of the BigTable
> paper, that this is how they describe they do 'ROOT': "The root tablet is
> just the first tablet in the METADATA table, but is treated specially -- it
> is never split..."
> This issue is for playing around with this notion to see what the problems
> are so can do a better description of this approach here, in the design:
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit?ts=606c120f#heading=h.ikbhxlcthjle
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25761) POC: hbase:meta,,1 as ROOT
[
https://issues.apache.org/jira/browse/HBASE-25761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17364071#comment-17364071
]
Francis Christopher Liu edited comment on HBASE-25761 at 6/16/21, 5:41 AM:
---
{quote}
If we do not split, it just perform like before, so for most cases where meta
is not too large, we can not test the new code. It is easy to finally lead the
feature to die as the most easy way to fix the problem is to not split meta...
We have purged several features in the past already...
{quote}
I see, reasonable concerns from a testing and feature adoption perspective. The
thought was to be backward compatible and have split meta off by default to be
rolling upgradeable from 2.x to 3.x as defined in the requirements section of
the [design
doc|https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit?ts=606c120f#heading=h.hlxgjlyq79ge].
If there is such a switch then we could also use that switch to split the
original first meta region (having a total of 2 meta regions) when it is turned
on. And for testing perhaps we could turn it on by default or turn it on in
cases that need it? Removing the switch would then end support for single meta
case.
{quote}
And for reviewing and checking in, I do not think it is a big deal, I've been
implementing almost the whole feature on a feature branch already, not very
hard to do it step by step.
{quote}
I see in this case you are comparing it against the
[master-local|https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit?ts=606c120f#heading=h.djjy98rqkyig]
implementation and not the [hbase:root
table|https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit?ts=606c120f#heading=h.9lp7l2j5mefg]
one.
was (Author: toffer):
{quote}
If we do not split, it just perform like before, so for most cases where meta
is not too large, we can not test the new code. It is easy to finally lead the
feature to die as the most easy way to fix the problem is to not split meta...
We have purged several features in the past already...
{quote}
I see, reasonable concerns from a testing and feature adoption perspective. The
thought was to be backward compatible and have split meta off by default to be
rolling upgradeable from 2.x to 3.x as defined in the [requirements
doc|https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit?ts=606c120f#heading=h.hlxgjlyq79ge].
If there is such a switch then we could also use that switch to split the
original first meta region (having a total of 2 meta regions) when it is turned
on. And for testing perhaps we could turn it on by default or turn it on in
cases that need it? Removing the switch would then end support for single meta.
{quote}
And for reviewing and checking in, I do not think it is a big deal, I've been
implementing almost the whole feature on a feature branch already, not very
hard to do it step by step.
{quote}
I see in this case you are comparing it against the
[master-local|https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit?ts=606c120f#heading=h.djjy98rqkyig]
implementation and not the [hbase:root
table|https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit?ts=606c120f#heading=h.9lp7l2j5mefg]
one.
> POC: hbase:meta,,1 as ROOT
> --
>
> Key: HBASE-25761
> URL: https://issues.apache.org/jira/browse/HBASE-25761
> Project: HBase
> Issue Type: Sub-task
>Reporter: Michael Stack
>Assignee: Francis Christopher Liu
>Priority: Major
>
> One of the proposals up in the split-meta design doc suggests a
> sleight-of-hand where the current hard-coded hbase:meta,,1 Region is
> leveraged to serve as first Region of a split hbase:meta but also does
> double-duty as 'ROOT'. This suggestion was put aside as a complicating
> recursion in chat but then Francis noticed on a re-read of the BigTable
> paper, that this is how they describe they do 'ROOT': "The root tablet is
> just the first tablet in the METADATA table, but is treated specially -- it
> is never split..."
> This issue is for playing around with this notion to see what the problems
> are so can do a better description of this approach here, in the design:
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit?ts=606c120f#heading=h.ikbhxlcthjle
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25761) POC: hbase:meta,,1 as ROOT
[
https://issues.apache.org/jira/browse/HBASE-25761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17364071#comment-17364071
]
Francis Christopher Liu commented on HBASE-25761:
-
{quote}
If we do not split, it just perform like before, so for most cases where meta
is not too large, we can not test the new code. It is easy to finally lead the
feature to die as the most easy way to fix the problem is to not split meta...
We have purged several features in the past already...
{quote}
I see, reasonable concerns from a testing and feature adoption perspective. The
thought was to be backward compatible and have split meta off by default to be
rolling upgradeable from 2.x to 3.x as defined in the [requirements
doc|https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit?ts=606c120f#heading=h.hlxgjlyq79ge].
If there is such a switch then we could also use that switch to split the
original first meta region (having a total of 2 meta regions) when it is turned
on. And for testing perhaps we could turn it on by default or turn it on in
cases that need it? Removing the switch would then end support for single meta.
{quote}
And for reviewing and checking in, I do not think it is a big deal, I've been
implementing almost the whole feature on a feature branch already, not very
hard to do it step by step.
{quote}
I see in this case you are comparing it against the
[master-local|https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit?ts=606c120f#heading=h.djjy98rqkyig]
implementation and not the [hbase:root
table|https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit?ts=606c120f#heading=h.9lp7l2j5mefg]
one.
> POC: hbase:meta,,1 as ROOT
> --
>
> Key: HBASE-25761
> URL: https://issues.apache.org/jira/browse/HBASE-25761
> Project: HBase
> Issue Type: Sub-task
>Reporter: Michael Stack
>Assignee: Francis Christopher Liu
>Priority: Major
>
> One of the proposals up in the split-meta design doc suggests a
> sleight-of-hand where the current hard-coded hbase:meta,,1 Region is
> leveraged to serve as first Region of a split hbase:meta but also does
> double-duty as 'ROOT'. This suggestion was put aside as a complicating
> recursion in chat but then Francis noticed on a re-read of the BigTable
> paper, that this is how they describe they do 'ROOT': "The root tablet is
> just the first tablet in the METADATA table, but is treated specially -- it
> is never split..."
> This issue is for playing around with this notion to see what the problems
> are so can do a better description of this approach here, in the design:
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit?ts=606c120f#heading=h.ikbhxlcthjle
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25761) POC: hbase:meta,,1 as ROOT
[ https://issues.apache.org/jira/browse/HBASE-25761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17364035#comment-17364035 ] Francis Christopher Liu edited comment on HBASE-25761 at 6/16/21, 3:30 AM: --- Thanks for your comment [~zhangduo]. >From what I can gather from the rough PoC that I did was that the salient >difference between this implementation and the root table implementation is >around the theme of backward compatibility. Some of which might seem valuable >to the community hence I wonder what the community thinks: # The compatibility code is built-in as part of the core implementation there is no extra code to support backward compatibility # It introduces no new major artifacts, such that: #* If hbase:meta region is not split it operationally will behave pretty much exactly the same way as before (no operator changes, user changes, etc). As opposed to having to be aware of the new hbase:root region from the get go. #* If hbase:meta region is split. Users/Operators only have to worry about hbase:meta regions (which one would have to in any split meta implementation), no new hbase:root table to worry about. Although yes one would need to be aware of the first meta region getting assigned first. # I think it might also be smoother to get reviewed and checked in #* Intuitively given 1 & 2, I think it might be easier to add things more piece meal vs root table was (Author: toffer): Thanks for your comment [~zhangduo]. >From what I can gather from the rough PoC that I did was that the salient >difference between this implementation and the root table implementation is >around the theme of backward compatibility. Some of which might seem valuable >to the community hence I wonder what the community thinks: 1. The compatibility code is built-in as part of the core implementation there is no extra code to support backward compatibility 2. It introduces no new major artifacts, such that: 2.a If hbase:meta region is not split it operationally will behave pretty much exactly the same way as before (no operator changes, user changes, etc). As opposed to having to be aware of the new hbase:root region from the get go. 2.b If hbase:meta region is split. Users/Operators only have to worry about hbase:meta regions (which one would have to in any split meta implementation), no new hbase:root table to worry about. Although yes one would need to be aware of the first meta region getting assigned first. 3. I think it might also be smoother to get reviewed and checked in 3.a Intuitively given 1 & 2, I think it might be easier to add things more piece meal vs root table > POC: hbase:meta,,1 as ROOT > -- > > Key: HBASE-25761 > URL: https://issues.apache.org/jira/browse/HBASE-25761 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Assignee: Francis Christopher Liu >Priority: Major > > One of the proposals up in the split-meta design doc suggests a > sleight-of-hand where the current hard-coded hbase:meta,,1 Region is > leveraged to serve as first Region of a split hbase:meta but also does > double-duty as 'ROOT'. This suggestion was put aside as a complicating > recursion in chat but then Francis noticed on a re-read of the BigTable > paper, that this is how they describe they do 'ROOT': "The root tablet is > just the first tablet in the METADATA table, but is treated specially -- it > is never split..." > This issue is for playing around with this notion to see what the problems > are so can do a better description of this approach here, in the design: > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit?ts=606c120f#heading=h.ikbhxlcthjle -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25761) POC: hbase:meta,,1 as ROOT
[ https://issues.apache.org/jira/browse/HBASE-25761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17364035#comment-17364035 ] Francis Christopher Liu edited comment on HBASE-25761 at 6/16/21, 3:27 AM: --- Thanks for your comment [~zhangduo]. >From what I can gather from the rough PoC that I did was that the salient >difference between this implementation and the root table implementation is >around the theme of backward compatibility. Some of which might seem valuable >to the community hence I wonder what the community thinks: 1. The compatibility code is built-in as part of the core implementation there is no extra code to support backward compatibility 2. It introduces no new major artifacts, such that: 2.a If hbase:meta region is not split it operationally will behave pretty much exactly the same way as before (no operator changes, user changes, etc). As opposed to having to be aware of the new hbase:root region from the get go. 2.b If hbase:meta region is split. Users/Operators only have to worry about hbase:meta regions (which one would have to in any split meta implementation), no new hbase:root table to worry about. Although yes one would need to be aware of the first meta region getting assigned first. 3. I think it might also be smoother to get reviewed and checked in 3.a Intuitively given 1 & 2, I think it might be easier to add things more piece meal vs root table was (Author: toffer): Thanks for your comment [~zhangduo]. >From what I can gather from the rough PoC that I did was that the salient >difference between this implementation and the root table implementation is >around the theme of backward compatibility. Some of which might seem valuable >to the community hence I wonder what the community thinks: 1. The compatibility code is built-in as part of the core implementation there is no extra code to support backward compatibility 2. It introduces no new major artifacts, such that: - If hbase:meta region is not split it operationally will behave pretty much exactly the same way as before (no operator changes, user changes, etc). As opposed to having to be aware of the new hbase:root region from the get go. - If hbase:meta region is split. Users/Operators only have to worry about hbase:meta regions (which one would have to in any split meta implementation), no new hbase:root table to worry about. Although yes one would need to be aware of the first meta region getting assigned first. 3. I think it might also be smoother to get reviewed and checked in - Intuitively given 1 & 2, I think it might be easier to add things more piece meal vs root table > POC: hbase:meta,,1 as ROOT > -- > > Key: HBASE-25761 > URL: https://issues.apache.org/jira/browse/HBASE-25761 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Assignee: Francis Christopher Liu >Priority: Major > > One of the proposals up in the split-meta design doc suggests a > sleight-of-hand where the current hard-coded hbase:meta,,1 Region is > leveraged to serve as first Region of a split hbase:meta but also does > double-duty as 'ROOT'. This suggestion was put aside as a complicating > recursion in chat but then Francis noticed on a re-read of the BigTable > paper, that this is how they describe they do 'ROOT': "The root tablet is > just the first tablet in the METADATA table, but is treated specially -- it > is never split..." > This issue is for playing around with this notion to see what the problems > are so can do a better description of this approach here, in the design: > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit?ts=606c120f#heading=h.ikbhxlcthjle -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25761) POC: hbase:meta,,1 as ROOT
[ https://issues.apache.org/jira/browse/HBASE-25761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17364035#comment-17364035 ] Francis Christopher Liu commented on HBASE-25761: - Thanks for your comment [~zhangduo]. >From what I can gather from the rough PoC that I did was that the salient >difference between this implementation and the root table implementation is >around the theme of backward compatibility. Some of which might seem valuable >to the community hence I wonder what the community thinks: 1. The compatibility code is built-in as part of the core implementation there is no extra code to support backward compatibility 2. It introduces no new major artifacts, such that: - If hbase:meta region is not split it operationally will behave pretty much exactly the same way as before (no operator changes, user changes, etc). As opposed to having to be aware of the new hbase:root region from the get go. - If hbase:meta region is split. Users/Operators only have to worry about hbase:meta regions (which one would have to in any split meta implementation), no new hbase:root table to worry about. Although yes one would need to be aware of the first meta region getting assigned first. 3. I think it might also be smoother to get reviewed and checked in - Intuitively given 1 & 2, I think it might be easier to add things more piece meal vs root table > POC: hbase:meta,,1 as ROOT > -- > > Key: HBASE-25761 > URL: https://issues.apache.org/jira/browse/HBASE-25761 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Assignee: Francis Christopher Liu >Priority: Major > > One of the proposals up in the split-meta design doc suggests a > sleight-of-hand where the current hard-coded hbase:meta,,1 Region is > leveraged to serve as first Region of a split hbase:meta but also does > double-duty as 'ROOT'. This suggestion was put aside as a complicating > recursion in chat but then Francis noticed on a re-read of the BigTable > paper, that this is how they describe they do 'ROOT': "The root tablet is > just the first tablet in the METADATA table, but is treated specially -- it > is never split..." > This issue is for playing around with this notion to see what the problems > are so can do a better description of this approach here, in the design: > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit?ts=606c120f#heading=h.ikbhxlcthjle -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25761) POC: hbase:meta,,1 as ROOT
[ https://issues.apache.org/jira/browse/HBASE-25761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17363394#comment-17363394 ] Francis Christopher Liu commented on HBASE-25761: - Attached are my findings, I've distilled them in a 1-pager design doc. I hope it's helpful. Comments are very much appreciated. > POC: hbase:meta,,1 as ROOT > -- > > Key: HBASE-25761 > URL: https://issues.apache.org/jira/browse/HBASE-25761 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Assignee: Francis Christopher Liu >Priority: Major > > One of the proposals up in the split-meta design doc suggests a > sleight-of-hand where the current hard-coded hbase:meta,,1 Region is > leveraged to serve as first Region of a split hbase:meta but also does > double-duty as 'ROOT'. This suggestion was put aside as a complicating > recursion in chat but then Francis noticed on a re-read of the BigTable > paper, that this is how they describe they do 'ROOT': "The root tablet is > just the first tablet in the METADATA table, but is treated specially -- it > is never split..." > This issue is for playing around with this notion to see what the problems > are so can do a better description of this approach here, in the design: > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit?ts=606c120f#heading=h.ikbhxlcthjle -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25761) POC: hbase:meta,,1 as ROOT
[ https://issues.apache.org/jira/browse/HBASE-25761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17342956#comment-17342956 ] Francis Christopher Liu commented on HBASE-25761: - Been working on this the past couple of weeks. I'm hacking my existing Root Table branch and converting it to first hbase:meta region as root. I've been making changes to get the bare minimum working for us to better understand the pros/cons. > POC: hbase:meta,,1 as ROOT > -- > > Key: HBASE-25761 > URL: https://issues.apache.org/jira/browse/HBASE-25761 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Assignee: Francis Christopher Liu >Priority: Major > > One of the proposals up in the split-meta design doc suggests a > sleight-of-hand where the current hard-coded hbase:meta,,1 Region is > leveraged to serve as first Region of a split hbase:meta but also does > double-duty as 'ROOT'. This suggestion was put aside as a complicating > recursion in chat but then Francis noticed on a re-read of the BigTable > paper, that this is how they describe they do 'ROOT': "The root tablet is > just the first tablet in the METADATA table, but is treated specially -- it > is never split..." > This issue is for playing around with this notion to see what the problems > are so can do a better description of this approach here, in the design: > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit?ts=606c120f#heading=h.ikbhxlcthjle -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-25761) POC: hbase:meta,,1 as ROOT
[ https://issues.apache.org/jira/browse/HBASE-25761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Christopher Liu reassigned HBASE-25761: --- Assignee: Francis Christopher Liu > POC: hbase:meta,,1 as ROOT > -- > > Key: HBASE-25761 > URL: https://issues.apache.org/jira/browse/HBASE-25761 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Assignee: Francis Christopher Liu >Priority: Major > > One of the proposals up in the split-meta design doc suggests a > sleight-of-hand where the current hard-coded hbase:meta,,1 Region is > leveraged to serve as first Region of a split hbase:meta but also does > double-duty as 'ROOT'. This suggestion was put aside as a complicating > recursion in chat but then Francis noticed on a re-read of the BigTable > paper, that this is how they describe they do 'ROOT': "The root tablet is > just the first tablet in the METADATA table, but is treated specially -- it > is never split..." > This issue is for playing around with this notion to see what the problems > are so can do a better description of this approach here, in the design: > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit?ts=606c120f#heading=h.ikbhxlcthjle -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244405#comment-17244405 ] Francis Christopher Liu edited comment on HBASE-11288 at 12/5/20, 5:38 AM: --- Good idea [~stack]. +1 Let's leave the baggage here so it'll be easier for the community to get involved. was (Author: toffer): Good idea [~stack]. +1 Let's leave the baggage here so it'll be easier for the community to get invovled. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244405#comment-17244405 ] Francis Christopher Liu commented on HBASE-11288: - Good idea [~stack]. +1 Let's leave the baggage here so it'll be easier for the community to get invovled. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25087) Splittable Meta: Generalize handling of meta regions to support splittable meta
Francis Christopher Liu created HBASE-25087: --- Summary: Splittable Meta: Generalize handling of meta regions to support splittable meta Key: HBASE-25087 URL: https://issues.apache.org/jira/browse/HBASE-25087 Project: HBase Issue Type: Sub-task Reporter: Francis Christopher Liu Assignee: Francis Christopher Liu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25056) Splittable Meta: foward port fixes from hbase-2 branch and stabilize unit tests
[ https://issues.apache.org/jira/browse/HBASE-25056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Christopher Liu resolved HBASE-25056. - Resolution: Fixed > Splittable Meta: foward port fixes from hbase-2 branch and stabilize unit > tests > --- > > Key: HBASE-25056 > URL: https://issues.apache.org/jira/browse/HBASE-25056 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25056) Splittable Meta: foward port fixes from hbase-2 branch and stabilize unit tests
[ https://issues.apache.org/jira/browse/HBASE-25056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200694#comment-17200694 ] Francis Christopher Liu commented on HBASE-25056: - Completed forward port a few tests are failing but acceptable enough to stabilize furter as part of the next task. > Splittable Meta: foward port fixes from hbase-2 branch and stabilize unit > tests > --- > > Key: HBASE-25056 > URL: https://issues.apache.org/jira/browse/HBASE-25056 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-25056) Splittable Meta: foward port fixes from hbase-2 branch and stabilize unit tests
[ https://issues.apache.org/jira/browse/HBASE-25056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Christopher Liu reassigned HBASE-25056: --- Assignee: Francis Christopher Liu > Splittable Meta: foward port fixes from hbase-2 branch and stabilize unit > tests > --- > > Key: HBASE-25056 > URL: https://issues.apache.org/jira/browse/HBASE-25056 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25056) Splittable Meta: foward port fixes from hbase-2 branch and stabilize unit tests
Francis Christopher Liu created HBASE-25056: --- Summary: Splittable Meta: foward port fixes from hbase-2 branch and stabilize unit tests Key: HBASE-25056 URL: https://issues.apache.org/jira/browse/HBASE-25056 Project: HBase Issue Type: Sub-task Reporter: Francis Christopher Liu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25028) Rebase Splittable Meta Master branch
[ https://issues.apache.org/jira/browse/HBASE-25028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Christopher Liu resolved HBASE-25028. - Resolution: Fixed > Rebase Splittable Meta Master branch > > > Key: HBASE-25028 > URL: https://issues.apache.org/jira/browse/HBASE-25028 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > > It's been a while since the branch was rebased and there were significant > changes that affected the patch that went in. I've been working on this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25028) Rebase Splittable Meta Master branch
[ https://issues.apache.org/jira/browse/HBASE-25028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197562#comment-17197562 ] Francis Christopher Liu commented on HBASE-25028: - Rebased and flattened the commits. > Rebase Splittable Meta Master branch > > > Key: HBASE-25028 > URL: https://issues.apache.org/jira/browse/HBASE-25028 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > > It's been a while since the branch was rebased and there were significant > changes that affected the patch that went in. I've been working on this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25028) Rebase Splittable Meta Master branch
Francis Christopher Liu created HBASE-25028: --- Summary: Rebase Splittable Meta Master branch Key: HBASE-25028 URL: https://issues.apache.org/jira/browse/HBASE-25028 Project: HBase Issue Type: Sub-task Reporter: Francis Christopher Liu Assignee: Francis Christopher Liu It's been a while since the branch was rebased and there were significant changes that affected the patch that went in. I've been working on this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195952#comment-17195952 ] Francis Christopher Liu commented on HBASE-11288: - Thanks [~apurtell] for clarifying that. I've filed HBASE-25027 to capture the discussion so we don't have to revisit it here again. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25027) Generalize MasterRegistry, etc to support both Master and Regionserver
Francis Christopher Liu created HBASE-25027: --- Summary: Generalize MasterRegistry, etc to support both Master and Regionserver Key: HBASE-25027 URL: https://issues.apache.org/jira/browse/HBASE-25027 Project: HBase Issue Type: Improvement Reporter: Francis Christopher Liu Assignee: Francis Christopher Liu Creating a jira to capture a discussion in [HBASE-11288|https://issues.apache.org/jira/browse/HBASE-11288?focusedCommentId=17194403=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17194403]. The discussion came out of avoiding the master from being part of the read/write path. As per [~andrew.purt...@gmail.com] "Generalizing the work to "bootstrap servers" that could be operating in either of master or regionserver roles seems fine but please let's make that a new JIRA or at least its own subtask." This is currently a place holder if HBASE-11288 implementation of split meta goes in as it would not make sense to add it otherwise. If it does I will get to implementing this then. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17194141#comment-17194141
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
If one solution will be used by more and more use cases, it will be not a
specialization and will be generalized for HBase.
{quote}
Understood this definition is different from how I used the term and they
aren't necessarily contradicting. I agree as well that it won’t be a
specialization (your definition) if it’s used by more use cases.
{quote}
Because the root table only have one region, it cannot scale horizontally, too.
If you use read region replica for this, the "master local region" can use read
region replica feature, too. In terms of scalability, "root table" is same with
"master local region".
{quote}
I see I mentioned master scalability with regards to using "master local
region" for other use cases such as replication, etc which might benefit from
horizontal scalability. Although yes replicas/caching will help with cluster
scalability for either implementation (although IMHO operationally “master
local region” requires it and it’s optional for “root table”). Having said that
it's also good to think about whether it's worth it to give the master (which
doesn't scale horizontally) more work when it doesn't need to do it.
{quote}
For me, the assignment framework "easy to read/understand" is more important.
{quote}
Off-hand I think both will be easy to read/understand. For "root table" the
code is basically an extension/intertwining of meta code so if you understand
meta code you will understand root code they’re almost the same.
{quote}
Now the core dependency in read/write path is the ZK. And I thought there are
exist issues about this: HBase should reduce ZK dependency in the future. So
when no ZK dependency, I thought that introduce master to read/write path is
acceptable.
{quote}
I see, I too would like to not expose ZK to the client. Some comments in this
jira is a discussion between me and Andy about using regionserver instead of
the master to provide the bootstrap information. I will work on implementing
that option only if root is not hosted in master as it would no longer make
sense if it is.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
> Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192765#comment-17192765 ] Francis Christopher Liu commented on HBASE-11288: - Apologies for the late reply [~zghao]. It seems our definition of specialization is different and one does not contradict the other. Using your definition of specialization (use cases storing data in “local master region”), in essence I agree it is not. I’ll try to be careful using the word as it seems to not help move the discussion forward. As a side note, I’m sure you thought of this but just wanted to chime in: in general I’d weigh storing data on the master vs other options as the master won’t scale horizontally among other reasons. Although I think that discussion would be case by case and better suited in their own jiras. In terms of simplicity, I think if you look at both from different perspectives either one can be seen as simpler than the other. Here’s two possible perspectives: The “local master region” in one perspective is simpler as it keeps the root logic in the master. The “root table” approach from an operations point of view is simpler as it’s only change is basically adding 1 more region to the cluster (worst case 2-tier assignment). Allow me to elaborate on the previous statement. “Root table” is operationally simpler compared to “local master region” as the latter adds new failure/operational modes, adds master to read/write path, etc. Then if caching is introduced in “local master region” the implementation becomes less simpler and operationally (depending how you look at it) more complex (although a bit more scalable and robust). In contrast based on our experience the “root table” approach will not need caching (except possibly for extreme cases we haven’t run into) hence operationally the change remains to be just the addition of 1 region, the root region. As for “assignment problems” I think we’ve proven through the successful ITBLL runs that hbase-2 is able to handle the extra tier of assignment? As for understanding the code the approach is mainly extending existing meta code not really adding new code paths. If a developer understands meta code they will easily be able to understand root code as it is merely an extension of the other. It would be great if we could come up with a solution that offers the simplicity of both. If not then it’d be great to come up with one that offers the best benefit factoring in any tradeoffs. I think this is where understanding different perspectives would be helpful. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192736#comment-17192736 ] Francis Christopher Liu commented on HBASE-11288: - In software projects sometimes the biggest bugs aren't in the code but in relationships, as well as possibly the most difficult to debug. Collaboration on this jira has failed and does not seem possible which is quite unfortunate. Based on my understanding of the discussions on this jira and in the docs it seems the root as a table approach has merit hence in the interest of letting the community decide what's best I will continue to work on this patch. I will be happy to continue objective discussions whether it is towards improving the patch or arguing against it. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183903#comment-17183903 ] Francis Christopher Liu commented on HBASE-11288: - I see thanks for confirming my understanding of the patch as well as elaborating on it more [~zhangduo]. Master Local Region is a specialization because it makes root no longer the same as hbase:meta. It is no longer is a table like meta table: local region, does not host region on a regionserver, etc. Solutions to root cannot always be generalized to hbase:meta. In itself Master Local Region implementation has its merits but when taking it as part of a whole: the catalog, I’m of the opinion we’d need a compelling reason to justify the specialization? I see sounds like you’re implying that a simple api could work for “general root table” as well? Assuming there is no compelling reason to go with Master Local Region implementation. For the purpose of understanding where this discussion is going. Earlier you mentioned if we could get a reasonable result from the ITBLL run you would change your mind. Is this still the case? > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177667#comment-17177667 ] Francis Christopher Liu commented on HBASE-11288: - I see apologies I thought you wanted my thoughts I missed the question. As mentioned root is half of Catalog its responsibilities are similar to that of the meta table. The benefit to root being a table just like meta is a table is when we need to do/apply something to Catalog we can come up with a generalized approach that can be done/applied to both. Having a generalized approach we avoid unneeded complexities hence my questions about it being worth it, etc. Does that answer your question? I see in your solution is root a table like meta table would the code paths be more or less the same? Would we be able to apply generalized solutions for both or would they be different code paths? Are these changes in the splittable-meta branch? Yes indeed it’s good to know the proc-v2 is stable and working well. Makes me look forward to deploying hbase-2 a bit more. I think the intent would be to pick one and stick with it for a long time. Hence the rigor in deliberation. In which case would adding new states in SCP still be a concern? I see I apologize if I sounded pushy I did not mean to. There could be other solutions that are good, but if the solution involves specializing then I’d also like to understand whether that tradeoff is compelling. There could be reasons that would make that tradeoff compelling I just don't know about it. If there was a compelling reason as mentioned I would change my current thinking on "local region on master". > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176265#comment-17176265 ] Francis Christopher Liu commented on HBASE-11288: - The run I mentioned in the previous comment completed successfully. I'll try to run using AsyncFSWAL and remove workaround #4. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176263#comment-17176263 ] Francis Christopher Liu commented on HBASE-11288: - Hi [~zhangduo], there's a lot of code for a few reasons: 1. The way I wrote the patch was to be easy to read how adding root is merely piggybacking/extending on a lot of the meta code hence there's a lot of copy paste, 2. The code to split meta will be basically touching existing code that's been touched already but will be generalized. 3. There's already some code that went in to address splittable meta. 4. There's some refactoring (renaming root to meta), generalization, etc. The production could changes would mainly be generalizing and extending existing code paths. I could try and cleanup the branch-2 patch so we can see what it would look like if that's a big concern? With regards to biasing root as a table. Root is half of the catalog it's responsibilities are sort of similar to that of the meta table hence I think there's some reasonable motivation with the bias. In which case I would like to understand why we are doing the specialized solution for root instead of coming up with a general one for the catalog as a whole? Also why is it worth it? Sorry if I sound like a broken record, I see that you responded to my question in a previous post. I just want to make sure I understand your current position (as some things might've changed these past 2 weeks)? > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175227#comment-17175227 ] Francis Christopher Liu edited comment on HBASE-11288 at 8/11/20, 4:59 AM: --- Apologies for the late reply. I got caught up dealing with the issues to get my setup running. Right now it's running and 4 of 5 iterations has finished. I'm using the command [~stack] shared. Here's a rundown of some of the things I dealt with so far. 1. Disabled netty native transport - regionservers JVM's would basically freeze. Couldn't even do jstack without "-F". I found out that internally we don't run with native transport for hadoop as it has a lot of issues apparently (wrong fds getting closed, etc). 2. Disabled security - there was some issue with TokenProvider throwing IllegalThreadStateException on startup causing the RS to abort 3. Hadoop-2.8 and zookeeper 3.4.x - made some quick changes so I could downgrade the dependencies as these are what we use internally. Shouldn't affect the validity of the test? 4. MonitorTaskImpl.prettyPrintJournal was throwing an NPE causing RS to abort - might be related to me using "filesystem" as the wal provider. I changed the code to swallow the exception for now as it seems unrelated. Will try and rerun the test use Async wal provider. 5. SCP bug introduced by patch - caused meta not to get assigned. Caught and fixed this quick and early. 6. [~stack] root executor patch - didn't run into this, but makes sense so I've applied the patch to the branch and have been running with it. was (Author: toffer): Apologies for the late reply. I got caught up dealing with the issues to get my setup running. Right now it's running and 4 of 5 iterations has finished. I'm using the command [~stack] shared. Here's a rundown of some of the things I dealt with so far. 1. Disabled netty native transport - regionservers JVM's would basically freeze. Couldn't even do jstack without "-F". I found out that internally we don't run with native transport for hadoop as it has a lot of issues apparently (wrong fds getting closed, etc). 2. Disabled security - there was some issue with TokenProvider throwing IllegalThreadStateException on startup causing the RS to abort 3. Hadoop-2.8 and zookeeper 3.4.x - made some quick changes so I could downgrade the dependencies as these are what we use internally. Shouldn't affect the validity of the test? 4. MonitorTaskImpl.prettyPrintJournal was throwing an NPE causing RS to abort - might be related to me using "filesystem" as the wal provider. I changed the code to swallow the exception for now as it seems unrelated. Will try and rerun the test use Async wal provider. 5. SCP bug introduced by patch - caused meta not to get assigned. Caught and fixed this quick and early. 6. [~stack] root executor patch - didn't run into this, but makes sense. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175229#comment-17175229 ] Francis Christopher Liu commented on HBASE-11288: - Apologies for the late reply [~zhangduo]. Yes it just adds root table (another tier). There is no split meta support. Did you want to test split meta as well? We could do that I'll just need some time to backport the changes that I've done on master as well as add the changes needed for failover (eg scp), etc. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175227#comment-17175227 ] Francis Christopher Liu commented on HBASE-11288: - Apologies for the late reply. I got caught up dealing with the issues to get my setup running. Right now it's running and 4 of 5 iterations has finished. I'm using the command [~stack] shared. Here's a rundown of some of the things I dealt with so far. 1. Disabled netty native transport - regionservers JVM's would basically freeze. Couldn't even do jstack without "-F". I found out that internally we don't run with native transport for hadoop as it has a lot of issues apparently (wrong fds getting closed, etc). 2. Disabled security - there was some issue with TokenProvider throwing IllegalThreadStateException on startup causing the RS to abort 3. Hadoop-2.8 and zookeeper 3.4.x - made some quick changes so I could downgrade the dependencies as these are what we use internally. Shouldn't affect the validity of the test? 4. MonitorTaskImpl.prettyPrintJournal was throwing an NPE causing RS to abort - might be related to me using "filesystem" as the wal provider. I changed the code to swallow the exception for now as it seems unrelated. Will try and rerun the test use Async wal provider. 5. SCP bug introduced by patch - caused meta not to get assigned. Caught and fixed this quick and early. 6. [~stack] root executor patch - didn't run into this, but makes sense. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171397#comment-17171397 ] Francis Christopher Liu commented on HBASE-11288: - Hi Guys, apologies for the delay. I've pushed up HBASE-11288.branch-2, which is a rebase of the master branch patch. The rebase is a little older than current HEAD as it didn't seem stable when I was rebasing onto it last weekend. I've disabled some unit tests which seems unneeded for the PoC, there are a few tests that are failing on my desktop but not my mac. I've deployed it on a test cluster and it comes up and basic stuff is working. I'll probably do another pass to see if there's anything else but having said that I think it's good enough to give it a try. I'll reach out to [~stack] tomorrow about running ITBLL. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164343#comment-17164343 ] Francis Christopher Liu commented on HBASE-11288: - A deadline sounds fine to me as well. And yes I would definitely appreciate the help [~stack] as well as another perspective when we're writing the assessment. I've not done ITBLL with procv2 assignment issues before so my estimates may be a bit off, off-hand one week sounds like a best case scenario, two weeks sounds more reasonable (assuming no surprises at work). Let me think about it more. What do you think [~stack]? I'm assuming this estimate excludes the rebasing on branch-2, although I'm hoping it would be quick if the differences aren't that big. Just trying to make sure we won’t be writing an assessment we’ll find lacking reading it year or so from now. As mentioned before if the ITBLL test shows procv2 is not ready for 2-tiered assignment then it’s clear “general root table” is not an option. I proposed this when 2-tiered assignment readiness seemed to be the most concerning. When it became apparent the concern was because of hot spotting on the root table, I asked a question similar to this: What value are we getting out of having a specialized solution for root instead of fixing it for both the catalog tables (root and meta) as a whole and is it worth it? If there is another answer that is compelling like 2-tiered assignment readiness. That would definitely be something that would change my current thinking on the “master local region” approach. Hopefully I addressed your concern? Nice article [~stack] will go through it. [~zyork] good to see that this is getting picked up again. What's the jira for this again? > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163398#comment-17163398 ] Francis Christopher Liu commented on HBASE-11288: - [~stack] and [~zhangduo] thank you for your suggestions of avoiding master branch and just rebasing on branch-2. I will go ahead and do that then. I will try to get as much information of course. I’ll do my best to get ITBLL to pass. [~zhangduo] Good to hear you are willing to change your mind with a reasonable result. As my intention has always been not only to prove/disprove it to myself but to you and [~stack] as well to help come to a consensus on a path forward. Looking at [~stack]’s recent comment the consensus seems to be that there was no consensus on a path forward. Although as [~stack] pointed out [~apurtell] did suggest we just do a reset so I missed the signs that we were doing that so my mistake. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162691#comment-17162691 ] Francis Christopher Liu commented on HBASE-11288: - I seem to have made a mistake and thought that we'd come to an agreement before moving forward so I've just been waiting. Looks like people are doing what they mentioned they wanted to do, so to move things forward on my end let me go ahead then and run ITBLL. 2-tiered assignment may not be the most contentious thing right now but it sounded like it was still important? So I plan to first start run ITBLL on unchanged master to get a baseline and then run with the patch. Has anyone run ITBLL on master? Is there a particular commit that I should be using, etc? > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162683#comment-17162683
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
I found a version in one of my local repo and pushed it back to HBASE-11288.
Please take a look to see if it is the newest one. Sorry again. Francis
Christopher Liu.
{quote}
No worries [~zhangduo] thanks for fixing it.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157846#comment-17157846 ] Francis Christopher Liu edited comment on HBASE-11288 at 7/15/20, 3:39 AM: --- I am all for “collaborate and work out a path forward”. How do we go about doing that? My previous proposal was trying to add some rigor by tackling the most pressing issue (in terms of picking one over the other) which seemed based on earlier discussions and the documentation was tiering. Any proposal? I have some technical responses to the recents posts here but will try to hold off until we can agree on a path forward. (78 words) was (Author: toffer): I am all for “collaborate and work out a path forward”. How do we go about doing that? My previous proposal was tackling the most pressing issue which seemed based on earlier discussions and the documentation was tiering. Any proposal? I have some technical responses to the recents posts here but will try to hold off until we can agree on a path forward. (64 words) > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157846#comment-17157846 ] Francis Christopher Liu commented on HBASE-11288: - I am all for “collaborate and work out a path forward”. How do we go about doing that? My previous proposal was tackling the most pressing issue which seemed based on earlier discussions and the documentation was tiering. Any proposal? I have some technical responses to the recents posts here but will try to hold off until we can agree on a path forward. (64 words) > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155797#comment-17155797 ] Francis Christopher Liu commented on HBASE-11288: - I think we are conflating the purpose of caching and that makes the evaluation a bit confusing. Here's my attempt at fleshing them out. And drawing some conclusions. There's essentially two cases: # Caching to prevent master to become part of read/write path - this use case is only presenet in "local master region". Caching here is used to reduce the effect of disadvantage. Bad enough to be required as part of any serious deployment(?). # Caching to avoid hot regions. - this is something potentially optional in "general root table". But arguably recommended in "local master region" given the sharing of master and root responsibilities in a single process. There is some nuance here as to why this caching use case is more important/recommended for "local master region" because of being colocated with the master especially when it comes to scaling(?). For 2, I think we are special casing root a little too much. For example in "general root table" implementation although splitting meta can reduce hot regions and help with scaling it does not eliminate it. In fact in our experience we run into hot meta regions more often than hot Root region. In which case I believe if we are considering caching as a solution for hot root region we need to think more holistically and think about a solution for Catalog Tables as a whole. Along those lines, although it's currently not a big deal for us, would it also make sense to consider availability for Catalog Tables as whole, which also can potentially be addressed with caching? The same can be said with the noisy neighbors argument does it really make sense to special case root instead of coming up with a holistic solution for Catalog Tables as a whole? It seems one potential advantage for "general root table" is being able to address issues with the Catalog Tables holistically potentially having a simpler and more elegant solution. The devil is in the details here of course and so the general question is how important is special casing root for any particular use case vs fixing it for catalog table as a whole. It at least seemed to me based on discussions and the design doc that the 2-tier was potentially that use case. What do you guys think? > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155192#comment-17155192
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
Do you agree with the above statements?
{quote}
>From what I can read in the doc and my understanding of the discussions so far
>the main difference between the two implementations is the tiering. “Master
>local region”’s only advantage is that it stays 1-tier for assignment,
>although at the cost of making compromises to avoid 2-tier. The caching as
>mentioned is not specific to one implementation.
{quote}
After rethinking, I think you misuderstood me. It seems that you are trying to
show that the 'general root table' solution can work. It definately can work as
you already have a cluster running with the solution. This is not my point. I
do not mean the 'general root table' can not work.
{quote}
I see so you agree it can work for HBase? Yes it seemed unclear wether you
thought it could work or not in general. Thanks for clarifying that. If you
could also clarify another thing, you seemed to imply there might be problems
with it for HBase currently (Eg ProcV2 is not mature enough) is this correct?
Just trying to make sure I understand your position.
{quote}
In general, I think both of the solution can work. Here we just want to choose
the 'better' solution.
{quote}
I see that’s good to know. So we can continue discussing on validating the main
difference (2-tier vs 1-tier)?
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154247#comment-17154247
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
Bigtable is designed way back in 2006, not everything in BigTable is correct.
And they do not give us their code so we do not know their framework on how to
deal with tablet assignment and tablet server crash.
{quote}
Yes of course my point was that it is not an unsolvable problem. It's unclear
wether you agree or disagree?
{quote}
What we have for now is procedure v2 in HBase, so our discussion should based
on procedure v2.
{quote}
Great we're on the same page then.
{quote}
And this is not only about adding another tier, how do you plan to distribute
the load of the general root table? Region replica is not stable enough, for
years.
{quote}
Let's follow Stack's previous comment? Let's not discuss replicas or caching as
they could be applied to either split meta implementation. Let's focus on the
tiering issue?
{quote}
This suggestion is weak. I'm afraid after running ITBLL we will sit here again
to argue that, 'This is only a POC, do not focus on polishing'.
{quote}
I'm missing something. It is a POC and its intent was to gather information and
help answer critical questions like this. So I'm not sure what the concern is
against POCs doing what POCs are supposed to be used for? Rest assured for this
particular case I am more concerned as to how we came to the decision than the
actual decision. It would be best for everyone if we could apply some rigor
for something so important.
{quote}
But from my understanding, ITBLL is for polishing. We should have a clear
design, and a clean code, and then we use ITBLL to find out the corner cases to
polish our code, make it more robust.
{quote}
It is definitely good for that. But what prevents us from using it for this
purpose? Wether it always passes, fails sometimes or fail always we learn
something and that is valuable for determining wether proc v2 can support
2-tier now, short term, long term or never. Then we can come to an informed
decision. For example we might decide we cannot wait that long for proc v2 to
mature so we go with 1-tier.
{quote}
And I want to know you opinion on 'master local region'. I've explain that,
with caching servers, master will not be on the critical path of normal client
read/write. Client just go to the cache servers, which is similar to read
replicas feature, but much easier to implement and more stable. Are there any
other concerns about this solution?
{quote}
It does help but it is still making a compromise to avoid the concerns of
2-tier. Which is why my main concern now is applying some rigor to help us come
to a well informed decision as to wether proc v2 cannot support it and we
should go with 1-tier. I think proc v2 is what was missing that prevented us
from succeeding in the past although it’s possible it may not be mature enough
at this stage.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154103#comment-17154103 ] Francis Christopher Liu commented on HBASE-11288: - Thanks for you response Duo. So it sounds to me the main reason for going with the “Root table on master” implementation is because of the strong disagreement towards the “General Root Table” implementation in particular the fact that it adds another tier in assignment. Given that “General Root Table” has proven itself in BigTable and other clones (eg Accumulo AFAIK). The concern it seems is specific to HBase’s implementation, there is a strong concern that we might fail again at getting it to work. Correct me if I’m wrong here. Regardless of which implementation the addition of split meta will not only be a significant change to HBase but also something that we will likely be living with for quite a while. Because of that I am proposing that we apply some rigor on deciding which implementation we will go with. My current thinking is given that the main reason it seems for picking “Root table on master” is because we want to avoid failing at tiering again, why don’t we come up with a way to validate that assertion? IMHO the situation now and then are very different especially with the addition of ProcV2 and the revamped Assignment. For validation, my understanding is we use ITBLL to test wether the current 1-tier assignment is working properly or not, why don’t we try and run ITBLL on the “General Root Table” POC to investigate and validate the 2-tier assignment concern? Thoughts? > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120925#comment-17120925
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
I added some comments on the PR. I think the PR is still far from merging...
{quote}
Thank you for taking a look and your comments. It seems your comments are about
polish? As mentioned this is a draft patch, does that mean the general
implementation is ok and I can work on polishing it?
{quote}
Usually the UT part will spend at least two hours to finish...
Thanks.
{quote}
Oh I see you're talking about the PR test result. I ran it locally with -fn so
I could get all the tests to run. Although I think I ran it without running IT
tests. I'll run it again after rebase.
{quote}
Lastly, you're working on implementing split meta while I am working on it too?
I wanted to know what your thoughts are where this is going? What happens when
you and I finish our feature branches?
{quote}
I'd really appreciate your response on this question. Having previous
discussions with you I think we both want split-meta and we both want to do it
right. So I think we both want the same thing, so maybe we can work a little
more closely on this?
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119482#comment-17119482
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
But you can see HBASE-24389, for now I've made all the UTs pass. This is very
important as we could do things step by step, on a feature branch. A feature
branch is not something that you could break everything until you finish the
last commit you fix everything, otherwise it does not make any difference with
a big fat patch...
{quote}
Yes I agree. And a few things: 1. As recommend by the community I put up a
draft patch to show the changes and get feedback before completing it. 2. The
tests that are passing are enough to show the concept of the draft patch. Tho
there are actually not much that are failing. 3. I've actually broken up the
changes into separate commits to make it easier for the reviewer. Did you
manage to take a look at the PR? Would appreciate your comments.
{quote}
Of course split meta will have more works, I've opened 5 sub tasks already and
HBASE-24388 and HBASE-24389 are only two of them. But as said above, I'm
optimistic since the current patch can pass all the existing UTs, and I think
the client side code is already enough for handling multiple meta regions.
{quote}
I agree you will get there. I was more talking about the approach making a
recommendation based on what I've gone through. Also it seems our work is
overlapping I've done some of the work that you already did. Perhaps I can help
out?
Lastly, you're working on implementing split meta while I am working on it too?
I wanted to know what your thoughts are where this is going? What happens when
you and I finish our feature branches?
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118523#comment-17118523
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
For root table, I thought the previous problems about meta table will become
the problems about root table. root table should be splitable? If root table
cannot splitable, which regionserver to host root table's region, a isolated
region or share with user regions? In our production use case (300+
RegionServer, 200K+ Regions, 1K+ tables), the meta region is isolated on one
regionserver, not share with the user regions. This is a waste of machine, but
for better stability. And we have many hack logic for meta regions, such as
assign meta first, donot replicate meta wal, balancer give a bigger weight for
meta region, the new root table need all these hack, too? You can found these
by grep isMetaTable or isMetaRegion calls in our codebase...
{quote}
Have you tried regionserver groups? In our deploy we isolated it by having a
system regionserver group which hosts all the system tables (root, meta, etc).
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118515#comment-17118515 ] Francis Christopher Liu edited comment on HBASE-11288 at 5/28/20, 10:01 AM: [~zhangduo][~stack] since you guys had big concerns with tiered assignment, would appreciate some comments in anything I might be doing undesirable in the PR as well as suggestions if you have any. Note the PR is a draft so there is likely room for improvement. Past that would a long run of ChaosMonkey be a way to validate that it's stable? was (Author: toffer): [~zhangduo][~stack] since you guys had big concerns with tiered assignment, would appreciate some comments in anything I might be doing undesirable in the PR as well as suggestions if you have any. Past that would a long run of ChaosMonkey be a way to validate that it's stable? > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118516#comment-17118516
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
By unify, I mean get as much reuse as possible in code and configuration.
{quote}
Cool thanks a lot [~apurtell] for your input!
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118515#comment-17118515 ] Francis Christopher Liu commented on HBASE-11288: - [~zhangduo][~stack] since you guys had big concerns with tiered assignment, would appreciate some comments in anything I might be doing undesirable in the PR as well as suggestions if you have any. Past that would a long run of ChaosMonkey be a way to validate that it's stable? > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118504#comment-17118504
]
Francis Christopher Liu edited comment on HBASE-11288 at 5/28/20, 9:54 AM:
---
{quote}
I will try to implement splittable meta by storing the root table in a 'master
local region', on the feature branch.
{quote}
I see is the intent to do a POC with master local region?
{quote}
HBASE24389, the goal is to move the location of meta to the 'master local
region', and also remove the assumption that there is only a single meta region
as much as possible in code, without actually split the meta table.
{quote}
I see, from my experience having written this patch a few times. Adding root
and then generalizing hbase:meta (eg moving away from single meta region)
regions is most of the code changes, the actual splitting meta code is not a
lot since it's basically just another region. As well as the true way to test
if you've done your job is to actually have more than one meta so it doesn't
really make sense to split generalizing meta handling and actually splitting
the table there might actually be more work if you do it that way.
was (Author: toffer):
{quote}
I will try to implement splittable meta by storing the root table in a 'master
local region', on the feature branch.
{quote}
I see is the intent to do a POC with master local region?
{quote}
HBASE24389, the goal is to move the location of meta to the 'master local
region', and also remove the assumption that there is only a single meta region
as much as possible in code, without actually split the meta table.
{quote}
I see, from my experience having written this patch a few times. Adding root
and then generalizing hbase:meta (eg moving away from single meta region)
regions is most of the code changes as well as the true way to test if you've
done your job is to actually have more than one meta so it doesn't really make
sense to split generalizing meta handling and actually splitting the table
there might actually be more work.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118514#comment-17118514 ] Francis Christopher Liu commented on HBASE-11288: - Thanks for the doc [~stack]. I've added some questions/risks in the doc (will fill in more info tomorrow) we should probably address them based on priority so we can answer the big questions first. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118504#comment-17118504
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
I will try to implement splittable meta by storing the root table in a 'master
local region', on the feature branch.
{quote}
I see is the intent to do a POC with master local region?
{quote}
HBASE24389, the goal is to move the location of meta to the 'master local
region', and also remove the assumption that there is only a single meta region
as much as possible in code, without actually split the meta table.
{quote}
I see, from my experience having written this patch a few times. Adding root
and then generalizing hbase:meta (eg moving away from single meta region)
regions is most of the code changes as well as the true way to test if you've
done your job is to actually have more than one meta so it doesn't really make
sense to split generalizing meta handling and actually splitting the table
there might actually be more work.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117616#comment-17117616 ] Francis Christopher Liu edited comment on HBASE-11288 at 5/27/20, 10:22 AM: Hi guys, just following up on what everyone's thoughts is for where meta region locations will actually be stored? Are we already sold on one direction over another? Do we need more discussion? Do we need investigation to figure out which is better (eg POC)? So far it's between storing it in the master and the root table. Correct me if I'm wrong or oversimplifying [~zhangduo] but it seems the main contention/concern seems to be between the complexity that tiering assignments add and putting the responsibility on the master? Also I see a meta location store for master in another feature branch (HBASE-11288.splittable-meta)? Is this a POC or is this the direction we are going? Let me know as I've rebased the root patch and been working on adding split meta support to HBASE-11288 feature branch. was (Author: toffer): Hi guys, just following up on what everyone's thoughts is for where meta region locations will actually be stored? Are we already sold on one direction over another? Do we need more discussion? Do we need investigation to figure out which is better (eg POC)? So far it's between storing it in the master and the root table. Correct me if I'm wrong or oversimplifying [~zhangduo] but it seems the main contention/concern seems to be between the complexity that tiering assignments add and putting the responsibility on the master? Also I see a meta location store for master in another feature branch (HBASE-11288.splittable-meta)? Is this a POC or is this the direction we are going? Let me know as I've rebased the root patch and working on adding split meta support to HBASE-11288 feature branch. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117616#comment-17117616 ] Francis Christopher Liu commented on HBASE-11288: - Hi guys, just following up on what everyone's thoughts is for where meta region locations will actually be stored? Are we already sold on one direction over another? Do we need more discussion? Do we need investigation to figure out which is better (eg POC)? So far it's between storing it in the master and the root table. Correct me if I'm wrong or oversimplifying [~zhangduo] but it seems the main contention/concern seems to be between the complexity that tiering assignments add and putting the responsibility on the master? Also I see a meta location store for master in another feature branch (HBASE-11288.splittable-meta)? Is this a POC or is this the direction we are going? Let me know as I've rebased the root patch and working on adding split meta support to HBASE-11288 feature branch. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117607#comment-17117607
]
Francis Christopher Liu commented on HBASE-11288:
-
Thanks [~apurtell] I see where you're coming from now.
{quote}
a different master oriented model was preferred to minimize operational and
configuration changes (especially if there is co-location of ZK and master
service)
{quote}
Since the config names between master and zookeeper registry are different what
configuration change is minimized? I see some of your users run 3 masters and
a 3 zookeeper ensemble on them as well. Tho operationally they shouldn't be
much different?
{quote}
If we had such a contribution I’d probably ask how difficult it would be to
unify them so it becomes the operators choice what hosts from either role to
include in the bootstrap string.
{quote}
We might go this route tho I'll have to get back to this after this patch is
done. Tho what do you mean by unify in this case? Unify the configs? And the
code? Eg reuse the rpcs etc? Do you want me to create a jira for this now? Or
when we decide on it?
{quote}
If you want to do more then the discussion becomes more nuanced but for
example if the configuration string is a single host name that resolves to
multiple address records then the registry client could be taught to handle
this case and it would then just work along with the case where multiple hosts
are listed in the string instead of discovered by DNS.
{quote}
Yeah we can definitely add that if people see a use of it. This approach is a
standard tool in our PE's toolbelt. Perhaps yours too? If so this would be
compelling to add? Also FWIW zookeeper already supports this tho it only loads
the mapping once during startup.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113080#comment-17113080
]
Francis Christopher Liu edited comment on HBASE-11288 at 5/21/20, 11:23 AM:
[~apurtell] thanks for the explanation, I'just have a few follow-up questions
if thats ok.
{quote}
In contrast, what if we start exporting a subset of regionservers. How does
that work? It's guaranteed to be different from how zk quorum strings worked.
{quote}
I pretty sure it's well thought out just understanding the thought process in
case we need to consider creating a regionserver registry so we understand the
tradeoff. My thinking was providing a static subset of regionservers the same
way that's being done with providing a static list of masters. Are we not able
to provide a static subset of masters in the current implementation?
{quote}
There are a lot more of them, they are randomly chosen , the connection info
has to dynamic instead of static (how is that collected? published to
clients?).
{quote}
I think there are a few ways to picking the static subset. Let me know if these
make sense. One way would be to pick regionservers that are part of the
regionserver group hosting the meta. Another way is to provide a seed list and
an api to allow the client to discover more if they need to someone mentioned
that his is how cassandra does it a while back when we were takling about a
regionserver registry. Another way is to provide just a single domain name and
that domain name contains a static subset list of regionserver ips which the
operator can choose to updated dynamically in case one is decomissioned or a
new on is added (this should work for the master approach too tho you would
need a resolver) this way the client config does not need to change.
was (Author: toffer):
[~apurtell] thanks for the explanation, I'just have a few follow-up questions
if thats ok.
{quote}
In contrast, what if we start exporting a subset of regionservers. How does
that work? It's guaranteed to be different from how zk quorum strings worked.
{quote}
I pretty sure it's well thought out just understanding the thought process in
case we need to consider creating a regionserver registry so we understand the
tradeoff. My thinking was providing a static subset of regionservers the same
way that's being done with providing a static list of masters. Are we not able
to provide a static subset of masters in the current implementation?
{quote}
There are a lot more of them, they are randomly chosen , the connection info
has to dynamic instead of static (how is that collected? published to
clients?).
{quote}
I think there are many ways to picking the static subset. One way would be to
pick regionservers that are part of the regionserver group hosting the meta.
Another way is to provide a seed list and an api to allow the client to
discover more if they need to someone mentioned that his is how cassandra does
it a while back when we were takling about a regionserver registry. Another way
is to provide just a single domain name and that domain name contains a static
subset list of regionserver ips which the operator can choose to updated
dynamically in case one is decomissioned or a new on is added (this should work
for the master approach too tho you would need a resolver) this way the client
config does not need to change.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113085#comment-17113085
]
Francis Christopher Liu edited comment on HBASE-11288 at 5/21/20, 11:11 AM:
{quote}
At least we have to provide an API to fecth all the content in root table, as
the cache service needs it. So for HBCK it could also uses this API. But this
API will not be part of the client facing API, for client they just need a
simple locateMeta method.
{quote}
I see. You would need mutation apis as well to correct the errors.
was (Author: toffer):
{quote}
At least we have to provide an API to fecth all the content in root table, as
the cache service needs it. So for HBCK it could also uses this API. But this
API will not be part of the client facing API, for client they just need a
simple locateMeta method.
{quote}
I see. You would need mutation apis as well.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113085#comment-17113085
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
At least we have to provide an API to fecth all the content in root table, as
the cache service needs it. So for HBCK it could also uses this API. But this
API will not be part of the client facing API, for client they just need a
simple locateMeta method.
{quote}
I see. You would need mutation apis as well.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113080#comment-17113080
]
Francis Christopher Liu commented on HBASE-11288:
-
[~apurtell] thanks for the explanation, I'just have a few follow-up questions
if thats ok.
{quote}
In contrast, what if we start exporting a subset of regionservers. How does
that work? It's guaranteed to be different from how zk quorum strings worked.
{quote}
I pretty sure it's well thought out just understanding the thought process in
case we need to consider creating a regionserver registry so we understand the
tradeoff. My thinking was providing a static subset of regionservers the same
way that's being done with providing a static list of masters. Are we not able
to provide a static subset of masters in the current implementation?
{quote}
There are a lot more of them, they are randomly chosen , the connection info
has to dynamic instead of static (how is that collected? published to
clients?).
{quote}
I think there are many ways to picking the static subset. One way would be to
pick regionservers that are part of the regionserver group hosting the meta.
Another way is to provide a seed list and an api to allow the client to
discover more if they need to someone mentioned that his is how cassandra does
it a while back when we were takling about a regionserver registry. Another way
is to provide just a single domain name and that domain name contains a static
subset list of regionserver ips which the operator can choose to updated
dynamically in case one is decomissioned or a new on is added (this should work
for the master approach too tho you would need a resolver) this way the client
config does not need to change.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113058#comment-17113058
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
For example, I'm not impressed by HBASE-18095 while lots of other committers
really love it. As in our deployments, we can also control how we deploy
zookeeper, not only HBase. Zookeeper supports a feature called observer node,
which can be considered as a cache of the zk cluster.
{quote}
I see, I agree with your sentiment and we do that too where appropriate. In
fact we were about to go down a similar route as you described with zookeeper
except for some limitations in early zookeeper versions prior to 3.5(?) which
did not have the "read only"(?) mode. The problem for us was that without that
mode there still had to be a quorum agreement for connections to the observer
even though the clients are doing only reads. This was unacceptable for us as
that was one of the things we observed was hammering the ensemble so we had to
go with a different approach.
More recenlty we also have implemented a Rest Registry because of security
requirements along the lines of what [~apurtell] described. Hence I too find
that it makes sense to abstract out zookeeper from the user so we have better
control of the attack surface. I suspect this requirement is going to become
more common.
{quote}
Then for me, it is also very easy to just implement a simple cache server, to
pull all the content in root from the active master and cache it in memory,
then to serve clients as the 'master' for locating meta. And I could also make
use of lvs, to spread load across multiple cache servers. And for users who can
not control things other than HBase, we could implement something like
HBASE-18095, to let backup masters to serve the locating meta requests.
{quote}
Yeah I think we would need an implemented solution for users as well. Or it
would seem a bit lopsided expecting users that worry about root traffic to
implement their own caching service.
{quote}
But if you use a hbase:root table, then you will expose the whole region API to
users. The API is powerful, but hard to simulate, so it will be really hard to
spread the load across multiple servers.
{quote}
If that is something you need we could still have root table but not expose the
region api (possibly like we already do for mutations) to them just the api you
described? Also I'm wondering if we don't expose the scan apis at all then do
we plan to have hbck for root where it is located (eg in the master)?
{quote}
Enable read replicas on root table? Seems a bit overkill and read replicas
itself is still not stable enough I suppose...
{quote}
It would seem overkill if we only did it for root. But we do it for meta today?
Or are people no longer using it for meta? There's no developer support in
getting it stable?
Please let me know what you think?
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112035#comment-17112035
]
Francis Christopher Liu edited comment on HBASE-11288 at 5/20/20, 10:33 AM:
{quote}
The solution is to make meta splittable... But this is not possible for root
table right?
{quote}
Right but there's being able to scale and there's getting hammered/spammed. For
the latter you could still run into issues and still need a solution.
{quote}
Well, for me I would say they are not the same... The location for root is on
zk while meta is in root, and root can not be split but meta can...
{quote}
True but they are both tables. Operationally that makes things more intuitive.
Eg load balancing across servers etc.
was (Author: toffer):
{quote}
The solution is to make meta splittable... But this is not possible for root
table right?
{quote}
There's being able to scale and there's getting hammered/spammed. For the
latter you could still run into issues and still need a solution.
{quote}
Well, for me I would say they are not the same... The location for root is on
zk while meta is in root, and root can not be split but meta can...
{quote}
True but they are both tables. Operationally that makes things more intuitive.
Eg load balancing across servers etc.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112035#comment-17112035
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
The solution is to make meta splittable... But this is not possible for root
table right?
{quote}
There's being able to scale and there's getting hammered/spammed. For the
latter you could still run into issues and still need a solution.
{quote}
Well, for me I would say they are not the same... The location for root is on
zk while meta is in root, and root can not be split but meta can...
{quote}
True but they are both tables. Operationally that makes things more intuitive.
Eg load balancing across servers etc.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112028#comment-17112028
]
Francis Christopher Liu edited comment on HBASE-11288 at 5/20/20, 10:25 AM:
{quote}
I think we should figure out how this will work. Can be done in parallel in
another issue. Baseline is old clients keep working.
{quote}
Yep let's make backward compatibility work. The simplest would be to just proxy
requests like rest or thrift server does now. If a request for meta hits the
regionserver hosting root. Should be fairly straightforward. Yeah another
branch on top of this one maybe just to not muck things up as we are trying
stuff.
{quote}
The tiering of assignment – root going out before anything else, then all of
the meta regions and then user-space regions. And that ROOT is just so ugly...
the way the rows are done; the crazy comparator.
{quote}
Tiering - yeah this is a tough one since root holds meta locationsit
invariable requires root to be available for writing. What is your concern
here...the complexity of the process?
Ugly root - The way the row keys are done? The nested thing? It's not that bad
at least the logic is being reused. Tho we probably don't need the
"hbase:meta," prefix on all rowstho I"m not sure what we gain?
{quote}
Currently we have meta and namespace and acls and we still don't get it right
(Duo helped us out here by merging namespace into meta table in master branch).
{quote}
ACLs I've never had an issue with? Namespace AFAIK needed attention and got
kicked around a bit? It's better it's in hbase:meta now...no one will ignore
that table as well as less specialized code. This is a bit different from
tiering assignment issue tho?
{quote}
We should do up a bit of a doc on how this will work; how it will work for old
and new clients (e.g. handing out the whole meta table when client does meta
location lookup); how the proxying will be done so old clients will work when
meta splits, etc.
{quote}
Sounds good. Do you want to brainstorm? Or should I write a proposal then we
work on it?
was (Author: toffer):
{quote}
I think we should figure out how this will work. Can be done in parallel in
another issue. Baseline is old clients keep working.
{quote}
Yep let's make backward compatibility work. The simplest would be to just proxy
requests like rest or thrift server does now. If a request for meta hits the
regionserver hosting root. Should be fairly straightforward. Yeah another
branch on top of this one maybe just to not muck things up as we are trying
stuff.
{quote}
The tiering of assignment – root going out before anything else, then all of
the meta regions and then user-space regions. And that ROOT is just so ugly...
the way the rows are done; the crazy comparator.
{quote}
Tiering - yeah this is a tough one since root holds meta locationsit
invariable requires root to be available for writing. What is your concern
here...the complexity of the process?
Ugly root - The way the row keys are done? The nested thing? It's not that bad
at least the logic is being reused. Tho we probably don't need the
"hbase:meta," prefix on all rowstho I"m not sure what we gain?
{quote}
Currently we have meta and namespace and acls and we still don't get it right
(Duo helped us out here by merging namespace into meta table in master branch).
{quote}
ACLs I've never had an issue with? Namespace AFAIK needed attention and got
kicked around a bit? It's better it's in hbase:meta now...no one will ignore
that table. This is a bit different from tiering assignment issue tho?
{quote}
We should do up a bit of a doc on how this will work; how it will work for old
and new clients (e.g. handing out the whole meta table when client does meta
location lookup); how the proxying will be done so old clients will work when
meta splits, etc.
{quote}
Sounds good. Do you want to brainstorm? Or should I write a proposal then we
work on it?
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112028#comment-17112028
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
I think we should figure out how this will work. Can be done in parallel in
another issue. Baseline is old clients keep working.
{quote}
Yep let's make backward compatibility work. The simplest would be to just proxy
requests like rest or thrift server does now. If a request for meta hits the
regionserver hosting root. Should be fairly straightforward. Yeah another
branch on top of this one maybe just to not muck things up as we are trying
stuff.
{quote}
The tiering of assignment – root going out before anything else, then all of
the meta regions and then user-space regions. And that ROOT is just so ugly...
the way the rows are done; the crazy comparator.
{quote}
Tiering - yeah this is a tough one since root holds meta locationsit
invariable requires root to be available for writing. What is your concern
here...the complexity of the process?
Ugly root - The way the row keys are done? The nested thing? It's not that bad
at least the logic is being reused. Tho we probably don't need the
"hbase:meta," prefix on all rowstho I"m not sure what we gain?
{quote}
Currently we have meta and namespace and acls and we still don't get it right
(Duo helped us out here by merging namespace into meta table in master branch).
{quote}
ACLs I've never had an issue with? Namespace AFAIK needed attention and got
kicked around a bit? It's better it's in hbase:meta now...no one will ignore
that table. This is a bit different from tiering assignment issue tho?
{quote}
We should do up a bit of a doc on how this will work; how it will work for old
and new clients (e.g. handing out the whole meta table when client does meta
location lookup); how the proxying will be done so old clients will work when
meta splits, etc.
{quote}
Sounds good. Do you want to brainstorm? Or should I write a proposal then we
work on it?
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112008#comment-17112008
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
No, because the registry addresses must be explicitly discoverable, at a
manageable scale for that purpose.
{quote}
Thanks for the explanation [~apurtell]! I should've followed up with you after
we discussed it briefly.
{quote}
Number of masters will be up to orders of magnitude less than regionservers. So
we can't directly serve the registry data out of regionservers and call it a
day.
{quote}
I agree publishing all the regionserver locations wouldn't make sense, but I'm
curious and probably missing something as to why can't we publish a subset of
regionserver locations?
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111992#comment-17111992
]
Francis Christopher Liu edited comment on HBASE-11288 at 5/20/20, 9:57 AM:
---
{quote}
Just do not see any difference if we just have a single root region on a region
server? For all the cases, the client will also harmmering the region server
which hosts the root region and bring down all the cluster...
{quote}
IMHO this type of workload is a regionserver responsibility and hence there are
advantages to it being done on the regionserver.
1. AFAIK hbase:meta today is getting hammered? If so, we'll likely want to fix
that if we haven't already? If so then whatever fix/enhancement we did the root
table can take advantage of as well? In which case it gives simplicity and code
reuse as we are applying one solution to two problems. Although we'd have to
add code for root getting assigned first etc, which I believe should be
relatively straightforward with procedures.
On the other hand if use the master and backup masters we will be giving them a
new specialized responsibility, which will introduce new specialized code. And
then we would need to introduce more code for the master(s) to not get
hammered.
2. From an operations perspective it would be easier for operations folks to
reason about things as the way both catalog tiers are handled are the same and
so the way to manage them if there are issues are the same.
3. In recovery scenarios the master is already busy doing it's master work, so
it would be better if it could focus it's resources on that as it does not
horizontally scale.
Please correct me if I'm making any wrong assumptions here. Let me know what
you think?
was (Author: toffer):
{quote}
Just do not see any difference if we just have a single root region on a region
server? For all the cases, the client will also harmmering the region server
which hosts the root region and bring down all the cluster...
{quote}
IMHO this type of workload is a regionserver responsibility and hence there are
advantages to it being done on the regionserver.
1. AFAIK hbase:meta today is getting hammered? If so, we'll likely want to fix
that if we haven't already? If so then whatever fix/enhancement we did the root
table can take advantage of as well? In which case it gives simplicity and code
reuse as we are applying one solution to two problems. Although we'd have to
add code for root getting assigned first etc, which I believe should be
relatively straightforward with procedures.
On the other hand if use the master and backup masters we will be giving them a
new specialized responsibility, which will introduce new specialized code. And
then we would need to introduce more code for the master(s) to not get
hammered.
2. From an operations perspective it would be easier for operations folks to
reason about things as the way both catalog tiers are handled are the same and
so the way to manage them if there are issues are the same.
Please correct me if I'm making any wrong assumptions here. Let me know what
you think?
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111992#comment-17111992
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
Just do not see any difference if we just have a single root region on a region
server? For all the cases, the client will also harmmering the region server
which hosts the root region and bring down all the cluster...
{quote}
IMHO this type of workload is a regionserver responsibility and hence there are
advantages to it being done on the regionserver.
1. AFAIK hbase:meta today is getting hammered? If so, we'll likely want to fix
that if we haven't already? If so then whatever fix/enhancement we did the root
table can take advantage of as well? In which case it gives simplicity and code
reuse as we are applying one solution to two problems. Although we'd have to
add code for root getting assigned first etc, which I believe should be
relatively straightforward with procedures.
On the other hand if use the master and backup masters we will be giving them a
new specialized responsibility, which will introduce new specialized code. And
then we would need to introduce more code for the master(s) to not get
hammered.
2. From an operations perspective it would be easier for operations folks to
reason about things as the way both catalog tiers are handled are the same and
so the way to manage them if there are issues are the same.
Please correct me if I'm making any wrong assumptions here. Let me know what
you think?
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111994#comment-17111994
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
Yes please, if all your changes are floating on upstream history this will be
nice and tidy.
{quote}
Will do!
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111060#comment-17111060
]
Francis Christopher Liu edited comment on HBASE-11288 at 5/19/20, 10:42 AM:
{quote}
We store the procedure data in a local HRegion on master.
{quote}
I see thanks for the explanation and the link.
{quote}
And on client accessing meta, I think for root region it is fine. We do not
mirror all the requests to region servers to master, when will only contact
master when locating meta, which should be rarely happen.
{quote}
I can think of a couple of cases were it might not be good/ideal:
1. Large batch jobs that does lookups on an hbase table in tasks would hit it
on startup. Same for large application restarts (eg storm topologies, etc).
2. In the event of failure degradation is not as graceful, as if clients are
not able to find meta regions they will start hammering the master when the
master is already busy trying to recover the cluster.
3. Generally just misbehaving actors (eg poorly written applications) or buggy
third party client implementations that users attempt to use.
{quote}
And now we even have a MasterRegistry on 2.x, where we move the locating meta
requests from zk to master.
{quote}
I think what we might do is add a resolver plugin so all master registry
requests just go to the backup masters. I've not looked at the code yet but
maybe we can have the backup masters serve out the local hregion content from
the backup masters? IMHO ideally the registry should've been served out of
the regionservers. Today we have a "system" regionserver group where all the
system tables served out of (including root). With the new setup we would have
a system group and an extra set of master servers for serving out root and
registry stuff and the extra operational nuance of backup masters not just
being backup but doing active work for the system.
Just laying out my concerns let me know what you guys think? Let me know if
it's reasonable/unreasonable.
was (Author: toffer):
{quote}
We store the procedure data in a local HRegion on master.
{quote}
I see thanks for the explanation and the link.
{quote}
And on client accessing meta, I think for root region it is fine. We do not
mirror all the requests to region servers to master, when will only contact
master when locating meta, which should be rarely happen.
{quote}
I can think of a couple of cases were it might not be good/ideal:
1. Large batch jobs that does lookups on an hbase table in tasks would hit it
on startup. Same for large application restarts (eg storm topologies, etc).
2. In the event of failure degradation is not as graceful, as if clients are
not able to find meta regions they will start hammering the master when the
master is already busy trying to recover the cluster.
3. Generally just misbehaving actors (eg poorly written applications) or buggy
third party client implementations that users attempt to use.
{quote}
And now we even have a MasterRegistry on 2.x, where we move the locating meta
requests from zk to master.
{quote}
I think what we might do is add a resolver plugin so all master registry
requests just go to the backup masters. I've not looked at the code yet but
maybe we can have the backup masters serve out the local hregion content from
the backup masters? IMHO ideally the registry should've been served out of
the regionservers. Today we have a "system" regionserver group where all the
system tables served out of (including root). With the new setup we would have
a system group and an extra set of master servers for serving out root and
registry stuff and the extra nuance of backup masters not just being backup but
doing active work for the system.
Just laying out my concerns let me know what you guys think?
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111060#comment-17111060
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
We store the procedure data in a local HRegion on master.
{quote}
I see thanks for the explanation and the link.
{quote}
And on client accessing meta, I think for root region it is fine. We do not
mirror all the requests to region servers to master, when will only contact
master when locating meta, which should be rarely happen.
{quote}
I can think of a couple of cases were it might not be good/ideal:
1. Large batch jobs that does lookups on an hbase table in tasks would hit it
on startup. Same for large application restarts (eg storm topologies, etc).
2. In the event of failure degradation is not as graceful, as if clients are
not able to find meta regions they will start hammering the master when the
master is already busy trying to recover the cluster.
3. Generally just misbehaving actors (eg poorly written applications) or buggy
third party client implementations that users attempt to use.
{quote}
And now we even have a MasterRegistry on 2.x, where we move the locating meta
requests from zk to master.
{quote}
I think what we might do is add a resolver plugin so all master registry
requests just go to the backup masters. I've not looked at the code yet but
maybe we can have the backup masters serve out the local hregion content from
the backup masters? IMHO ideally the registry should've been served out of
the regionservers. Today we have a "system" regionserver group where all the
system tables served out of (including root). With the new setup we would have
a system group and an extra set of master servers for serving out root and
registry stuff and the extra nuance of backup masters not just being backup but
doing active work for the system.
Just laying out my concerns let me know what you guys think?
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111049#comment-17111049
]
Francis Christopher Liu commented on HBASE-11288:
-
Thanks for the feedback [~stack]!
{quote}
Can we split meta in a way such that older clients keep working? Perhaps old
clients get a proxying Region that fields requests to the actual split meta
table so they keep working?
{quote}
We have rest and thrift proxies. I don't see why we can't do this. Do you want
it implemented now as a PoC before we go forward with completing the feature?
{quote}
Can we NOT do a ROOT table w/ its crazy meta-meta comparator?
{quote}
What are your concerns with it? Trying to understand what needs to be
addressed?
{quote}
We've not been good at tier's of assign; i.e. ROOT first, then META regions.
{quote}
Do you remember any of the big issues? I think procedures have helped in that
regard. We've not have any problems AFAIK with 1.3 that uses SCPs, etc.
ChaosMonkey should suss any other issues out?
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111044#comment-17111044
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
For one, it makes evaluating the split meta patch plus recommended changes like
master local region possible.
{quote}
Sounds good. I've created a branch named HBASE-11288. Also by the sound of it I
should be rebasing when I pull in stuff from upstream and not merging? If there
are any other best practices I should follow please let me know or point me in
the right direction. Thanks!
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110036#comment-17110036 ] Francis Christopher Liu commented on HBASE-11288: - Thanks a lot [~zhangduo] and [~apurtell] for taking a look! [~apurtell] I'll put up a feature branch. Tho I'm not sure how it helps reviewing the code changes? With regards to making the patch smaller. There are things that are sort of broken in trunk but functional because meta is not split (eg comparator changes). I could break those out and get those merged in first? Beyond that I'm open to suggestions? [~zhangduo] Let me take a look at the "master local region" any pointer would be helpful. Although off-hand given your description it sounds to me like requests for locating meta reigions will be served by the master? If so at least in 1.x, I felt that having clients not accessing the master for data access was an important feature. We'll run into a lot of trouble if clients start hammering the master etc. I'd like to hear your thoughts as I'm not sure how things are in 2.x now, but that would be a concern that I'd have if my understanding is correct? > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073594#comment-17073594 ] Francis Christopher Liu edited comment on HBASE-11288 at 4/2/20, 10:35 AM: --- Hi, I pushed up my current WIP/Draft patch for feedback as PR on [github|https://github.com/apache/hbase/pull/1418]. Please let me know what you guys think. was (Author: toffer): Hi, I pushed up my current WIP/Draft patch for feedback as PR on [github|https://github.com/apache/hbase/pull/1418]. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073594#comment-17073594 ] Francis Christopher Liu commented on HBASE-11288: - Hi, I pushed up my current WIP/Draft patch for feedback as PR on [github|https://github.com/apache/hbase/pull/1418]. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942317#comment-16942317
]
Francis Christopher Liu edited comment on HBASE-11288 at 10/1/19 9:11 PM:
--
{quote}
I'd like to see old clients keep working though meta has split under them.
Perhaps the serverName in the MetaRegionServer znode proxies requests when the
region asked for is named 'hbase:meta,,1.1588230740' to actual meta table whose
regions could be elsewhere on the cluster. I suppose 'hbase:meta,,1.1588230740'
would have to be the old ROOT too?
{quote}
Oh interesting idea. How about instead of proxying when a requests for meta
hits the RS with the ROOT region, we could throw a RegionMovedException to
redirect the client to the proper meta region RS . On the RS holding the meta
region we translate the 'hbase:meta,,1.1588230740' region info to the
appropriate meta region info if split. This way we can avoid proxying which
seems to make things a bit more complicated?
{quote}
Lets do a one-pager.
{quote}
Sounds good.
was (Author: toffer):
{quote}
I'd like to see old clients keep working though meta has split under them.
Perhaps the serverName in the MetaRegionServer znode proxies requests when the
region asked for is named 'hbase:meta,,1.1588230740' to actual meta table whose
regions could be elsewhere on the cluster. I suppose 'hbase:meta,,1.1588230740'
would have to be the old ROOT too?
{quote}
Oh interesting idea. How about instead of proxying we could throw a
RegionMovedException to redirect to the proper meta region RS when a requests
for meta hits the RS with the ROOT region. On the RS holding the meta region we
translate the 'hbase:meta,,1.1588230740' region info to the appropriate meta
region info if split. This way we can avoid proxying which seems to make things
a bit more complicated?
{quote}
Lets do a one-pager.
{quote}
Sounds good.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Sub-task
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942317#comment-16942317
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
I'd like to see old clients keep working though meta has split under them.
Perhaps the serverName in the MetaRegionServer znode proxies requests when the
region asked for is named 'hbase:meta,,1.1588230740' to actual meta table whose
regions could be elsewhere on the cluster. I suppose 'hbase:meta,,1.1588230740'
would have to be the old ROOT too?
{quote}
Oh interesting idea. How about instead of proxying we could throw a
RegionMovedException to the proper meta region destination when a requests for
meta hits the RS with the ROOT region. On the RS holding the meta region we
translate the 'hbase:meta,,1.1588230740' region info to the appropriate meta
region info if split. This way we can avoid proxying which seems to make things
a bit more complicated?
{quote}
Lets do a one-pager.
{quote}
Sounds good.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Sub-task
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942317#comment-16942317
]
Francis Christopher Liu edited comment on HBASE-11288 at 10/1/19 9:10 PM:
--
{quote}
I'd like to see old clients keep working though meta has split under them.
Perhaps the serverName in the MetaRegionServer znode proxies requests when the
region asked for is named 'hbase:meta,,1.1588230740' to actual meta table whose
regions could be elsewhere on the cluster. I suppose 'hbase:meta,,1.1588230740'
would have to be the old ROOT too?
{quote}
Oh interesting idea. How about instead of proxying we could throw a
RegionMovedException to redirect to the proper meta region RS when a requests
for meta hits the RS with the ROOT region. On the RS holding the meta region we
translate the 'hbase:meta,,1.1588230740' region info to the appropriate meta
region info if split. This way we can avoid proxying which seems to make things
a bit more complicated?
{quote}
Lets do a one-pager.
{quote}
Sounds good.
was (Author: toffer):
{quote}
I'd like to see old clients keep working though meta has split under them.
Perhaps the serverName in the MetaRegionServer znode proxies requests when the
region asked for is named 'hbase:meta,,1.1588230740' to actual meta table whose
regions could be elsewhere on the cluster. I suppose 'hbase:meta,,1.1588230740'
would have to be the old ROOT too?
{quote}
Oh interesting idea. How about instead of proxying we could throw a
RegionMovedException to the proper meta region destination when a requests for
meta hits the RS with the ROOT region. On the RS holding the meta region we
translate the 'hbase:meta,,1.1588230740' region info to the appropriate meta
region info if split. This way we can avoid proxying which seems to make things
a bit more complicated?
{quote}
Lets do a one-pager.
{quote}
Sounds good.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Sub-task
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941446#comment-16941446
]
Francis Christopher Liu edited comment on HBASE-11288 at 10/1/19 1:21 AM:
--
{quote}
How many rows in your biggest hbase:root? And how many regions in hbase:meta
have you gotten to?
{quote}
We have a cluster that has 1.04 million regions. That same cluster has 119
hbase:meta regions. We configured the split size to be small.
{quote}
A lot has changed since 1.3 but patch will help.
{quote}
Yes, it shows two important things: 1. The approach of adding a splittable
hbase:meta is fairly straightforward and can be applied to other branches (even
tho a lot has changed), 2. The approach works :-). Next question is where do we
go from here? A patch for master?
was (Author: toffer):
{quote}
How many rows in your biggest hbase:root? And how many regions in hbase:meta
have you gotten to?
{quote}
We have a cluster that has 1.04 million regions. That same cluster has 119
hbase:meta regions. We configured the split size to be small.
{quote}
A lot has changed since 1.3 but patch will help.
{quote}
Yes, it shows two important things: 1. The approach of adding hbase:meta is
fairly straightforward and can be applied to other branches (even tho a lot has
changed), 2. The approach works :-). Next question is where do we go from here?
A patch for master?
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Sub-task
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941448#comment-16941448 ] Francis Christopher Liu commented on HBASE-11288: - BTW let me know if the document needs more information. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941446#comment-16941446
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
How many rows in your biggest hbase:root? And how many regions in hbase:meta
have you gotten to?
{quote}
We have a cluster that has 1.04 million regions. That same cluster has 119
hbase:meta regions. We configured the split size to be small.
{quote}
A lot has changed since 1.3 but patch will help.
{quote}
Yes, it shows two important things: 1. The approach of adding hbase:meta is
fairly straightforward and can be applied to other branches (even tho a lot has
changed), 2. The approach works :-). Next question is where do we go from here?
A patch for master?
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Sub-task
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941439#comment-16941439
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
Since now META can split all the journal entries that is been added for a
region split will now be added to ROOT and the failure of the meta split will
be treated similar to a normal region, correct?
{quote}
In 1.3 Journals are not persisted perhaps in newer 1.x versions? Tho yes
regionstate information for hbase:meta regions are persisted in the hbase:root
table. Meta region split failures will be handled in the same manner that is
correct. Tho there are cases that hbase:meta region operations will take
priority over user regions but I'll have to double check which parts those are.
It's been a while.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Sub-task
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940663#comment-16940663 ] Francis Christopher Liu edited comment on HBASE-11288 at 9/30/19 6:06 AM: -- Apologies for the delay here's a short [writeup|https://docs.google.com/document/d/1cs_sRC5xbK2JPdw99kqUqUGMJ0yHier5QPx21al619g/edit]. Let me know if it needs more detail or feel free to comment. BTW I had to rebase the branch. was (Author: toffer): Apologies for the delay here's a short [writeup|[https://docs.google.com/document/d/1cs_sRC5xbK2JPdw99kqUqUGMJ0yHier5QPx21al619g/edit]] . Let me know if it needs more detail or feel free to comment. BTW I had to rebase the branch. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940663#comment-16940663 ] Francis Christopher Liu commented on HBASE-11288: - Apologies for the delay here's a short [writeup|[https://docs.google.com/document/d/1cs_sRC5xbK2JPdw99kqUqUGMJ0yHier5QPx21al619g/edit]]. Let me know if it needs more detail or feel free to comment. BTW I had to rebase the branch. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940663#comment-16940663 ] Francis Christopher Liu edited comment on HBASE-11288 at 9/30/19 6:04 AM: -- Apologies for the delay here's a short [writeup|[https://docs.google.com/document/d/1cs_sRC5xbK2JPdw99kqUqUGMJ0yHier5QPx21al619g/edit]] . Let me know if it needs more detail or feel free to comment. BTW I had to rebase the branch. was (Author: toffer): Apologies for the delay here's a short [writeup|[https://docs.google.com/document/d/1cs_sRC5xbK2JPdw99kqUqUGMJ0yHier5QPx21al619g/edit]]. Let me know if it needs more detail or feel free to comment. BTW I had to rebase the branch. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
