[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244405#comment-17244405 ] Francis Christopher Liu commented on HBASE-11288: - Good idea [~stack]. +1 Let's leave the baggage here so it'll be easier for the community to get invovled. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244386#comment-17244386 ] Duo Zhang commented on HBASE-11288: --- +1 on opening a new issue. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244353#comment-17244353 ] Michael Stack commented on HBASE-11288: --- Chatting w/ [~toffer] and [~zhangduo] on wednesday, the suggestion was made that we just start the split meta project over; that we go back to requirements and get agreement on these first and from there, move forward to an agreed-upon design and from there to implementation. We'd do this to make this project more open to collaboration and to help ensure the resulting product will be more broadly acceptible to the community. (The thought is that we will probably end up using a lot of what has already been written here – text and code – but only that which aligns w/ requirements, and so on) In the spirit of the above sentiment, I think we should leave this issue behind and open a new one. This issue is full of great discussion but there is a lot of not-so-great discussion too. Either way, it has gotten too long for any but the most determined to make sense of. Shout if you disagree and think we should keep all of split meta here, including the reset, otherwise I'll open a new issue over the weekend and start by putting up a location to catch requirements in. Thanks. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195952#comment-17195952 ] Francis Christopher Liu commented on HBASE-11288: - Thanks [~apurtell] for clarifying that. I've filed HBASE-25027 to capture the discussion so we don't have to revisit it here again. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17194403#comment-17194403 ] Andrew Kyle Purtell commented on HBASE-11288: - bq. I too would like to not expose ZK to the client. The HBASE-23305 master registry work has been committed and released starting with 2.3.0, so it's not really optional at this point. Committing something that returns a hard dependency on ZK in the client for region location would be a regression. bq. Some comments in this jira is a discussion between me and Andy about using regionserver instead of the master to provide the bootstrap information. Generalizing the work to "bootstrap servers" that could be operating in either of master or regionserver roles seems fine but please let's make that a new JIRA or at least its own subtask. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17194141#comment-17194141
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
If one solution will be used by more and more use cases, it will be not a
specialization and will be generalized for HBase.
{quote}
Understood this definition is different from how I used the term and they
aren't necessarily contradicting. I agree as well that it won’t be a
specialization (your definition) if it’s used by more use cases.
{quote}
Because the root table only have one region, it cannot scale horizontally, too.
If you use read region replica for this, the "master local region" can use read
region replica feature, too. In terms of scalability, "root table" is same with
"master local region".
{quote}
I see I mentioned master scalability with regards to using "master local
region" for other use cases such as replication, etc which might benefit from
horizontal scalability. Although yes replicas/caching will help with cluster
scalability for either implementation (although IMHO operationally “master
local region” requires it and it’s optional for “root table”). Having said that
it's also good to think about whether it's worth it to give the master (which
doesn't scale horizontally) more work when it doesn't need to do it.
{quote}
For me, the assignment framework "easy to read/understand" is more important.
{quote}
Off-hand I think both will be easy to read/understand. For "root table" the
code is basically an extension/intertwining of meta code so if you understand
meta code you will understand root code they’re almost the same.
{quote}
Now the core dependency in read/write path is the ZK. And I thought there are
exist issues about this: HBase should reduce ZK dependency in the future. So
when no ZK dependency, I thought that introduce master to read/write path is
acceptable.
{quote}
I see, I too would like to not expose ZK to the client. Some comments in this
jira is a discussion between me and Andy about using regionserver instead of
the master to provide the bootstrap information. I will work on implementing
that option only if root is not hosted in master as it would no longer make
sense if it is.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
> Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192925#comment-17192925
]
Guanghao Zhang commented on HBASE-11288:
{quote}Using your definition of specialization (use cases storing data in
“local master region”)
{quote}
If one solution will be used by more and more use cases, it will be not a
specialization and will be generalized for HBase.
{quote}the master won’t scale horizontally among other reasons
{quote}
Because the root table only have one region, it cannot scale horizontally, too.
If you use read region replica for this, the "master local region" can use read
region replica feature, too. In terms of scalability, "root table" is same with
"master local region".
{quote}In terms of simplicity, I think if you look at both from different
perspectives either one can be seen as simpler than the other.
{quote}
For me, the assignment framework "easy to read/understand" is more important.
{quote}bq. adds master to read/write path, etc
{quote}
Now the core dependency in read/write path is the ZK. And I thought there are
exist issues about this: HBase should reduce ZK dependency in the future. So
when no ZK dependency, I thought that introduce master to read/write path is
acceptable.
{quote}As for “assignment problems” I think we’ve proven through the successful
ITBLL runs that hbase-2 is able to handle the extra tier of assignment
{quote}
Successful ITBLL is great. Sorry for miss this.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
> Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192765#comment-17192765 ] Francis Christopher Liu commented on HBASE-11288: - Apologies for the late reply [~zghao]. It seems our definition of specialization is different and one does not contradict the other. Using your definition of specialization (use cases storing data in “local master region”), in essence I agree it is not. I’ll try to be careful using the word as it seems to not help move the discussion forward. As a side note, I’m sure you thought of this but just wanted to chime in: in general I’d weigh storing data on the master vs other options as the master won’t scale horizontally among other reasons. Although I think that discussion would be case by case and better suited in their own jiras. In terms of simplicity, I think if you look at both from different perspectives either one can be seen as simpler than the other. Here’s two possible perspectives: The “local master region” in one perspective is simpler as it keeps the root logic in the master. The “root table” approach from an operations point of view is simpler as it’s only change is basically adding 1 more region to the cluster (worst case 2-tier assignment). Allow me to elaborate on the previous statement. “Root table” is operationally simpler compared to “local master region” as the latter adds new failure/operational modes, adds master to read/write path, etc. Then if caching is introduced in “local master region” the implementation becomes less simpler and operationally (depending how you look at it) more complex (although a bit more scalable and robust). In contrast based on our experience the “root table” approach will not need caching (except possibly for extreme cases we haven’t run into) hence operationally the change remains to be just the addition of 1 region, the root region. As for “assignment problems” I think we’ve proven through the successful ITBLL runs that hbase-2 is able to handle the extra tier of assignment? As for understanding the code the approach is mainly extending existing meta code not really adding new code paths. If a developer understands meta code they will easily be able to understand root code as it is merely an extension of the other. It would be great if we could come up with a solution that offers the simplicity of both. If not then it’d be great to come up with one that offers the best benefit factoring in any tradeoffs. I think this is where understanding different perspectives would be helpful. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192736#comment-17192736 ] Francis Christopher Liu commented on HBASE-11288: - In software projects sometimes the biggest bugs aren't in the code but in relationships, as well as possibly the most difficult to debug. Collaboration on this jira has failed and does not seem possible which is quite unfortunate. Based on my understanding of the discussions on this jira and in the docs it seems the root as a table approach has merit hence in the interest of letting the community decide what's best I will continue to work on this patch. I will be happy to continue objective discussions whether it is towards improving the patch or arguing against it. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184027#comment-17184027 ] Guanghao Zhang commented on HBASE-11288: IMO, the "master local region" solution is not a specialization now. Firstly, we already store the procedure stuff to master local region now. Secondly, in other discussions, we also thought other system sutffs can use the "master local region", too, such as the meta data of hbase replication. More impormantly, the "master local region" solution is easy to understand and implement. It reuse a lot of "HRegion" code. If I am not wrong, Duo remove many code (which only few people understand.) when implement Region Procedure Store. It make our code base simple. KISS is great design principle, forever. And for assignment framework, it already too complicated now. We meet too many "assign problem" previously. It was a huge pain for many hbase users. More system tables introduced, the assign framework more complicated. So I hope the solution should not introduce more complexity to HBase. Thanks. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183943#comment-17183943
]
Duo Zhang commented on HBASE-11288:
---
{quote}
Master Local Region is a specialization because it makes root no longer the
same as hbase:meta. It is no longer is a table like meta table: local region,
does not host region on a regionserver, etc. Solutions to root cannot always be
generalized to hbase:meta.
In itself Master Local Region implementation has its merits but when taking it
as part of a whole: the catalog, I’m of the opinion we’d need a compelling
reason to justify the specialization?
{quote}
I'm really really tierd here. I've posted my opinion many times on this but no
matter what I say, you still just ignore it and repeat the same words many many
times. 'My solutoin is general, you solution is specialized'.
{quote}
For the purpose of understanding where this discussion is going. Earlier you
mentioned if we could get a reasonable result from the ITBLL run you would
change your mind. Is this still the case?
{quote}
I've already replied above. This is not a fair play to me. Why I said these
words, was just because I wanted to make progress and also get the same
response back, on how you could change your mind. This is your answer:
{quote}
What value are we getting out of having a specialized solution for root instead
of fixing it for both the catalog tables (root and meta) as a whole and is it
worth it? If there is another answer that is compelling like 2-tiered
assignment readiness. That would definitely be something that would change my
current thinking on the “master local region” approach. Hopefully I addressed
your concern?
{quote}
And there is a trap that, for passing ITBLL, you do not need to convence
anyone, but your response here, means I need to convence you. So things get
back to what I said above, though I explained a lot on what is 'general' and
what is 'specialized', you just ignoreed all the my posts and repeated your
words, and then using the ITBLL result to force me change my mind. I think I've
been fooled here. So my answer is, I will never change my mind. Just
introducing a root table but even do not support splitting meta, do not need
too much code, for my solution it is also easy to pass ITBLL with so little new
feature. So what is value here?
Based on the recent activity, I do not think we can reach an agreement here.
You can do what you want here and I will open another issue to land my solution.
Thanks.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
> Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183903#comment-17183903 ] Francis Christopher Liu commented on HBASE-11288: - I see thanks for confirming my understanding of the patch as well as elaborating on it more [~zhangduo]. Master Local Region is a specialization because it makes root no longer the same as hbase:meta. It is no longer is a table like meta table: local region, does not host region on a regionserver, etc. Solutions to root cannot always be generalized to hbase:meta. In itself Master Local Region implementation has its merits but when taking it as part of a whole: the catalog, I’m of the opinion we’d need a compelling reason to justify the specialization? I see sounds like you’re implying that a simple api could work for “general root table” as well? Assuming there is no compelling reason to go with Master Local Region implementation. For the purpose of understanding where this discussion is going. Earlier you mentioned if we could get a reasonable result from the ITBLL run you would change your mind. Is this still the case? > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17181039#comment-17181039 ] Duo Zhang commented on HBASE-11288: --- And another thing is what I have said in the past, exposing only a simple locating API to client will give us more flexibility. For now we will use master local region because it is easy to implement, but later we could also change to use a general root table. Client will not see any difference as it does not care whether you access a master local region or a general table at master side, just return what the client want is enough. And in HBASE-24765, we add the ability to return a list of Endpoint to client for getting the bootstrap information. For now we will return all the master nodes, maybe we could change it in the future to return a list of region server holding the root region replicas so we could make use of region servers to distribute the load. But if we go with general root table and expose the table interface to client, it will be hard for us to switch to other solutions in the future. Wire compatibility is very important even between major releases, and locating root is one of the necessary step of all client read/write requests. Thanks. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17179070#comment-17179070
]
Duo Zhang commented on HBASE-11288:
---
{quote}
I see in your solution is root a table like meta table would the code paths be
more or less the same? Would we be able to apply generalized solutions for both
or would they be different code paths? Are these changes in the splittable-meta
branch?
{quote}
Ahha, so you never looked at my patch, but then made a statement that my patch
is 'specialized', while your patch is 'generalized'.
Well, let me explain to you so you could save some time.
First, we already have a master local region, for storing the procedure data.
We could just add another family to it, for storing the root 'table'. It
requires less code and less complexities comparing to your 'general root
table'. As we do not need to introduce new InitRootProcedure, do not need to
modify WALFactory to add a special WALProvider for root table, and also do not
need to modify SCP to deal with the specialized WAL file for root table. And
since it is a region, we can reuse most of the code for accessing meta. You can
see the code in RegionStateStore, only at the last step we need to determine
whether we should go to the root region or the meta table.
Second, on distributing the load of root table, we are reusing the framework
introduced by HBASE-18095. As I said before many times, this is not a
'specialized' solution. And if consider the information in root as the
'bootstrap' information, it is very natural to use this solution. And now, we
are actually building a 'specialized' read replica implementation for meta
table, as we can not make use of the replication framework to spread the data
of meta table. We are stilling discussing, please see HBASE-18070 for more
details. Anyway, if this can be done, you can reuse this framework to
distributed the load, but I could also make use of the framework in HBASE-18095
to distributed the load. And what's more, HBASE-18095 has already been done,
and you can see the PR for HBASE-24459, after refactoring, we only need 130+
more lines of code to support distrbuting the load of root, and the solution is
also straigt-forward.
Do you have any other concerns? [~toffer].
Thanks.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
> Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177667#comment-17177667 ] Francis Christopher Liu commented on HBASE-11288: - I see apologies I thought you wanted my thoughts I missed the question. As mentioned root is half of Catalog its responsibilities are similar to that of the meta table. The benefit to root being a table just like meta is a table is when we need to do/apply something to Catalog we can come up with a generalized approach that can be done/applied to both. Having a generalized approach we avoid unneeded complexities hence my questions about it being worth it, etc. Does that answer your question? I see in your solution is root a table like meta table would the code paths be more or less the same? Would we be able to apply generalized solutions for both or would they be different code paths? Are these changes in the splittable-meta branch? Yes indeed it’s good to know the proc-v2 is stable and working well. Makes me look forward to deploying hbase-2 a bit more. I think the intent would be to pick one and stick with it for a long time. Hence the rigor in deliberation. In which case would adding new states in SCP still be a concern? I see I apologize if I sounded pushy I did not mean to. There could be other solutions that are good, but if the solution involves specializing then I’d also like to understand whether that tradeoff is compelling. There could be reasons that would make that tradeoff compelling I just don't know about it. If there was a compelling reason as mentioned I would change my current thinking on "local region on master". > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176644#comment-17176644 ] Duo Zhang commented on HBASE-11288: --- But you still do not answer my question directly right? Why root must be a table? In my implementation, root is still stored as a ‘region’, so we could still reuse most of the code right? And I’ve done a refactoring on master for generalizing a CatalogFamilyFormat, for putting these methods. And now we already have a framework to distributed the load of root, so there is no such ‘specialized solution’, we just use an existing solution, for distributed ‘cluster bootstrap information’. Passing ITBLL is good, even without splitting support, this means our proc-v2 framework is stable enough after the several years polishing, to support complicated logic, and introducing root table is generally fine. Though I still do not like that we introduce a lot of new state in SCP. It is easy to add new states but hard to delete them. But I’m very disappointed that, starting from the first comment, you never changed anything. Though I explained a lot, you just did not see and kept saying root as table is the only good way, all other ways are compromise. You just kept pushing your solution, I do not think this is a good way for collaboration. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176265#comment-17176265 ] Francis Christopher Liu commented on HBASE-11288: - The run I mentioned in the previous comment completed successfully. I'll try to run using AsyncFSWAL and remove workaround #4. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176263#comment-17176263 ] Francis Christopher Liu commented on HBASE-11288: - Hi [~zhangduo], there's a lot of code for a few reasons: 1. The way I wrote the patch was to be easy to read how adding root is merely piggybacking/extending on a lot of the meta code hence there's a lot of copy paste, 2. The code to split meta will be basically touching existing code that's been touched already but will be generalized. 3. There's already some code that went in to address splittable meta. 4. There's some refactoring (renaming root to meta), generalization, etc. The production could changes would mainly be generalizing and extending existing code paths. I could try and cleanup the branch-2 patch so we can see what it would look like if that's a big concern? With regards to biasing root as a table. Root is half of the catalog it's responsibilities are sort of similar to that of the meta table hence I think there's some reasonable motivation with the bias. In which case I would like to understand why we are doing the specialized solution for root instead of coming up with a general one for the catalog as a whole? Also why is it worth it? Sorry if I sound like a broken record, I see that you responded to my question in a previous post. I just want to make sure I understand your current position (as some things might've changed these past 2 weeks)? > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175937#comment-17175937
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288.splittable-meta
[build #22 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/22/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/22/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/22/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/22/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/22//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
> Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175238#comment-17175238 ] Duo Zhang commented on HBASE-11288: --- I’m just a bit surprised that with so many code in place we still do not support meta split yet? I tried to write a UT to test split based on your branch and found that the region server will reject the request. And what do you think of my post above? I think there is a bias in our discussion, that ROOT 'must be a table'. If we take a look from the client interface, you can see that, now in ConnectionRegistry, we have a method to get the location of the single meta region, it is considered as the 'bootstrap information' of a hbase cluster. After meta can be split, the only difference is that, we should allow users to get addresses for different meta regions. We could still consider this as the 'bootstrap information'. From the client side, we do not care how to store the data of ROOT. If we think from this direction, distributing the load of ROOT, is exactly the same with distributing the load of the 'bootstrap information', which is exactly what we have done in HBASE-18095. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175229#comment-17175229 ] Francis Christopher Liu commented on HBASE-11288: - Apologies for the late reply [~zhangduo]. Yes it just adds root table (another tier). There is no split meta support. Did you want to test split meta as well? We could do that I'll just need some time to backport the changes that I've done on master as well as add the changes needed for failover (eg scp), etc. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175227#comment-17175227 ] Francis Christopher Liu commented on HBASE-11288: - Apologies for the late reply. I got caught up dealing with the issues to get my setup running. Right now it's running and 4 of 5 iterations has finished. I'm using the command [~stack] shared. Here's a rundown of some of the things I dealt with so far. 1. Disabled netty native transport - regionservers JVM's would basically freeze. Couldn't even do jstack without "-F". I found out that internally we don't run with native transport for hadoop as it has a lot of issues apparently (wrong fds getting closed, etc). 2. Disabled security - there was some issue with TokenProvider throwing IllegalThreadStateException on startup causing the RS to abort 3. Hadoop-2.8 and zookeeper 3.4.x - made some quick changes so I could downgrade the dependencies as these are what we use internally. Shouldn't affect the validity of the test? 4. MonitorTaskImpl.prettyPrintJournal was throwing an NPE causing RS to abort - might be related to me using "filesystem" as the wal provider. I changed the code to swallow the exception for now as it seems unrelated. Will try and rerun the test use Async wal provider. 5. SCP bug introduced by patch - caused meta not to get assigned. Caught and fixed this quick and early. 6. [~stack] root executor patch - didn't run into this, but makes sense. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175122#comment-17175122 ] Duo Zhang commented on HBASE-11288: --- Ping [~toffer]. Mind answering the question above? Thanks. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17173736#comment-17173736 ] Duo Zhang commented on HBASE-11288: --- Hi [~toffer], so your patch just adds a root table, without splitting meta support? > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17173693#comment-17173693 ] Michael Stack commented on HBASE-11288: --- After adding the priority patch, my ITBLL run from last night completed all 5 cycles of 1B each. branch-2 w/ HBASE-11288.branch-2+priority does as 'well' as branch-2 w/o the patch. Will do some more runs to see if can find any more deadlocks, etc. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17173529#comment-17173529 ] Duo Zhang commented on HBASE-11288: --- Thank you guys, great progress here. >From my experience, we met the same situation when testing TRSP with ITBLL. >That is, no data loss, but it is really hard to finish a whole ITBLL run. It >will always hit rpc timeout when putting or verifying. I think this is easy to understand. We do not change the WAL implementation, and do not change the actually WAL splitting either(we just change how we schedule WAL splitting), so it is not likely to introduce data loss. It is not easy to introduce data lost even for a double assign, it will just introduce FileNotFoundException when the stale region compacts HFiles and then fail the requests(it can be fixed by a RS restarting). What we change here is the critical procedures for dealing with region assign, and also master startup. When there are corner cases we do not handle correctly, the result is a cluster hang. Sometimes the cluster will hang for a very long time and then recover by itself, and usually you will not sit there to see the ITBLL right? So when you come back, you will just see a healthy cluster and a rpc timeout in the ITBLL log. And what you can see is that a region can not online for a very long time but finally it is online, so you will think this is just because the cluster is overloaded. But actually, if you run ITBLL enough times, you will finally get a chance to see the cluster in hanging state, and then find out the root cause. I still remember that we had done a lot of works to deal with the corner cases but still could not fully solve the problem until I noticed that we were on the wrong direction and introduced HBASE-22074. It is really not easy to fix all the corner cases. Thanks. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Attachments: jstack20200807_bad_rpc_priority.txt, root_priority.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17173524#comment-17173524 ] Michael Stack commented on HBASE-11288: --- On latest ITBLL run, I ran into a deadlock, a combination of chaos monkey DDOS'ing the 'default' RPC handlers and then hbase:root failing to report successful open because its 'priority' was default rather than priority – it was locked out. Will attach jstack and patch to make it so hbase:root ops get priority. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17173205#comment-17173205
]
Michael Stack commented on HBASE-11288:
---
13 servers in the cluster. Here are some details on what the ITBLL runs are
like from the launch script (the script is neat and tidy because it [~ndimiduk]
's, not mine):
{code:java}
ITBLL_LOOP_ITERATIONS='5'
ITBLL_LOOP_NUM_MAPPERS='2' # 13 region servers / node managers in the cluster
ITBLL_LOOP_NUM_NODES_PER_MAPPER='5' # multiple of 25x10^6
ITBLL_LOOP_OUTPUT_DIR="./itbll-output-$(date -u --rfc-3339=seconds | tr ' ' 'T'
| tr ':' '-' | cut -d+ -f1)"
ITBLL_LOOP_NUM_REDUCERS='13'env JAVA_HOME="${JAVA_HOME}" \
HBASE_CLASSPATH='.' \
HBASE_OPTS="${HBASE_OPTS[*]}" \
hbase \
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList \
"${CONFIGS[@]}" \
-monkeyProps "${MONKEY_PROPERTIES}" \
--monkey serverKilling \
-ncc \
loop \
"${ITBLL_LOOP_ITERATIONS}" \
"${ITBLL_LOOP_NUM_MAPPERS}" \
"${ITBLL_LOOP_NUM_NODES_PER_MAPPER}" \
"${ITBLL_LOOP_OUTPUT_DIR}" \
"${ITBLL_LOOP_NUM_REDUCERS}"
$ more monkey.properties
rolling.batch.suspend.rs.ratio = 0.0{code}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17173190#comment-17173190
]
Michael Stack commented on HBASE-11288:
---
Update: I have been running the HBASE-11288.branch-2 on our little ITBLL rig
here of 12 nodes or so. Doing the baseline I ran 5 cycles of 1B rows on each
run – which is what we've been doing to 'check' if a version is basically
sound. With root in the mix, all seems to basically work. On my first run, we
stalled after the second cycle after two runs checked out w/ all REFERENCES in
place – it seems to be an issue w/ my rig rather than hbase. On second run over
last night, we did one cycle and then stalled cluster seems healthy so
again seems like issue w/ the rig rather than hbase but let me run again.
{code:java}
hbase(main):001:0> scan 'hbase:root'
ROW COLUMN+CELL
hbase:meta,,1
column=info:regioninfo, timestamp=2020-08-07T09:30:28.921Z,
value=PBUF\x08\x01\x12\x0D\x0A\x05hbase\x12\x04meta\x1A\x00"\x00(\x000\x008\x00
hbase:meta,,1
column=info:seqnumDuringOpen, timestamp=2020-08-07T09:30:28.921Z,
value=\x00\x00\x00\x00\x00\x00M\xB1
hbase:meta,,1
column=info:server, timestamp=2020-08-07T09:30:28.921Z,
value=hbasedn192.example.org:16020
hbase:meta,,1
column=info:serverstartcode, timestamp=2020-08-07T09:30:28.921Z,
value=\x00\x00\x01s\xC8/g\x87
hbase:meta,,1 column=info:sn,
timestamp=2020-08-07T09:30:28.632Z,
value=hbasedn192.example.org,16020,1596791416711
hbase:meta,,1 column=info:state,
timestamp=2020-08-07T09:30:28.921Z, value=OPEN
hbase:meta,,1 column=info:v,
timestamp=2020-08-06T18:53:08.596Z, value=\x00\x01
1 row(s)
Took 0.2852 seconds
{code}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171746#comment-17171746 ] Michael Stack commented on HBASE-11288: --- Started baseline running branch-2 at place where you added patch [~toffer] Will report back... > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171489#comment-17171489 ] Duo Zhang commented on HBASE-11288: --- Waiting for your good news. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171397#comment-17171397 ] Francis Christopher Liu commented on HBASE-11288: - Hi Guys, apologies for the delay. I've pushed up HBASE-11288.branch-2, which is a rebase of the master branch patch. The rebase is a little older than current HEAD as it didn't seem stable when I was rebasing onto it last weekend. I've disabled some unit tests which seems unneeded for the PoC, there are a few tests that are failing on my desktop but not my mac. I've deployed it on a test cluster and it comes up and basic stuff is working. I'll probably do another pass to see if there's anything else but having said that I think it's good enough to give it a try. I'll reach out to [~stack] tomorrow about running ITBLL. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164474#comment-17164474
]
Michael Stack commented on HBASE-11288:
---
I think ITBLL run needs a week at least; always takes longer than expected
(smile).
Zach is talking about HBASE-24749
{quote}There is no such 'general' solution yet, but we already have a framework
to support the 'specialized' solution
{quote}
True. There are some parts and some tough problems to solve still before we
have a 'general' solution. We need one though reading [link Discussion: ROOT &
‘hotspotting’|https://docs.google.com/document/d/12B1BOMLx_esFycILdHmIYh0NKOzWUJ1I9XuvL75Euug/edit?ts=5f16ea73#heading=h.7lpctjymn8et]
The 'specialized' solution cannot be generalized; it is ROOT only.
{quote}bq. I think there is a bias in our discussion, that ROOT 'must be a
table'.
{quote}
I think this observation helpful. Duo would classify _hbase:meta_ locations the
same as cluster id or the location of active master – i.e. cluster metadata. If
the ROOT data small (1M ? Perhaps ~1k locations) then it doesn't take a stretch
thinking of it as 'bootstrap information'.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164372#comment-17164372
]
Duo Zhang commented on HBASE-11288:
---
{quote}
What value are we getting out of having a specialized solution for root instead
of fixing it for both the catalog tables (root and meta) as a whole and is it
worth it?
{quote}
I think I have answered this question many times... There is no such 'general'
solution yet, but we already have a framework to support the 'specialized'
solution, and it has already been implemented in HBASE-24459, and the long term
solution is HBASE-24753.
I think there is a bias in our discussion, that ROOT 'must be a table'. If we
take a look from the client interface, you can see that, now in
ConnectionRegistry, we have a method to get the location of the single meta
region, it is considered as the 'bootstrap information' of a hbase cluster.
After meta can be split, the only difference is that, we should allow users to
get addresses for different meta regions. We could still consider this as the
'bootstrap information'. From the client side, we do not care how to store the
data of ROOT. If we think from this direction, distributing the load of ROOT,
is exactly the same with distributing the load of the 'bootstrap information',
which is exactly what we have done in HBASE-18095.
Anyway, let's wait for the ITBLL result first. And then we could discuss again.
Thanks.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164343#comment-17164343 ] Francis Christopher Liu commented on HBASE-11288: - A deadline sounds fine to me as well. And yes I would definitely appreciate the help [~stack] as well as another perspective when we're writing the assessment. I've not done ITBLL with procv2 assignment issues before so my estimates may be a bit off, off-hand one week sounds like a best case scenario, two weeks sounds more reasonable (assuming no surprises at work). Let me think about it more. What do you think [~stack]? I'm assuming this estimate excludes the rebasing on branch-2, although I'm hoping it would be quick if the differences aren't that big. Just trying to make sure we won’t be writing an assessment we’ll find lacking reading it year or so from now. As mentioned before if the ITBLL test shows procv2 is not ready for 2-tiered assignment then it’s clear “general root table” is not an option. I proposed this when 2-tiered assignment readiness seemed to be the most concerning. When it became apparent the concern was because of hot spotting on the root table, I asked a question similar to this: What value are we getting out of having a specialized solution for root instead of fixing it for both the catalog tables (root and meta) as a whole and is it worth it? If there is another answer that is compelling like 2-tiered assignment readiness. That would definitely be something that would change my current thinking on the “master local region” approach. Hopefully I addressed your concern? Nice article [~stack] will go through it. [~zyork] good to see that this is getting picked up again. What's the jira for this again? > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164059#comment-17164059 ] Zach York commented on HBASE-11288: --- Since (as Duo pointed out) this is relevant to the work to store File locations in meta, I will get up to speed on the discussion and current status. I won't impede the process, but will be happy to help with reviews/discussions. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164023#comment-17164023 ] Michael Stack commented on HBASE-11288: --- A deadline seems reasonable to me for making a call on how to proceed. I offer to help w/ the ITBLL eval if that'd be of use (on branch-2, what I know). As an aside, pardon if you've seen this already, but I liked this post on the merits of the design process and of the value documenting decisions: [https://www.industrialempathy.com/posts/design-docs-at-google/] > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163989#comment-17163989
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288.splittable-meta
[build #18 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/18/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/18/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/18/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/18/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163415#comment-17163415
]
Duo Zhang commented on HBASE-11288:
---
{quote}
I’ll do my best to get ITBLL to pass.
{quote}
You do not need to get ITBLL to pass. I do not think the work can be done
within a reasonable time. [~ndimiduk] has spent several month to archive this.
Instead, set a deadline, 1 week or 2 weeks, then we come back to analyze the
result, on how to explain the failures, and also the complexity of the code to
archive the current state. Of course it will be good if you can pass the ITBLL.
And an interesting thing is that, I offer that, I will change my mind if you
can get a reasonable result, then what can make you change your mind? We both
have a solution, I do not think it is fair to me that, only I offer how to
change my mind, but you do not offer anything to change your mind.
Thanks.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163398#comment-17163398 ] Francis Christopher Liu commented on HBASE-11288: - [~stack] and [~zhangduo] thank you for your suggestions of avoiding master branch and just rebasing on branch-2. I will go ahead and do that then. I will try to get as much information of course. I’ll do my best to get ITBLL to pass. [~zhangduo] Good to hear you are willing to change your mind with a reasonable result. As my intention has always been not only to prove/disprove it to myself but to you and [~stack] as well to help come to a consensus on a path forward. Looking at [~stack]’s recent comment the consensus seems to be that there was no consensus on a path forward. Although as [~stack] pointed out [~apurtell] did suggest we just do a reset so I missed the signs that we were doing that so my mistake. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163154#comment-17163154 ] Duo Zhang commented on HBASE-11288: --- I have to explain a bit here so people will not misunderstand me. The ITBLL stuff is proposed by you [~toffer], so I do not spend much time on you as I think you will accept it. The cache stuff is considered as off topic many times here so I spent a lot of time to convience others and finally [~stack] promised to open a separated issue to discuss it, and he did lots of work to open a design doc and collect feedbacks. And I've already done implementing HBASE-24459 to show how we can implement cache server for master local region. It is a bit surprise to me that now you say we do not have an consensus yet so you are just waiting. Maybe it is my fault, I should ask you more times on it, though it is proposed by you. And back to ITBLL, yes, I had a different opinion at the first place as I did not think it would provide any useful feedbacks, but what I also said is that, you are free to run it as you like, you prove your statement. [~stack] sir, what I mean is, I do not think the RESULT can change my mind, as [~toffer] also said in the latest reply, master itself is not stable, so the base line is a problem. It does mean, I will never change my mind, no matter what you do. This is completely different. If you can get a reasonable result from the ITBLL run, I will change my mind. And on technical suggestion on the ITBLL, since it is only a POC, and the assignment part is almost the same on master and branch-2, you could try to rebase your currect POC to branch-2? [~stack] and [~ndimiduk] have done a lot of work to stabilize branch-2, so I think the base line of branch-2 will be much better. And when failure happens, you need to find out the root cause, it is because the coded added by you or not. Of course it will good if we can pass ITBLL. And I want to speak again, let's make progress here, this is an very important feature. Recently the AWS guys want to add more things(the storefiles list) to meta table, which will make it more larger. So [~toffer], please do not say that you are just waiting, you are part of the community. You should show others your plan on how to make progress here. Thanks. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162937#comment-17162937
]
Michael Stack commented on HBASE-11288:
---
{quote}I seem to have made a mistake and thought that we'd come to an agreement
before moving forward so I've just been waiting. Looks like people are doing
what they mentioned they wanted to do, so to move things forward on my end let
me go ahead then and run ITBLL. 2-tiered assignment may not be the most
contentious thing right now but it sounded like it was still important? So I
plan to first start run ITBLL on unchanged master to get a baseline and then
run with the patch. Has anyone run ITBLL on master? Is there a particular
commit that I should be using, etc?
{quote}
Understood. There was no agreement on a path forward, just a reset, listing of
what we want from this issue as suggested by [~apurtell]
[~zhangduo] repeated a suggestion he'd made a few times. There was no sign-off.
I did the piece that had been suggested of me – ROOT load distribution
discussion -- in an effort at moving this project forward since this was
posited a blocker (though I thought otherwise). The ITBLL testing, you
suggested you'd do to demonstrate tiered assign is (perhaps) not a problem, the
main objection to the ROOT-as-general-Region approach, would seem like a useful
exercise but it seems whatever the result, it won't change Duo's mind (as he
states above – correct me if I have this wrong) which seems like a problem.
On ITBLL against Master, I'm not sure it even passes. It works on branch-2 most
of the time there is an odd failure that needs looking into. I've not tried
Master.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162691#comment-17162691 ] Francis Christopher Liu commented on HBASE-11288: - I seem to have made a mistake and thought that we'd come to an agreement before moving forward so I've just been waiting. Looks like people are doing what they mentioned they wanted to do, so to move things forward on my end let me go ahead then and run ITBLL. 2-tiered assignment may not be the most contentious thing right now but it sounded like it was still important? So I plan to first start run ITBLL on unchanged master to get a baseline and then run with the patch. Has anyone run ITBLL on master? Is there a particular commit that I should be using, etc? > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162683#comment-17162683
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
I found a version in one of my local repo and pushed it back to HBASE-11288.
Please take a look to see if it is the newest one. Sorry again. Francis
Christopher Liu.
{quote}
No worries [~zhangduo] thanks for fixing it.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17160834#comment-17160834
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288.splittable-meta
[build #17 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/17/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/17/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/17/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/17/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17160442#comment-17160442 ] Duo Zhang commented on HBASE-11288: --- Posted a patch for HBASE-24459. https://github.com/apache/hbase/pull/2095 PTAL. Thanks. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17159008#comment-17159008
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288
[build #3 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288/3/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288/3/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288/3/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288/3/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288/3/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158891#comment-17158891 ] Michael Stack commented on HBASE-11288: --- I put up a discussion doc on distributing ROOT load: [https://docs.google.com/document/d/12B1BOMLx_esFycILdHmIYh0NKOzWUJ1I9XuvL75Euug/edit?usp=sharing] TL;DR asks if concern for ROOT loading is the proper focus; rather should we be solving hotspotting/meta access prioritization/etc for the general case. If ROOT loading is critical, has some starts on possible implementations to get the discussion going. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158755#comment-17158755
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288.splittable-meta
[build #16 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/16/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/16/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/16/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/16/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158180#comment-17158180 ] Duo Zhang commented on HBASE-11288: --- I found a version in one of my local repo and pushed it back to HBASE-11288. Please take a look to see if it is the newest one. Sorry again. [~toffer]. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158152#comment-17158152
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288
[build #2 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288/2/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288/2/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288/2/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288/2/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157932#comment-17157932
]
Duo Zhang commented on HBASE-11288:
---
Oh, shit. [~toffer] Sorry that when rebasing HBASE-11288.splittable-meta for
implementing HBASE-24459, I accidentally pushed to HBASE-11288 instead of
HBASE-11288.splittable-meta. Do you still have a local branch for it? You can
do a force push to get it back.
And please see the proposal above, let's make some progress here.
{quote}
Anyway, I think a possible way to make progress is what I proposed above,
Francis Christopher Liu go with ITBLL on his POC to see if we can get some
useful results, and Michael Stack to file an issue to brainstorm on how to
distributed the load of both meta and root, or maybe even a general table(I
offer my help to polish it), and I will try to implement at least a POC for
HBASE-24459(actually I used to plan to leave this issue to Bharath Vissapragada
as this is his area) to show my plan on distribute the load of root table.
{quote}
Thanks.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157846#comment-17157846 ] Francis Christopher Liu commented on HBASE-11288: - I am all for “collaborate and work out a path forward”. How do we go about doing that? My previous proposal was tackling the most pressing issue which seemed based on earlier discussions and the documentation was tiering. Any proposal? I have some technical responses to the recents posts here but will try to hold off until we can agree on a path forward. (64 words) > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156810#comment-17156810 ] Michael Stack commented on HBASE-11288: --- Linked to issue for figuring out a distributed cache for the non-splittable root location issue. Suggest we move further discussion of cache there. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156783#comment-17156783 ] Michael Stack commented on HBASE-11288: --- In this issue, I want a split meta – not fixes for hotspotting nor to develop a cache for the 'meta' table. There are alternative approaches splitting meta – one that has a version in production at scale on hbase1 with multiple attempts at landing the large patch updated for hbase2 and another that is being made up on the fly via PRs making use of some nice new features that have emerged of late. Given alternatives, I want us to collaborate and work out a path forward; an agreed upon design with our decisions. (95 words not including these). > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156766#comment-17156766 ] Duo Zhang commented on HBASE-11288: --- Thanks [~apurtell]. I just want to make progress here as I think this is an important issue. I reviewed the comments above again, at the first place I tried to discuss, I mentioned many times about how to distribute the load of root table, but got no answer. And then I started to implement the splittable meta to show how I can finish the work, then I was told(offline) to stop, then I stopped and went back to discuss again. But the situation is still the same, no one wants to answer the question about how to distribute the load of root table... I'm really dispirited that after two months, we made little progress. I'm still asking the same question and still getting the same answer(actually no answer)... Anyway, I think a possible way to make progress is what I proposed above, [~toffer] go with ITBLL on his POC to see if we can get some useful results, and [~stack] to file an issue to brainstorm on how to distributed the load of both meta and root, or maybe even a general table(I offer my help to polish it), and I will try to implement at least a POC for HBASE-24459(actually I used to plan to leave this issue to [~bharathv] as this is his area) to show my plan on distribute the load of root table. What do you guys think? Thanks. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156347#comment-17156347 ] Andrew Kyle Purtell commented on HBASE-11288: - Here is my suggestion: Reset this discussion. All parties propose what you want to do. Do not use more than 100 words to make your proposal. Spend effort to get maximum economy of language. In this case discussion is not the best tool. Focus on code. Focus on collaboration - by pull request. Review step by step. So maybe start again with the feature branch and take it step by step. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156345#comment-17156345 ] Andrew Kyle Purtell commented on HBASE-11288: - Glad for the candid language and transparency, it is helpful. With my PMC hat on I am not interested in technical details at this point. This is my concern: bq. so if you guys have a design on how to deal with the hot spot, I offer to help that jira first and leave this jira to you. I just want to do the most important thing to help our users. Duo, I don’t think the actions of the others were intended to play games. Especially the design doc is an honest attempt to resolve discussion of alternatives. Unfortunately as you point out conflict resolution on this project requires discussion and debate and the language issues are a real problem. I think us English speakers can acknowledge you have been very patient up to now. Sometimes discussion must go at length to resolve disputes. An outcome where one person is worn down, though, is not a good result. What I have quoted from you above is a good suggestion, at least there would still be some collaboration, but I wonder if more can be done to rebuild trust. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156240#comment-17156240
]
Duo Zhang commented on HBASE-11288:
---
{quote}
I can file a sub-issue to address as a follow-on – I like the Francis
suggestion that we might broaden the cache discussion to cover more than ROOT
only – but it is out-of-scope for this Jira.
{quote}
Please, file a jira for this. But this is not a follow-on, it is a blocker for
the 'general root table', and it is even more important than splittable meta.
Most users do not need to serve 1M regions but their meta region can also be
hammered. So if you guys have a design on how to deal with the hot spot, I
offer to help that jira first and leave this jira to you. I just want to do the
most important thing to help our users.
But I hope you guys truly have a design, or at least a workable idea, on how to
deal with hot spot regions. If not, a..., OK, I quit. It is enough for me
to play the 'this is good for the community' game. There is another rule for me
is, I will treat you like how you treat me.
I asked you to create a feature branch to land your big patch piece by piece,
and then you create a feature branch without telling anyone and made a lot of
commits to it, so I had to open sub tasks to land my solution too. At least I
always asked others to review before committing a patch.
I'm asked to stop because we haven't reached an agreement yet, OK, I stopped,
and filled the design doc, and started to discuss with you guys again. And then
I found that my words on design doc were removed many times, and the discussion
here did not make any progress, you guys kept ignoring the disadvantages of
your solution and refused to give answers, just using debating skills to shut
me up. English is not my mother language, I'm not good at debating in English.
if we are speaking Chinese I'm willing to play with you guys more but since
we're talking with English, I quit, you win, fair enough. My only way to fight
back is my coding skill and my undstanding of the architectures of HBase.
That's all. Sincerely.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156170#comment-17156170 ] Duo Zhang commented on HBASE-11288: --- I’m really confused. Why you guys keep rejecting to give a clear design on the caching stuff? When I was asked to give a clear design, I even filled the design doc with an implementation plan to explain what I thought, to help others understand. Caching is what I said, one of the advantage of ‘master local region’, and you guys first say that this is not an advantage of ‘master local region’, and remove it from the design doc. So I asked how do you plan to implement the caching for ‘general root table’, if it is the same with ‘master local region’ then the pros for ‘general root table’ are all gone. Then you guys claimed that it is not the same. So I asked again, please give a clear design. This time you guys said this should be addressed as a whole, not only for root table, but still, refuse to give a clear design, and even refused to discuss this in this jira and define it as ‘off topic’. The Pros for ‘general root table’ in the design are even written by me, but the Pros for my solution are always being ignored and removed from the design doc. In the discussion here, one of the advantages is defined as a ‘compromise’, and the other is defined as ‘off topic’. What do you guys think if you are treating like this? Not a double standard? So I will try to be constructive the last time, please give a design on how you plan to deal with hot spot regions as a whole in the HBase and how we can use it to deal with the hot spot of ‘general root table’. We should not rely on something does not exist yet, as the hot spot for root table is a real problem. If you guys still keep refusing to answer the question but trying to fight me by back by setting the rules on what can be talked, I will not discuss here any more. Just waste of time. I will give the jira back to you guys and you can do what ever you like and I will file another jira to complete my implementation. I will try to get help for others in the community. Sincerely. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156138#comment-17156138
]
Michael Stack commented on HBASE-11288:
---
{quote}I think this is a important now as [~toffer] keep saying that it should
be the same as 'master local region' but obviously, your argument later that we
should not involve masters means it is a different solition.
{quote}
Pardon. Asking if cache should be considered in here was a mistake. It has
taken us off-topic. This is an issue on splitting meta. That we still have
hot-spotting post-split-meta is moot. I can file a sub-issue to address as a
follow-on – I like the Francis suggestion that we might broaden the cache
discussion to cover more than ROOT only – but it is out-of-scope for this Jira.
The only reason we are talking caching in the first place – caching the ROOT
Table when we don't have caching for the even hotter hbase:meta Region -- is as
an amelioration for a downside of in-Master ROOT.
{quote}
Please give a clear design. I was attacked at the first place because I did not
give out a detailed design, it hurts my feelings because it seems like a double
standard.
{quote}
I missed where you were attacked (If anything, you seem to be the one on
'attack' on a reread of this issue). If double standard, please educate me so I
can undo my misbehavior – off-JIRA.
{quote}So let's go back to the plan above?
{quote}
Plan? I suggest we finish the design. The ITBLL work will help. Me doing a
cache design will not (Odd is your insistence that caching is only possible for
in-Master Region though client has a simple API hosted in a client-side
pluggable Interface). You pressing ahead and 'implementing' some Master-based
caching for Master Local Region comes across as you just being done w/ this
whole process.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155806#comment-17155806
]
Duo Zhang commented on HBASE-11288:
---
{quote}
Cache tier implementation is not part of this discussion unless you think it
needs to be. That the API locating ROOT content is simple and so easy to back
w/ a general cache is sufficient to our purposes here? Or do you think we need
to say more on this topic? (As to the cache implementation, I'd think we'd want
to avoid involving Masters in any caching solution if ROOT is located elsewhere
– involving Masters would undo a 'pro' of the 'general root table' direction as
you point out subsequently in your comment).
{quote}
I think this is a important now as [~toffer] keep saying that it should be the
same as 'master local region' but obviously, your argument later that we should
not involve masters means it is a different solition. Please give a clear
design. I was attacked at the first place because I did not give out a detailed
design, it hurts my feelings because it seems like a double standard.
So let's go back to the plan above? [~stack] please give a clear design on how
to implement the cache server for 'general root table' which do not need to
involve masters, and [~toffer] please go with your ITBLL on the POC to see if
we can get a useful result, and I will implement HBASE-24459 to show how to
deal with the caching for 'master local region'.
Thanks.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155801#comment-17155801 ] Duo Zhang commented on HBASE-11288: --- Generally speaking, caching is for solving hot root region, as a side effect, it could reduce the problem of master on critical path. And please stop saying things which do not exist yet. For addressing hot spot region, there are mainly two ways in HBase, one is split the hot region, the other is read replicas. The former one is not suitable for root region as it is not splittable, the latter one is not stable enough so we do not consider it. If you think you can fix them as a whole, please give a clear solution. Hot spot is a problem when HBase was born more than 10 years ago, and we still do not have a perfect solution to solve it, and even this issue is partly for solving the hot spot on meta region. You can not simply throw out a statement that ‘we should address them as a whole’ but do not give a solution, irresponsible. Thanks. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155797#comment-17155797 ] Francis Christopher Liu commented on HBASE-11288: - I think we are conflating the purpose of caching and that makes the evaluation a bit confusing. Here's my attempt at fleshing them out. And drawing some conclusions. There's essentially two cases: # Caching to prevent master to become part of read/write path - this use case is only presenet in "local master region". Caching here is used to reduce the effect of disadvantage. Bad enough to be required as part of any serious deployment(?). # Caching to avoid hot regions. - this is something potentially optional in "general root table". But arguably recommended in "local master region" given the sharing of master and root responsibilities in a single process. There is some nuance here as to why this caching use case is more important/recommended for "local master region" because of being colocated with the master especially when it comes to scaling(?). For 2, I think we are special casing root a little too much. For example in "general root table" implementation although splitting meta can reduce hot regions and help with scaling it does not eliminate it. In fact in our experience we run into hot meta regions more often than hot Root region. In which case I believe if we are considering caching as a solution for hot root region we need to think more holistically and think about a solution for Catalog Tables as a whole. Along those lines, although it's currently not a big deal for us, would it also make sense to consider availability for Catalog Tables as whole, which also can potentially be addressed with caching? The same can be said with the noisy neighbors argument does it really make sense to special case root instead of coming up with a holistic solution for Catalog Tables as a whole? It seems one potential advantage for "general root table" is being able to address issues with the Catalog Tables holistically potentially having a simpler and more elegant solution. The devil is in the details here of course and so the general question is how important is special casing root for any particular use case vs fixing it for catalog table as a whole. It at least seemed to me based on discussions and the design doc that the 2-tier was potentially that use case. What do you guys think? > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155781#comment-17155781
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288.splittable-meta
[build #15 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/15/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/15/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/15/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/15/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155573#comment-17155573
]
Michael Stack commented on HBASE-11288:
---
{quote}You mean that, with 'general root table', we will also only expose a
very simple API for locating meta?
{quote}
Yes. We could do this.
{quote}Client will not go to the root region directly but the cache on master
and backup master?
{quote}
Cache tier implementation is not part of this discussion unless you think it
needs to be. That the API locating ROOT content is simple and so easy to back
w/ a general cache is sufficient to our purposes here? Or do you think we need
to say more on this topic? (As to the cache implementation, I'd think we'd want
to avoid involving Masters in any caching solution if ROOT is located elsewhere
– involving Masters would undo a 'pro' of the 'general root table' direction as
you point out subsequently in your comment).
Your addition to the design doc seems good – I undid the bolded red (smile) –
but it seems we can't have this new addition and the point above it both stand
at the same time: revisit?
While extra-tier vs Master-in-line-for-meta-lookups are main pivot point
figuring where to locate ROOT, there are others like Master is writing an
in-process Region rather than remote RPC'ing and it'll have no 'noisy
neighbors'... I updated the design.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155225#comment-17155225
]
Duo Zhang commented on HBASE-11288:
---
I've updated the design doc to mention this.
Thanks.
{quote}
In section 2.1 we claim that we could also have cache servers for ‘general root
table’ by only providing a limited set of locating meta API, but if we go this
direction, all the above Pros are gone. It is not the traditional way of
locating meta through table API, we can not reuse the code of locating normal
regions to locating meta, hbck2 will also need to execute at master side only.
{quote}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155220#comment-17155220
]
Duo Zhang commented on HBASE-11288:
---
{quote}
I didn't remove it. Caching for ROOT migrated further down the doc. to its own
section. ROOT caching does not seem particular to any one of the two
suggestions for ROOT location; with simple location API for hbase:meta Regions
in ConnectionRegistry API, plugging in a ROOT cache tier seems doable in either
case. Thanks.
{quote}
I'm a bit confused. You mean that, with 'general root table', we will also only
expose a very simple API for locating meta? Then why do we still need a
'general root table'? And how do you plan to only expose a simple locating API
on top of a 'general root table'? Client will not go to the root region
directly but the cache on master and backup master?
If we go with this direction, why do we need a 'general root table'? All the 3
advantages listed in the design doc are completely gone. Comparing to 'master
local region', it only has a disadvantage that it needs 2-tier.
So if you both think the caching should work like this, then the discussion is
over. I do not see any advantages of 'general root table' then. Let's just go
with 'master local region'.
Thanks.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155200#comment-17155200
]
Duo Zhang commented on HBASE-11288:
---
{quote}
I see that’s good to know. So we can continue discussing on validating the main
difference (2-tier vs 1-tier)?
{quote}
I do not think 1-tier is a compromise, it is an advantage. No matter whether
2-tier can work, 2-tier is much more complicated than 1-tier right? If we can
go with a simple solution then why do we need to go with a much more
complicated solution?
I'm really confused that you just ignore the disadvantage of your solution and
even use your disadvantage to comparing with the advantage of my solution?
What's your point?
{quote}
The caching as mentioned is not specific to one implementation.
{quote}
Again. Please give a clear design doc.
Thanks.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155192#comment-17155192
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
Do you agree with the above statements?
{quote}
>From what I can read in the doc and my understanding of the discussions so far
>the main difference between the two implementations is the tiering. “Master
>local region”’s only advantage is that it stays 1-tier for assignment,
>although at the cost of making compromises to avoid 2-tier. The caching as
>mentioned is not specific to one implementation.
{quote}
After rethinking, I think you misuderstood me. It seems that you are trying to
show that the 'general root table' solution can work. It definately can work as
you already have a cluster running with the solution. This is not my point. I
do not mean the 'general root table' can not work.
{quote}
I see so you agree it can work for HBase? Yes it seemed unclear wether you
thought it could work or not in general. Thanks for clarifying that. If you
could also clarify another thing, you seemed to imply there might be problems
with it for HBase currently (Eg ProcV2 is not mature enough) is this correct?
Just trying to make sure I understand your position.
{quote}
In general, I think both of the solution can work. Here we just want to choose
the 'better' solution.
{quote}
I see that’s good to know. So we can continue discussing on validating the main
difference (2-tier vs 1-tier)?
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154723#comment-17154723
]
Michael Stack commented on HBASE-11288:
---
{quote}We have discussed this a lot in the design doc but seems [~stack]
removed most of the content.
{quote}
I didn't remove it. Caching for ROOT migrated further down the doc. to its own
section. ROOT caching does not seem particular to any one of the two
suggestions for ROOT location; with simple location API for hbase:meta Regions
in ConnectionRegistry API, plugging in a ROOT cache tier seems doable in either
case. Thanks.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154675#comment-17154675 ] Duo Zhang commented on HBASE-11288: --- After rethinking, I think you misuderstood me. It seems that you are trying to show that the 'general root table' solution can work. It definately can work as you already have a cluster running with the solution. This is not my point. I do not mean the 'general root table' can not work. In general, I think both of the solution can work. Here we just want to choose the 'better' solution. Thanks. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154294#comment-17154294 ] Duo Zhang commented on HBASE-11288: --- Let me put up the problems and the solutions for the two solutions again. For general root table, there are two problems: 1. '2-tier' of assignment/initialzation/bootstraping on master start up and fail recovery. A 'possible' solution is: run ITBLL to see if it works. Time cost: 2. how to distribute load of the general root table. Read replicas feature has been dropped. The current statement is that we could also implement cache servers for a general root table, but we need a clear design. Time cost: For master local region, there is one problem: 1. Master is on the critical path of client normal read/write. The solution is to introduce cache servers for root table content. We have discussed this a lot in the design doc but seems [~stack] removed most of the content. Anyway, there is a issue HBASE-24459 for it, I can start working on it soon to show how to implement cache servers. Time cost: within two weeks. Do you agree with the above statements? [~toffer]. Thanks. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154253#comment-17154253
]
Duo Zhang commented on HBASE-11288:
---
{quote}
Let's follow Stack's previous comment? Let's not discuss replicas or caching as
they could be applied to either split meta implementation. Let's focus on the
tiering issue?
{quote}
I do not know how can we make a cache server for a general root table. Please
give a clear design on how to do it.
{quote}
I'm missing something. It is a POC and its intent was to gather information and
help answer critical questions like this. So I'm not sure what the concern is
against POCs doing what POCs are supposed to be used for? Rest assured for this
particular case I am more concerned as to how we came to the decision than the
actual decision. It would be best for everyone if we could apply some rigor for
something so important.
{quote}
It is you that suggestted we run ITBLL against the POC, not me. Correct me if
I'm wrong.
{quote}
It is definitely good for that. But what prevents us from using it for this
purpose? Wether it always passes, fails sometimes or fail always we learn
something and that is valuable for determining wether proc v2 can support
2-tier now, short term, long term or never. Then we can come to an informed
decision. For example we might decide we cannot wait that long for proc v2 to
mature so we go with 1-tier.
{quote}
I have shown my opinion above. I do not think this will help. But you're free
to run ITBLL by yourself and post the result here, no one can stop you doing
this right?
{quote}
It does help but it is still making a compromise to avoid the concerns of
2-tier. Which is why my main concern now is applying some rigor to help us come
to a well informed decision as to wether proc v2 cannot support it and we
should go with 1-tier. I think proc v2 is what was missing that prevented us
from succeeding in the past although it’s possible it may not be mature enough
at this stage.
{quote}
I'm a bit concerned. Why do you think if proc-v2 can support 2-tier than we
should use 2-tier? Please focus on the problem of the 'master local region'
itself? Or your point is that 'master local region' does not use 2-tier so it
is not a good solution? This does not make sense to me.
Thanks.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154247#comment-17154247
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
Bigtable is designed way back in 2006, not everything in BigTable is correct.
And they do not give us their code so we do not know their framework on how to
deal with tablet assignment and tablet server crash.
{quote}
Yes of course my point was that it is not an unsolvable problem. It's unclear
wether you agree or disagree?
{quote}
What we have for now is procedure v2 in HBase, so our discussion should based
on procedure v2.
{quote}
Great we're on the same page then.
{quote}
And this is not only about adding another tier, how do you plan to distribute
the load of the general root table? Region replica is not stable enough, for
years.
{quote}
Let's follow Stack's previous comment? Let's not discuss replicas or caching as
they could be applied to either split meta implementation. Let's focus on the
tiering issue?
{quote}
This suggestion is weak. I'm afraid after running ITBLL we will sit here again
to argue that, 'This is only a POC, do not focus on polishing'.
{quote}
I'm missing something. It is a POC and its intent was to gather information and
help answer critical questions like this. So I'm not sure what the concern is
against POCs doing what POCs are supposed to be used for? Rest assured for this
particular case I am more concerned as to how we came to the decision than the
actual decision. It would be best for everyone if we could apply some rigor
for something so important.
{quote}
But from my understanding, ITBLL is for polishing. We should have a clear
design, and a clean code, and then we use ITBLL to find out the corner cases to
polish our code, make it more robust.
{quote}
It is definitely good for that. But what prevents us from using it for this
purpose? Wether it always passes, fails sometimes or fail always we learn
something and that is valuable for determining wether proc v2 can support
2-tier now, short term, long term or never. Then we can come to an informed
decision. For example we might decide we cannot wait that long for proc v2 to
mature so we go with 1-tier.
{quote}
And I want to know you opinion on 'master local region'. I've explain that,
with caching servers, master will not be on the critical path of normal client
read/write. Client just go to the cache servers, which is similar to read
replicas feature, but much easier to implement and more stable. Are there any
other concerns about this solution?
{quote}
It does help but it is still making a compromise to avoid the concerns of
2-tier. Which is why my main concern now is applying some rigor to help us come
to a well informed decision as to wether proc v2 cannot support it and we
should go with 1-tier. I think proc v2 is what was missing that prevented us
from succeeding in the past although it’s possible it may not be mature enough
at this stage.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154128#comment-17154128
]
Duo Zhang commented on HBASE-11288:
---
{quote}
Given that “General Root Table” has proven itself in BigTable and other clones
(eg Accumulo AFAIK).
{quote}
Bigtable is designed way back in 2006, not everything in BigTable is correct.
And they do not give us their code so we do not know their framework on how to
deal with tablet assignment and tablet server crash. What we have for now is
procedure v2 in HBase, so our discussion should based on procedure v2. If you
have other better solution, please file another issue to replace procedure v2
and then we go back here.
And this is not only about adding another tier, how do you plan to distribute
the load of the general root table? Region replica is not stable enough, for
years.
{quote}
For validation, my understanding is we use ITBLL to test wether the current
1-tier assignment is working properly or not, why don’t we try and run ITBLL on
the “General Root Table” POC to investigate and validate the 2-tier assignment
concern? Thoughts?
{quote}
This suggestion is weak. I'm afraid after running ITBLL we will sit here again
to argue that, 'This is only a POC, do not focus on polishing'. But from my
understanding, ITBLL is for polishing. We should have a clear design, and a
clean code, and then we use ITBLL to find out the corner cases to polish our
code, make it more robust.
And I want to know you opinion on 'master local region'. I've explain that,
with caching servers, master will not be on the critical path of normal client
read/write. Client just go to the cache servers, which is similar to read
replicas feature, but much easier to implement and more stable. Are there any
other concerns about this solution?
Thanks.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154103#comment-17154103 ] Francis Christopher Liu commented on HBASE-11288: - Thanks for you response Duo. So it sounds to me the main reason for going with the “Root table on master” implementation is because of the strong disagreement towards the “General Root Table” implementation in particular the fact that it adds another tier in assignment. Given that “General Root Table” has proven itself in BigTable and other clones (eg Accumulo AFAIK). The concern it seems is specific to HBase’s implementation, there is a strong concern that we might fail again at getting it to work. Correct me if I’m wrong here. Regardless of which implementation the addition of split meta will not only be a significant change to HBase but also something that we will likely be living with for quite a while. Because of that I am proposing that we apply some rigor on deciding which implementation we will go with. My current thinking is given that the main reason it seems for picking “Root table on master” is because we want to avoid failing at tiering again, why don’t we come up with a way to validate that assertion? IMHO the situation now and then are very different especially with the addition of ProcV2 and the revamped Assignment. For validation, my understanding is we use ITBLL to test wether the current 1-tier assignment is working properly or not, why don’t we try and run ITBLL on the “General Root Table” POC to investigate and validate the 2-tier assignment concern? Thoughts? > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152090#comment-17152090 ] Duo Zhang commented on HBASE-11288: --- Any updates here? Thanks. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151444#comment-17151444
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288.splittable-meta
[build #14 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/14/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/14/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/14/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/14/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17150418#comment-17150418
]
Michael Stack commented on HBASE-11288:
---
{quote}The key thing is store the ROOT table as a "general hbase table" VS
"master local region", right? We need to agree on this design firstly.
{quote}
Yes sir. The doc _tries_ to list +/- of each approach.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17150390#comment-17150390 ] Duo Zhang commented on HBASE-11288: --- OK, I posted some comments in the design doc and also changed it a bit. Waiting for your reply. Thanks. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149947#comment-17149947 ] Guanghao Zhang commented on HBASE-11288: The key thing is store the ROOT table as a "general hbase table" VS "master local region", right? We need to agree on this design firstly. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149895#comment-17149895 ] Michael Stack commented on HBASE-11288: --- Lets not split the issue. Lets keep the history in one place. Lets figure how to work together on this thing and put it behind us – its been dragging on a long time. Lets finish up design and sign off on it and then move forward (Lets bypass discussion of Read Replica; not directly pertinent). Thanks.. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149764#comment-17149764
]
Duo Zhang commented on HBASE-11288:
---
{quote}
Thank you for taking a look and your comments. It seems your comments are about
polish? As mentioned this is a draft patch, does that mean the general
implementation is ok and I can work on polishing it?
{quote}
I do not understand. While I was talking about the design, and how we could
make the feature availble step by step, you told me to review your patch, and
after I reviewed your patch you said this is just a POC so do not focus on
polishing? Then what do you expect me to do? What I can see is that there are
full of branch-1 code in the patch and it can not even pass the small UTs.
{quote}
Oh I see you're talking about the PR test result. I ran it locally with -fn so
I could get all the tests to run. Although I think I ran it without running IT
tests. I'll run it again after rebase.
{quote}
Of course I'm telling the PR result, I can not see your local result right?
{quote}
I'd really appreciate your response on this question. Having previous
discussions with you I think we both want split-meta and we both want to do it
right. So I think we both want the same thing, so maybe we can work a little
more closely on this?
{quote}
I'm OK with us both implementing the feature. No problem. If you want I will
open another issue and move all the sub tasks there, and give back this issue
to you. What I can say is that, I strongly disagree with implementing a general
ROOT table, we should not tumble at the same place twice. I do not trust the
read replicas features either, it has been there for years and till now, we
still have fellas in the community try to stabilize it. That's really not a
good sign and we should avoid relying on it critically. If you are fine to
implement based on the master local region approach you can pick up the issues
you like to implement them. I do not think there will be big difference for the
split/merge procedures in the two approaches.
Thanks.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149725#comment-17149725
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288.splittable-meta
[build #13 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/13/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/13/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/13/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/13/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149614#comment-17149614 ] Michael Stack commented on HBASE-11288: --- [~zhangduo] you seem to be pressing on w/ your notion of how split should be done ahead of agreement around design and plan. Mind responding above to [~toffer] questions and giving design another look. Advantages accredited to one approach -- narrow API on ROOT -- no longer seem exclusive to one approach only and so ditto for caching approach. Thanks. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148932#comment-17148932
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288.splittable-meta
[build #12 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/12/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/12/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/12/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/12/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148216#comment-17148216
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288.splittable-meta
[build #11 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/11/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/11/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/11/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/11/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148171#comment-17148171 ] Michael Stack commented on HBASE-11288: --- Some cleanup in the design doc: https://docs.google.com/document/d/1OYrCfpmmLPkSa5-AepQw8hv09zNcNPaFzSXezhcMS7E/edit?ts=5ef1c14c# There is a pivot point in the middle as to where/how the ROOT table is done. One approach is actively being worked on in a branch w/ impl steps noted in the doc. The other has a partial impl attached here as a PR based on a hbase1-based prod deploy. Input appreciated > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147131#comment-17147131
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288.splittable-meta
[build #10 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/10/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/10/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/10/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/10/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17145897#comment-17145897
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288.splittable-meta
[build #9 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/9/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/9/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/9/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/9/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17144459#comment-17144459
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288.splittable-meta
[build #8 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/8/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/8/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/8/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/8/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17141640#comment-17141640
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288.splittable-meta
[build #7 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/7/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/7/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/7/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/7//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17129893#comment-17129893
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288.splittable-meta
[build #6 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/6/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/6/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/6/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/6/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128671#comment-17128671
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288.splittable-meta
[build #5 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/5/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/5/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/5/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/5/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17127075#comment-17127075
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288.splittable-meta
[build #4 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/4/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/4/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/4/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/4/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124427#comment-17124427
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288.splittable-meta
[build #3 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/3/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/3/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/3/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/3/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17121078#comment-17121078
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288.splittable-meta
[build #2 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/2/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/2/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/2/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/2/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120925#comment-17120925
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
I added some comments on the PR. I think the PR is still far from merging...
{quote}
Thank you for taking a look and your comments. It seems your comments are about
polish? As mentioned this is a draft patch, does that mean the general
implementation is ok and I can work on polishing it?
{quote}
Usually the UT part will spend at least two hours to finish...
Thanks.
{quote}
Oh I see you're talking about the PR test result. I ran it locally with -fn so
I could get all the tests to run. Although I think I ran it without running IT
tests. I'll run it again after rebase.
{quote}
Lastly, you're working on implementing split meta while I am working on it too?
I wanted to know what your thoughts are where this is going? What happens when
you and I finish our feature branches?
{quote}
I'd really appreciate your response on this question. Having previous
discussions with you I think we both want split-meta and we both want to do it
right. So I think we both want the same thing, so maybe we can work a little
more closely on this?
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120921#comment-17120921
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288
[build #1 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288/1/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288/1/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288/1/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288/1/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288/1/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
