[
https://issues.apache.org/jira/browse/HBASE-21743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16750358#comment-16750358
]
Sergey Shelukhin commented on HBASE-21743:
------------------------------------------
AMv1 chief problems in my view were too many persistent places to store state
(meta, ZK, in addition to implicit cluster state) so it was not stateless :)
And multi-threaded bugs with lots of custom code partying on that state. And
the fact that code was not well isolated, so you needed to know it all to
reason about it.
Looks like procv2 solved the last problem, and also improved on the state issue
by removing ZK (although there can still be issues with persistent proc state),
but the MT issue was not addressed. Isolated code does make it easier to
investigate! But lots of races are still there and it seems to be a systematic
issue to me... when procedures don't coordinate there are bugs, but if they
coordinate too much in future there's risk of devolving back into tightly
coupled model.
For the race condition problem, long time ago I was proposing smth like hybrid
actor model with non blocking single threaded amnesiac AM farming out async
operations, updating state and (more or less) forgetting about them, but at
this point it would be rather radical... making a pluggable AM might be a model
to allow for experimentation like this in future.
For now, given that master startup already acts almost like an "actor" (it's
one thread doing steps in sequence) we can also add an option to make it
"amnesiac". That should make it more resilient to consequences of race
condition bugs, and manual intervention, and also reduce bugs.
Let me think about this and get feedback, I'll try to start a thread in a
couple days.
> stateless assignment
> --------------------
>
> Key: HBASE-21743
> URL: https://issues.apache.org/jira/browse/HBASE-21743
> Project: HBase
> Issue Type: Bug
> Reporter: Sergey Shelukhin
> Priority: Major
>
> Running HBase for only a few weeks we found dozen(s?) of bugs with assignment
> that all seem to have the same nature - split brain between 2 procedures; or
> between procedure and master startup (meta replica bugs); or procedure and
> master shutdown (HBASE-21742); or procedure and something else (when SCP had
> incorrect region list persisted, don't recall the bug#).
> To me, it starts to look like a pattern where, like in AMv1 where concurrent
> interactions were unclear and hard to reason about, despite the cleaner
> individual pieces in AMv2 the problem of unclear concurrent interactions has
> been preserved and in fact increased because of the operation state
> persistence and isolation.
> Procedures are great for multi-step operations that need rollback and stuff
> like that, e.g. creating a table or snapshot, or even region splitting.
> However I'm not so sure about assignment.
> We have the persisted information - region state in meta (incl transition
> states like opening, or closing), server list as WAL directory list.
> Procedure state is not any more reliable then those (we can argue that meta
> update can fail, but so can procv2 WAL flush, so we have to handle cases of
> out of date information regardless). So, we don't need any extra state to
> decide on assignment, whether for recovery and balancing. In fact, as
> mentioned in some bugs, deleting procv2 WAL is often the best way to recover
> the cluster, because master can already figure out what to do without
> additional state.
> I think there should be an option for stateless assignment that does that.
> It can either be as a separate pluggable assignment procedure; or an option
> that will not recover SCP, RITs etc from WAL but always derive recovery
> procedures from the existing cluster state.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
