[
https://issues.apache.org/jira/browse/HBASE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794915#comment-13794915
]
Jonathan Hsieh edited comment on HBASE-5487 at 10/15/13 5:54 AM:
-----------------------------------------------------------------
FYI, I've been looking at our support cases, and have been thinking and writing
up a clean slate design for a master redesign with the problems we've faced in
the field in mind. I focus a bit more on invariants necessary in the different
states, state transitions with master interactions, extensibility of the model,
and on the recovery strategy. It basically takes a pessimistic view of the
world and if I had to summarize its spirit, I'd call it the
"hbck-all-the-time".master.
It is currently durable storage agnostic but requires atomic CAS operations
(single row or single znode should be sufficient). When I re-read this thread
it could use either of the implementation details described here (zk vs meta,
etc). It sounds like being based in hbase is preferred so a little more
thought is going in that direction. I'm working currently on examples of how
to extend it for new features currently (like fast write recovery aka
distributed log replay) and proving to myself that it would be immune from
problems we've encountered before like double assignments, conflicting
concurrent operations (especially during recovery), and regions stuck in
transitions in the face of failures, hangs or juliet pauses.
I read Sergey's doc after my first cut and while there are some similarities it
deviates in other places. (I definitely want more on the error recovery and
error prevention mechanics). My hope is to share it sometime this later week so
that folks can read, discuss and compare the different designs presented at the
upcoming dev meeting. Before any jirae are file for implementation the design
should also consider things like upgrades, compatibility and performance.
I'm also hoping I'll have time to take a look at the accumulo master's design
as well for the discussion.
was (Author: jmhsieh):
FYI, I've been looking at our support cases, and have been thinking and writing
up a clean slate design for a master redesign with the problems we've faced in
the field in mind. I focus a bit more on invariants necessary in the different
states, state transitions with master interactions, extensibility of the model,
and on the recovery strategy. It basically takes a pessimistic view of the
world and if I had to summarize its spirit, I'd call it the
"hbck-all-the-time".master.
It is currently durable storage agnostic but requires atomic CAS operations
(single row or single znode should be sufficient). When I re-read this thread
it could use either of the implementation details described here (zk vs meta,
etc). It sounds like being based in hbase is preferred so a little more
thought is going in that direction. I'm working currently on examples of how
to extend it for new features currently (like fast write recovery aka
distributed log replay) and proving to myself that it would be immune from
problems we've encountered before like double assignments, conflicting
concurrent operations (especially during recovery), and regions stuck in
transitions in the face of failures, hangs or juliet pauses.
I read Sergey's doc after my first cut and while there are some similarities it
deviates in other places. (I definitely want more on the error recovery and
error prevention mechanics). My hope is to share it sometime this later week so
that folks can read, discuss and compare the different designs presented at the
upcoming dev meeting. Before and jirae are file for implementation also
consider things like upgrades, compatibility and performance.
I'm also hoping I'll have time to take a look at the accumulo master's design
as well for the discussion.
> Generic framework for Master-coordinated tasks
> ----------------------------------------------
>
> Key: HBASE-5487
> URL: https://issues.apache.org/jira/browse/HBASE-5487
> Project: HBase
> Issue Type: New Feature
> Components: master, regionserver, Zookeeper
> Affects Versions: 0.94.0
> Reporter: Mubarak Seyed
> Priority: Critical
> Attachments: Region management in Master.pdf
>
>
> Need a framework to execute master-coordinated tasks in a fault-tolerant
> manner.
> Master-coordinated tasks such as online-scheme change and delete-range
> (deleting region(s) based on start/end key) can make use of this framework.
> The advantages of framework are
> 1. Eliminate repeated code in Master, ZooKeeper tracker and Region-server for
> master-coordinated tasks
> 2. Ability to abstract the common functions across Master -> ZK and RS -> ZK
> 3. Easy to plugin new master-coordinated tasks without adding code to core
> components
--
This message was sent by Atlassian JIRA
(v6.1#6144)
