[
https://issues.apache.org/jira/browse/HBASE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14614:
--------------------------
Release Note:
Replaces the AssignmentManager with a new procedurev2-based AssignmentManager
h1. AMv2
Puts AssignmentManager up on top of the ProcedureV2 state machine with
persistence engine. Each assignment atom is now a Procedure implementation;
e.g. an AssignProcedure and an UnassignProcedure. Molecules of aggregated
Procedures are used to do more involved assignment steps: e.g. the move region
procedure is made of an Unassign followed by an Assign subprocedure.
AMv2 is 1500 lines. Old AM was near 4000. Functionality has been moved out to
Procedures. In-memory states of regions and servers has been cleaned up stored
in new RegionStates implementation. RegionStateStore takes care of publishing
final region state out to the hbase:meta table.
New RemoteProcedureDispatcher/RSProcedureDispatcher runs the Procedure-based
assignments ‘remotely’. Knows about ‘servers’. Does aggregation of assignments
by time on a time/count basis so can send procedures in batches rather than one
per RPC. Procedure status comes back on the back of the RegionServer heartbeat
reporting online regions. The response is passed to the AMv2 to ‘process’. It
will check against the in-memory state. If there is a mismatch, it fences out
the RegionServer on the assumption that something went wrong on the RS
side.Timeouts trigger retries. The Procedure machine ensures only one operation
at a time on any one region/table using locking and smarts about what is serial
and what can be run concurrently.
New accounting of RegionServer version will be used running rolling restarts.
‘States’ -- OPENING, CLOSING, etc. -- are now in-memory in-the-master only
serialized out to the ProcedureV2 WAL. They are no longer persisted to
ZooKeeper.
h2. Assign Detail
The Assign starts by pushing the "assign" operation to the AssignmentManager
and then will go into a “waiting" state. The AM will batch the "assign"
requests and ask the Balancer where to put the region (the various policies
will be respected: retain, round-robin, random). Once the AM and the balancer
have found a place for the region, the procedure will be resumed and an "open
region" request will be placed in the Remote Dispatcher queue, and the
procedure once again will go into a "waiting state". The Remote Dispatcher
will batch the various requests for that server and they will be sent to the RS
for execution. The RS will complete the open operation by calling
master.reportRegionStateTransition(). The AM will intercept the transition
report, and notify the procedure. The procedure will finish the assignment by
publishing to new state on hbase:meta or it will retry the assignment.
h3. Unassign Detail
The Unassign starts by placing a "close region" request in the Remote
Dispatcher queue, and the procedure will then go into a "waiting state". The
Remote Dispatcher will batch the various requests for that server and they will
be sent to the RS for execution. The RS will complete the open operation by
calling master.reportRegionStateTransition(). The AM will intercept the
transition report, and notify the procedure. The procedure will finish the
unassign by publishing its new state on meta or it will retry the unassign.
h1. New Configs
* "hbase.procedure.remote.dispatcher.threadpool.size" defaults 128
* "hbase.procedure.remote.dispatcher.delay.msec" default 150ms
* "hbase.procedure.remote.dispatcher.max.queue.size" with default 32
* "hbase.regionserver.rpc.startup.waittime" with default 60 seconds.
h1. TODO
As of this writing.
Put up a model diagram.
* Handle region migration
* Handle meta assignment first
* Handle sys table assignment first (e.g. acl, namespace)
* Handle table priorities
* Do we report same AM metrics as we used too? We do it all in here now.
INCOMPATIBLE
A known incompatible is that because splits and merges are now run from the
master, Coprocessors that used to watch for merge/split from a RegionObserver
now no longer work; to watch split/merges, you need to have an observer on the
Master instead.
was:
Replaces the AssignmentManager with a new procedurev2-based AssignmentManager
h1. AMv2
Puts AssignmentManager up on top of the ProcedureV2 state machine with
persistence engine. Each assignment atom is now a Procedure implementation;
e.g. an AssignProcedure and an UnassignProcedure. Molecules of aggregated
Procedures are used to do more involved assignment steps: e.g. the move region
procedure is made of an Unassign followed by an Assign subprocedure.
AMv2 is 1500 lines. Old AM was near 4000. Functionality has been moved out to
Procedures. In-memory states of regions and servers has been cleaned up stored
in new RegionStates implementation. RegionStateStore takes care of publishing
final region state out to the hbase:meta table.
New RemoteProcedureDispatcher/RSProcedureDispatcher runs the Procedure-based
assignments ‘remotely’. Knows about ‘servers’. Does aggregation of assignments
by time on a time/count basis so can send procedures in batches rather than one
per RPC. Procedure status comes back on the back of the RegionServer heartbeat
reporting online regions. The response is passed to the AMv2 to ‘process’. It
will check against the in-memory state. If there is a mismatch, it fences out
the RegionServer on the assumption that something went wrong on the RS
side.Timeouts trigger retries. The Procedure machine ensures only one operation
at a time on any one region/table using locking and smarts about what is serial
and what can be run concurrently.
New accounting of RegionServer version will be used running rolling restarts.
‘States’ -- OPENING, CLOSING, etc. -- are now in-memory in-the-master only
serialized out to the ProcedureV2 WAL. They are no longer persisted to
ZooKeeper.
h2. Assign Detail
The Assign starts by pushing the "assign" operation to the AssignmentManager
and then will go into a “waiting" state. The AM will batch the "assign"
requests and ask the Balancer where to put the region (the various policies
will be respected: retain, round-robin, random). Once the AM and the balancer
have found a place for the region, the procedure will be resumed and an "open
region" request will be placed in the Remote Dispatcher queue, and the
procedure once again will go into a "waiting state". The Remote Dispatcher
will batch the various requests for that server and they will be sent to the RS
for execution. The RS will complete the open operation by calling
master.reportRegionStateTransition(). The AM will intercept the transition
report, and notify the procedure. The procedure will finish the assignment by
publishing to new state on hbase:meta or it will retry the assignment.
h3. Unassign Detail
The Unassign starts by placing a "close region" request in the Remote
Dispatcher queue, and the procedure will then go into a "waiting state". The
Remote Dispatcher will batch the various requests for that server and they will
be sent to the RS for execution. The RS will complete the open operation by
calling master.reportRegionStateTransition(). The AM will intercept the
transition report, and notify the procedure. The procedure will finish the
unassign by publishing its new state on meta or it will retry the unassign.
h1. New Configs
* "hbase.procedure.remote.dispatcher.threadpool.size" defaults 128
* "hbase.procedure.remote.dispatcher.delay.msec" default 150ms
* "hbase.procedure.remote.dispatcher.max.queue.size" with default 32
* "hbase.regionserver.rpc.startup.waittime" with default 60 seconds.
h1. TODO
As of this writing.
Put up a model diagram.
* Handle region migration
* Handle meta assignment first
* Handle sys table assignment first (e.g. acl, namespace)
* Handle table priorities
* Do we report same AM metrics as we used too? We do it all in here now.
> Procedure v2: Core Assignment Manager
> -------------------------------------
>
> Key: HBASE-14614
> URL: https://issues.apache.org/jira/browse/HBASE-14614
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2
> Affects Versions: 2.0.0
> Reporter: Stephen Yuan Jiang
> Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: HBASE-14614.master.003.patch,
> HBASE-14614.master.004.patch, HBASE-14614.master.005.patch,
> HBASE-14614.master.006.patch, HBASE-14614.master.007.patch,
> HBASE-14614.master.008.patch, HBASE-14614.master.009.patch,
> HBASE-14614.master.010.patch, HBASE-14614.master.014.patch,
> HBASE-14614.master.015.patch, HBASE-14614.master.017.patch,
> HBASE-14614.master.018.patch, HBASE-14614.master.019.patch,
> HBASE-14614.master.020.patch, HBASE-14614.master.022.patch,
> HBASE-14614.master.023.patch, HBASE-14614.master.024.patch,
> HBASE-14614.master.025.patch, HBASE-14614.master.026.patch,
> HBASE-14614.master.027.patch, HBASE-14614.master.028.patch,
> HBASE-14614.master.029.patch, HBASE-14614.master.030.patch,
> HBASE-14614.master.033.patch, HBASE-14614.master.038.patch,
> HBASE-14614.master.039.patch, HBASE-14614.master.040.patch,
> HBASE-14614.master.041.patch, HBASE-14614.master.042.patch,
> HBASE-14614.master.043.patch, HBASE-14614.master.044.patch,
> HBASE-14614.master.045.patch, HBASE-14614.master.045.patch,
> HBASE-14614.master.046.patch, HBASE-14614.master.047.patch,
> HBASE-14614.master.048.patch, HBASE-14614.master.049.patch,
> HBASE-14614.master.050.patch, HBASE-14614.master.051.patch
>
>
> New AssignmentManager implemented using proc-v2.
> - AssignProcedure handle assignment operation
> - UnassignProcedure handle unassign operation
> - MoveRegionProcedure handle move/balance operation
> Concurrent Assign operations are batched together and sent to the balancer
> Concurrent Assign and Unassign operation ready to be sent to the RS are
> batched together
> This patch is an intermediate state where we add the new AM as
> AssignmentManager2() to the master, to be reached by tests. but the new AM
> will not be integrated with the rest of the system. Only new am unit-tests
> will exercise the new assigment manager. The integration with the master code
> is part of HBASE-14616
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
