Interdiff from the ganeti-sandbox review: diff --git a/doc/design-hroller.rst b/doc/design-hroller.rst index 547708e..5305c71 100644 --- a/doc/design-hroller.rst +++ b/doc/design-hroller.rst @@ -26,6 +26,10 @@ reboots). Proposed changes ================
+ +Calculating rolling maintenances +-------------------------------- + In order to perform rolling maintenance we need to migrate instances off the nodes before a reboot. How this can be done depends on the instance's disk template: @@ -43,16 +47,15 @@ them (citation needed). As such we'll implement for now just the In order to do that we can use the following algorithm: -1) Compute node sets that don't share both a primary and a secondary for -any instance. This can be done already by the current hroller graph -coloring algorithm: nodes are in the same set (color) if no edge -(instance) exists between them (see the :manpage:`hroller(1)` manpage -for more details). -2) Inside each node set calculate subsets that don't share any secondary -node (this can be done by coloring the graph created by putting an edge -between each two nodes at distance 2 in the original graph, since if two -nodes have an instance connected to the same secondary they will be -themselves connected in the distance-2 graph) +1) Compute node sets that don't contain both the primary and the +secondary for any instance. This can be done already by the current +hroller graph coloring algorithm: nodes are in the same set (color) if +and only if no edge (instance) exists between them (see the +:manpage:`hroller(1)` manpage for more details). +2) Inside each node set calculate subsets that don't have any secondary +node in common (this can be done by creating a graph of nodes that are +connected if and only if an instance on both has the same secondary +node, and coloring that graph) 3) It is then possible to migrate in parallel all nodes in a subset created at step 2, and then reboot/perform maintenance on them 4) All instances will be migrated back on their original primaries, @@ -74,9 +77,9 @@ nodegroup). In order to do the job we can either: handled in parallel. - Perform migrations on one node at a time, but without waiting for the first node to come back before proceeding. This allows us to continue, - restricting the cluster, until capacity in the nodegroup is available, - and then having to wait for some nodes to come back so that capacity - is available again for the last few nodes. + restricting the cluster, until no more capacity in the nodegroup is + available, and then having to wait for some nodes to come back so that + capacity is available again for the last few nodes. - Pre-Calculate sets of nodes that can be migrated together (probably with a greedy algorithm) and parallelize between them, with the migrate-back approach discussed for DRBD to perform the calculation @@ -88,6 +91,29 @@ might be safe. This perhaps would be a good reason to consider managing better RBD pools, if those are implemented on top of nodes storage, rather than on dedicated storage machines. +Executing rolling maintenances +------------------------------ + +With the help of restricted commands, hroller will also be able to +execute maintenances. This will be done the following way: + +1) A command (--check-cmd) will be called on all selected online nodes to check +whether a node needs maintenance. Hroller will proceed only on nodes +that respond positively to this invocation. +2) A command (--maint-cmd) will be called on a node to do the actual +maintenance operation, when it has no primary instances on. If this +succeeds it should do any operation needed to perform the operation, +including triggering the actual reboot. +3) A command (--verify-cmd) will be called to check that the operation +was successful, if it was not hroller will stop and not proceed with +other nodes. +4) The master node will be kept last, but will not otherwise be treated +specially. If hroller was running on the master node, care must be +exercised as its maintenance will have interrupted the software itself, +and as such the verification step will not happen. This will not +automatically be taken care of, in the first version. + + Future work =========== @@ -101,6 +127,11 @@ If/when RBD pools can be managed inside Ganeti, care can be taken so that the pool is evacuated as well from a node before it's put into maintenance. This is equivalent to evacuating DRBD secondaries. +Master failovers during the maintenance should be performed by hroller. +This requires RPC/RAPI support for master failover. Hroller should also +be modified to better support running on the master itself and +continuing on the new master. + .. vim: set textwidth=72 : .. Local Variables: .. mode: rst On Fri, Feb 8, 2013 at 2:40 PM, Guido Trotter <[email protected]> wrote: > Thanks. Also sent to: > https://code.google.com/p/ganeti-sandbox/source/detail?r=fe753129919e3d94d172690897b401b525c5a85d&name=hroller&repo=design-reviews > for graphical review. ;) > > Guido > > > On Fri, Feb 8, 2013 at 2:32 PM, Iustin Pop <[email protected]> wrote: >> On Fri, Feb 08, 2013 at 02:25:54PM +0100, Guido Trotter wrote: >>> Hroller in 2.7 was implemented (as an experiment) but never explicitly >>> designed. This design mentions the current functionality, and explains >>> the changes that will happen for the next release to support live >>> maintenance scheduling (rolling reboot). >> >> Did a quick read and looks good, so LGTM. >> >> thanks, >> iustin > > > > -- > Guido Trotter > Ganeti engineering > Google Germany -- Guido Trotter Ganeti engineering Google Germany
