diff --git a/doc/design-hroller.rst b/doc/design-hroller.rst index 5305c71..7799c14 100644 --- a/doc/design-hroller.rst +++ b/doc/design-hroller.rst @@ -12,10 +12,10 @@ Current state and shortcomings ==============================
To enable automating cluster-wide reboots a new htool, called HRoller, -was added from Ganeti 2.7. This tool helps parallelizing cluster offline -maintenances by calculating which nodes are not both primary and -secondary for a DRBD instance, and thus can be rebooted at the same -time, when all instances are down. +was added to Ganeti starting from version 2.7. This tool helps +parallelizing cluster offline maintenances by calculating which nodes +are not both primary and secondary for a DRBD instance, and thus can be +rebooted at the same time, when all instances are down. The way this is done is documented in the :manpage:`hroller(1)` manpage. @@ -32,7 +32,15 @@ Calculating rolling maintenances In order to perform rolling maintenance we need to migrate instances off the nodes before a reboot. How this can be done depends on the -instance's disk template: +instance's disk template and status: + +Down instances +++++++++++++++ + +If an instance was shutdown when the maintenance started it will be +ignored. This allows avoiding nedlessly moving its primary around, since +it won't suffer a downtime anyway. + DRBD ++++ @@ -57,11 +65,11 @@ node in common (this can be done by creating a graph of nodes that are connected if and only if an instance on both has the same secondary node, and coloring that graph) 3) It is then possible to migrate in parallel all nodes in a subset -created at step 2, and then reboot/perform maintenance on them -4) All instances will be migrated back on their original primaries, -which allows the original computation to be used for each following -subset without N+1 failures being triggered, if none were present -before. +created at step 2, and then reboot/perform maintenance on them, and +migrate back their original primaries, which allows the computation +above to be reused for each following subset without N+1 failures being +triggered, if none were present before. See below about the actual +execution of the maintenance. Non-DRBD ++++++++ @@ -85,33 +93,40 @@ nodegroup). In order to do the job we can either: migrate-back approach discussed for DRBD to perform the calculation only once. -Note that for non-DRBD disks that still use local storage (eg. RBD) -redundancy might break anyway, and nothing except the first algorithm -might be safe. This perhaps would be a good reason to consider managing -better RBD pools, if those are implemented on top of nodes storage, -rather than on dedicated storage machines. +Note that for non-DRBD disks that still use local storage (eg. RBD and +plain) redundancy might break anyway, and nothing except the first +algorithm might be safe. This perhaps would be a good reason to consider +managing better RBD pools, if those are implemented on top of nodes +storage, rather than on dedicated storage machines. Executing rolling maintenances ------------------------------ -With the help of restricted commands, hroller will also be able to -execute maintenances. This will be done the following way: - -1) A command (--check-cmd) will be called on all selected online nodes to check -whether a node needs maintenance. Hroller will proceed only on nodes -that respond positively to this invocation. -2) A command (--maint-cmd) will be called on a node to do the actual -maintenance operation, when it has no primary instances on. If this -succeeds it should do any operation needed to perform the operation, -including triggering the actual reboot. +Hroller accepts commands to run to do maintenance automatically. These +are going to be run on the machine hroller runs on, and take a node name +as input. They have then to gain access to the target node (via ssh, +restricted commands, or some other means) and perform their duty. + +1) A command (--check-cmd) will be called on all selected online nodes +to check whether a node needs maintenance. Hroller will proceed only on +nodes that respond positively to this invocation. +FIXME: decide about -D +2) Hroller will evacuate the node of all primary instances. +3) A command (--maint-cmd) will be called on a node to do the actual +maintenance operation. It should do any operation needed to perform the +maintenance including triggering the actual reboot. 3) A command (--verify-cmd) will be called to check that the operation -was successful, if it was not hroller will stop and not proceed with -other nodes. +was successful, it has to wait until the target node is back up (and +decide after how long it should give up) and perform the verification. +If it's not successful hroller will stop and not proceed with other +nodes. 4) The master node will be kept last, but will not otherwise be treated specially. If hroller was running on the master node, care must be exercised as its maintenance will have interrupted the software itself, and as such the verification step will not happen. This will not -automatically be taken care of, in the first version. +automatically be taken care of, in the first version. An additional flag +to just skip the master node will be present as well, in case that's +preferred. Future work On Tue, Feb 12, 2013 at 11:58 AM, Guido Trotter <[email protected]> wrote: > Current interdiff by the way: > http://code.google.com/p/ganeti-sandbox/source/diff?path=/doc/design-hroller.rst&repo=design-reviews&format=side&name=hroller&r=389dc00f52c5eaa45669dbe28d3c68f9c3d885c4&old_path=/doc/design-hroller.rst&old=84939610bcb404481cd2f6580ad45c378ab73a35 > > On Tue, Feb 12, 2013 at 11:51 AM, Guido Trotter <[email protected]> wrote: >> On Mon, Feb 11, 2013 at 12:49 PM, Iustin Pop <[email protected]> wrote: >>> On Mon, Feb 11, 2013 at 12:48:33PM +0100, Guido Trotter wrote: >>>> On Mon, Feb 11, 2013 at 12:45 PM, Iustin Pop <[email protected]> wrote: >>>> > On Mon, Feb 11, 2013 at 11:55:20AM +0100, Guido Trotter wrote: >>>> >> Interdiff from the ganeti-sandbox review: >>>> >> >>>> >> diff --git a/doc/design-hroller.rst b/doc/design-hroller.rst >>>> >> index 547708e..5305c71 100644 >>>> >> --- a/doc/design-hroller.rst >>>> >> +++ b/doc/design-hroller.rst >>>> >> @@ -26,6 +26,10 @@ reboots). >>>> >> Proposed changes >>>> >> ================ >>>> >> >>>> >> + >>>> >> +Calculating rolling maintenances >>>> >> +-------------------------------- >>>> >> + >>>> >> In order to perform rolling maintenance we need to migrate instances >>>> >> off >>>> >> the nodes before a reboot. How this can be done depends on the >>>> >> instance's disk template: >>>> >> @@ -43,16 +47,15 @@ them (citation needed). As such we'll implement >>>> >> for now just the >>>> >> >>>> >> In order to do that we can use the following algorithm: >>>> >> >>>> >> -1) Compute node sets that don't share both a primary and a secondary >>>> >> for >>>> >> -any instance. This can be done already by the current hroller graph >>>> >> -coloring algorithm: nodes are in the same set (color) if no edge >>>> >> -(instance) exists between them (see the :manpage:`hroller(1)` manpage >>>> >> -for more details). >>>> >> -2) Inside each node set calculate subsets that don't share any >>>> >> secondary >>>> >> -node (this can be done by coloring the graph created by putting an edge >>>> >> -between each two nodes at distance 2 in the original graph, since if >>>> >> two >>>> >> -nodes have an instance connected to the same secondary they will be >>>> >> -themselves connected in the distance-2 graph) >>>> >> +1) Compute node sets that don't contain both the primary and the >>>> >> +secondary for any instance. This can be done already by the current >>>> >> +hroller graph coloring algorithm: nodes are in the same set (color) if >>>> >> +and only if no edge (instance) exists between them (see the >>>> >> +:manpage:`hroller(1)` manpage for more details). >>>> >> +2) Inside each node set calculate subsets that don't have any secondary >>>> >> +node in common (this can be done by creating a graph of nodes that are >>>> >> +connected if and only if an instance on both has the same secondary >>>> >> +node, and coloring that graph) >>>> >> 3) It is then possible to migrate in parallel all nodes in a subset >>>> >> created at step 2, and then reboot/perform maintenance on them >>>> >> 4) All instances will be migrated back on their original primaries, >>>> >> @@ -74,9 +77,9 @@ nodegroup). In order to do the job we can either: >>>> >> handled in parallel. >>>> >> - Perform migrations on one node at a time, but without waiting for the >>>> >> first node to come back before proceeding. This allows us to >>>> >> continue, >>>> >> - restricting the cluster, until capacity in the nodegroup is >>>> >> available, >>>> >> - and then having to wait for some nodes to come back so that capacity >>>> >> - is available again for the last few nodes. >>>> >> + restricting the cluster, until no more capacity in the nodegroup is >>>> >> + available, and then having to wait for some nodes to come back so >>>> >> that >>>> >> + capacity is available again for the last few nodes. >>>> >> - Pre-Calculate sets of nodes that can be migrated together (probably >>>> >> with a greedy algorithm) and parallelize between them, with the >>>> >> migrate-back approach discussed for DRBD to perform the calculation >>>> >> @@ -88,6 +91,29 @@ might be safe. This perhaps would be a good reason >>>> >> to consider managing >>>> >> better RBD pools, if those are implemented on top of nodes storage, >>>> >> rather than on dedicated storage machines. >>>> >> >>>> >> +Executing rolling maintenances >>>> >> +------------------------------ >>>> >> + >>>> >> +With the help of restricted commands, hroller will also be able to >>>> >> +execute maintenances. This will be done the following way: >>>> >> + >>>> >> +1) A command (--check-cmd) will be called on all selected online nodes >>>> >> to check >>>> >> +whether a node needs maintenance. Hroller will proceed only on nodes >>>> >> +that respond positively to this invocation. >>>> >> +2) A command (--maint-cmd) will be called on a node to do the actual >>>> >> +maintenance operation, when it has no primary instances on. If this >>>> >> +succeeds it should do any operation needed to perform the operation, >>>> >> +including triggering the actual reboot. >>>> >> +3) A command (--verify-cmd) will be called to check that the operation >>>> >> +was successful, if it was not hroller will stop and not proceed with >>>> >> +other nodes. >>>> >> +4) The master node will be kept last, but will not otherwise be treated >>>> >> +specially. If hroller was running on the master node, care must be >>>> >> +exercised as its maintenance will have interrupted the software itself, >>>> >> +and as such the verification step will not happen. This will not >>>> >> +automatically be taken care of, in the first version. >>>> > >>>> > I don't think this should depend on restricted commands. What's wrong >>>> > with just using SSH? >>>> > >>>> >>>> Well, restricted commands is just an rpc call. Do we have a good ssh >>>> library for haskell? >>> >>> RunCmd? :) >>> >>>> > With restricted commands, this would make hroller dependent on this >>>> > feature which might not always be enabled (and which can be a security >>>> > problem). Whereas ssh is much easier to enable/disable, if needed. >>>> > >>>> >>>> Ack. Perhaps we can start with ssh then, and in the future give both >>>> options. >>>> Adding more tools requiring ssh to the nodes seemed to go against what >>>> we were trying to do (limit the need for ssh access) >>> >>> Well, yes, but making tools dependent on a feature purely designed for >>> end-users seems 10x worse, IMHO :) >>> >>> +1 for flexible options. >>> >> >> Alternative option: how about (at least for now) no option at all, and >> just running local commands >> (on the machine hroller runs on). These can ssh in, use restricted >> commands, or even phone an operator for what we care... :) >> >> What do you think? >> >> Guido > > > > -- > Guido Trotter > Ganeti engineering > Google Germany -- Guido Trotter Ganeti engineering Google Germany
