On Fri, Feb 15, 2013 at 10:21:17AM +0100, Guido Trotter wrote: > Hi, > > Can I get an LGTM on this, please? I'd like to commit this version, > even if there is a meeting about it tonight, as any changes for the > meeting can be discussed as a new patch, rather than keeping > interdiffing this.
Mmm, I'm now confused which version we're discussing. I only had one comment on the initial design: nack on tying hroller to restricted commands, and after that LGTM. iustin > On Tue, Feb 12, 2013 at 11:59 AM, Guido Trotter <[email protected]> wrote: > > diff --git a/doc/design-hroller.rst b/doc/design-hroller.rst > > index 5305c71..7799c14 100644 > > --- a/doc/design-hroller.rst > > +++ b/doc/design-hroller.rst > > @@ -12,10 +12,10 @@ Current state and shortcomings > > ============================== > > > > To enable automating cluster-wide reboots a new htool, called HRoller, > > -was added from Ganeti 2.7. This tool helps parallelizing cluster offline > > -maintenances by calculating which nodes are not both primary and > > -secondary for a DRBD instance, and thus can be rebooted at the same > > -time, when all instances are down. > > +was added to Ganeti starting from version 2.7. This tool helps > > +parallelizing cluster offline maintenances by calculating which nodes > > +are not both primary and secondary for a DRBD instance, and thus can be > > +rebooted at the same time, when all instances are down. > > > > The way this is done is documented in the :manpage:`hroller(1)` manpage. > > > > @@ -32,7 +32,15 @@ Calculating rolling maintenances > > > > In order to perform rolling maintenance we need to migrate instances off > > the nodes before a reboot. How this can be done depends on the > > -instance's disk template: > > +instance's disk template and status: > > + > > +Down instances > > +++++++++++++++ > > + > > +If an instance was shutdown when the maintenance started it will be > > +ignored. This allows avoiding nedlessly moving its primary around, since > > +it won't suffer a downtime anyway. > > + > > > > DRBD > > ++++ > > @@ -57,11 +65,11 @@ node in common (this can be done by creating a > > graph of nodes that are > > connected if and only if an instance on both has the same secondary > > node, and coloring that graph) > > 3) It is then possible to migrate in parallel all nodes in a subset > > -created at step 2, and then reboot/perform maintenance on them > > -4) All instances will be migrated back on their original primaries, > > -which allows the original computation to be used for each following > > -subset without N+1 failures being triggered, if none were present > > -before. > > +created at step 2, and then reboot/perform maintenance on them, and > > +migrate back their original primaries, which allows the computation > > +above to be reused for each following subset without N+1 failures being > > +triggered, if none were present before. See below about the actual > > +execution of the maintenance. > > > > Non-DRBD > > ++++++++ > > @@ -85,33 +93,40 @@ nodegroup). In order to do the job we can either: > > migrate-back approach discussed for DRBD to perform the calculation > > only once. > > > > -Note that for non-DRBD disks that still use local storage (eg. RBD) > > -redundancy might break anyway, and nothing except the first algorithm > > -might be safe. This perhaps would be a good reason to consider managing > > -better RBD pools, if those are implemented on top of nodes storage, > > -rather than on dedicated storage machines. > > +Note that for non-DRBD disks that still use local storage (eg. RBD and > > +plain) redundancy might break anyway, and nothing except the first > > +algorithm might be safe. This perhaps would be a good reason to consider > > +managing better RBD pools, if those are implemented on top of nodes > > +storage, rather than on dedicated storage machines. > > > > Executing rolling maintenances > > ------------------------------ > > > > -With the help of restricted commands, hroller will also be able to > > -execute maintenances. This will be done the following way: > > - > > -1) A command (--check-cmd) will be called on all selected online nodes to > > check > > -whether a node needs maintenance. Hroller will proceed only on nodes > > -that respond positively to this invocation. > > -2) A command (--maint-cmd) will be called on a node to do the actual > > -maintenance operation, when it has no primary instances on. If this > > -succeeds it should do any operation needed to perform the operation, > > -including triggering the actual reboot. > > +Hroller accepts commands to run to do maintenance automatically. These > > +are going to be run on the machine hroller runs on, and take a node name > > +as input. They have then to gain access to the target node (via ssh, > > +restricted commands, or some other means) and perform their duty. > > + > > +1) A command (--check-cmd) will be called on all selected online nodes > > +to check whether a node needs maintenance. Hroller will proceed only on > > +nodes that respond positively to this invocation. > > +FIXME: decide about -D > > +2) Hroller will evacuate the node of all primary instances. > > +3) A command (--maint-cmd) will be called on a node to do the actual > > +maintenance operation. It should do any operation needed to perform the > > +maintenance including triggering the actual reboot. > > 3) A command (--verify-cmd) will be called to check that the operation > > -was successful, if it was not hroller will stop and not proceed with > > -other nodes. > > +was successful, it has to wait until the target node is back up (and > > +decide after how long it should give up) and perform the verification. > > +If it's not successful hroller will stop and not proceed with other > > +nodes. > > 4) The master node will be kept last, but will not otherwise be treated > > specially. If hroller was running on the master node, care must be > > exercised as its maintenance will have interrupted the software itself, > > and as such the verification step will not happen. This will not > > -automatically be taken care of, in the first version. > > +automatically be taken care of, in the first version. An additional flag > > +to just skip the master node will be present as well, in case that's > > +preferred. > > > > > > Future work > > > > On Tue, Feb 12, 2013 at 11:58 AM, Guido Trotter <[email protected]> > > wrote: > >> Current interdiff by the way: > >> http://code.google.com/p/ganeti-sandbox/source/diff?path=/doc/design-hroller.rst&repo=design-reviews&format=side&name=hroller&r=389dc00f52c5eaa45669dbe28d3c68f9c3d885c4&old_path=/doc/design-hroller.rst&old=84939610bcb404481cd2f6580ad45c378ab73a35 > >> > >> On Tue, Feb 12, 2013 at 11:51 AM, Guido Trotter <[email protected]> > >> wrote: > >>> On Mon, Feb 11, 2013 at 12:49 PM, Iustin Pop <[email protected]> wrote: > >>>> On Mon, Feb 11, 2013 at 12:48:33PM +0100, Guido Trotter wrote: > >>>>> On Mon, Feb 11, 2013 at 12:45 PM, Iustin Pop <[email protected]> wrote: > >>>>> > On Mon, Feb 11, 2013 at 11:55:20AM +0100, Guido Trotter wrote: > >>>>> >> Interdiff from the ganeti-sandbox review: > >>>>> >> > >>>>> >> diff --git a/doc/design-hroller.rst b/doc/design-hroller.rst > >>>>> >> index 547708e..5305c71 100644 > >>>>> >> --- a/doc/design-hroller.rst > >>>>> >> +++ b/doc/design-hroller.rst > >>>>> >> @@ -26,6 +26,10 @@ reboots). > >>>>> >> Proposed changes > >>>>> >> ================ > >>>>> >> > >>>>> >> + > >>>>> >> +Calculating rolling maintenances > >>>>> >> +-------------------------------- > >>>>> >> + > >>>>> >> In order to perform rolling maintenance we need to migrate > >>>>> >> instances off > >>>>> >> the nodes before a reboot. How this can be done depends on the > >>>>> >> instance's disk template: > >>>>> >> @@ -43,16 +47,15 @@ them (citation needed). As such we'll implement > >>>>> >> for now just the > >>>>> >> > >>>>> >> In order to do that we can use the following algorithm: > >>>>> >> > >>>>> >> -1) Compute node sets that don't share both a primary and a > >>>>> >> secondary for > >>>>> >> -any instance. This can be done already by the current hroller graph > >>>>> >> -coloring algorithm: nodes are in the same set (color) if no edge > >>>>> >> -(instance) exists between them (see the :manpage:`hroller(1)` > >>>>> >> manpage > >>>>> >> -for more details). > >>>>> >> -2) Inside each node set calculate subsets that don't share any > >>>>> >> secondary > >>>>> >> -node (this can be done by coloring the graph created by putting an > >>>>> >> edge > >>>>> >> -between each two nodes at distance 2 in the original graph, since > >>>>> >> if two > >>>>> >> -nodes have an instance connected to the same secondary they will be > >>>>> >> -themselves connected in the distance-2 graph) > >>>>> >> +1) Compute node sets that don't contain both the primary and the > >>>>> >> +secondary for any instance. This can be done already by the current > >>>>> >> +hroller graph coloring algorithm: nodes are in the same set (color) > >>>>> >> if > >>>>> >> +and only if no edge (instance) exists between them (see the > >>>>> >> +:manpage:`hroller(1)` manpage for more details). > >>>>> >> +2) Inside each node set calculate subsets that don't have any > >>>>> >> secondary > >>>>> >> +node in common (this can be done by creating a graph of nodes that > >>>>> >> are > >>>>> >> +connected if and only if an instance on both has the same secondary > >>>>> >> +node, and coloring that graph) > >>>>> >> 3) It is then possible to migrate in parallel all nodes in a subset > >>>>> >> created at step 2, and then reboot/perform maintenance on them > >>>>> >> 4) All instances will be migrated back on their original primaries, > >>>>> >> @@ -74,9 +77,9 @@ nodegroup). In order to do the job we can either: > >>>>> >> handled in parallel. > >>>>> >> - Perform migrations on one node at a time, but without waiting for > >>>>> >> the > >>>>> >> first node to come back before proceeding. This allows us to > >>>>> >> continue, > >>>>> >> - restricting the cluster, until capacity in the nodegroup is > >>>>> >> available, > >>>>> >> - and then having to wait for some nodes to come back so that > >>>>> >> capacity > >>>>> >> - is available again for the last few nodes. > >>>>> >> + restricting the cluster, until no more capacity in the nodegroup > >>>>> >> is > >>>>> >> + available, and then having to wait for some nodes to come back so > >>>>> >> that > >>>>> >> + capacity is available again for the last few nodes. > >>>>> >> - Pre-Calculate sets of nodes that can be migrated together > >>>>> >> (probably > >>>>> >> with a greedy algorithm) and parallelize between them, with the > >>>>> >> migrate-back approach discussed for DRBD to perform the > >>>>> >> calculation > >>>>> >> @@ -88,6 +91,29 @@ might be safe. This perhaps would be a good reason > >>>>> >> to consider managing > >>>>> >> better RBD pools, if those are implemented on top of nodes storage, > >>>>> >> rather than on dedicated storage machines. > >>>>> >> > >>>>> >> +Executing rolling maintenances > >>>>> >> +------------------------------ > >>>>> >> + > >>>>> >> +With the help of restricted commands, hroller will also be able to > >>>>> >> +execute maintenances. This will be done the following way: > >>>>> >> + > >>>>> >> +1) A command (--check-cmd) will be called on all selected online > >>>>> >> nodes to check > >>>>> >> +whether a node needs maintenance. Hroller will proceed only on > >>>>> >> nodes > >>>>> >> +that respond positively to this invocation. > >>>>> >> +2) A command (--maint-cmd) will be called on a node to do the actual > >>>>> >> +maintenance operation, when it has no primary instances on. If this > >>>>> >> +succeeds it should do any operation needed to perform the operation, > >>>>> >> +including triggering the actual reboot. > >>>>> >> +3) A command (--verify-cmd) will be called to check that the > >>>>> >> operation > >>>>> >> +was successful, if it was not hroller will stop and not proceed with > >>>>> >> +other nodes. > >>>>> >> +4) The master node will be kept last, but will not otherwise be > >>>>> >> treated > >>>>> >> +specially. If hroller was running on the master node, care must be > >>>>> >> +exercised as its maintenance will have interrupted the software > >>>>> >> itself, > >>>>> >> +and as such the verification step will not happen. This will not > >>>>> >> +automatically be taken care of, in the first version. > >>>>> > > >>>>> > I don't think this should depend on restricted commands. What's wrong > >>>>> > with just using SSH? > >>>>> > > >>>>> > >>>>> Well, restricted commands is just an rpc call. Do we have a good ssh > >>>>> library for haskell? > >>>> > >>>> RunCmd? :) > >>>> > >>>>> > With restricted commands, this would make hroller dependent on this > >>>>> > feature which might not always be enabled (and which can be a security > >>>>> > problem). Whereas ssh is much easier to enable/disable, if needed. > >>>>> > > >>>>> > >>>>> Ack. Perhaps we can start with ssh then, and in the future give both > >>>>> options. > >>>>> Adding more tools requiring ssh to the nodes seemed to go against what > >>>>> we were trying to do (limit the need for ssh access) > >>>> > >>>> Well, yes, but making tools dependent on a feature purely designed for > >>>> end-users seems 10x worse, IMHO :) > >>>> > >>>> +1 for flexible options. > >>>> > >>> > >>> Alternative option: how about (at least for now) no option at all, and > >>> just running local commands > >>> (on the machine hroller runs on). These can ssh in, use restricted > >>> commands, or even phone an operator for what we care... :) > >>> > >>> What do you think? > >>> > >>> Guido > >> > >> > >> > >> -- > >> Guido Trotter > >> Ganeti engineering > >> Google Germany > > > > > > > > -- > > Guido Trotter > > Ganeti engineering > > Google Germany > > > > -- > Guido Trotter > Ganeti engineering > Google Germany
