On Mon, Feb 11, 2013 at 12:49 PM, Iustin Pop <[email protected]> wrote: > On Mon, Feb 11, 2013 at 12:48:33PM +0100, Guido Trotter wrote: >> On Mon, Feb 11, 2013 at 12:45 PM, Iustin Pop <[email protected]> wrote: >> > On Mon, Feb 11, 2013 at 11:55:20AM +0100, Guido Trotter wrote: >> >> Interdiff from the ganeti-sandbox review: >> >> >> >> diff --git a/doc/design-hroller.rst b/doc/design-hroller.rst >> >> index 547708e..5305c71 100644 >> >> --- a/doc/design-hroller.rst >> >> +++ b/doc/design-hroller.rst >> >> @@ -26,6 +26,10 @@ reboots). >> >> Proposed changes >> >> ================ >> >> >> >> + >> >> +Calculating rolling maintenances >> >> +-------------------------------- >> >> + >> >> In order to perform rolling maintenance we need to migrate instances off >> >> the nodes before a reboot. How this can be done depends on the >> >> instance's disk template: >> >> @@ -43,16 +47,15 @@ them (citation needed). As such we'll implement >> >> for now just the >> >> >> >> In order to do that we can use the following algorithm: >> >> >> >> -1) Compute node sets that don't share both a primary and a secondary for >> >> -any instance. This can be done already by the current hroller graph >> >> -coloring algorithm: nodes are in the same set (color) if no edge >> >> -(instance) exists between them (see the :manpage:`hroller(1)` manpage >> >> -for more details). >> >> -2) Inside each node set calculate subsets that don't share any secondary >> >> -node (this can be done by coloring the graph created by putting an edge >> >> -between each two nodes at distance 2 in the original graph, since if two >> >> -nodes have an instance connected to the same secondary they will be >> >> -themselves connected in the distance-2 graph) >> >> +1) Compute node sets that don't contain both the primary and the >> >> +secondary for any instance. This can be done already by the current >> >> +hroller graph coloring algorithm: nodes are in the same set (color) if >> >> +and only if no edge (instance) exists between them (see the >> >> +:manpage:`hroller(1)` manpage for more details). >> >> +2) Inside each node set calculate subsets that don't have any secondary >> >> +node in common (this can be done by creating a graph of nodes that are >> >> +connected if and only if an instance on both has the same secondary >> >> +node, and coloring that graph) >> >> 3) It is then possible to migrate in parallel all nodes in a subset >> >> created at step 2, and then reboot/perform maintenance on them >> >> 4) All instances will be migrated back on their original primaries, >> >> @@ -74,9 +77,9 @@ nodegroup). In order to do the job we can either: >> >> handled in parallel. >> >> - Perform migrations on one node at a time, but without waiting for the >> >> first node to come back before proceeding. This allows us to continue, >> >> - restricting the cluster, until capacity in the nodegroup is available, >> >> - and then having to wait for some nodes to come back so that capacity >> >> - is available again for the last few nodes. >> >> + restricting the cluster, until no more capacity in the nodegroup is >> >> + available, and then having to wait for some nodes to come back so that >> >> + capacity is available again for the last few nodes. >> >> - Pre-Calculate sets of nodes that can be migrated together (probably >> >> with a greedy algorithm) and parallelize between them, with the >> >> migrate-back approach discussed for DRBD to perform the calculation >> >> @@ -88,6 +91,29 @@ might be safe. This perhaps would be a good reason >> >> to consider managing >> >> better RBD pools, if those are implemented on top of nodes storage, >> >> rather than on dedicated storage machines. >> >> >> >> +Executing rolling maintenances >> >> +------------------------------ >> >> + >> >> +With the help of restricted commands, hroller will also be able to >> >> +execute maintenances. This will be done the following way: >> >> + >> >> +1) A command (--check-cmd) will be called on all selected online nodes >> >> to check >> >> +whether a node needs maintenance. Hroller will proceed only on nodes >> >> +that respond positively to this invocation. >> >> +2) A command (--maint-cmd) will be called on a node to do the actual >> >> +maintenance operation, when it has no primary instances on. If this >> >> +succeeds it should do any operation needed to perform the operation, >> >> +including triggering the actual reboot. >> >> +3) A command (--verify-cmd) will be called to check that the operation >> >> +was successful, if it was not hroller will stop and not proceed with >> >> +other nodes. >> >> +4) The master node will be kept last, but will not otherwise be treated >> >> +specially. If hroller was running on the master node, care must be >> >> +exercised as its maintenance will have interrupted the software itself, >> >> +and as such the verification step will not happen. This will not >> >> +automatically be taken care of, in the first version. >> > >> > I don't think this should depend on restricted commands. What's wrong >> > with just using SSH? >> > >> >> Well, restricted commands is just an rpc call. Do we have a good ssh >> library for haskell? > > RunCmd? :) > >> > With restricted commands, this would make hroller dependent on this >> > feature which might not always be enabled (and which can be a security >> > problem). Whereas ssh is much easier to enable/disable, if needed. >> > >> >> Ack. Perhaps we can start with ssh then, and in the future give both options. >> Adding more tools requiring ssh to the nodes seemed to go against what >> we were trying to do (limit the need for ssh access) > > Well, yes, but making tools dependent on a feature purely designed for > end-users seems 10x worse, IMHO :) > > +1 for flexible options. >
Alternative option: how about (at least for now) no option at all, and just running local commands (on the machine hroller runs on). These can ssh in, use restricted commands, or even phone an operator for what we care... :) What do you think? Guido
