On Mon, Feb 11, 2013 at 12:48:33PM +0100, Guido Trotter wrote: > On Mon, Feb 11, 2013 at 12:45 PM, Iustin Pop <[email protected]> wrote: > > On Mon, Feb 11, 2013 at 11:55:20AM +0100, Guido Trotter wrote: > >> Interdiff from the ganeti-sandbox review: > >> > >> diff --git a/doc/design-hroller.rst b/doc/design-hroller.rst > >> index 547708e..5305c71 100644 > >> --- a/doc/design-hroller.rst > >> +++ b/doc/design-hroller.rst > >> @@ -26,6 +26,10 @@ reboots). > >> Proposed changes > >> ================ > >> > >> + > >> +Calculating rolling maintenances > >> +-------------------------------- > >> + > >> In order to perform rolling maintenance we need to migrate instances off > >> the nodes before a reboot. How this can be done depends on the > >> instance's disk template: > >> @@ -43,16 +47,15 @@ them (citation needed). As such we'll implement > >> for now just the > >> > >> In order to do that we can use the following algorithm: > >> > >> -1) Compute node sets that don't share both a primary and a secondary for > >> -any instance. This can be done already by the current hroller graph > >> -coloring algorithm: nodes are in the same set (color) if no edge > >> -(instance) exists between them (see the :manpage:`hroller(1)` manpage > >> -for more details). > >> -2) Inside each node set calculate subsets that don't share any secondary > >> -node (this can be done by coloring the graph created by putting an edge > >> -between each two nodes at distance 2 in the original graph, since if two > >> -nodes have an instance connected to the same secondary they will be > >> -themselves connected in the distance-2 graph) > >> +1) Compute node sets that don't contain both the primary and the > >> +secondary for any instance. This can be done already by the current > >> +hroller graph coloring algorithm: nodes are in the same set (color) if > >> +and only if no edge (instance) exists between them (see the > >> +:manpage:`hroller(1)` manpage for more details). > >> +2) Inside each node set calculate subsets that don't have any secondary > >> +node in common (this can be done by creating a graph of nodes that are > >> +connected if and only if an instance on both has the same secondary > >> +node, and coloring that graph) > >> 3) It is then possible to migrate in parallel all nodes in a subset > >> created at step 2, and then reboot/perform maintenance on them > >> 4) All instances will be migrated back on their original primaries, > >> @@ -74,9 +77,9 @@ nodegroup). In order to do the job we can either: > >> handled in parallel. > >> - Perform migrations on one node at a time, but without waiting for the > >> first node to come back before proceeding. This allows us to continue, > >> - restricting the cluster, until capacity in the nodegroup is available, > >> - and then having to wait for some nodes to come back so that capacity > >> - is available again for the last few nodes. > >> + restricting the cluster, until no more capacity in the nodegroup is > >> + available, and then having to wait for some nodes to come back so that > >> + capacity is available again for the last few nodes. > >> - Pre-Calculate sets of nodes that can be migrated together (probably > >> with a greedy algorithm) and parallelize between them, with the > >> migrate-back approach discussed for DRBD to perform the calculation > >> @@ -88,6 +91,29 @@ might be safe. This perhaps would be a good reason > >> to consider managing > >> better RBD pools, if those are implemented on top of nodes storage, > >> rather than on dedicated storage machines. > >> > >> +Executing rolling maintenances > >> +------------------------------ > >> + > >> +With the help of restricted commands, hroller will also be able to > >> +execute maintenances. This will be done the following way: > >> + > >> +1) A command (--check-cmd) will be called on all selected online nodes to > >> check > >> +whether a node needs maintenance. Hroller will proceed only on nodes > >> +that respond positively to this invocation. > >> +2) A command (--maint-cmd) will be called on a node to do the actual > >> +maintenance operation, when it has no primary instances on. If this > >> +succeeds it should do any operation needed to perform the operation, > >> +including triggering the actual reboot. > >> +3) A command (--verify-cmd) will be called to check that the operation > >> +was successful, if it was not hroller will stop and not proceed with > >> +other nodes. > >> +4) The master node will be kept last, but will not otherwise be treated > >> +specially. If hroller was running on the master node, care must be > >> +exercised as its maintenance will have interrupted the software itself, > >> +and as such the verification step will not happen. This will not > >> +automatically be taken care of, in the first version. > > > > I don't think this should depend on restricted commands. What's wrong > > with just using SSH? > > > > Well, restricted commands is just an rpc call. Do we have a good ssh > library for haskell?
RunCmd? :) > > With restricted commands, this would make hroller dependent on this > > feature which might not always be enabled (and which can be a security > > problem). Whereas ssh is much easier to enable/disable, if needed. > > > > Ack. Perhaps we can start with ssh then, and in the future give both options. > Adding more tools requiring ssh to the nodes seemed to go against what > we were trying to do (limit the need for ssh access) Well, yes, but making tools dependent on a feature purely designed for end-users seems 10x worse, IMHO :) +1 for flexible options. iustin
