Current interdiff by the way: http://code.google.com/p/ganeti-sandbox/source/diff?path=/doc/design-hroller.rst&repo=design-reviews&format=side&name=hroller&r=389dc00f52c5eaa45669dbe28d3c68f9c3d885c4&old_path=/doc/design-hroller.rst&old=84939610bcb404481cd2f6580ad45c378ab73a35
On Tue, Feb 12, 2013 at 11:51 AM, Guido Trotter <[email protected]> wrote: > On Mon, Feb 11, 2013 at 12:49 PM, Iustin Pop <[email protected]> wrote: >> On Mon, Feb 11, 2013 at 12:48:33PM +0100, Guido Trotter wrote: >>> On Mon, Feb 11, 2013 at 12:45 PM, Iustin Pop <[email protected]> wrote: >>> > On Mon, Feb 11, 2013 at 11:55:20AM +0100, Guido Trotter wrote: >>> >> Interdiff from the ganeti-sandbox review: >>> >> >>> >> diff --git a/doc/design-hroller.rst b/doc/design-hroller.rst >>> >> index 547708e..5305c71 100644 >>> >> --- a/doc/design-hroller.rst >>> >> +++ b/doc/design-hroller.rst >>> >> @@ -26,6 +26,10 @@ reboots). >>> >> Proposed changes >>> >> ================ >>> >> >>> >> + >>> >> +Calculating rolling maintenances >>> >> +-------------------------------- >>> >> + >>> >> In order to perform rolling maintenance we need to migrate instances off >>> >> the nodes before a reboot. How this can be done depends on the >>> >> instance's disk template: >>> >> @@ -43,16 +47,15 @@ them (citation needed). As such we'll implement >>> >> for now just the >>> >> >>> >> In order to do that we can use the following algorithm: >>> >> >>> >> -1) Compute node sets that don't share both a primary and a secondary for >>> >> -any instance. This can be done already by the current hroller graph >>> >> -coloring algorithm: nodes are in the same set (color) if no edge >>> >> -(instance) exists between them (see the :manpage:`hroller(1)` manpage >>> >> -for more details). >>> >> -2) Inside each node set calculate subsets that don't share any secondary >>> >> -node (this can be done by coloring the graph created by putting an edge >>> >> -between each two nodes at distance 2 in the original graph, since if two >>> >> -nodes have an instance connected to the same secondary they will be >>> >> -themselves connected in the distance-2 graph) >>> >> +1) Compute node sets that don't contain both the primary and the >>> >> +secondary for any instance. This can be done already by the current >>> >> +hroller graph coloring algorithm: nodes are in the same set (color) if >>> >> +and only if no edge (instance) exists between them (see the >>> >> +:manpage:`hroller(1)` manpage for more details). >>> >> +2) Inside each node set calculate subsets that don't have any secondary >>> >> +node in common (this can be done by creating a graph of nodes that are >>> >> +connected if and only if an instance on both has the same secondary >>> >> +node, and coloring that graph) >>> >> 3) It is then possible to migrate in parallel all nodes in a subset >>> >> created at step 2, and then reboot/perform maintenance on them >>> >> 4) All instances will be migrated back on their original primaries, >>> >> @@ -74,9 +77,9 @@ nodegroup). In order to do the job we can either: >>> >> handled in parallel. >>> >> - Perform migrations on one node at a time, but without waiting for the >>> >> first node to come back before proceeding. This allows us to continue, >>> >> - restricting the cluster, until capacity in the nodegroup is available, >>> >> - and then having to wait for some nodes to come back so that capacity >>> >> - is available again for the last few nodes. >>> >> + restricting the cluster, until no more capacity in the nodegroup is >>> >> + available, and then having to wait for some nodes to come back so that >>> >> + capacity is available again for the last few nodes. >>> >> - Pre-Calculate sets of nodes that can be migrated together (probably >>> >> with a greedy algorithm) and parallelize between them, with the >>> >> migrate-back approach discussed for DRBD to perform the calculation >>> >> @@ -88,6 +91,29 @@ might be safe. This perhaps would be a good reason >>> >> to consider managing >>> >> better RBD pools, if those are implemented on top of nodes storage, >>> >> rather than on dedicated storage machines. >>> >> >>> >> +Executing rolling maintenances >>> >> +------------------------------ >>> >> + >>> >> +With the help of restricted commands, hroller will also be able to >>> >> +execute maintenances. This will be done the following way: >>> >> + >>> >> +1) A command (--check-cmd) will be called on all selected online nodes >>> >> to check >>> >> +whether a node needs maintenance. Hroller will proceed only on nodes >>> >> +that respond positively to this invocation. >>> >> +2) A command (--maint-cmd) will be called on a node to do the actual >>> >> +maintenance operation, when it has no primary instances on. If this >>> >> +succeeds it should do any operation needed to perform the operation, >>> >> +including triggering the actual reboot. >>> >> +3) A command (--verify-cmd) will be called to check that the operation >>> >> +was successful, if it was not hroller will stop and not proceed with >>> >> +other nodes. >>> >> +4) The master node will be kept last, but will not otherwise be treated >>> >> +specially. If hroller was running on the master node, care must be >>> >> +exercised as its maintenance will have interrupted the software itself, >>> >> +and as such the verification step will not happen. This will not >>> >> +automatically be taken care of, in the first version. >>> > >>> > I don't think this should depend on restricted commands. What's wrong >>> > with just using SSH? >>> > >>> >>> Well, restricted commands is just an rpc call. Do we have a good ssh >>> library for haskell? >> >> RunCmd? :) >> >>> > With restricted commands, this would make hroller dependent on this >>> > feature which might not always be enabled (and which can be a security >>> > problem). Whereas ssh is much easier to enable/disable, if needed. >>> > >>> >>> Ack. Perhaps we can start with ssh then, and in the future give both >>> options. >>> Adding more tools requiring ssh to the nodes seemed to go against what >>> we were trying to do (limit the need for ssh access) >> >> Well, yes, but making tools dependent on a feature purely designed for >> end-users seems 10x worse, IMHO :) >> >> +1 for flexible options. >> > > Alternative option: how about (at least for now) no option at all, and > just running local commands > (on the machine hroller runs on). These can ssh in, use restricted > commands, or even phone an operator for what we care... :) > > What do you think? > > Guido -- Guido Trotter Ganeti engineering Google Germany
