On Wednesday, July 10, 2013 4:57:50 AM UTC-4, Klaus Aehlig wrote: > > Signed-off-by: Klaus Aehlig <[email protected] <javascript:>> > --- > Makefile.am | 1 + > doc/design-draft.rst | 1 + > doc/design-optables.rst | 181 > ++++++++++++++++++++++++++++++++++++++++++++++++ > 3 files changed, 183 insertions(+) > create mode 100644 doc/design-optables.rst > > diff --git a/Makefile.am b/Makefile.am > index dd9dbd2..3101d87 100644 > --- a/Makefile.am > +++ b/Makefile.am > @@ -449,6 +449,7 @@ docinput = \ > doc/design-oob.rst \ > doc/design-ovf-support.rst \ > doc/design-opportunistic-locking.rst \ > + doc/design-optables.rst \ > doc/design-partitioned.rst \ > doc/design-query-splitting.rst \ > doc/design-query2.rst \ > diff --git a/doc/design-draft.rst b/doc/design-draft.rst > index 0e454cd..df9f2dc 100644 > --- a/doc/design-draft.rst > +++ b/doc/design-draft.rst > @@ -20,6 +20,7 @@ Design document drafts > design-internal-shutdown.rst > design-glusterfs-ganeti-support.rst > design-openvswitch.rst > + design-optables.rst > > .. vim: set textwidth=72 : > .. Local Variables: > diff --git a/doc/design-optables.rst b/doc/design-optables.rst > new file mode 100644 > index 0000000..3828d85 > --- /dev/null > +++ b/doc/design-optables.rst > @@ -0,0 +1,181 @@ > +========================================== > +Filtering of jobs for the ganeti job queue > +========================================== > + > +.. contents:: :depth: 4 > + > +This is a design document detailing the semantics of the filtering > +of jobs in ganeti. For the implementation there will be a separate > +design document that also describes the vision for the ganeti daemon > +structure. > + > + > +Current state and shortcomings > +============================== > + > +Control of the ganeti job queue is quite limited. There is a single > +status bit, the "drained flag". If set, no new jobs are accepted to > +the queue. This is too coarse for some use cases. > + > +- The queue might be required to be drained for several reasons, > + initiated by different persons or automatic programs. Each one > + should be able to indicate that his reasons for draining are over > + without affecting the others. > + > +- There is no support for partial drains. For example, one might want > + to allow all jobs belonging to a manual (or externally coordinated) > + maintenance, while disallowing all other jobs. > + > +- There is no support for blocking jobs by their op-codes, e.g., > + disallowing all jobs that bring new instances to a cluster. This might > + be part of a maintenance preparation. > + > +- There is no support for a soft version of draining, where all > + jobs currently in the queue are finished, while new jobs entering > + the queue are delayed until the drain is over. > + > + > +Proposed changes > +================ > + > +We propose to add filters on the job queue. These will be part of the > +configuration and as such are persisted with it. Conceptionally, the > +filters are always processed when a job enters the queue and while it > +is still in the queue. Of course, in the implementation, reevaluation > +is only carried out, if something could make the result change, e.g., > +a new job is entered to the queue, or the filter rules are changed. > +There is no distinction between filter processing when a job is about > +to enter the queue and while it is in the queue, as this can be > +expressed by the filter rules themselves (see predicates below). > + > +Format of a Filter rule > +----------------------- > + > +Filter rules are given be the following data. > + > +- A UUID. This ensures that there can be different filter rules > + that otherwise have all parameters equal. In this way, multiple > + drains for different reasons are possible. The UUID is used to > + address the filter rule, in particular for deletion. > + > + If no UUID is provided at rule addition, ganeti will create one. > + > +- The watermark. This is the highest job id ever used, as valid in > + the moment when the filter was added. This data will be added > + automatically upon addition of the filter. > + > +- A priority. This is a non-negative integer. Filters are processed > + in order of increasing priority until a rule applies. While there > + is a well-defined order in which rules of the same priority are > + evaluated (increasing watermark, then the uuid, are taken as tie > + breakers), rules of the same priority should not overlap, if they > + have different actions associated. > + > +- A list of predicates. The rule fires, if all of them hold true > + for the job. > + > +- An action. For the time being, one of the following, but more > + actions might be added in the future (in particular, future > + implementations might add an action making filtering continue with > + a different filter chain). > + > + - ACCEPT. The job will be accepted; no further filter rules > + are applied. > + > + - PAUSE. The job will be accepted to the queue and remain there; > + however, it is not executed. If an opcode is currently running, > + it continues, but the next opcode will not be started. A paused > + job has to release all locks it might have acquired as soon as > + possible, at the latest when the currently running opcode has > + finished. > + > + - REJECT. The job is rejected. If it is already in the queue, > + it will be removed as failed for policy reasons. > + > +- A reason trail, in the same format as reason trails for opcodes. > + This allows to find out, which maintenance (or other reason) caused > + the addition of this filter rule. > + > +Predicates available for the filter rules > +----------------------------------------- > + > +A predicate is a list, with the first element being the name of the > +predicate and the rest being parameters suitable for that predicate. > +In most cases, the name of the predicate will be a field of a job, > +and there will be a single parameter, which is a filter in the sense > +of the ganeti query language. However, no assumption should be made > +that all predicates are of this shape. More predicates may be added > +in the future. > + > +- ``jobid``. Only parameter is a filter. For this filter, there is only > + one field available, ``id``, which represents the id the job to be > + filtered. In all value positions, the string ``watermark`` will be > + replaced by the value of the watermark. > + > +- ``opcode``. Only parameter is a filter. For this filter, ``OP_ID`` > + and all other fields present in the opcode are available. This > predicate > + will hold true, if the filter is true for at least one opcode in > + the job. > + > +- ``reason``. Only parameter is a filter. For this filter, the three > + fields ``source``, ``reason``, ``timestamp`` of reason trail entries > + are available. This predicate is true, if one of the entries of one > + of the opcodes in this job satisfies the filter. > + > + > +Examples > +======== > + > +Draining the queue. > +:: > + > + {'priority': 0, > + 'predicates': [['jobid', ['>', 'id', 'watermark']]], > + 'action': 'REJECT'} > + > +Soft draining could be achieved by replacing ``REJECT`` by ``PAUSE`` in > the > +above example. > + > +Pausing all new jobs not belonging to a specific maintenance. > +:: > + > + {'priority': 1, > + 'predicates': [['jobid', ['>', 'id', 'watermark']], > + ['reason', ['!', ['=~', 'reason', 'maintenance pink > bunny']]]], > + 'action': 'PAUSE'} > + > +Canceling all queued instance creations and disallowing new such jobs. > +:: > + > + {'priority': 1, > + 'predicates': [['opcode', ['=', 'OP_ID', 'OP_INSTANCE_CREATE']]], > + 'action': 'REJECT'} > + > + > + > +Interface > +========= > + > +Since queue control is intended to be used by external > maintenance-handling > +tools as well, the official interface for manipulating queue filters is > the > +:doc:`rapi`. For convenience, a command-line interface might be added > later. >
It seems like this functionality should be available from both the rapi and commandline. If it's a matter of prioritizing work, then I agree the rapi is more important. > + > +The following resources will be added. > + > +- /2/filters/ > + - GET returns the list of all currently set filters > + - POST adds a new filter > + > +- /2/filters/[uuid] > + - GET returns the description of the specified filter > + - DELETE removes the specified The "Format of a Filter Rule" section suggests that one can provide a UUID at filter creation time. If that's true, there should be a POST method on this resource. > + > +Security considerations > +======================= > + > +Filtering of jobs is not a security feature! It merely serves the purpose > +of coordinating efforts and avoiding accidental conflicting > +jobs. Everybody with appropriate credentials can modify the filter > +rules, not just the originator of a rule. To avoid accidental > +lock-out, requests modifying the queue are executed directly and not > +going through the queue themselves. > -- > 1.8.3 > >
