jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/382480 )
Change subject: Documentation: convert Markdown to reStructuredText ...................................................................... Documentation: convert Markdown to reStructuredText * PyPi renders only reStructuredText READMEs, while GitHub renders both formats. Converting all Markdown files to reStructuredText to simplify it's inclusion in PyPi and future documentation. * Convert README, CHANGELOG and TODO to reStructuredText Bug: T159308 Change-Id: I38da8998602fde68edbc8646d14961e061dd12d9 --- D CHANGELOG.md A CHANGELOG.rst D README.md A README.rst D TODO.md A TODO.rst 6 files changed, 710 insertions(+), 551 deletions(-) Approvals: jenkins-bot: Verified Volans: Looks good to me, approved diff --git a/CHANGELOG.md b/CHANGELOG.md deleted file mode 100644 index addf31d..0000000 --- a/CHANGELOG.md +++ /dev/null @@ -1,224 +0,0 @@ -# CUMIN CHANGELOG - - -## v1.2.1 (2017-09-27) - -### New features: -* OpenStack backend: allow to set default query params in the configuration - ([T176314](https://phabricator.wikimedia.org/T176314)): - - Allow to set arbitrary default query params in the configuration for the OpenStack backend. This is useful for - example if Cumin is installed inside an OpenStack project to automatically search only within the instances of the - current project. See the example in the provided `doc/examples/config.yaml` file. - -### Bug Fixes: -* Configuration: do not raise on empty configuration or aliases. Moved the check of required parameters where needed, - in order to raise explicit exceptions with a more meaningful message for the user. -* Exceptions: convert remaining spurious exceptions to CuminError or improve their error message. - - -## v1.1.1 (2017-09-26) - -### Bug Fixes: -* OpenStack: limit grammar to not overlap with the global one - - -## v1.1.0 (2017-09-21) - -### New features: -* Backends: add OpenStack backend ([T175711](https://phabricator.wikimedia.org/T175711)) - -### Bug Fixes: -* CLI: fix --version option -* Installation: fix data_files installation directory ([T174008](https://phabricator.wikimedia.org/T174008)) -* Transports: better handling of empty list ([T174911](https://phabricator.wikimedia.org/T174911)): - * BaseWorker: accept an empty list in the command setter. It's its default value, there is no point in forbidding a - client to set it to the same value. - * ClusterShellWorker: return immediately if there are no target hosts. -* Clustershell: make call to tqdm.write() explicit where to send the output, not relying on its default. - - -## v1.0.0 (2017-08-23) - -### CLI breaking changes: -* CLI: migrate to timeout per command ([T164838](https://phabricator.wikimedia.org/T164838)): - * the global timeout command line options changes from `-t/--timeout` to `--global-timeout`. - * the `-t/--timeout` option is now used to set the timeout for each command in each host independently. - -### Configuration breaking changes: -* Query: add multi-query support ([T170394](https://phabricator.wikimedia.org/T170394)): - * Remove the `backend` configuration key as it is not anymore used. - * Add a new optional `default_backend` configuration key. If set the query will be first executed with the default - backend, and if failing the parsing it will be executed with the global multi-query grammar. This allow to keep - backward compatibility with the query that were executed with previous versions of Cumin. - -### API breaking changes: -* PuppetDB backend: consistently use `InvalidQueryError` ([T162151](https://phabricator.wikimedia.org/T162151)). -* Transports: refactor command handling to support new features ([T164838](https://phabricator.wikimedia.org/T164838)), - ([T164833](https://phabricator.wikimedia.org/T164833)) and ([T171679](https://phabricator.wikimedia.org/T171679)): - * Transports: move `BaseWorker` helper methods to module functions. - * Transports: add `Command` class - * Transports: use the new `Command` class in `BaseWorker`, moving from a list of strings to a list of `Command` - objects. - * Transports: maintain backward compatibility and easy of usage automatically converting a list of strings to a list - of `Command` objects when setting the commands property. - * Allow to set the `ok_codes` property of the `transports.Command` class to an empty list to consider any return code - as successful. The case in which no return code should be treated successful has no practical use. - * ClusterShell: adapt the calls to commands for the new `Command` objects. -* Configuration: move configuration loader from the `cli` module to the main `cumin` module - ([T169640](https://phabricator.wikimedia.org/T169640)): - * add a `cumin.Config` class - * move the `parse_config` helper to cumin's main module from the `cli` one, to allow to easily load the configuration - also when it's used as a Python library. -* `QueryBuilder`: move query string to `build()` method. The constructor of the `QueryBuilder` was changed to not - accept anymore a query string directly, but just the configuration and the optional logger. The query string is now a - required parameter of the `build()` method. This properly split configuration and parameters, allowing to easily - `build()` multiple queries with the same `QueryBuilder` instance. -* Transports: convert hosts to ClusterShell's `NodeSet` ([T170394](https://phabricator.wikimedia.org/T170394)): - * in preparation for the multi-query support, start moving the transports to accept a ClusterShell's `NodeSet` - instead of a list of nodes. With the new multi-query support the backends too will return only NodeSets. -* Query: add multi-query support ([T170394](https://phabricator.wikimedia.org/T170394)): - * Aliases are now global and must use the global grammar syntax. - * `Query` class: the public `build()` method has become private and now is sufficient to call the - `execute(query_string)` method. Example usage: - ``` - config = cumin.Config(args.config) - hosts = query.Query(config, logger=logger).execute(query_string) - ``` - * `Query` class: the public methods `open_subgroup()` and `close_subgroup()` have become private, `_open_subgroup()` - and `_close_subgroup()` respectively. -* Transports: improve target management ([T171684](https://phabricator.wikimedia.org/T171684)): - * Add a `Target` class to handle all the target-related configuration. - * Let the `BaseWorker` require an instance of the `Target` class and delegate to it for all the target-related - configuration. - * This changes the `BaseWorker` constructor signature and removes the `hosts`, `batch_size` and `batch_sleep` - setters/getters. - -### New features: -* CLI: automatically set dry-run mode when no commands are specified - ([T161887](https://phabricator.wikimedia.org/T161887)). -* ClusterShell transport: output directly when only a single host is targeted. When the commands are executed against - only one host, print the output directly as it comes, to give the user an immediate feedback. There is no advantage - to collect the output for de-duplication in this case ([T164827](https://phabricator.wikimedia.org/T164827)). -* Transports: allow to specify a timeout per `Command` ([T164838](https://phabricator.wikimedia.org/T164838)). -* Transports: allow to specify exit codes per `Command` ([T164833](https://phabricator.wikimedia.org/T164833)). Allow - to specify for each `Command` object a list of exit codes to be considered successful when executing its specific - command. -* ClusterShell backend: allow to specify exit codes per `Command` - ([T164833](https://phabricator.wikimedia.org/T164833)). -* ClusterShell backend: allow to set a timeout per `Command` ([T164838](https://phabricator.wikimedia.org/T164838)). -* CLI: add `-i/--interactive` option. When set, this option drops into a Python shell (REPL) after the execution, - allowing the user to manipulate the results with the full power of Python. In this first iteration it can be used - only when one command is specified. ([T165838](https://phabricator.wikimedia.org/T165838)). -* CLI: add `-o/--output` to get the output in different formats. Allow to have `txt` and `json` output when only one - command is specified. In this first iteration the formatted output will be printed after the standard output with a - separator, in a next iteration the standard output will be suppressed - ([T165842](https://phabricator.wikimedia.org/T165842)). -* Query and grammar: add support for aliases ([T169640](https://phabricator.wikimedia.org/T169640)): - * Allow aliases of the form `A:alias_name` into the grammar. - * Automatically replace recursively all the aliases directly in the `QueryBuilder`, to make it completely transparent - for the backends. -* Configuration: automatically load aliases from file ([T169640](https://phabricator.wikimedia.org/T169640)). When - loading the configuration, automatically load also any aliases present in the `aliases.yaml` file in the same - directory of the configuration file, if present. -* Query: add multi-query support ([T170394](https://phabricator.wikimedia.org/T170394)): - * Each backend has now its own grammar and parsing rules as they are completely independent from each other. - * Add a new global grammar that allows to execute blocks of queries with different backends and aggregate the - results. -* CLI: add an option to ignore exit codes of commands ([T171679](https://phabricator.wikimedia.org/T171679)). Add the - `-x/--ignore-exit-codes` option to consider any executed command as successful, ignoring the returned exit codes. - This can be useful for a cleaner output and the usage of batches when running troubleshooting commands for which the - return code might be ignored (i.e. grep). - -### Minor improvements: -* CLI: improve configuration error handling ([T158747](https://phabricator.wikimedia.org/T158747)). -* Fix Pylint and other validation tools reported errors ([T154588](https://phabricator.wikimedia.org/T154588)). -* Package metadata and testing tools improvements ([T154588](https://phabricator.wikimedia.org/T154588)): - * Fill `setup.py` with all the parameters, suitable for a future submission to PyPI. - * Autodetect the version from Git tags and expose it in the module using `setuptools_scm`. - * CLI: add a `--version` option to print the current version and exit. - * Tests: use `pytest` to run the tests. - * Tests: convert tests from `unittest` to `pytest`. - * Tests: make `tox` use the dependencies in `setup.py`, removing the now unnecessary requirements files. - * Tests: add security analyzer `Bandit` to `tox`. - * Tests: add `Prospector` to `tox`, that in turns runs multiple additional tools: `dodgy`, `mccabe`, `pep257`, - `pep8`, `profile-validator`, `pyflakes`, `pylint`, `pyroma`, `vulture`. -* Tests: simplify and improve parametrized tests. Take advantage of `pytest.mark.parametrize` to run the same test - multiple times with different parameters instead of looping inside the same test. This not only simplifies the code - but also will make each parametrized test fail independently allowing an easier debugging. -* CLI: simplify imports and introspection -* Logging: add a custom `trace()` logging level: - * Add an additional custom logging level after `DEBUG` called `TRACE` mainly for development debugging. - * Fail in case the same log level is already set with a different name. This could happen when used as a library. - * CLI: add the `--trace` option to enable said logging level. -* Tests: improved tests fixture usage and removed usage of the example configuration present in the documentation from - the tests. -* Transports: improve command list validation of the `transports.Command` class to not allow an empty list for the - commands property ([T171679](https://phabricator.wikimedia.org/T171679)). - -### Bug Fixes: -* PuppetDB backend: do not auto upper case the first character when the query is a regex - ([T161730](https://phabricator.wikimedia.org/T161730)). -* PuppetDB backend: forbid resource's parameters regex as PuppetDB API v3 do not support regex match for resource's - parameters ([T162151](https://phabricator.wikimedia.org/T162151)). -* ClusterShell transport: fix set of list options ([T164824](https://phabricator.wikimedia.org/T164824)). -* Transports: fix `success_threshold` getter when set to `0` ([T167392](https://phabricator.wikimedia.org/T167392)). -* Transports: fix `ok_codes` getter for empty list ([T167394](https://phabricator.wikimedia.org/T167394)). -* `QueryBuilder`: fix subgroup close at the end of query. When a query was having subgroups that were closed at the end - of the query, QueryBuilder was not calling the `close_subgroup()` method of the related backend as it should have. - For example in a query like `host1* and (R:Class = Foo or R:Class = Bar)`. -* Fix test dependency issue. Due to a braking API change in the latest version of `Vulture`, `Prospector` is not - working anymore with the installed version of `Vulture` due to missing constraint in their `setup.py`. Adding - temporary the `Vulture` dependency here as a workaround. See - [the related issue](https://github.com/landscapeio/prospector/issues/230) for more details. - - -## v0.0.2 (2017-03-15) - -### Configuration breaking changes: -* Add support for batch processing ([T159968](https://phabricator.wikimedia.org/T159968)): - * Moved the `environment` block in the configuration file to the top level from within a specific transport. - -### API breaking changes: -* Add support for batch processing ([T159968](https://phabricator.wikimedia.org/T159968)): - * Refactored the `BaseWorker` class (and the `ClusterShellWorker` accordingly) to avoid passing a lot of parameters - to the execute() method, moving them to setters and getters with validation and default values, respectively. - * Add state machine for a transport's node state. - * Add CuminError exception and make all custom exceptions inherit from it to allow to easily catch only Cumin's - exceptions. -* ClusterShell transport: always require an event handler ([T159968](https://phabricator.wikimedia.org/T159968)): - * Since the addition of the batch capability running without an event handler doesn't really work because only the - first batch will be scheduled. - * Updated the CLI to work transparently and set the mode to `sync` when there is only one command. - * Unify the reporting lines format and logic between `sync` and `async` modes for coherence. - -### New features: -* Add support for `not` in simple hosts selection queries ([T158748](https://phabricator.wikimedia.org/T158748)). -* Add support for batch processing ([T159968](https://phabricator.wikimedia.org/T159968)): - * It's now possible to specify a `batch_size` and a `batch_sleep` parameters to define the size of a sliding batch - and an optional sleep between hosts executions. - * ClusterShell transport: the batches behaves accordingly to the specified mode when multiple commands are specified: - * `sync`: the first command is executed in a sliding batch until executed on all hosts or aborted due unmet success - ratio. Then the execution of the second command will start if the success ratio is reached. - * `async`: all the commands are executed in series in the first batch, and then will proceed with the next hosts - with a sliding batch, if the success ratio is met. - * Improves logging for backends and transport. - * CLI: updated to use the batch functionality, use the transport return value as return code on exit. - * Improves test coverage. -* PuppetDB backend: automatically upper case the first character in resource names - ([T159970](https://phabricator.wikimedia.org/T159970)). - -### Minor improvements: -* Moved `config.yaml` to a `doc/examples/` directory. It simplify the ship of the example file when packaging. -* Allow to ignore selected `urllib3` warnings ([T158758](https://phabricator.wikimedia.org/T158758)). -* Add codecov and codacy config and badges. -* Fixing minor issues reported by codacy ([T158967](https://phabricator.wikimedia.org/T158967)). -* Add integration tests for ClusterShell transport using Docker ([T159969](https://phabricator.wikimedia.org/T159969)). - -### Bug Fixes: -* Match the whole string for hosts regex matching ([T158746](https://phabricator.wikimedia.org/T158746)). - - -## v0.0.1 (2017-02-17) - -* First released version ([T154588](https://phabricator.wikimedia.org/T154588)). diff --git a/CHANGELOG.rst b/CHANGELOG.rst new file mode 100644 index 0000000..e64b9e0 --- /dev/null +++ b/CHANGELOG.rst @@ -0,0 +1,315 @@ +############### +CUMIN CHANGELOG +############### + +`v1.2.1`_ (2017-09-27) +====================== + +New features +------------ + +* OpenStack backend: allow to set default query params in the configuration (`T176314`_): + Allow to set arbitrary default query params in the configuration for the OpenStack backend. This is useful for + example if Cumin is installed inside an OpenStack project to automatically search only within the instances of the + current project. See the example in the provided ``doc/examples/config.yaml`` file. + +Bug Fixes +--------- + +* Configuration: do not raise on empty configuration or aliases. Moved the check of required parameters where needed, + in order to raise explicit exceptions with a more meaningful message for the user. +* Exceptions: convert remaining spurious exceptions to CuminError or improve their error message. + +`v1.1.1`_ (2017-09-26) +====================== + +Bug Fixes +--------- + +* OpenStack: limit grammar to not overlap with the global one. + +`v1.1.0`_ (2017-09-21) +====================== + +New features +------------ + +* Backends: add OpenStack backend (`T175711`_). + +Bug Fixes +--------- + +* CLI: fix --version option. +* Installation: fix ``data_files`` installation directory (`T174008`_) +* Transports: better handling of empty list (`T174911`_): + + * BaseWorker: accept an empty list in the command setter. It's its default value, there is no point in forbidding a + client to set it to the same value. + * ClusterShellWorker: return immediately if there are no target hosts. + +* Clustershell: make call to tqdm.write() explicit where to send the output, not relying on its default. + +`v1.0.0`_ (2017-08-23) +====================== + +CLI breaking changes: +--------------------- + +* CLI: migrate to timeout per command (`T164838`_): + + * the global timeout command line options changes from ``-t/--timeout`` to ``--global-timeout``. + * the ``-t/--timeout`` option is now used to set the timeout for each command in each host independently. + +Configuration breaking changes: +------------------------------- + +* Query: add multi-query support (`T170394`_): + + * Remove the ``backend`` configuration key as it is not anymore used. + * Add a new optional ``default_backend`` configuration key. If set the query will be first executed with the default + backend, and if failing the parsing it will be executed with the global multi-query grammar. This allow to keep + backward compatibility with the query that were executed with previous versions of Cumin. + +API breaking changes: +--------------------- + +* PuppetDB backend: consistently use ``InvalidQueryError`` (`T162151`_). +* Transports: refactor command handling to support new features (`T164838`_), (`T164833`_) and (`T171679`_): + + * Transports: move ``BaseWorker`` helper methods to module functions. + * Transports: add ``Command`` class. + * Transports: use the new ``Command`` class in ``BaseWorker``, moving from a list of strings to a list of ``Command`` + objects. + * Transports: maintain backward compatibility and easy of usage automatically converting a list of strings to a list + of ``Command`` objects when setting the commands property. + * Allow to set the ``ok_codes`` property of the ``transports.Command`` class to an empty list to consider any return + code as successful. The case in which no return code should be treated successful has no practical use. + * ClusterShell: adapt the calls to commands for the new ``Command`` objects. + +* Configuration: move configuration loader from the ``cli`` module to the main ``cumin`` module (`T169640`_): + + * add a ``cumin.Config`` class. + * move the ``parse_config`` helper to cumin's main module from the ``cli`` one, to allow to easily load the + configuration also when it's used as a Python library. + +* ``QueryBuilder``: move query string to ``build()`` method. The constructor of the ``QueryBuilder`` was changed to not + accept anymore a query string directly, but just the configuration and the optional logger. The query string is now a + required parameter of the ``build()`` method. This properly split configuration and parameters, allowing to easily + ``build()`` multiple queries with the same ``QueryBuilder`` instance. +* Transports: convert hosts to ClusterShell's ``NodeSet`` (`T170394`_): + +* in preparation for the multi-query support, start moving the transports to accept a ClusterShell's ``NodeSet`` + instead of a list of nodes. With the new multi-query support the backends too will return only NodeSets. + +* Query: add multi-query support (`T170394`_): + + * Aliases are now global and must use the global grammar syntax. + * ``Query`` class: the public ``build()`` method has become private and now is sufficient to call the + ``execute(query_string)`` method. Example usage:: + + config = cumin.Config(args.config) + hosts = query.Query(config, logger=logger).execute(query_string) + + * ``Query`` class: the public methods ``open_subgroup()`` and ``close_subgroup()`` have become private, + ``_open_subgroup()`` and ``_close_subgroup()`` respectively. + +* Transports: improve target management (`T171684`_): + + * Add a ``Target`` class to handle all the target-related configuration. + * Let the ``BaseWorker`` require an instance of the ``Target`` class and delegate to it for all the target-related + configuration. + * This changes the ``BaseWorker`` constructor signature and removes the ``hosts``, ``batch_size`` and ``batch_sleep`` + setters/getters. + +New features +------------ + +* CLI: automatically set dry-run mode when no commands are specified (`T161887`_). +* ClusterShell transport: output directly when only a single host is targeted. When the commands are executed against + only one host, print the output directly as it comes, to give the user an immediate feedback. There is no advantage + to collect the output for de-duplication in this case (`T164827`_). +* Transports: allow to specify a timeout per ``Command`` (`T164838`_). +* Transports: allow to specify exit codes per ``Command`` (`T164833`_). Allow to specify for each ``Command`` object a + list of exit codes to be considered successful when executing its specific command. +* ClusterShell backend: allow to specify exit codes per ``Command`` (`T164833`_). +* ClusterShell backend: allow to set a timeout per ``Command`` (`T164838`_). +* CLI: add ``-i/--interactive`` option (`T165838`_). When set, this option drops into a Python shell (REPL) after the + execution, allowing the user to manipulate the results with the full power of Python. In this first iteration it can + be used only when one command is specified. +* CLI: add ``-o/--output`` to get the output in different formats (`T165842`_). Allow to have ``txt`` and ``json`` + output when only one command is specified. In this first iteration the formatted output will be printed after the + standard output with a separator, in a next iteration the standard output will be suppressed. +* Query and grammar: add support for aliases (`T169640`_): + + * Allow aliases of the form ``A:alias_name`` into the grammar. + * Automatically replace recursively all the aliases directly in the ``QueryBuilder``, to make it completely + transparent for the backends. + +* Configuration: automatically load aliases from file (`T169640`_). When loading the configuration, automatically load + also any aliases present in the ``aliases.yaml`` file in the same directory of the configuration file, if present. +* Query: add multi-query support (`T170394`_): + + * Each backend has now its own grammar and parsing rules as they are completely independent from each other. + * Add a new global grammar that allows to execute blocks of queries with different backends and aggregate the + results. + +* CLI: add an option to ignore exit codes of commands (`T171679`_). Add the ``-x/--ignore-exit-codes`` option to + consider any executed command as successful, ignoring the returned exit codes. This can be useful for a cleaner + output and the usage of batches when running troubleshooting commands for which the return code might be ignored + (i.e. grep). + +Minor improvements: +------------------- + +* CLI: improve configuration error handling (`T158747`_). +* Fix Pylint and other validation tools reported errors (`T154588`_). +* Package metadata and testing tools improvements (`T154588`_): + + * Fill ``setup.py`` with all the parameters, suitable for a future submission to PyPI. + * Autodetect the version from Git tags and expose it in the module using ``setuptools_scm``. + * CLI: add a ``--version`` option to print the current version and exit. + * Tests: use ``pytest`` to run the tests. + * Tests: convert tests from ``unittest`` to ``pytest``. + * Tests: make ``tox`` use the dependencies in ``setup.py``, removing the now unnecessary requirements files. + * Tests: add security analyzer ``Bandit`` to ``tox``. + * Tests: add ``Prospector`` to ``tox``, that in turns runs multiple additional tools: ``dodgy``, ``mccabe``, + ``pep257``, ``pep8``, ``profile-validator``, ``pyflakes``, ``pylint``, ``pyroma``, ``vulture``. + +* Tests: simplify and improve parametrized tests. Take advantage of ``pytest.mark.parametrize`` to run the same test + multiple times with different parameters instead of looping inside the same test. This not only simplifies the code + but also will make each parametrized test fail independently allowing an easier debugging. +* CLI: simplify imports and introspection. +* Logging: add a custom ``trace()`` logging level: + + * Add an additional custom logging level after ``DEBUG`` called ``TRACE`` mainly for development debugging. + * Fail in case the same log level is already set with a different name. This could happen when used as a library. + * CLI: add the ``--trace`` option to enable said logging level. + +* Tests: improved tests fixture usage and removed usage of the example configuration present in the documentation from + the tests. +* Transports: improve command list validation of the ``transports.Command`` class to not allow an empty list for the + commands property (`T171679`_). + +Bug Fixes +--------- + +* PuppetDB backend: do not auto upper case the first character when the query is a regex (`T161730`_). +* PuppetDB backend: forbid resource's parameters regex as PuppetDB API v3 do not support regex match for resource's + parameters (`T162151`_). +* ClusterShell transport: fix set of list options (`T164824`_). +* Transports: fix ``success_threshold`` getter when set to ``0`` (`T167392`_). +* Transports: fix ``ok_codes`` getter for empty list (`T167394`_). +* ``QueryBuilder``: fix subgroup close at the end of query. When a query was having subgroups that were closed at the + end of the query, QueryBuilder was not calling the ``close_subgroup()`` method of the related backend as it should + have. For example in a query like ``host1* and (R:Class = Foo or R:Class = Bar)``. +* Fix test dependency issue. Due to a braking API change in the latest version of ``Vulture``, ``Prospector`` is not + working anymore with the installed version of ``Vulture`` due to missing constraint in their ``setup.py``. See + `Prospector issue #230`_ for more details. + +`v0.0.2`_ (2017-03-15) +====================== + +Configuration breaking changes: +------------------------------- + +* Add support for batch processing (`T159968`_): + + * Moved the ``environment`` block in the configuration file to the top level from within a specific transport. + +API breaking changes: +--------------------- + +* Add support for batch processing (`T159968`_): + + * Refactored the ``BaseWorker`` class (and the ``ClusterShellWorker`` accordingly) to avoid passing a lot of + parameters to the execute() method, moving them to setters and getters with validation and default values, + respectively. + * Add state machine for a transport's node state. + * Add CuminError exception and make all custom exceptions inherit from it to allow to easily catch only Cumin's + exceptions. + +* ClusterShell transport: always require an event handler (`T159968`_): + + * Since the addition of the batch capability running without an event handler doesn't really work because only the + first batch will be scheduled. + * Updated the CLI to work transparently and set the mode to ``sync`` when there is only one command. + * Unify the reporting lines format and logic between ``sync`` and ``async`` modes for coherence. + +New features +------------ + +* Add support for ``not`` in simple hosts selection queries (`T158748`_). +* Add support for batch processing (`T159968`_): + + * It's now possible to specify a ``batch_size`` and a ``batch_sleep`` parameters to define the size of a sliding + batch and an optional sleep between hosts executions. + * ClusterShell transport: the batches behaves accordingly to the specified mode when multiple commands are specified: + + * ``sync``: the first command is executed in a sliding batch until executed on all hosts or aborted due unmet + success ratio. Then the execution of the second command will start if the success ratio is reached. + * ``async``: all the commands are executed in series in the first batch, and then will proceed with the next hosts + with a sliding batch, if the success ratio is met. + + * Improves logging for backends and transport. + * CLI: updated to use the batch functionality, use the transport return value as return code on exit. + * Improves test coverage. + +* PuppetDB backend: automatically upper case the first character in resource names (`T159970`_). + +Minor improvements: +------------------- + +* Moved ``config.yaml`` to a ``doc/examples/`` directory. It simplify the ship of the example file when packaging. +* Allow to ignore selected ``urllib3`` warnings (`T158758`_). +* Add codecov and codacy config and badges. +* Fixing minor issues reported by codacy (`T158967`_). +* Add integration tests for ClusterShell transport using Docker (`T159969`_). + +Bug Fixes +--------- + +* Match the whole string for hosts regex matching (`T158746`_). + +`v0.0.1`_ (2017-02-17) +====================== + +* First released version (`T154588`_). + + +.. _`Prospector issue #230`: https://github.com/landscapeio/prospector/issues/230 + +.. _`T154588`: https://phabricator.wikimedia.org/T154588 +.. _`T158746`: https://phabricator.wikimedia.org/T158746 +.. _`T158747`: https://phabricator.wikimedia.org/T158747 +.. _`T158748`: https://phabricator.wikimedia.org/T158748 +.. _`T158758`: https://phabricator.wikimedia.org/T158758 +.. _`T158967`: https://phabricator.wikimedia.org/T158967 +.. _`T159968`: https://phabricator.wikimedia.org/T159968 +.. _`T159969`: https://phabricator.wikimedia.org/T159969 +.. _`T159970`: https://phabricator.wikimedia.org/T159970 +.. _`T161730`: https://phabricator.wikimedia.org/T161730 +.. _`T161887`: https://phabricator.wikimedia.org/T161887 +.. _`T162151`: https://phabricator.wikimedia.org/T162151 +.. _`T164824`: https://phabricator.wikimedia.org/T164824 +.. _`T164827`: https://phabricator.wikimedia.org/T164827 +.. _`T164833`: https://phabricator.wikimedia.org/T164833 +.. _`T164838`: https://phabricator.wikimedia.org/T164838 +.. _`T165838`: https://phabricator.wikimedia.org/T165838 +.. _`T165842`: https://phabricator.wikimedia.org/T165842 +.. _`T167392`: https://phabricator.wikimedia.org/T167392 +.. _`T167394`: https://phabricator.wikimedia.org/T167394 +.. _`T169640`: https://phabricator.wikimedia.org/T169640 +.. _`T170394`: https://phabricator.wikimedia.org/T170394 +.. _`T171679`: https://phabricator.wikimedia.org/T171679 +.. _`T171684`: https://phabricator.wikimedia.org/T171684 +.. _`T174008`: https://phabricator.wikimedia.org/T174008 +.. _`T174911`: https://phabricator.wikimedia.org/T174911 +.. _`T175711`: https://phabricator.wikimedia.org/T175711 +.. _`T176314`: https://phabricator.wikimedia.org/T176314 + +.. _`v0.0.1`: https://github.com/wikimedia/cumin/releases/tag/v0.0.1 +.. _`v0.0.2`: https://github.com/wikimedia/cumin/releases/tag/v0.0.2 +.. _`v1.0.0`: https://github.com/wikimedia/cumin/releases/tag/v1.0.0 +.. _`v1.1.0`: https://github.com/wikimedia/cumin/releases/tag/v1.1.0 +.. _`v1.1.1`: https://github.com/wikimedia/cumin/releases/tag/v1.1.1 +.. _`v1.2.1`: https://github.com/wikimedia/cumin/releases/tag/v1.2.1 diff --git a/README.md b/README.md deleted file mode 100644 index 6af6f73..0000000 --- a/README.md +++ /dev/null @@ -1,275 +0,0 @@ -# Cumin - An automation and orchestration framework - -[](https://travis-ci.org/wikimedia/cumin) -[](https://coveralls.io/github/wikimedia/cumin) -[](https://codecov.io/github/wikimedia/cumin) -[](https://www.codacy.com/app/volans-/cumin) -[](https://github.com/wikimedia/cumin/blob/master/LICENSE) - -## Summary - -Cumin provides a flexible and scalable automation framework to execute multiple commands on multiple hosts in parallel. -It allows to easily perform complex selections of hosts through a user-friendly query language which can interface -with different backend modules and combine their results for a fine grained selection. -The transport layer can also be selected, and can provide multiple execution strategies. -It can be used both via its command line interface (CLI) and as a Python library. - -More details of `cumin` usage in the Wikimedia Foundation are available on -[Cumin's Wikitech page]( https://wikitech.wikimedia.org/wiki/Cumin). - - -## Main components - -#### Query language - -Cumin provides a user-friendly generic query language that allows to combine the results of subqueries for multiple -backends. - -- Each query part can be composed with the others using boolean operators (`and`, `or`, `and not`, `xor`). -- Multiple query parts can be grouped together with parentheses (`(`, `)`). -- Specific backend query (`I{backend-specific query syntax}`, where `I` is an identifier for the specific backend). -- Alias replacement, according to aliases defined in the configuration file (`A:group1`). -- The identifier `A` is reserved for the aliases replacement and cannot be used to identify a backend. -- If a default_backend is set in the configuration, it will try to execute the query first directly with the default - backend and only if the query is not parsable with that backend it will try to execute it with the main grammar. -- A complex query example: `(D{host1 or host2} and (P{R:Class = Role::MyClass} and not A:group1)) or D{host3}` - -``` -Backus-Naur form (BNF) of the grammar: - <grammar> ::= <item> | <item> <boolean> <grammar> - <item> ::= <backend_query> | <alias> | "(" <grammar> ")" - <backend_query> ::= <backend> "{" <query> "}" - <alias> ::= A:<alias_name> - <boolean> ::= "and not" | "and" | "xor" | "or" -``` - -Given that the `pyparsing` library defines the grammar in a BNF-like style, for the details of the tokens not -specified above check directly the source code in `cumin/grammar.py`. - -The `Query` class defined in `cumin/query.py` is the one taking care of replacing the aliases, building and executing -the query parts with their respective backends and aggregating the results. Once a query is executed, it returns a -[ClusterShell NodeSet object](http://clustershell.readthedocs.io/en/latest/api/NodeSet.html#ClusterShell.NodeSet.NodeSet). -with the FQDN of the hosts that matches the selection. - -#### Backends - -All the backends share a minimal common interface that is defined in the `BaseQuery` class defined in -`cumin/backends/__init__.py` and they are instantiated by the `Query` class defined in `cumin/query.py` when building -and executing the query. Each backend module need to define a `query_class` module variable that -is a pointer to the backend class for dynamic instantiation and a `GRAMMAR_PREFIX` constant string that is the -identifier to be used in the main query syntax to identify the backend. The `GRAMMAR_PREFIX` `A` is reserved by the -main grammar for aliases. - -##### PuppetDB - -This backend uses the PuppetDB API to perform the query. The specific query language has this features: - -- Each query part can be composed with the others using boolean operators (`and`, `or`, `not`) -- Multiple query parts can be grouped together with parentheses (`(`, `)`). -- A query part can be of two different types: - - `Hostname matching`: this is a simple string that be used to match directly the hostname of the hosts in the - selected backend. It allows for glob expansion (`*`) and the use of the powerful - [ClusterShell NodeSet syntax](http://clustershell.readthedocs.io/en/latest/api/NodeSet.html#ClusterShell.NodeSet.NodeSet). - - `Category matching`: an identifier composed by a category, a colon and a key, followed by a comparison operator and - a value, as in `F:key = value`. - -The available categories are: - -- `F`: for querying facts -- `R`: for querying resources - -The available operators are: - -- `=`: equality -- `>=`: greater than or equal to -- `<=`: less than or equal to -- `<`: less than -- `>`: greater than -- `~`: regexp match - -Some query examples: - -- All hosts: `*` -- Hosts globbing: `host10*` -- [ClusterShell NodeSet syntax](http://clustershell.readthedocs.io/en/latest/api/NodeSet.html#ClusterShell.NodeSet.NodeSet) - for hosts expansion: `host10[10-42].domain` -- Category based key-value selection: - - `R:Resource::Name`: query all the hosts that have a resource of type `Resource::Name`. - - `R:Resource::Name = 'resource-title'`: query all the hosts that have a resource of type `Resource::Name` whose - title is `resource-title`. For example `R:Class = MyModule::MyClass`. - - `R:Resource::Name@field = 'some-value'`: query all the hosts that have a resource of type `Resource::Name` whose - field `field` has the value `some-value`. The valid fields are: `tag`, `certname`, `type`, `title`, `exported`, - `file`, `line`. The previous syntax is a shortcut for this one with the field `title`. - - `R:Resource::Name%param = 'some-value'`: query all the hosts that have a resource of type `Resource::Name` whose - parameter `param` has the value `some-value`. - - Mixed facts/resources queries are not supported, but the same result can be achieved by the main grammar using - multiple subqueries. -- A complex selection for facts: - `host10[10-42].*.domain or (not F:key1 = value1 and host10*) or (F:key2 > value2 and F:key3 ~ '^value[0-9]+')` - -``` -Backus-Naur form (BNF) of the grammar: - <grammar> ::= <item> | <item> <and_or> <grammar> - <item> ::= [<neg>] <query-token> | [<neg>] "(" <grammar> ")" - <query-token> ::= <token> | <hosts> - <token> ::= <category>:<key> [<operator> <value>] -``` - -Given that the `pyparsing` library used to define the grammar uses a BNF-like style, for the details of the tokens not -specified above see directly the code in `cumin/backends/puppetdb.py`. - - -##### OpenStack - -This backend uses the OpenStack APIs to perform the query. The specific query language has this features: - -- Each query can specify multiple parameters to filter the hosts selection in the form `key:value`. -- The special `project` key allow to filter by the OpenStack project name: `project:project_name`. If not specified - all the visible and enabled projects will be queried. -- Any other `key:value` pair will be passed as is to the - [OpenStack list-servers API](https://developer.openstack.org/api-ref/compute/#list-servers). - Multiple filters can be added separated by space. The value can be enclosed in single or double quotes: - `name:"host1.*\.domain" image:UUID` -- By default the filters `status:ACTIVE` and `vm_state:ACTIVE` are also added, but will be overridden if specified in - the query. -- To mix multiple selections the general grammar must be used with multiple subqueries: - `O{project:project1} or O{project:project2}` -- The special query `*` is a shortcut to select all hosts in all OpenStack projects. -- See the example configuration in `doc/examples/config.yaml` for all the OpenStack-related parameters that can be set. - -``` -Backus-Naur form (BNF) of the grammar: - <grammar> ::= "*" | <items> - <items> ::= <item> | <item> <whitespace> <items> - <item> ::= <key>:<value> -``` - -Given that the `pyparsing` library used to define the grammar uses a BNF-like style, for the details of the tokens not -specified above see directly the code in `cumin/backends/openstack.py`. - - -##### Direct - -The `direct` backend allow to use Cumin without any external dependency for the hosts selection. -It allow to write arbitrarily complex queries with subgroups and boolean operators, but each item must be either the -hostname itself, or the using host expansion using the powerful ClusterShell NodeSet syntax, see: - https://clustershell.readthedocs.io/en/latest/api/NodeSet.html - -The typical usage for the `direct` backend is as a reliable alternative in cases in which the primary host -selection mechanism is not working and also for testing the transports without any external backend dependency. - -Some query examples: - -- Simple selection: `host1.domain` -- ClusterShell syntax for hosts expansion: `host10[10-42].domain,host2010.other-domain` -- A complex selection: - `host100[1-5].domain or (host10[30-40].domain and (host10[10-42].domain and not host33.domain))` - -``` -Backus-Naur form (BNF) of the grammar: - <grammar> ::= <item> | <item> <boolean> <grammar> - <item> ::= <hosts> | "(" <grammar> ")" - <boolean> ::= "and not" | "and" | "xor" | "or" -``` - -Given that the `pyparsing` library used to define the grammar usesa BNF-like style, for the details of the tokens not -specified above check directly the source code in `cumin/backends/direct.py`. - - -#### Transports - -The transport layer is the one used to convey the commands to be executed into the selected hosts. -The transport abstraction allow to specify a mode to choose the execution plan, an event handler class and a success -threshold. Those can be used by the chosen transport to customize the behavior of the execution plan. - -All the transports share a common interface that is defined in the `BaseWorker` class defined in -`cumin/transports/__init__.py` and they are instantiated through the `Transport` factory class defined in -`cumin/transport.py`. Each backend module need to define a `worker_class` module variable that is a pointer to the -transport class for dynamic instantiation. - -##### ClusterShell - -This transport uses the [ClusterShell](https://github.com/cea-hpc/clustershell) Python library to connect to the -selected hosts and execute a list of commands. This transport accept the following customizations: -- `sync` execution mode: given a list of commands, the first one will be executed on all the hosts, then, if the - success ratio is reached, the second one will be executed on all hosts where the first one was successful, and so on -- `async` execution mode: given a list of commands, on each hosts the commands will be executed sequentially, - interrupting the execution on any single host at the first command that fails. The execution on the hosts is - independent between each other. -- custom execution mode: can be achieved creating a custom event handler class that extends the `BaseEventHandler` - class defined in `cumin/transports/clustershell.py`, implementing its abstract methods and setting to this class - object the handler to the transport. - -## Installation - -From the source code in the `master` branch: - - python setup.py install - -Is it also possible to build a Debian package using the `debian` branch, for example with `git-buildpackage`. - -## Configuration - -The default configuration file for `cumin` CLI is expected to be found at `/etc/cumin/config.yaml`; the path can be -changed via a command-line switch, `--config`. A commented example configuration is available in -`doc/examples/config.yaml`. - -Cumin will also automatically load any aliases defined in a `aliases.yaml` file, if present in the same directory of -the main configuration file. An aliases example file is available in `doc/examples/aliases.yaml` - -## CLI - -### Usage -``` -cumin [OPTIONS] HOSTS COMMAND [COMMAND ...] -``` -### OPTIONS - -For the full list of available optional arguments see `cumin --help`. - -#### Mode - -The `-m/--mode` argument is required when multiple COMMANDS are specified and defines the mode of execution: - -* `sync`: execute the first command on all hosts, then proceed with the next one only if `-s/--success-percentage` is -reached. -* `async`: execute on each host, independently from each other, the list of commands, aborting the execution on any given -host at the first command that fails. - -### Positional arguments - -#### HOSTS - -A host selection query according to a custom grammar. The hosts selection query is executed against the configured -backend to extract the list of hosts to use as target. - - -#### COMMAND -A command to be executed on all the target hosts in parallel, according to the configuration and options selected. - -Multiple commands will be executed sequentially. - - -## Running tests - -The `tox` utility, a wrapper around virtualenv, is used to run the test. To list the available environements: -``` -tox -l -``` -To run one: -``` -tox -e flake8 -``` -You can pass extra arguments to the underlying command: -``` -# Run only tests in a specific file: -tox -e unit -- -k test_puppetdb.py - -# Run only one specific test: -tox -e unit -- -k test_invalid_grammars -``` -Also integration tests are available, but not run by default by tox. They depends on a running Docker instance. -To run them: -``` -tox -e integration -``` diff --git a/README.rst b/README.rst new file mode 100644 index 0000000..47aeb9c --- /dev/null +++ b/README.rst @@ -0,0 +1,321 @@ +################################################# +Cumin - An automation and orchestration framework +################################################# + +| |Build Status| +| |Coveralls Coverage| +| |Codcov Coverage| +| |Codacy| +| |Licence| + +Summary +======= + +Cumin provides a flexible and scalable automation framework to execute multiple commands on multiple hosts in parallel. + +It allows to easily perform complex selections of hosts through a user-friendly query language which can +interface with different backend modules and combine their results for a fine grained selection. The transport layer +can also be selected, and can provide multiple execution strategies. + +It can be used both via its command line interface (CLI) and as a Python library. + +More details on the usage of Cumin in the Wikimedia Foundation are available on `Cumin's Wikitech page`_. + +Main components +=============== + +Query language +-------------- + +Cumin provides a user-friendly generic query language that allows to combine the results of subqueries for multiple +backends. + +- Each query part can be composed with the others using boolean operators (``and``, ``or``, ``and not``, ``xor``). +- Multiple query parts can be grouped together with parentheses (``(``, ``)``). +- Specific backend query (``I{backend-specific query syntax}``, where ``I`` is an identifier for the specific backend). +- Alias replacement, according to aliases defined in the configuration file (``A:group1``). +- The identifier ``A`` is reserved for the aliases replacement and cannot be used to identify a backend. +- If a ``default_backend`` is set in the configuration, it will try to execute the query first directly with the + default backend and only if the query is not parsable with that backend it will try to execute it with the main + grammar. +- A complex query example: + ``(D{host1 or host2} and (P{R:Class = Role::MyClass} and not A:group1)) or D{host3}`` + +:: + + Backus-Naur form (BNF) of the grammar: + <grammar> ::= <item> | <item> <boolean> <grammar> + <item> ::= <backend_query> | <alias> | "(" <grammar> ")" + <backend_query> ::= <backend> "{" <query> "}" + <alias> ::= A:<alias_name> + <boolean> ::= "and not" | "and" | "xor" | "or" + +Given that the ``pyparsing`` library defines the grammar in a BNF-like style, for the details of the tokens not +specified above check directly the source code in ``cumin/grammar.py``. + +The ``Query`` class defined in ``cumin/query.py`` is the one taking care of replacing the aliases, building and +executing the query parts with their respective backends and aggregating the results. Once a query is executed, it +returns a `ClusterShell NodeSet`_ with the FQDN of the hosts that matches the selection. + +Backends +-------- + +All the backends share a minimal common interface that is defined in the ``BaseQuery`` class defined in +``cumin/backends/__init__.py`` and they are instantiated by the ``Query`` class defined in ``cumin/query.py`` when +building and executing the query. Each backend module need to define a ``query_class`` module variable that is a +pointer to the backend class for dynamic instantiation and a ``GRAMMAR_PREFIX`` constant string that is the +identifier to be used in the main query syntax to identify the backend. The ``GRAMMAR_PREFIX`` ``A`` is reserved by +the main grammar for aliases. + +PuppetDB +^^^^^^^^ + +This backend uses the PuppetDB API to perform the query. The specific query language has this features: + +- Each query part can be composed with the others using boolean operators (``and``, ``or``, ``not``) +- Multiple query parts can be grouped together with parentheses (``(``, ``)``). +- A query part can be of two different types: + + - ``Hostname matching``: this is a simple string that be used to match directly the hostname of the hosts in the + selected backend. It allows for glob expansion (``*``) and the use of the powerful `ClusterShell NodeSet`_. + - ``Category matching``: an identifier composed by a category, a colon and a key, followed by a comparison operator + and a value, as in ``F:key = value``. + +The available categories are: + +- ``F``: for querying facts +- ``R``: for querying resources + +The available operators are: + +- ``=``: equality +- ``>=``: greater than or equal to +- ``<=``: less than or equal to +- ``<``: less than +- ``>``: greater than +- ``~``: regexp match + +Some query examples: + +- All hosts: ``*`` +- Hosts globbing: ``host10*`` +- `ClusterShell NodeSet`_ for hosts expansion: ``host10[10-42].domain`` +- Category based key-value selection: + + - ``R:Resource::Name``: query all the hosts that have a resource of type ``Resource::Name``. + - ``R:Resource::Name = 'resource-title'``: query all the hosts that have a resource of type ``Resource::Name`` + whose title is ``resource-title``. For example ``R:Class = MyModule::MyClass``. + - ``R:Resource::Name@field = 'some-value'``: query all the hosts that have a resource of type ``Resource::Name`` + whose field ``field`` has the value ``some-value``. The valid fields are: ``tag``, ``certname``, ``type``, + ``title``, ``exported``, ``file``, ``line``. The previous syntax is a shortcut for this one with the field + ``title``. + - ``R:Resource::Name%param = 'some-value'``: query all the hosts that have a resource of type ``Resource::Name`` + whose parameter ``param`` has the value ``some-value``. + +- Mixed facts/resources queries are not supported, but the same result can be achieved by the main grammar using + multiple subqueries. +- A complex selection for facts: + ``host10[10-42].*.domain or (not F:key1 = value1 and host10*) or (F:key2 > value2 and F:key3 ~ '^value[0-9]+')`` + +:: + + Backus-Naur form (BNF) of the grammar: + <grammar> ::= <item> | <item> <and_or> <grammar> + <item> ::= [<neg>] <query-token> | [<neg>] "(" <grammar> ")" + <query-token> ::= <token> | <hosts> + <token> ::= <category>:<key> [<operator> <value>] + +Given that the ``pyparsing`` library used to define the grammar uses a BNF-like style, for the details of the tokens +not specified above see directly the code in ``cumin/backends/puppetdb.py``. + +OpenStack +^^^^^^^^^ + +This backend uses the OpenStack APIs to perform the query. The specific query language has this features: + +- Each query can specify multiple parameters to filter the hosts selection in the form ``key:value``. +- The special ``project`` key allow to filter by the OpenStack project name: ``project:project_name``. If not + specified all the visible and enabled projects will be queried. +- Any other ``key:value`` pair will be passed as is to the `OpenStack list-servers API`_. Multiple filters can be + added separated by space. The value can be enclosed in single or double quotes: ``name:"host1.*\.domain" image:UUID`` +- By default the filters ``status:ACTIVE`` and ``vm_state:ACTIVE`` are also added, but will be overridden if specified + in the query. +- To mix multiple selections the general grammar must be used with multiple subqueries: + ``O{project:project1} or O{project:project2}`` +- The special query ``*`` is a shortcut to select all hosts in all OpenStack projects. +- See the example configuration in ``doc/examples/config.yaml`` for all the OpenStack-related parameters that can be + set. + +:: + + Backus-Naur form (BNF) of the grammar: + <grammar> ::= "*" | <items> + <items> ::= <item> | <item> <whitespace> <items> + <item> ::= <key>:<value> + +Given that the ``pyparsing`` library used to define the grammar uses a BNF-like style, for the details of the tokens +not specified above see directly the code in ``cumin/backends/openstack.py``. + +Direct +^^^^^^ + +The ``direct`` backend allow to use Cumin without any external dependency for the hosts selection. It allow to write +arbitrarily complex queries with subgroups and boolean operators, but each item must be either the hostname itself, +or the using host expansion using the powerful `ClusterShell NodeSet`_ syntax. + +The typical usage for the ``direct`` backend is as a reliable alternative in cases in which the primary host +selection mechanism is not working and also for testing the transports without any external backend dependency. + +Some query examples: + +- Simple selection: ``host1.domain`` +- ClusterShell syntax for hosts expansion: ``host10[10-42].domain,host2010.other-domain`` +- A complex selection: + ``host100[1-5].domain or (host10[30-40].domain and (host10[10-42].domain and not host33.domain))`` + +:: + + Backus-Naur form (BNF) of the grammar: + <grammar> ::= <item> | <item> <boolean> <grammar> + <item> ::= <hosts> | "(" <grammar> ")" + <boolean> ::= "and not" | "and" | "xor" | "or" + +Given that the ``pyparsing`` library used to define the grammar usesa BNF-like style, for the details of the tokens +not specified above check directly the source code in ``cumin/backends/direct.py``. + +Transports +---------- + +The transport layer is the one used to convey the commands to be executed into the selected hosts. The transport +abstraction allow to specify a mode to choose the execution plan, an event handler class and a success threshold. +Those can be used by the chosen transport to customize the behavior of the execution plan. + +All the transports share a common interface that is defined in the ``BaseWorker`` class defined in +``cumin/transports/__init__.py`` and they are instantiated through the ``Transport`` factory class defined in +``cumin/transport.py``. Each backend module need to define a ``worker_class`` module variable that is a pointer to +the transport class for dynamic instantiation. + +ClusterShell +^^^^^^^^^^^^ + +This transport uses the `ClusterShell <https://github.com/cea-hpc/clustershell>`__ Python library to connect to the +selected hosts and execute a list of commands. This transport accept the following customizations: + +- ``sync`` execution mode: given a list of commands, the first one will be executed on all the hosts, then, if the + success ratio is reached, the second one will be executed on all hosts where the first one was successful, and so on +- ``async`` execution mode: given a list of commands, on each hosts the commands will be executed sequentially, + interrupting the execution on any single host at the first command that fails. The execution on the hosts is + independent between each other. +- custom execution mode: can be achieved creating a custom event handler class that extends the ``BaseEventHandler`` + class defined in ``cumin/transports/clustershell.py``, implementing its abstract methods and setting to this class + object the handler to the transport. + +Installation +============ + +From the source code in the ``master`` branch: + +:: + + python setup.py install + +Is it also possible to build a Debian package using the ``debian`` branch, for example with ``gbp buildpackage``. + +Configuration +============= + +The default configuration file for ``cumin`` CLI is expected to be found at ``/etc/cumin/config.yaml``; the path can +be changed via a command-line switch, ``--config``. A commented example configuration is available in +``doc/examples/config.yaml``. + +Cumin will also automatically load any aliases defined in a ``aliases.yaml`` file, if present in the same directory +of the main configuration file. An aliases example file is available in ``doc/examples/aliases.yaml`` + +CLI +=== + +Usage +----- + +:: + + cumin [OPTIONS] HOSTS COMMAND [COMMAND ...] + +OPTIONS +^^^^^^^ + +For the full list of available optional arguments see ``cumin --help``. + +Mode +'''' + +The ``-m/--mode`` argument is required when multiple COMMANDS are specified and defines the mode of execution: + +- ``sync``: execute the first command on all hosts, then proceed with the next one only if ``-s/--success-percentage`` + is reached. +- ``async``: execute on each host, independently from each other, the list of commands, aborting the execution on any + given host at the first command that fails. + +Positional arguments +^^^^^^^^^^^^^^^^^^^^ + +HOSTS +''''' + +A host selection query according to a custom grammar. The hosts selection query is executed against the configured +backend to extract the list of hosts to use as target. + +COMMAND +''''''' + +A command to be executed on all the target hosts in parallel, according to the configuration and options selected. +Multiple commands will be executed sequentially. + +Running tests +============= + +The ``tox`` utility, a wrapper around virtualenv, is used to run the +test. To list the available environements: + +:: + + tox -l + +To run one: + +:: + + tox -e flake8 + +You can pass extra arguments to the underlying command: + +:: + + # Run only tests in a specific file: + tox -e unit -- -k test_puppetdb.py + + # Run only one specific test: + tox -e unit -- -k test_invalid_grammars + +Also integration tests are available, but not run by default by tox. They depends on a running Docker instance. To run +them: + +:: + + tox -e integration + + +.. |Build Status| image:: https://travis-ci.org/wikimedia/cumin.svg?branch=master + :target: https://travis-ci.org/wikimedia/cumin +.. |Coveralls Coverage| image:: https://coveralls.io/repos/github/wikimedia/cumin/badge.svg?branch=master + :target: https://coveralls.io/github/wikimedia/cumin +.. |Codcov Coverage| image:: https://codecov.io/github/wikimedia/cumin/coverage.svg?branch=master + :target: https://codecov.io/github/wikimedia/cumin +.. |Codacy| image:: https://api.codacy.com/project/badge/Grade/73d9a429dc7343eb935471bf05826fc0 + :target: https://www.codacy.com/app/volans-/cumin +.. |Licence| image:: https://img.shields.io/badge/license-GPLv3%2B-blue.svg + :target: https://github.com/wikimedia/cumin/blob/master/LICENSE + +.. _`Cumin's Wikitech page`: https://wikitech.wikimedia.org/wiki/Cumin +.. _`ClusterShell NodeSet`: https://clustershell.readthedocs.io/en/v1.7.3/api/NodeSet.html#ClusterShell.NodeSet.NodeSet +.. _`OpenStack list-servers API`: https://developer.openstack.org/api-ref/compute/#list-servers diff --git a/TODO.md b/TODO.md deleted file mode 100644 index fd61dbf..0000000 --- a/TODO.md +++ /dev/null @@ -1,52 +0,0 @@ -TODO ----- - -Tracking ideas to improve Cumin. They are in no particular order inside each section -and there is no guarantee that any item listed will be implemented in the nearby future. - -## On the masters - -##### Internal improvements / bug fixes -* global: add a man page [T159308](https://phabricator.wikimedia.org/T159308) -* CLI: fix progress bar interaction with ctrl+c and sigint_handler() -* CLI: suppress normal output when `-o/--output` is used -* clustershell transport: decouple the output handling from the event handlers -* clustershell transport: improve test coverage for partial/total failures and timeouts -* clustershell transport: improve and extend integration tests - -##### Small improvements -* global: allow to log the whole output to a specific file, to allow multiple people follow the progress -* global: allow to randomize the list hosts before execution [T164587](https://phabricator.wikimedia.org/T164587) -* CLI: `--batch-size` allow to specify percentage too -* CLI: improve the dry-run mode to show what would have been done -* CLI: add a `--color` or a `--no-color` option to manage the output color -* CLI: read commands from a file, one per line -* CLI: add `--limit` to randomly select N hosts within a broader selection -* puppetdb backend: improve globbing support, check if fnmatch could be used without conflicting with ClusterShell NodeSet -* puppetdb backend: allow to specify boolean values for resource parameters [T161545](https://phabricator.wikimedia.org/T161545) - -##### New Features -* global: connection timeout/failure should be treated differently than normal failures - * don't consider them for the success threshold by default, add a `--fail-always` option for that - * if the first command executed on a host fails with exit code 255, try to run `/bin/true`, if it fails too it should be considered a connection timeout/failure -* global: allow to notify the user who launched the execution on failure/termination trough IRC/email. Useful for long running jobs -* global: allow to differentiate the command to execute on a per-host basis, i.e. passing a different parameter for each host. -* global: allow to have an external audit log and/or announce commands execution on IRC -* transports: add an output-only transport to nicely print the matching hosts and some related count -* transports: allow to specify a rollback strategy to be executed in each host on failure -* transports: add parallel execution of local commands on the master for each targeted host with the host as a parameter. Needs a new local transport with ExecWorker to shell out in parallel. -* backends: generalize backends to allow to return other data too, not only the host certnames -* backends: add a new backend to support conftool -* backends: add a new backed to query the known hosts file format -* puppetdb backend: add support for API v4 -* CLI: when `-i/--interactive` is used and no command or query is specified, drop into a REPL session allowing to easily setup them. - - -## On the targets - -#### Future plans - -* Create a single entry point to allow the execution of idempotent _modules_ -* Create a safe and reliable sync up mechanism for the modules -* Allow to handle timeouts and failures locally within the module (local rollback and/or cleanup) -* Allow to drop privileges into a different user diff --git a/TODO.rst b/TODO.rst new file mode 100644 index 0000000..a4c04b1 --- /dev/null +++ b/TODO.rst @@ -0,0 +1,74 @@ +########## +CUMIN TODO +########## + +Tracking ideas to improve Cumin. They are in no particular order inside each section and there is no guarantee that +any item listed will be implemented in the nearby future. + +On the masters +============== + +Internal improvements / bug fixes +--------------------------------- + +* global: add a man page `T159308`_. +* CLI: fix progress bar interaction with ctrl+c and ``sigint_handler()``. +* CLI: suppress normal output when ``-o/--output`` is used. +* clustershell transport: decouple the output handling from the event handlers. +* clustershell transport: improve test coverage for partial/total failures and timeouts. +* clustershell transport: improve and extend integration tests. + +Small improvements +------------------ + +* global: allow to log the whole output to a specific file, to allow multiple people follow the progress. +* global: allow to randomize the list hosts before execution `T164587`_. +* CLI: ``--batch-size`` allow to specify percentage too. +* CLI: improve the dry-run mode to show what would have been done. +* CLI: add a ``--color`` or a ``--no-color`` option to manage the output color. +* CLI: read commands from a file, one per line. +* CLI: add ``--limit`` to randomly select N hosts within a broader selection. +* puppetdb backend: improve globbing support, check if fnmatch could be used without conflicting with ClusterShell + NodeSet. +* puppetdb backend: allow to specify boolean values for resource parameters `T161545`_. + +New Features +------------ + +* global: connection timeout/failure should be treated differently than normal failures: + + * don't consider them for the success threshold by default, add a ``--fail-always`` option for that + * if the first command executed on a host fails with exit code 255, try to run ``/bin/true``, if it fails too it + should be considered a connection timeout/failure + +* global: allow to notify the user who launched the execution on failure/termination trough IRC/email. Useful for long + running jobs. +* global: allow to differentiate the command to execute on a per-host basis, i.e. passing a different parameter for + each host. +* global: allow to have an external audit log and/or announce commands execution on IRC. +* transports: add an output-only transport to nicely print the matching hosts and some related count. +* transports: allow to specify a rollback strategy to be executed in each host on failure. +* transports: add parallel execution of local commands on the master for each targeted host with the host as a + parameter. Needs a new local transport with ExecWorker to shell out in parallel. +* backends: generalize backends to allow to return other data too, not only the host certnames. +* backends: add a new backend to support conftool. +* backends: add a new backed to query the known hosts file format. +* puppetdb backend: add support for API v4. +* CLI: when ``-i/--interactive`` is used and no command or query is specified, drop into a REPL session allowing to + easily setup them. + +On the targets +============== + +Future plans +------------ + +* Create a single entry point to allow the execution of idempotent modules. +* Create a safe and reliable sync up mechanism for the modules. +* Allow to handle timeouts and failures locally within the module (local rollback and/or cleanup). +* Allow to drop privileges into a different user. + + +.. _`T159308`: https://phabricator.wikimedia.org/T159308 +.. _`T161545`: https://phabricator.wikimedia.org/T161545 +.. _`T164587`: https://phabricator.wikimedia.org/T164587 -- To view, visit https://gerrit.wikimedia.org/r/382480 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I38da8998602fde68edbc8646d14961e061dd12d9 Gerrit-PatchSet: 3 Gerrit-Project: operations/software/cumin Gerrit-Branch: master Gerrit-Owner: Volans <[email protected]> Gerrit-Reviewer: Faidon Liambotis <[email protected]> Gerrit-Reviewer: Gehel <[email protected]> Gerrit-Reviewer: Giuseppe Lavagetto <[email protected]> Gerrit-Reviewer: Volans <[email protected]> Gerrit-Reviewer: jenkins-bot <> _______________________________________________ MediaWiki-commits mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
