Greetings, We wanted to give an update on the progress and direction of the Matrix project, which Ben announced previously[1]. To recap, Matrix is a new bundle testing tool that augments the existing testing tools, such as bundletester and mojo, by differing in the following core philosophies:
- Juju provides a set of operational primitives that are generally expected to apply to any bundle and charms. Given that, if we test those primitives against a bundle, we can gain a level of confidence in the fitness of that bundle, and provide feedback to the bundle and charm authors, while both shifting the burden of testing away from the charm author and toward the stewards of the testing tool as well as ensuring that the tests cover cases which might not be considered by the charm author. - Unpredictability and failure are inherent properties of running big software, especially in the cloud; thus, the testing should include somewhat chaotic and potentially destructive mutations to the deployment. This will both uncover bugs (in Juju, charms, and the software the charms manage) as well as provide insight into areas for improving reliability of the bundle in production. Ben, Pete, and I have been working to get the first version finished and polished up, and have begun working with the Juju QA team to transfer ownership over to Torsten, Seman, and Aaron for maintainership of the default set of test suite. Our goal to finish this transition is December 16th. The current default test suite can be run against a bundle and verify that it deploys and stabilizes (within the somewhat limited ability currently available for charms to report health, i.e., the agent and workload status at the unit and application level), and then runs a random set of mutations (a.k.a. “glitch”) to the deployment and watches the health as that progresses. (This is similar to Netflix’s idea of a “chaos monkey.”) The glitch plan that is generated is saved and can be replayed to reproduce a failure. The bundle under test can additionally provide a custom set of suites in a tests/matrix.yaml file, as well as custom tasks to test things that are non-generic and specific to the bundle being tested or in more depth than can be tested by the generic framework. The custom suites can also override the tests built into the Matrix, if for some reason they need to be tweaked, or to test new functionality before it is moved into the generic test suites. We will continue to work with the QA team to support them improving the built-in suites, and to add features to improve the tests that are done. We are confident that the QA team will provide great insights moving forward as they take ownership of the Matrix project. Once again, the code for Matrix lives at https://github.com/juju-solutions/matrix and reviews, comments, issues, and contributions are welcome. [1] https://lists.ubuntu.com/archives/juju/2016-November/008108.html
-- Juju mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
