On Thu, Mar 17, 2016 at 8:24 AM Merlijn Sebrechts < [email protected]> wrote:
> As an aside; is there a good write-up somewhere about charm unit testing. > I'd like to do this but I'm not sure how to do this. I am completely new to > unit testing so I'm having a hard time to see how a good unittest for a > Charm would look like and what exactly should be tested. > This is a great question. There's a bug currently open against the reactive framework to figure out how to build a good set of unittest helpers to make writing tests really straight forward. As far as a current examples, I don't have any at the moment - but I will try to find some and if I can't I will make some! > 2016-03-17 1:52 GMT+01:00 Marco Ceppi <[email protected]>: > >> Hello everyone! >> >> This is an email I've been meaning to write for a while, and have >> rewritten a few times now. With 2.0 on the horizon and the charm ecosystem >> rapidly growing, I couldn't keep the idea to myself any longer. >> >> # tl;dr: >> >> We should stop writing Amulet tests in charms and instead only write them >> Bundles and force charms to do unit-testing (when possible) and promote >> that all charms be included in bundles in the store. >> >> # Problem >> >> Without making this a novel, charm-testing and amulet started before >> bundles were even a construct in Juju with a spec written before Juju 1.0. >> Since then, many new comers to the ecosystem have remarked how odd it is to >> be writing deployment validations at the charm level. Indeed, as years have >> gone by and new tools have sprung up it's become clear that; having an >> author try to model all the permutations of a charms deployment and do the >> physical deploys at that charm level are tedious and incomplete at best. >> >> With the explosion of layers and improvements to uniting test in charms >> at that component level, I feel that continuing to create these bespoke >> "bundles" via amulet in a single charm will not be a robust solution going >> forward. As we sprint closer to Juju 2.0 we're seeing a higher demand for >> assurance of working scenarios, and a sharp focus on quality at every >> level. As such I'd like to propose the following policy changes: >> >> - All bundles must have tests before promulgation to the store >> - All charms need to have comprehensive tests (unit or amulet) >> - All charms should be included in a bundle >> >> I'll break down my reasoning and examples in the following sections: >> >> # All bundles must have tests before promulgation to the store >> >> Writing bundle tests with Amulet is actually a more compelling story >> today than writing an Amulet test case for a charm. As an example, there's >> a new ELK stack bundle being produced, here's what the test for that bundle >> looks like: >> https://github.com/juju-solutions/bundle-elk-stack/blob/master/tests/10-test-bundle >> >> This makes a lot of sense because it's asserting that the bundle is >> working as expected by the Author who put the bundle together. It's also >> loading the bundle.yaml as the deployment spec meaning as the bundle >> evolves the tests will make sure they continue to run as expected. Also, >> this could potentially be used in future smoke tests for charms being >> updated if a CI process swaps out, say elasticsearch, for a newer version >> of a charm being reviewed. We can assert that both the unittests in >> elasticsearch work and it operates properly in an existing real world >> solution a la the bundle. >> >> Additional examples: >> - >> https://github.com/juju-solutions/bundle-realtime-syslog-analytics/blob/master/tests/01-bundle.py >> - >> https://github.com/juju-solutions/bundle-apache-core-batch-processing/blob/master/tests/01-bundle.py >> >> # All charms need to have comprehensive tests (unit or amulet) >> >> This is just a clarification and more strongly typed policy change that >> require charms have (preferred) unit tests or, if not applicable, then an >> Amulet test. Bash doesn't really allow for unittesting, so in those >> scenarios, Amulet tests would function as a valid testing case. >> >> There are also some charms which will not make sense as a bundle. One >> example is the recently promulgated Fiche charm: >> http://bazaar.launchpad.net/~charmers/charms/trusty/fiche/trunk/view/head:/tests/10-deploy >> It's >> a standalone pastebin, but it's an awesome service that provides deployment >> validation with an Amulet test. The test stands up the charm, exercises >> configuration, and validates the service responds in an expected way. For >> scenarios where a charm does not have a bundle an Amulet test would be >> required. >> >> Any charm that currently includes an Amulet test is welcome to continue >> keeping such a test. >> >> # All charms should be included in a bundle >> >> This last one is to underscore that charms need to serve a purpose. This >> policy is written as not an absolute, but instead a strongly worded >> suggestion as there are always charms that are exceptions to the rules. One >> such example is the aforementioned Fiche charm which as a bundle would not >> make as much sense, but is still a purposeful charm. >> >> That being said, most users coming to consume Juju are looking to solve a >> problem. Bundles underscore solutions to problems that people can consume, >> and get started quicker. >> >> As such, when new applications are charmed a test of "is this application >> something that serves a clear purpose" having a bundle submitted alongside >> the charm validates that claim and provides users a way to immediately get >> started with a solution. >> >> # Conclusion >> >> These policy changes, once accepted, will be targeted at all charms and >> bundles in Xenial as well as any new charm submitted after policy >> acceptance date for trusty, and finally any charm currently under review >> will be encouraged to adhere to the new policy but won't be required. >> >> # Action items >> >> I'm seeking feedback on this concept and welcome suggestions for >> improvements, questions, dissenting opinions, and any other remarks as well >> as votes from ~charmers and feedback from the community at large. >> >> Thanks, >> Marco Ceppi >> >> -- >> Juju mailing list >> [email protected] >> Modify settings or unsubscribe at: >> https://lists.ubuntu.com/mailman/listinfo/juju >> >>
-- Juju mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
