I tend to agree with Ryan, I think the ideas are reasonably sound, although I'm not sure about the "every charm should be part of a bundle" policy, but I certainly don't think you should discourage testing at charm level the encapsulation can be useful, and you can never have too many tests!
I'm +0.5 for charms part of a bundle expectation -1 for discouraging charm tests My 2 cents. Tom -------------- Director Meteorite.bi - Saiku Analytics Founder Tel: +44(0)5603641316 (Thanks to the Saiku community we reached our Kickstart <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/> goal, but you can always help by sponsoring the project <http://www.meteorite.bi/products/saiku/sponsorship>) On 17 March 2016 at 04:38, Ryan Beisner <[email protected]> wrote: > Good evening, > > I really like the notion of a bundle possessing functional tests as an > enhancement to test coverage. I agree with almost all of those ideas. :-) > tldr; I would suggest that we consider bundle tests 'in addition to' and > not 'as a replacement of' individual charm tests, because: > > > *# Coverage and relevance* > Any given charm may have many different modes of operation -- features > which are enabled in some bundles but not in others. A bundle test will > likely only exercise that charm in the context of its configuration as it > pertains to that bundle. However, those who propose changes to the > individual charm should know (via tests) if they've functionally broken the > wider set of its knobs, bells and levers, which may be irrelevant to, or > not testable in the bundle's amulet test due to its differing perspective. > This opens potential functional test coverage gaps if we lean solely on the > bundle for the test. > > There are numerous cases where a charm can shift personalities and use > cases, but not always on-the-fly in an already-deployed model. In those > cases, it may take a completely different and new deployment topology and > configuration (bundle) to be able to exercise the relevant functional > tests. Without integrated amulet tests within the charm, one would have to > publish multiple bundles, each containing separate amulet tests. For > low-dev-velocity charms, for simple charms, or for charms that aren't > likely to be involved in complex workloads, this may be manageable. But I > don't think we should discourage or stop looking for individual charm > amulet tests even there. > > A charm's integrated amulet test can be both more focused and more > expansive in what it exercises, as it can contain multiple deployment > topologies and configurations (equivalent to cycling multiple unique > bundles). For example: charm-xyz with and without SSL; or in HA and > without HA; or IPv4 vs. IPv6; or IPv4 HA vs. IPv6 HA, multicast vs. > unicast; [IPv6 + HA + SSL] vs [IPv4 + HA + SSL]; or mysql deploying mysql > proper vs. mysql deploying a variant; and you can see the gist of the > coverage explosion which translates to having a whole load of bundles to > produce and maintain. > > > *# Dev and test: cost, scale and velocity* > Individual charm amulet tests are an important piece in testing large or > complex models. I'll share some bits of what we do for OpenStack charms as > an example. No bias. :-) > > Each of the OpenStack charms contain amulet test definitions. We lean > heavily on those tests to deploy fractions of a full OpenStack bundle as > the core of our CI development gate. With [27 charms] x [stable + dev] x > [8 Ubuntu/OpenStack Release Combos], there are currently* ~432 *possible > variations of amulet tests (derived bundles of fractional OpenStacks). A > subset of those are executed in gate, depending on relevance to the > developer's proposed change. This allows us to endure a high velocity of > focused testing on development in these very active charms. Because the > derived models are much smaller than the reference bundle, we can give > developers rapid and automated feedback, plus they can iterate on > development outside of our CI without having to be able to deploy a full > OpenStack. > > That is not to say that we don't have acceptance and integration tests for > full OpenStack bundles. We do that in the form of mojo specs which > dynamically deploy any number of full OpenStack bundle topologies and > configurations against multiple Ubuntu+OpenStack release combos, using > either the dev or the stable set of OpenStack charms. It basically takes > what I've described above for amulet and allows us to pivot entire bundles > into different models automatically. There are currently *84* such > OpenStack mojo specs with tests (bundle equivalents) > > Fear not, this is mostly accomplished with bundle inheritance, yaml foo, > and shared test libraries. We're not actually maintaining ~*516 bundles*. > But if we were to achieve the current level of coverage with bundles, > that's approximately how many there would need to be. This includes the > upcoming Xenial and Mitaka releases. Reduce by ~12% when Juno EOLs. Add > 12% when we hit Newton B1, and so on. > > > *# How I'd like to use the proposed ideas* > There are some OpenStack reference bundles in the charm store. My > suggested approach would be to continue to leverage individual charm amulet > tests while adding functional tests to the existing charm store bundles. > That would increase test coverage, and provide a mechanism to validate > proposed changes to those specific bundles, such as to re-validate the > bundles when charm versions are revved within them. > > > To summarize, I am: > > -1 to stopping or discouraging individual charm amulet tests > > +1 for every charm containing amulet tests > > +1 for every charm containing unit tests > > +1 for every charm having amulet coverage in at least 1 bundle > > +1 for every bundle possessing amulet tests > > > Also open to feedback, discussion, suggestions, kicks in the shin. > > Thanks for all the great tooling and thought leadership. We leverage the > everloving *stuff* out of Amulet. > > Charm on! > > -- > Ryan Beisner > QA Engineer, Ubuntu OpenStack Engineering, Canonical, Ltd. > irc:beisner gh/gerrit:ryan-beisner lp:~1chb1n > > > > On Wed, Mar 16, 2016 at 7:52 PM, Marco Ceppi <[email protected]> > wrote: > >> Hello everyone! >> >> This is an email I've been meaning to write for a while, and have >> rewritten a few times now. With 2.0 on the horizon and the charm ecosystem >> rapidly growing, I couldn't keep the idea to myself any longer. >> >> # tl;dr: >> >> We should stop writing Amulet tests in charms and instead only write them >> Bundles and force charms to do unit-testing (when possible) and promote >> that all charms be included in bundles in the store. >> >> # Problem >> >> Without making this a novel, charm-testing and amulet started before >> bundles were even a construct in Juju with a spec written before Juju 1.0. >> Since then, many new comers to the ecosystem have remarked how odd it is to >> be writing deployment validations at the charm level. Indeed, as years have >> gone by and new tools have sprung up it's become clear that; having an >> author try to model all the permutations of a charms deployment and do the >> physical deploys at that charm level are tedious and incomplete at best. >> >> With the explosion of layers and improvements to uniting test in charms >> at that component level, I feel that continuing to create these bespoke >> "bundles" via amulet in a single charm will not be a robust solution going >> forward. As we sprint closer to Juju 2.0 we're seeing a higher demand for >> assurance of working scenarios, and a sharp focus on quality at every >> level. As such I'd like to propose the following policy changes: >> >> - All bundles must have tests before promulgation to the store >> - All charms need to have comprehensive tests (unit or amulet) >> - All charms should be included in a bundle >> >> I'll break down my reasoning and examples in the following sections: >> >> # All bundles must have tests before promulgation to the store >> >> Writing bundle tests with Amulet is actually a more compelling story >> today than writing an Amulet test case for a charm. As an example, there's >> a new ELK stack bundle being produced, here's what the test for that bundle >> looks like: >> https://github.com/juju-solutions/bundle-elk-stack/blob/master/tests/10-test-bundle >> >> This makes a lot of sense because it's asserting that the bundle is >> working as expected by the Author who put the bundle together. It's also >> loading the bundle.yaml as the deployment spec meaning as the bundle >> evolves the tests will make sure they continue to run as expected. Also, >> this could potentially be used in future smoke tests for charms being >> updated if a CI process swaps out, say elasticsearch, for a newer version >> of a charm being reviewed. We can assert that both the unittests in >> elasticsearch work and it operates properly in an existing real world >> solution a la the bundle. >> >> Additional examples: >> - >> https://github.com/juju-solutions/bundle-realtime-syslog-analytics/blob/master/tests/01-bundle.py >> - >> https://github.com/juju-solutions/bundle-apache-core-batch-processing/blob/master/tests/01-bundle.py >> >> # All charms need to have comprehensive tests (unit or amulet) >> >> This is just a clarification and more strongly typed policy change that >> require charms have (preferred) unit tests or, if not applicable, then an >> Amulet test. Bash doesn't really allow for unittesting, so in those >> scenarios, Amulet tests would function as a valid testing case. >> >> There are also some charms which will not make sense as a bundle. One >> example is the recently promulgated Fiche charm: >> http://bazaar.launchpad.net/~charmers/charms/trusty/fiche/trunk/view/head:/tests/10-deploy >> It's >> a standalone pastebin, but it's an awesome service that provides deployment >> validation with an Amulet test. The test stands up the charm, exercises >> configuration, and validates the service responds in an expected way. For >> scenarios where a charm does not have a bundle an Amulet test would be >> required. >> >> Any charm that currently includes an Amulet test is welcome to continue >> keeping such a test. >> >> # All charms should be included in a bundle >> >> This last one is to underscore that charms need to serve a purpose. This >> policy is written as not an absolute, but instead a strongly worded >> suggestion as there are always charms that are exceptions to the rules. One >> such example is the aforementioned Fiche charm which as a bundle would not >> make as much sense, but is still a purposeful charm. >> >> That being said, most users coming to consume Juju are looking to solve a >> problem. Bundles underscore solutions to problems that people can consume, >> and get started quicker. >> >> As such, when new applications are charmed a test of "is this application >> something that serves a clear purpose" having a bundle submitted alongside >> the charm validates that claim and provides users a way to immediately get >> started with a solution. >> >> # Conclusion >> >> These policy changes, once accepted, will be targeted at all charms and >> bundles in Xenial as well as any new charm submitted after policy >> acceptance date for trusty, and finally any charm currently under review >> will be encouraged to adhere to the new policy but won't be required. >> >> # Action items >> >> I'm seeking feedback on this concept and welcome suggestions for >> improvements, questions, dissenting opinions, and any other remarks as well >> as votes from ~charmers and feedback from the community at large. >> >> Thanks, >> Marco Ceppi >> >> -- >> Juju mailing list >> [email protected] >> Modify settings or unsubscribe at: >> https://lists.ubuntu.com/mailman/listinfo/juju >> >> > > -- > Juju mailing list > [email protected] > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju > >
-- Juju mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
