On Tue, Jan 17, 2017 at 9:54 AM, Merlijn Sebrechts <[email protected]> wrote: > Hi all > > > The testing docs need some love. The whole Charm testing ecosystem is very > complex. The docs should focus on guiding users to a best-practice workflow. > There is just too much choice to let the user figure it out himself. > > We want charmers to write Charm and Bundle tests that are compatible with > > a) `bundletester` + (LXD | a cloud) > b) the review queue > > c) the cwr-ci bundle > > > The docs should provide two best-practice workflows; one for testing charms > and one for testing bundles. Each workflow needs a working example. > > The example includes a `tests.yaml` file that is compatible with a,b and c, > and it should include commands to deploy the tests locally. The example > should hide as much advanced config as possible. It's nice that you can > specify a custom glob pattern for the test files, but this is something for > advanced uses. > > Some questions that the docs should answer: > > - What tools should I use? (Bundletester + Amulet. cwr-ci if you want a ci > pipeline.) > - What's the best way to deploy a Charm/Bundle using amulet? > (`deployment.add` for a Charm and `find_bundle` for a bundle) > - Which options should I use in tests.yaml? > - Reset the environment or not? Let bundletester do cleanup, or clean it up > using Amulet? > - Deploy bundle by bundletester or by amulet? > - Where can I find Charms/Bundles that have working best-practice tests? > > I think we need some discussion around what the best-practice workflows for > testing are, and then we need to standardize everything around these > workflows. An example of a choice that needs to be discussed is whether > bundle tests should first execute the individual Charm tests. > > - The default behavior is to do this, so that implies "yes, bundle tests > should first execute charm tests" > - Charms and bundles are promulgated independently, and cwr-ci bundle is > also going this approach. This implies that the Charms are already tested > (either by cwr-ci or the review queue). This implies "no, it's useless to > first execute charm tests because we know they've succeeded". >
Thanks for bringing up the points, and the areas we should be sure we should cover in our testing docs. I agree we should make some best practice recommendations and examples for developers to leverage when getting started. With cwr-ci we wanted to be sure and include the goal of charms which is a solution context or bundles. Thus, we developed the tooling around charms being part of a bundle and even having a reference bundle that testing can occur around. There is also the introduction of Matrix to improve the operational testing of the applications in a distributed manner. A lot of good info to be documented. Ideally we take the documentation at: https://jujucharms.com/docs/stable/developer-testing Along with your key points we update it to reflect the readme at: https://jujucharms.com/u/juju-solutions/cwr-ci/ Expanding on key tools such as - CWR - Bundletester - Matrix - Libjuju Be also nice to have some recommended workflows. I'll have a draft ready by the Charmer Summit that I will send to the list here and invite you and any others interested in charm/bundle testing. Perhaps if folks are able to attend the summit in person we can discuss further face-to-face. Thoughts? -Antonio > > Kind regards > Merlijn > -- Juju mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
