Dan, thanks for elaborating on this third option. I think it might be a good idea. I think the author of the test should know what the dependencies are. A few concerns:
* I guess it's a very early stage proposal, but I'd come up with a tag naming convention that clearly indicates what those tags do, like @role-echo (which marries the whole thing to Vagrant and its nomenclature) or @extension-echo. * How would puppet alter default cucumber behavior? * If we don't want to confuse people why some tests don't run, maybe we could store the list of available tags, check for available roles before running cucumber and show a message saying that tests requiring following (disabled) roles, will not run? -- Juliusz On Tue, Jul 29, 2014 at 5:17 PM, Dan Duvall <[email protected]> wrote: > There is another alternative that might be worth getting opinions on—Juliusz > and I both thought it seemed a bit complicated, but it would have some > additional benefits, so here goes. > > Firstly, we'd add cucumber tags to indicate the additional role dependencies > of a given feature (or individual scenario). For instance, editor_ve.feature > would include a `@visualeditor` tag and notification.feature would include > an `@echo` tag. Secondly, we'd restrict the default behavior of cucumber > (via puppet) to run only those tests that are tagged with any of the > currently enabled roles. > > The major upside to this approach would be that only those tests suited to > the current development environment get run—ostensibly these are the only > tests the developer would care to run—translating to far fewer false > negatives and more confidence (and utility) in the test suite. With the > dual-roles approach, it's highly likely that developers will continue to run > into these soft-dependency issues. And, as already stated, a single-role > approach is less flexible and will bloat the provision and git-update times. > > The biggest downside to this approach is pretty obvious, I think: It would > require the author of the test to know and include any role dependencies in > the list of tags. Is this a reasonable expectation of developers and/or > project managers (who might one day be writing features)? Another possible > downside is that altering cucumber's default behavior might confuse people > as to why some tests are being skipped. > > > On Tue, Jul 22, 2014 at 10:56 AM, Arthur Richards <[email protected]> > wrote: >> >> I agree with Max - or alternatively (or in addition) setting up a >> mobilefrontend-production role to set things up as close as possible to >> prod. >> >> >> On Fri, Jul 18, 2014 at 9:22 PM, Max Semenik <[email protected]> >> wrote: >>> >>> I personally would prefer 2, because buncing everything into one role >>> would be too inflexible. I don't see much maintenance overhead if compared >>> with 1. >>> >>> >>> On Fri, Jul 18, 2014 at 6:19 PM, Juliusz Gonera <[email protected]> >>> wrote: >>>> >>>> I talked to Dan Duvall today and he said that many mobile browser >>>> tests fail on default vagrant instance because other extensions that >>>> are not hard dependencies of MF are not activated by default when >>>> activating the mobilefrontend role (extensions such as Echo, GeoData, >>>> VisualEditor). There are two things we can do: >>>> >>>> 1. Make other extensions that mobile uses dependencies of >>>> mobilefrontend role in vagrant >>>> 2. Create a separate role, e.g. mobilefrontend-browsertests, that will >>>> list mobilefrontend and all those extensions as its dependencies >>>> >>>> I was wondering if there is anyone who uses vagrant and would like to >>>> have MF enabled, but not all the other extensions. If not, 1. seems >>>> like a simpler solution. >>>> >>>> Thoughts? >>>> >>>> -- >>>> Juliusz >>> >>> >> >> >> >> -- >> Arthur Richards >> Team Practices Lead >> [[User:Awjrichards]] >> IRC: awjr >> +1-415-839-6885 x6687 > > > > > -- > Dan Duvall > Automation Engineer > Wikimedia Foundation _______________________________________________ Mobile-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mobile-l
