@Shane Just how much time are we talking about, on average? I don't think many in the community have had much exposure to running the e2e tests in their current form. It might still be worth it in the short term.
On Tue, Oct 2, 2018 at 10:20 AM Shane Ardell <shane.m.ard...@gmail.com> wrote: > The protractor-flake package should catch and re-run false failures, so > people shouldn't get failing tests when they are done running. I just meant > that we often re-run flaky tests with protractor-flake, so it can take a > while to run and could increase the build time considerably. > > On Tue, Oct 2, 2018, 18:00 Casey Stella <ceste...@gmail.com> wrote: > > > Are the tests so brittle that, even with flaky, people will run upon > false > > failures as part of contributing a PR? If so, do we have a list of the > > brittle ones (and the things that would disambiguate a true failure from > a > > false failure) that we can add to the documentation? > > > > On Tue, Oct 2, 2018 at 11:58 AM Shane Ardell <shane.m.ard...@gmail.com> > > wrote: > > > > > I also would like to eventually have these tests automated. There are a > > > couple hurdles to setting up our e2e tests to run with our build. I > think > > > the biggest hurdle is setting up a dedicated server with data for the > e2e > > > tests to use. I would assume this requires funding, engineering > support, > > > obfuscated data, etc. I also think we should migrate our e2e tests to > > > Cypress first because Protractor lacks debugging tools that would make > > our > > > life much easier if, for example, we had a failure in our CI build but > > > could not reproduce locally. In addition, our current Protractor tests > > are > > > brittle and extremely slow. > > > > > > All that said, it seems we agree that we could add another PR checklist > > > item in the meantime. Clarifying those e2e test instructions should be > > part > > > of that task. > > > > > > On Mon, Oct 1, 2018 at 2:36 PM Casey Stella <ceste...@gmail.com> > wrote: > > > > > > > I'd also like to make sure that clear instructions are provided (or > > > linked > > > > to) about how to run them. Also, we need to make sure the > instructions > > > are > > > > rock-solid for running them. > > > > Looking at > > > > > > > > > > > > > > https://github.com/apache/metron/tree/master/metron-interface/metron-alerts#e2e-tests > > > > , > > > > would someone who doesn't have much or any knowledge of the UI be > able > > to > > > > run that without assistance? > > > > > > > > For instance, we use full-dev, do we need to stop data from being > > played > > > > into full-dev for the tests to work? > > > > > > > > Casey > > > > > > > > On Mon, Oct 1, 2018 at 8:29 AM Casey Stella <ceste...@gmail.com> > > wrote: > > > > > > > > > I'm not super keen on expanding the steps to contribute, especially > > in > > > an > > > > > avenue that should be automated. > > > > > That being said, I think that until we get to the point of > automating > > > the > > > > > e2e tests, it's sensible to add them to the checklist. > > > > > So, I would support it, but I would also urge us to move forward > the > > > > > efforts of running these tests as part of the CI build. > > > > > > > > > > What is the current gap there? > > > > > > > > > > Casey > > > > > > > > > > On Mon, Oct 1, 2018 at 7:41 AM Shane Ardell < > > shane.m.ard...@gmail.com> > > > > > wrote: > > > > > > > > > >> Hello everyone, > > > > >> > > > > >> In another discussion thread from July, I briefly mentioned the > idea > > > of > > > > >> adding a step to the pull request checklist asking contributors to > > run > > > > the > > > > >> UI end-to-end tests. Since we aren't running e2e tests as part of > > the > > > CI > > > > >> build, it's easy for contributors to unintentionally break these > > > tests. > > > > >> Reminding contributors to run these tests will hopefully help > catch > > > > >> situations like this before opening a pull request. > > > > >> > > > > >> Does this make sense to everyone? > > > > >> > > > > >> Regards, > > > > >> Shane > > > > >> > > > > > > > > > > > > > > >
