I think we'll have non manual rest/web deployment soon regardless of this discussion.
On Wed, Mar 22, 2017 at 2:00 PM, Ryan Merriman <merrim...@gmail.com> wrote: > I don't think a cluster installed by ansible is a prerequisite to using > ansible to integration test. They would be completely separate modules > except maybe sharing some property or inventory files. Just need to run > scripts and hit rest endpoints right? Just an idea, maybe it's overkill. > I'm cool with rolling our own. > > On Wed, Mar 22, 2017 at 1:49 PM, Casey Stella <ceste...@gmail.com> wrote: > >> Maybe, but I'd argue that we would want this to be run against a >> non-ansible installed cluster. For a first pass, I'd recommend just a set >> of shell scripts utilizing the REPL and the REST API along with shell >> commands. Most of our capabilities are quite scriptable. >> >> On Wed, Mar 22, 2017 at 2:47 PM, Ryan Merriman <merrim...@gmail.com> >> wrote: >> >> > Bumping this thread. Looks like we have several +1s so I propose we >> move >> > to the next step. I'm anxious to get this done because these tests >> would >> > have saved me time over the last couple weeks. The management UI in >> > https://github.com/apache/incubator-metron/pull/484 has a set of e2e >> tests >> > being maintained in another branch so those could also be included in >> this >> > test suite when the UI makes it into master. >> > >> > Ideas for an "Acceptance Testing Framework"? Could Ansible be good fit >> for >> > this since we already have it in our stack? >> > >> > On Mon, Mar 6, 2017 at 1:01 PM, Michael Miklavcic < >> > michael.miklav...@gmail.com> wrote: >> > >> > > Ok, yes I agree. In my experience with e2e/acceptance tests, they're >> best >> > > kept general with an emphasis on verifying that all the plumbing works >> > > together. So yes, there are definite edge cases I think we'll want to >> > test >> > > here, but I say that with the caveat that I think we should ideally >> cover >> > > as many non-happy-path cases in unit and integration tests as >> possible. >> > As >> > > an example, I don't think it makes sense to cover most of the profiler >> > > windowing DSL language edge cases in acceptance tests instead of or in >> > > addition to unit/integration tests unless there is something specific >> to >> > > the integration with a given an environment that we think could be >> > > problematic. >> > > >> > > M >> > > >> > > On Mon, Mar 6, 2017 at 11:32 AM, Casey Stella <ceste...@gmail.com> >> > wrote: >> > > >> > > > No, I'm saying that they shouldn't be restricted to real-world >> > use-cases. >> > > > The E2E tests I laid out weren't real-world, but they did exercise >> the >> > > > components similar to real-world use-cases. They should also be >> able >> > to >> > > be >> > > > able to tread outside of the happy-path for those use-cases. >> > > > >> > > > On Mon, Mar 6, 2017 at 6:30 PM, Michael Miklavcic < >> > > > michael.miklav...@gmail.com> wrote: >> > > > >> > > > > "I don't think acceptance tests should loosely associate with real >> > > uses, >> > > > > but they should >> > > > > be free to delve into weird non-happy-pathways." >> > > > > >> > > > > Not following - are you saying they should *tightly* associate >> with >> > > real >> > > > > uses and additonally include non-happy-path? >> > > > > >> > > > > On Fri, Mar 3, 2017 at 12:57 PM, Casey Stella <ceste...@gmail.com >> > >> > > > wrote: >> > > > > >> > > > > > It is absolutely not a naive question, Matt. We don't have a >> lot >> > (or >> > > > > any) >> > > > > > docs about our integration tests; it's more of a "follow the >> lead" >> > > type >> > > > > of >> > > > > > thing at the moment, but that should be rectified. >> > > > > > >> > > > > > The integration tests spin up and down infrastructure >> in-process, >> > > some >> > > > of >> > > > > > which are real and some of which are mock versions of the >> services. >> > > > > These >> > > > > > are good for catching some types of bugs, but often things sneak >> > > > through, >> > > > > > like: >> > > > > > >> > > > > > - Hbase and storm can't exist in the same JVM, so HBase is >> > mocked >> > > in >> > > > > > those cases. >> > > > > > - The FileSystem that we get for Hadoop is the >> > LocalRawFileSystem, >> > > > not >> > > > > > truly HDFS. There are differences and we've run into >> > > > > them..hilariously >> > > > > > at >> > > > > > times. ;) >> > > > > > - Things done statically in a bolt are shared across all >> bolts >> > > > because >> > > > > > they all are threads in the same process >> > > > > > >> > > > > > It's good, it catches bugs, it lets us debug things easily, it >> runs >> > > > with >> > > > > > every single build automatically via travis. >> > > > > > It's bad because it's awkward to get the dependencies isolated >> > > > > sufficiently >> > > > > > for all of these components to get them to play nice in the same >> > JVM. >> > > > > > >> > > > > > Acceptance tests would be run against a real cluster, so they >> > would: >> > > > > > >> > > > > > - run against real components, not testing or mock components >> > > > > > - run against multiple nodes >> > > > > > >> > > > > > I can imagine a world where we can unify the two to a certain >> > degree >> > > in >> > > > > > many cases if we could spin up a docker version of Metron to >> run as >> > > > part >> > > > > of >> > > > > > the build, but I think in the meantime, we should focus on >> > providing >> > > > > both. >> > > > > > >> > > > > > I suspect the reference application is possibly inspiring my >> > > > suggestions >> > > > > > here, but I think the main difference here is that the reference >> > > > > > application is intended to be informational from a end-user >> > > > perspective: >> > > > > > it's detailing a use-case that users will understand. I don't >> > think >> > > > > > acceptance tests should loosely associate with real uses, but >> they >> > > > should >> > > > > > be free to delve into weird non-happy-pathways. >> > > > > > >> > > > > > On Fri, Mar 3, 2017 at 2:16 PM, Matt Foley <ma...@apache.org> >> > wrote: >> > > > > > >> > > > > > > Automating stuff that now has to be done manually gets a big >> +1. >> > > > > > > >> > > > > > > But, Casey, could you please clarify the relationship between >> > what >> > > > you >> > > > > > > plan to do and the current “integration test” framework? Will >> > this >> > > > be >> > > > > in >> > > > > > > the form of additional integration tests? Or a different test >> > > > > framework? >> > > > > > > Can it be done in the integration test framework, rather than >> > > > creating >> > > > > > new >> > > > > > > mechanism? >> > > > > > > >> > > > > > > BTW, if that’s a naïve question, forgive me, but I could find >> > zero >> > > > > > > documentation for the existing integration test capability, >> > neither >> > > > > wiki >> > > > > > > pages nor READMEs nor Jiras. If there are any docs, please >> point >> > > me >> > > > at >> > > > > > > them. Or even archived email threads. >> > > > > > > >> > > > > > > There is also something called the “Reference Application” >> > > > > > > https://cwiki.apache.org/confluence/display/METRON/ >> > > > > > > Metron+Reference+Application which sounds remarkably like what >> > you >> > > > > > > propose to automate. Is there / can there / should there be a >> > > > > > relationship? >> > > > > > > >> > > > > > > Thanks, >> > > > > > > --Matt >> > > > > > > >> > > > > > > On 3/3/17, 7:40 AM, "Otto Fowler" <ottobackwa...@gmail.com> >> > wrote: >> > > > > > > >> > > > > > > +1 >> > > > > > > >> > > > > > > I agree with Justin’s points. >> > > > > > > >> > > > > > > >> > > > > > > On March 3, 2017 at 08:41:37, Justin Leet ( >> > > justinjl...@gmail.com >> > > > ) >> > > > > > > wrote: >> > > > > > > >> > > > > > > +1 to both. Having this would especially ease a lot of >> > testing >> > > > that >> > > > > > > hits >> > > > > > > multiple areas (which there is a fair amount of, given >> that >> > > we're >> > > > > > > building >> > > > > > > pretty quickly). >> > > > > > > >> > > > > > > I do want to point out that adding this type of thing >> makes >> > the >> > > > > speed >> > > > > > > of >> > > > > > > our builds and tests more important, because they already >> > take >> > > > up a >> > > > > > > good >> > > > > > > amount of time. There are obviously tickets to optimize >> these >> > > > > things, >> > > > > > > but >> > > > > > > I would like to make sure we don't pile too much on to >> every >> > > > > testing >> > > > > > > cycle >> > > > > > > before a PR. Having said that, I think the testing >> proposed >> > is >> > > > > > > absolutely >> > > > > > > valuable enough to go forward with. >> > > > > > > >> > > > > > > Justin >> > > > > > > >> > > > > > > On Fri, Mar 3, 2017 at 8:33 AM, Casey Stella < >> > > ceste...@gmail.com >> > > > > >> > > > > > > wrote: >> > > > > > > >> > > > > > > > I also propose, once this is done, that we modify the >> > > developer >> > > > > > > bylaws >> > > > > > > and >> > > > > > > > the github PR script to ensure that PR authors: >> > > > > > > > >> > > > > > > > - Update the acceptance tests where appropriate >> > > > > > > > - Run the tests as a smoketest >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > On Fri, Mar 3, 2017 at 8:21 AM, Casey Stella < >> > > > ceste...@gmail.com >> > > > > > >> > > > > > > wrote: >> > > > > > > > >> > > > > > > > > Hi All, >> > > > > > > > > >> > > > > > > > > After doing METRON-744, where I had to walk through a >> > > manual >> > > > > test >> > > > > > > of >> > > > > > > > every >> > > > > > > > > place that Stellar touched, it occurred to me that we >> > > should >> > > > > > script >> > > > > > > this. >> > > > > > > > > It also occurred to me that some scripts that are run >> by >> > > the >> > > > PR >> > > > > > > author >> > > > > > > to >> > > > > > > > > ensure no regressions and, eventually maybe, even run >> on >> > an >> > > > > INFRA >> > > > > > > > instance >> > > > > > > > > of Jenkins would give all of us some peace of mind. >> > > > > > > > > >> > > > > > > > > I am certain that this, along with a couple other >> manual >> > > > tests >> > > > > > from >> > > > > > > other >> > > > > > > > > PRs, could form the basis of a really great regression >> > > > > > > acceptance-test >> > > > > > > > > suite and I'd like to propose that we do that, as a >> > > > community. >> > > > > > > > > >> > > > > > > > > What I'd like to see from such a suite has the >> following >> > > > > > > characteristics: >> > > > > > > > > >> > > > > > > > > - Can be run on any Metron cluster, including but not >> > > limited >> > > > > to >> > > > > > > > > - Vagrant >> > > > > > > > > - AWS >> > > > > > > > > - An existing deployment >> > > > > > > > > - Can be *deployed* from ansible, but must be able to >> be >> > > > > deployed >> > > > > > > > > manually >> > > > > > > > > - With instructions in the readme >> > > > > > > > > - Tests should be idempotent and independent >> > > > > > > > > - Tear down what you set up >> > > > > > > > > >> > > > > > > > > I think between the Stellar REPL and the fundamental >> > > > > > scriptability >> > > > > > > of >> > > > > > > the >> > > > > > > > > Hadoop services, we can accomplish these tests with a >> > > > > combination >> > > > > > > of >> > > > > > > > shell >> > > > > > > > > scripts and python. >> > > > > > > > > >> > > > > > > > > I propose we break this into the following parts: >> > > > > > > > > >> > > > > > > > > - Acceptance Testing Framework with a small smoketest >> > > > > > > > > - Baseline Metron Test >> > > > > > > > > - Send squid data through the squid topology >> > > > > > > > > - Add an threat triage alert >> > > > > > > > > - Ensure it gets through to the other side with alerts >> > > > > preserved >> > > > > > > > > - + Enrichment >> > > > > > > > > - Add an enrichment in the enrichment pipeline to the >> > above >> > > > > > > > > - + Profiler >> > > > > > > > > - Add a profile with a tick of 1 minute to count per >> > > > > destination >> > > > > > > > > address >> > > > > > > > > - Base PCap test >> > > > > > > > > - Something like the manual test for METRON-743 ( >> > > > > > > > > https://github.com/apache/incubator-metron/pull/467# >> > > > > > > > issue-210285324 >> > > > > > > > > <https://github.com/apache/incubator-metron/pull/467# >> > > > > > > > issue-210285324> >> > > > > > > > > ) >> > > > > > > > > >> > > > > > > > > Thoughts? >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > Best, >> > > > > > > > > >> > > > > > > > > Casey >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > >
