I think this is a neat idea, however I have a couple concerns with this: 1. The build is really a tree of modules. With our recent de-coupling from Storm, this has become all the more exaggerated. This is a good thing, however the benefit of the parent pom approach we currently have is that it does a pretty good job of guaranteeing all of the leaf modules get built as well. Specifying the modules individually is going to turn this into a very manual process any time a new module is added or refactored, and is more likely to result in bugs. One other option to aid in solving that problem is to use aggregator modules, and then have the full local build parent pom use those aggregators instead of referring to the individual modules the way it does now. You still get the top-down tree, but there's less of a gap between Travis and our primary Maven structure. 2. I actually like the current limits because it has forced contributors and reviewers to consider the overall build time as they design and add new tests. Especially with the current set of upgrades and proposed integration test and Docker changes, I think we should be extra vigilant on getting those build/test times *down*, rather than enabling them to increase. Sure, there are other options for solving that problem, but Travis is a simple equalizer because it's impartial to whatever local hardware engineers are using.
On Wed, May 22, 2019 at 7:48 AM Nick Allen <n...@nickallen.org> wrote: > FYI - Here is a POC build of this concept running. This only runs the > Parser and Enrichment integration tests, but as separate jobs in Travis. > > https://travis-ci.org/nickwallen/metron/builds/535791047 > > > > > On Wed, May 22, 2019 at 9:20 AM Nick Allen <n...@nickallen.org> wrote: > > > > > > Justin Leet said > > <https://github.com/apache/metron/pull/1417#issuecomment-494464795> > [1]: Looking > > at our build times, I'm actually concerned that if we kill the caches our > > builds won't complete. The integration tests take >45 minutes and it's > very > > possible redownloading everything goes over our remaining time. > > > > To recap, our current Travis CI build is composed of multiple jobs. > Travis' > > time limit of 50 minutes > > <https://docs.travis-ci.com/user/customizing-the-build/#build-timeouts> > [2] > > is per job, rather than the total build time. While our total build time > > is on the order of 2.5 hours, it is only our integration-test job which > is > > coming close to that 50 minute limit. > > > > We could try splitting our integration tests into multiple jobs. Each of > > these would have the opportunity to run in parallel (given whatever > > resources Travis can allocate to us), but more importantly each one has > its > > own 50 minute limit. For example... > > > > - Parser Integration Tests: > > - `time mvn surefire:test@integration-tests -pl > > "metron-platform/metron-parsing/metron-parsing-storm, > > metron-platform/metron-parsing/metron-parsers, > > metron-platform/metron-parsing/metron-parsers-common/"` > > - Enrichment Integration Tests > > - `time mvn surefire:test@integration-tests -pl > > > > "metron-platform/metron-enrichment/metron-enrichment-common/,metron-platform/metron-enrichment/metron-enrichment-storm"` > > - etc, etc > > > > > > We would just need to determine how to logically split the tests up. If > > this sounds reasonable, I've already got a start on a POC. > > > > --- > > [1] https://github.com/apache/metron/pull/1417#issuecomment-494464795 > > [2] > https://docs.travis-ci.com/user/customizing-the-build/#build-timeouts > > >
