Great ideas. I want to see a daily signal for anything that could prevent a release from happening, and precommits that are fast and reliable for areas that are commonly broken by code changes.
We are now running the java quickstarts daily on a cron schedule, using direct, dataflow, and local spark and flink in the beam_PostRelease_NightlySnapshot job, see https://github.com/apache/beam/blob/master/release/build.gradle This should provide a good signal for the examples integration tests against these runners. As Kenn noted, the java_maveninstall also runs lots of tests. It would be good to be more clear and intentional about which tests run when, and to consider implementing additional "always up" environments for use by the tests. Having the nexmark smoke tests run regularly and stored in a database would really enhance our efforts, perhaps starting with directrunner for the performance tests. What area would have the most immediate impact? Nexmark smoke tests? On Fri, Mar 9, 2018 at 12:57 PM Kenneth Knowles <k...@google.com> wrote: > On Fri, Mar 9, 2018 at 3:08 AM Etienne Chauchot <echauc...@apache.org> > wrote: > >> Hi guys, >> >> I was looking at the various jenkins jobs and I wanted to submit a >> proposition: >> >> - Validates runner tests: currently run at PostCommit for all the >> runners. I think it is the quickest way to see >> regressions. So keep it that way >> > > We've also toyed with precommit for runners where it is fast. > > >> - Integration tests: AFAIK we only run the ones in examples module and >> only on demand. What about running all the IT (in >> particular IO IT) as a cron job on a daily basis with direct runner? >> Please note that it will require some always up >> backend infrastructure. >> > > I like this idea. We actually run more, but in postcommit. You can see the > goal here: > https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PostCommit_Java_MavenInstall.groovy#L47 > > There's no infrastructure set up that I see. It is only DirectRunner and > DataflowRunner currently, as they are "always up". But so could be local > Flink and Spark. Do the ITs spin up local versions of what they are > connecting to? > > If we have adequate resources, I also think ValidatesRunner on a real > cluster would add value, once we have the cluster set up / tear down or > "always up". > > >> - Performance tests: what about running Nexmark SMOKE test suite in batch >> and streaming modes with all the runners on a >> daily basis and store the running times in a RRD database (to see >> performance regressions)? > > > I like this idea, too. I think we could do DirectRunner (and probably > local Flink) as postcommit without being too expensive. > > Kenn > > > >> Please note that not all the >> queries run in all the runners in all the modes right now. Also, we have >> some streaming pipelines termination issues >> (see https://issues.apache.org/jira/browse/BEAM-2847) >> >> I know that Stephen Sisk use to work on these topics. I also talked to >> guys from Polidea. But As I understood, they >> launch mainly integration tests on Dataflow runner. >> >> WDYT? >> >> Etienne >> >> >>
