On Fri, Mar 9, 2018 at 3:08 AM Etienne Chauchot <echauc...@apache.org> wrote:
> Hi guys, > > I was looking at the various jenkins jobs and I wanted to submit a > proposition: > > - Validates runner tests: currently run at PostCommit for all the runners. > I think it is the quickest way to see > regressions. So keep it that way > We've also toyed with precommit for runners where it is fast. > - Integration tests: AFAIK we only run the ones in examples module and > only on demand. What about running all the IT (in > particular IO IT) as a cron job on a daily basis with direct runner? > Please note that it will require some always up > backend infrastructure. > I like this idea. We actually run more, but in postcommit. You can see the goal here: https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PostCommit_Java_MavenInstall.groovy#L47 There's no infrastructure set up that I see. It is only DirectRunner and DataflowRunner currently, as they are "always up". But so could be local Flink and Spark. Do the ITs spin up local versions of what they are connecting to? If we have adequate resources, I also think ValidatesRunner on a real cluster would add value, once we have the cluster set up / tear down or "always up". > - Performance tests: what about running Nexmark SMOKE test suite in batch > and streaming modes with all the runners on a > daily basis and store the running times in a RRD database (to see > performance regressions)? I like this idea, too. I think we could do DirectRunner (and probably local Flink) as postcommit without being too expensive. Kenn > Please note that not all the > queries run in all the runners in all the modes right now. Also, we have > some streaming pipelines termination issues > (see https://issues.apache.org/jira/browse/BEAM-2847) > > I know that Stephen Sisk use to work on these topics. I also talked to > guys from Polidea. But As I understood, they > launch mainly integration tests on Dataflow runner. > > WDYT? > > Etienne > > >
