Good ideas ! Validates runner tests and Integration tests should be nightly executed.
For the Performance tests, it's a great idea, but not sure daily basis is required. Maybe two times per week ? As these tests could be long, we should avoid to block executors that could impact our PR build and master build. Maybe we can add Jenkins executors dedicated to PerfTest.
Regards JB On 09/03/2018 12:08, Etienne Chauchot wrote:
Hi guys, I was looking at the various jenkins jobs and I wanted to submit a proposition: - Validates runner tests: currently run at PostCommit for all the runners. I think it is the quickest way to see regressions. So keep it that way - Integration tests: AFAIK we only run the ones in examples module and only on demand. What about running all the IT (in particular IO IT) as a cron job on a daily basis with direct runner? Please note that it will require some always up backend infrastructure. - Performance tests: what about running Nexmark SMOKE test suite in batch and streaming modes with all the runners on a daily basis and store the running times in a RRD database (to see performance regressions)? Please note that not all the queries run in all the runners in all the modes right now. Also, we have some streaming pipelines termination issues (see https://issues.apache.org/jira/browse/BEAM-2847) I know that Stephen Sisk use to work on these topics. I also talked to guys from Polidea. But As I understood, they launch mainly integration tests on Dataflow runner. WDYT? Etienne
