> > - Integration tests: AFAIK we only run the ones in examples module and > only on demand. What about running all the IT (in > particular IO IT) as a cron job on a daily basis with direct runner? > Please note that it will require some always up > backend infrastructure. >
Running IOITs on Direct runner is fairly easy now - all the testing infrastructure is there and the only thing needed is switching the runner to Direct so this is nice and low effort. +1 @Kenneth: currently we spin up required databases using Kubernetes (postgres, mongo on it's way on my branch). We also added a hdfs cluster setup but no Jenkins tests are fired on it on regular basis (yet). We also had some problems running IOITs on Flink and Spark, see BEAM-3370 and BEAM-3371 so this area may need some more development. > > - Performance tests: what about running Nexmark SMOKE test suite in batch > and streaming modes with all the runners on a > daily basis and store the running times in a RRD database (to see > performance regressions)? Please note that not all the > queries run in all the runners in all the modes right now. Also, we have > some streaming pipelines termination issues > (see https://issues.apache.org/jira/browse/BEAM-2847) > +1 too. Currently Performance Tests store results in BigQuery. Do you guys think it's a good idea to store all the tests results (Nexmark and IOIT) in one database (not separately)? Or maybe think otherwise? 2018-03-10 6:59 GMT+01:00 Jean-Baptiste Onofré <j...@nanthrax.net>: > Good ideas ! > > Validates runner tests and Integration tests should be nightly executed. > > For the Performance tests, it's a great idea, but not sure daily basis is > required. Maybe two times per week ? As these tests could be long, we > should avoid to block executors that could impact our PR build and master > build. Maybe we can add Jenkins executors dedicated to PerfTest. > > Regards > JB > > > On 09/03/2018 12:08, Etienne Chauchot wrote: > >> Hi guys, >> >> I was looking at the various jenkins jobs and I wanted to submit a >> proposition: >> >> - Validates runner tests: currently run at PostCommit for all the >> runners. I think it is the quickest way to see >> regressions. So keep it that way >> >> - Integration tests: AFAIK we only run the ones in examples module and >> only on demand. What about running all the IT (in >> particular IO IT) as a cron job on a daily basis with direct runner? >> Please note that it will require some always up >> backend infrastructure. >> >> - Performance tests: what about running Nexmark SMOKE test suite in batch >> and streaming modes with all the runners on a >> daily basis and store the running times in a RRD database (to see >> performance regressions)? Please note that not all the >> queries run in all the runners in all the modes right now. Also, we have >> some streaming pipelines termination issues >> (see https://issues.apache.org/jira/browse/BEAM-2847) >> >> I know that Stephen Sisk use to work on these topics. I also talked to >> guys from Polidea. But As I understood, they >> launch mainly integration tests on Dataflow runner. >> >> WDYT? >> >> Etienne >> >> >>
