Thanks everyone for your comments and support. Le vendredi 09 mars 2018 à 21:28 +0000, Alan Myrvold a écrit : > Great ideas. I want to see a daily signal for anything that could prevent a > release from happening, and precommits > that are fast and reliable for areas that are commonly broken by code changes. > > We are now running the java quickstarts daily on a cron schedule, using > direct, dataflow, and local spark and flink in > the beam_PostRelease_NightlySnapshot job, see > https://github.com/apache/beam/blob/master/release/build.gradle This > should provide a good signal for the examples integration tests against these > runners. > > As Kenn noted, the java_maveninstall also runs lots of tests. It would be > good to be more clear and intentional about > which tests run when, and to consider implementing additional "always up" > environments for use by the tests. > > Having the nexmark smoke tests run regularly and stored in a database would > really enhance our efforts, perhaps > starting with directrunner for the performance tests. Yes > > What area would have the most immediate impact? Nexmark smoke tests? Yes IMHO I think that Nexmark smoke tests would have a great return on investment. By just scheduling some of them (at first), we enable deep confidence in the runners on real user pipelines. In the past Nexmark has allowed to discover regressions in performance before a release and also to discover some bugs in some runners. But, please note that, for this last ability, Nexmark is limited currently: it only detects failures if an exception is thrown, there is no check of the correctness of the output PCollection because the aim was performance tests and there is no point adding a slow test for correctness. Nevertheless, if we store the output size (as I suggested in this thread), we can get a hint on a failure if the output size is different from the last stored output sizes. Etienne > > >
> On Fri, Mar 9, 2018 at 12:57 PM Kenneth Knowles <k...@google.com> wrote:> > > On Fri, Mar 9, 2018 at 3:08 AM Etienne Chauchot <echauc...@apache.org> > wrote:> > > Hi guys, > > > > > > > > > I was looking at the various jenkins jobs and I wanted to submit a > > > proposition: > > > > > > > > > - Validates runner tests: currently run at PostCommit for all the > > > runners. I think it is the quickest way to see > > > > > > regressions. So keep it that way> > > > We've also toyed with precommit > > > for runners where it is fast. > > > > > > > > - Integration tests: AFAIK we only run the ones in examples module and > > > only on demand. What about running all the IT (in > > > > > > particular IO IT) as a cron job on a daily basis with direct runner? > > > Please note that it will require some always up > > > > > > backend infrastructure.> > > > I like this idea. We actually run more, > > > but in postcommit. You can see the goal here: > > > https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PostCommit_Java_MavenInstall.groovy#L47 > > > > There's no infrastructure set up that I see. > > It is only > > > > DirectRunner and DataflowRunner currently, as they are "always up". But > > > > so could be local Flink and Spark. Do the ITs spin up local versions of > > > > what they are connecting to? > > > > If we have adequate resources, I also think ValidatesRunner on a real > > > > cluster would add value, once we have the cluster set up / tear down or > > > > "always up". > > > > > > > > - Performance tests: what about running Nexmark SMOKE test suite in batch > > > and streaming modes with all the runners on a > > > > > > daily basis and store the running times in a RRD database (to see > > > performance regressions)?> > > > I like this idea, too. I think we could do DirectRunner (and probably local > > Flink) as postcommit without being too expensive. > > > > Kenn > > > > > > > Please note that not all the > > > > > > queries run in all the runners in all the modes right now. Also, we have > > > some streaming pipelines termination issues > > > > > > (see https://issues.apache.org/jira/browse/BEAM-2847) > > > > > > > > > I know that Stephen Sisk use to work on these topics. I also talked to > > > guys from Polidea. But As I understood, they > > > > > > launch mainly integration tests on Dataflow runner. > > > > > > > > > WDYT? > > > > > > > > > Etienne > > > > > > > > > > > >
