Hi,
I would suggest to prepare a Maven profile to perform nexmark runs.
Then, I can setup a job (seed/manual) in Jenkins to run this.
Regards
JB
On 15/03/2018 22:13, Etienne Chauchot wrote:
So what next? Shall we schedule nexmark runs and add a Bigquery sink to
nexmark output?
Le lundi 12 mars 2018 à 10:30 +0100, Etienne Chauchot a écrit :
Thanks everyone for your comments and support.
Le vendredi 09 mars 2018 à 21:28 +0000, Alan Myrvold a écrit :
Great ideas. I want to see a daily signal for anything that could
prevent a release from happening, and precommits that are fast and
reliable for areas that are commonly broken by code changes.
We are now running the java quickstarts daily on a cron schedule,
using direct, dataflow, and local spark and flink in
the beam_PostRelease_NightlySnapshot job, see
https://github.com/apache/beam/blob/master/release/build.gradle This
should provide a good signal for the examples integration tests
against these runners.
As Kenn noted, the java_maveninstall also runs lots of tests. It
would be good to be more clear and intentional about which tests run
when, and to consider implementing additional "always up"
environments for use by the tests.
Having the nexmark smoke tests run regularly and stored in a database
would really enhance our efforts, perhaps starting with directrunner
for the performance tests.
Yes
What area would have the most immediate impact? Nexmark smoke tests?
Yes IMHO I think that Nexmark smoke tests would have a great return on
investment. By just scheduling some of them (at first), we enable
deep confidence in the runners on real user pipelines. In the past
Nexmark has allowed to discover regressions in performance before a
release and also to discover some bugs in some runners. But, please
note that, for this last ability, Nexmark is limited currently: it
only detects failures if an exception is thrown, there is no check of
the correctness of the output PCollection because the aim was
performance tests and there is no point adding a slow test for
correctness. Nevertheless, if we store the output size (as I suggested
in this thread), we can get a hint on a failure if the output size is
different from the last stored output sizes.
Etienne
On Fri, Mar 9, 2018 at 12:57 PM Kenneth Knowles <k...@google.com
<mailto:k...@google.com>> wrote:
On Fri, Mar 9, 2018 at 3:08 AM Etienne Chauchot
<echauc...@apache.org <mailto:echauc...@apache.org>> wrote:
Hi guys,
I was looking at the various jenkins jobs and I wanted to submit a
proposition:
- Validates runner tests: currently run at PostCommit for all the
runners. I think it is the quickest way to see
regressions. So keep it that way
We've also toyed with precommit for runners where it is fast.
- Integration tests: AFAIK we only run the ones in examples module
and only on demand. What about running all the IT (in
particular IO IT) as a cron job on a daily basis with direct
runner? Please note that it will require some always up
backend infrastructure.
I like this idea. We actually run more, but in postcommit. You can
see the goal here:
https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PostCommit_Java_MavenInstall.groovy#L47
There's no infrastructure set up that I see. It is only DirectRunner
and DataflowRunner currently, as they are "always up". But so could
be local Flink and Spark. Do the ITs spin up local versions of what
they are connecting to?
If we have adequate resources, I also think ValidatesRunner on a
real cluster would add value, once we have the cluster set up / tear
down or "always up".
- Performance tests: what about running Nexmark SMOKE test suite in
batch and streaming modes with all the runners on a
daily basis and store the running times in a RRD database (to see
performance regressions)?
I like this idea, too. I think we could do DirectRunner (and
probably local Flink) as postcommit without being too expensive.
Kenn
Please note that not all the
queries run in all the runners in all the modes right now. Also, we
have some streaming pipelines termination issues
(see https://issues.apache.org/jira/browse/BEAM-2847)
I know that Stephen Sisk use to work on these topics. I also talked
to guys from Polidea. But As I understood, they
launch mainly integration tests on Dataflow runner.
WDYT?
Etienne
