apilloud commented on code in PR #24477:
URL: https://github.com/apache/beam/pull/24477#discussion_r1051203200
##########
.test-infra/jenkins/NexmarkBuilder.groovy:
##########
@@ -174,10 +181,79 @@ class NexmarkBuilder {
}
}
+ static void pythonSuite(context, String title, Runner runner, SDK sdk,
Map<String, Object> options) {
+ InfluxDBCredentialsHelper.useCredentials(context)
+
+ for (int i = 0; i <= 12; i ++) {
+ if (
+ // https://github.com/apache/beam/issues/24678
+ i == 1 ||
+ // https://github.com/apache/beam/issues/24679
+ i == 4 || i == 6 || i == 9 ||
+ // https://github.com/apache/beam/issues/24680
+ i == 12) {
+ continue
+ }
+ pythonTest(context, title, i, runner, sdk, options)
+ }
+ }
+
+ static void pythonTest(context, String title, int query, Runner runner, SDK
sdk, Map<String, Object> options) {
+ context.steps {
+ shell("echo \"*** GENERATE events for ${title} query ${query} with
Python***\"")
+
+ options.put('query', query)
+
+ // Matches defaults in NexmarkSuite.java
+ if (query == 4 || query == 6 || query == 9) {
+ options.put('numEvents', 10000)
+ } else {
+ options.put('numEvents', 100000)
+ }
+
+ String eventFile = options.get('tempLocation') +
"/eventFiles/\${BUILD_TAG}/query${query}-"
+ options.remove('input')
+ options.put('generateEventFilePathPrefix', eventFile)
+
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':sdks:java:testing:nexmark:run')
+ commonJobProperties.setGradleSwitches(delegate)
+ switches("-Pnexmark.runner=:runners:direct-java")
+ switches("-Pnexmark.args=\"${parseOptions(options)}\"")
+ }
Review Comment:
Moving this into the python task is non-trivial, it is running a Java job
with different properties than the python job. Now that Java transforms are
supported in Python, we would be better off using that to just include the
Nexmark Java event generator in the Python job graph. Either option requires
significant work, I'm not going to do it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
