kkucharc commented on a change in pull request #12542:
URL: https://github.com/apache/beam/pull/12542#discussion_r469981965
##########
File path: .test-infra/jenkins/job_LoadTests_coGBK_Python.groovy
##########
@@ -25,7 +25,7 @@ import InfluxDBCredentialsHelper
def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
-def loadTestConfigurations = { datasetName ->
+def loadTestConfigurations = { mode, datasetName ->
Review comment:
I think we should agree on the common name of this param. In combine I
put `jobType` (mainly because Java test has `jobType`), but there is a
possibility to change it since combine isn't merged yet.
##########
File path: .test-infra/jenkins/job_LoadTests_coGBK_Python.groovy
##########
@@ -147,25 +147,30 @@ def loadTestConfigurations = { datasetName ->
autoscaling_algorithm: 'NONE'
]
],
- ].each { test -> test.pipelineOptions.putAll(additionalPipelineArgs) }
+ ]
+ .each { test -> test.pipelineOptions.putAll(additionalPipelineArgs) }
+ .each { test -> (mode) != 'streaming' ?: addStreamingOptions(test) }
}
-def batchLoadTestJob = { scope, triggeringContext ->
- scope.description('Runs Python CoGBK load tests on Dataflow runner in batch
mode')
- commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 240)
+def addStreamingOptions(test) {
+ // Use highmem workers to prevent out of memory issues.
+ test.pipelineOptions << [streaming: null,
+ worker_machine_type: 'n1-highmem-4'
Review comment:
Why this type of machine?
##########
File path: .test-infra/jenkins/job_LoadTests_coGBK_Python.groovy
##########
@@ -34,12 +34,12 @@ def loadTestConfigurations = { datasetName ->
pipelineOptions: [
project : 'apache-beam-testing',
region : 'us-central1',
- job_name : 'load-tests-python-dataflow-batch-cogbk-1-' +
now,
+ job_name : "load-tests-python-dataflow-${mode}-cogbk-1-" +
now,
Review comment:
Now you can also ass `now` as ${now} :D
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
