[beam] branch master updated: [BEAM-7347] Pass dpb_service_zone flag to benchmarks to fix breakage
This is an automated email from the ASF dual-hosted git repository.
markliu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 42ada03 [BEAM-7347] Pass dpb_service_zone flag to benchmarks to fix
breakage
new 15deadf Merge pull request #8602 from markflyhigh/fix-perf
42ada03 is described below
commit 42ada03f5758ff17d203add68c389194bde6581e
Author: Mark Liu
AuthorDate: Fri May 17 10:11:21 2019 -0700
[BEAM-7347] Pass dpb_service_zone flag to benchmarks to fix breakage
---
.test-infra/jenkins/CommonJobProperties.groovy | 10 --
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/.test-infra/jenkins/CommonJobProperties.groovy
b/.test-infra/jenkins/CommonJobProperties.groovy
index cb4a4eb..2dddaad 100644
--- a/.test-infra/jenkins/CommonJobProperties.groovy
+++ b/.test-infra/jenkins/CommonJobProperties.groovy
@@ -236,13 +236,19 @@ class CommonJobProperties {
project: 'apache-beam-testing',
dpb_log_level: 'INFO',
bigquery_table: 'beam_performance.pkb_results',
- k8s_get_retry_count: 36, // wait up to 6 minutes for K8s LoadBalancer
+ // wait up to 6 minutes for K8s LoadBalancer
+ k8s_get_retry_count: 36,
k8s_get_wait_interval: 10,
temp_dir: '$WORKSPACE',
// Use source cloned by Jenkins and not clone it second time
(redundantly).
beam_location: '$WORKSPACE/src',
// Publishes results with official tag, for use in dashboards.
- official: 'true'
+ official: 'true',
+ // dpb_service_zone is required in Perfkit BaseDpbService which Beam
Perfkit benchmarks
+ // depends on. However, it doesn't get used in Beam. Passing a fake
value from here is to
+ // avoid breakage.
+ // TODO(BEAM-7347): Remove this flag after dpb_service_zone is not
required.
+ dpb_service_zone: 'fake_zone',
]
// Note: in case of key collision, keys present in ArgMap win.
LinkedHashMap joinedArgs = standardArgs.plus(argMap)
[beam] branch master updated: [BEAM-6908] Refactor Python performance test groovy file for easy configuration (#8518)
This is an automated email from the ASF dual-hosted git repository.
markliu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new d77474d [BEAM-6908] Refactor Python performance test groovy file for
easy configuration (#8518)
d77474d is described below
commit d77474d16734cf63095765cbced97ecd4b016b54
Author: Mark Liu
AuthorDate: Thu May 16 17:31:55 2019 -0700
[BEAM-6908] Refactor Python performance test groovy file for easy
configuration (#8518)
Refactor Python performance test groovy file for easy configuration.
---
.../jenkins/job_PerformanceTests_Python.groovy | 158 -
.../jenkins/job_Performancetests_Python35.groovy | 63
2 files changed, 120 insertions(+), 101 deletions(-)
diff --git a/.test-infra/jenkins/job_PerformanceTests_Python.groovy
b/.test-infra/jenkins/job_PerformanceTests_Python.groovy
index 5354d46..732272f 100644
--- a/.test-infra/jenkins/job_PerformanceTests_Python.groovy
+++ b/.test-infra/jenkins/job_PerformanceTests_Python.groovy
@@ -18,46 +18,128 @@
import CommonJobProperties as commonJobProperties
-// This job runs the Beam Python performance tests on PerfKit Benchmarker.
-job('beam_PerformanceTests_Python'){
- // Set default Beam job properties.
- commonJobProperties.setTopLevelMainJobProperties(delegate)
-
- // Run job in postcommit every 6 hours, don't trigger every push.
- commonJobProperties.setAutoJob(
- delegate,
- 'H */6 * * *')
-
- // Allows triggering this build against pull requests.
- commonJobProperties.enablePhraseTriggeringFromPullRequest(
- delegate,
- 'Python SDK Performance Test',
- 'Run Python Performance Test')
-
- def pipelineArgs = [
- project: 'apache-beam-testing',
- staging_location: 'gs://temp-storage-for-end-to-end-tests/staging-it',
- temp_location: 'gs://temp-storage-for-end-to-end-tests/temp-it',
- output: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
- ]
+
+class PerformanceTestConfigurations {
+ // Name of the Jenkins job
+ String jobName
+ // Description of the Jenkins job
+ String jobDescription
+ // Phrase to trigger this Jenkins job
+ String jobTriggerPhrase
+ // Frequency of the job build, default to every 6 hours
+ String buildSchedule = 'H */6 * * *'
+ // A benchmark defined flag, will pass to benchmark as "--benchmarkName"
+ String benchmarkName = 'beam_integration_benchmark'
+ // A benchmark defined flag, will pass to benchmark as "--bigqueryTable"
+ String resultTable
+ // A benchmark defined flag, will pass to benchmark as "--beam_it_class"
+ String itClass
+ // A benchmark defined flag, will pass to benchmark as "--beam_it_module".
+ // It's a Gradle project that defines 'integrationTest' task. This task is
executed by Perfkit
+ // Beam benchmark launcher and can be added by enablePythonPerformanceTest()
defined in
+ // BeamModulePlugin.
+ String itModule
+ // A benchmark defined flag, will pass to benchmark as
"--beam_python_sdk_location".
+ // It's the location of Python SDK distribution archive which is required
for TestDataflowRunner.
+ String pythonSdkLocation = ''
+ // A benchmark defined flag, will pass to benchmark as "--beam_runner"
+ String runner = 'TestDataflowRunner'
+ // A benchmark defined flag, will pass to benchmark as "--beam_it_timeout"
+ Integer itTimeoutSec = 1200
+ // A benchmark defined flag, will pass to benchmark as "--beam_it_args"
+ Map extraPipelineArgs
+}
+
+// Common pipeline args for Dataflow job.
+def dataflowPipelineArgs = [
+project : 'apache-beam-testing',
+staging_location: 'gs://temp-storage-for-end-to-end-tests/staging-it',
+temp_location : 'gs://temp-storage-for-end-to-end-tests/temp-it',
+]
+
+
+// Configurations of each Jenkins job.
+def testConfigurations = [
+new PerformanceTestConfigurations(
+jobName : 'beam_PerformanceTests_WordCountIT_Py27',
+jobDescription: 'Python SDK Performance Test - Run WordCountIT in
Py27',
+jobTriggerPhrase : 'Run Python27 WordCountIT Performance Test',
+resultTable : 'beam_performance.wordcount_py27_pkb_results',
+itClass :
'apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it',
+itModule : 'sdks/python',
+extraPipelineArgs : dataflowPipelineArgs + [
+output: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
+],
+),
+new PerformanceTestConfigurations(
+jobName : 'beam_PerformanceTests_WordCountIT_Py35',
+jobDescription: 'Python SDK Performance Test - Run WordCountIT in
Py35',
+jobTriggerPhrase : 'Run Python35 WordCountIT Performance Test',
+res
[beam] branch master updated: Fix project path in beam_PerformanceTests_Python35
This is an automated email from the ASF dual-hosted git repository.
markliu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new c36913e Fix project path in beam_PerformanceTests_Python35
new bc47d72 Merge pull request #8510 from markflyhigh/fix-py-perf
c36913e is described below
commit c36913ee40b126bf2bb3d8bf7021ccfcfbac6955
Author: Mark Liu
AuthorDate: Mon May 6 12:05:15 2019 -0700
Fix project path in beam_PerformanceTests_Python35
---
.test-infra/jenkins/job_Performancetests_Python35.groovy | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/.test-infra/jenkins/job_Performancetests_Python35.groovy
b/.test-infra/jenkins/job_Performancetests_Python35.groovy
index 842df97..41e0fcc 100644
--- a/.test-infra/jenkins/job_Performancetests_Python35.groovy
+++ b/.test-infra/jenkins/job_Performancetests_Python35.groovy
@@ -51,9 +51,9 @@ job('beam_PerformanceTests_Python35'){
benchmarks : 'beam_integration_benchmark',
bigquery_table : 'beam_performance.wordcount_py35_pkb_results',
beam_it_class:
'apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it',
- beam_it_module : 'sdks/python/test-suites/dataflow/py3',
+ beam_it_module : 'sdks/python/test-suites/dataflow/py35',
beam_prebuilt: 'true', // skip beam prebuild
- beam_python_sdk_location :
'test-suites/dataflow/py3/build/apache-beam.tar.gz',
+ beam_python_sdk_location :
'test-suites/dataflow/py35/build/apache-beam.tar.gz',
beam_runner : 'TestDataflowRunner',
beam_it_timeout : '1200',
beam_it_args : pipelineArgsJoined,
[beam] branch master updated: [BEAM-6908] New Jenkins branch for Python35 benchmark
This is an automated email from the ASF dual-hosted git repository. markliu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 86b5b19 [BEAM-6908] New Jenkins branch for Python35 benchmark new d9c2de8 Merge pull request #8393 from markflyhigh/py35-benchmark 86b5b19 is described below commit 86b5b191ee50d215bed846bebaf10a56fe8453bc Author: Mark Liu AuthorDate: Wed Apr 24 14:08:31 2019 -0700 [BEAM-6908] New Jenkins branch for Python35 benchmark --- .test-infra/jenkins/README.md | 1 + .../jenkins/job_Performancetests_Python35.groovy | 63 ++ 2 files changed, 64 insertions(+) diff --git a/.test-infra/jenkins/README.md b/.test-infra/jenkins/README.md index f478d31..775569e 100644 --- a/.test-infra/jenkins/README.md +++ b/.test-infra/jenkins/README.md @@ -84,6 +84,7 @@ Beam Jenkins overview page: [link](https://builds.apache.org/view/A-D/view/Beam/ | beam_PerformanceTests_ManyFiles_TextIOIT | [cron](https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT/), [hdfs_cron](https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/) | `Run Java ManyFilesTextIO Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT) [, [hdfs_cron](https://builds.apache.org/job/beam_PerformanceTests_ParquetIOIT_HDFS/) | `Run Java ParquetIO Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_ParquetIOIT) [ | `Run Python Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_Python) | +| beam_PerformanceTests_Python35 | [cron](https://builds.apache.org/job/beam_PerformanceTests_Python35/) | `Run Python35 Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_Python35) | | beam_PerformanceTests_Spark | [cron](https://builds.apache.org/job/beam_PerformanceTests_Spark/) | `Run Spark Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_Spark) | | beam_PerformanceTests_TFRecordIOIT | [cron](https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/) | `Run Java JdbcIO Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT) | | beam_PerformanceTests_TextIOIT | [cron](https://builds.apache.org/job/beam_PerformanceTests_TextIOIT/), [hdfs_cron](https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/) | `Run Java TextIO Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_TextIOIT) [](https://builds.apa [...] diff --git a/.test-infra/jenkins/job_Performancetests_Python35.groovy b/.test-infra/jenkins/job_Performancetests_Python35.groovy new file mode 100644 index 000..842df97 --- /dev/null +++ b/.test-infra/jenkins/job_Performancetests_Python35.groovy @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import CommonJobProperties as commonJobProperties + +// This job runs the Beam Python35 performance benchmark on PerfKi
[beam] branch master updated: Fix Python wordcount benchmark by specifiying beam_it_module
This is an automated email from the ASF dual-hosted git repository.
markliu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new ad14cb9 Fix Python wordcount benchmark by specifiying beam_it_module
new 8ea9a8e Merge pull request #8389 from markflyhigh/fix-py-benchmark
ad14cb9 is described below
commit ad14cb90638effd4446784ae9c0ffc0a57ebbf15
Author: Mark Liu
AuthorDate: Wed Apr 24 11:02:34 2019 -0700
Fix Python wordcount benchmark by specifiying beam_it_module
---
.test-infra/jenkins/job_PerformanceTests_Python.groovy | 1 +
1 file changed, 1 insertion(+)
diff --git a/.test-infra/jenkins/job_PerformanceTests_Python.groovy
b/.test-infra/jenkins/job_PerformanceTests_Python.groovy
index be734ba..5354d46 100644
--- a/.test-infra/jenkins/job_PerformanceTests_Python.groovy
+++ b/.test-infra/jenkins/job_PerformanceTests_Python.groovy
@@ -51,6 +51,7 @@ job('beam_PerformanceTests_Python'){
benchmarks : 'beam_integration_benchmark',
bigquery_table : 'beam_performance.wordcount_py_pkb_results',
beam_it_class:
'apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it',
+ beam_it_module : 'sdks/python',
beam_prebuilt: 'true', // skip beam prebuild
beam_python_sdk_location : 'build/apache-beam.tar.gz',
beam_runner : 'TestDataflowRunner',
[beam] branch master updated: [BEAM-6908] Support Python3 performance benchmarks - part 2
This is an automated email from the ASF dual-hosted git repository.
markliu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 43be51f [BEAM-6908] Support Python3 performance benchmarks - part 2
new c97e391 Merge pull request #8324 from markflyhigh/py3-perf
43be51f is described below
commit 43be51f1d64ffe3ae856570ac68348ab99ea5946
Author: Mark Liu
AuthorDate: Tue Apr 16 14:28:38 2019 -0700
[BEAM-6908] Support Python3 performance benchmarks - part 2
---
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 57 ++
.../apache_beam/io/gcp/gcsio_integration_test.py | 4 +-
sdks/python/build.gradle | 31 +---
sdks/python/scripts/run_integration_test.sh| 25 +-
sdks/python/test-suites/dataflow/py3/build.gradle | 1 +
5 files changed, 75 insertions(+), 43 deletions(-)
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 6614954..4b6c75d 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -235,6 +235,24 @@ class BeamModulePlugin implements Plugin {
String itModule = System.getProperty('itModule')
}
+ // Reads and contains all necessary performance test parameters
+ class PythonPerformanceTestConfiguration {
+// Fully qualified name of the test to run.
+String tests = System.getProperty('tests')
+
+// Attribute tag that can filter the test set.
+String attribute = System.getProperty('attr')
+
+// Extra test options pass to nose.
+String[] extraTestOptions = ["--nocapture"]
+
+// Name of Cloud KMS encryption key to use in some tests.
+String kmsKeyName = System.getProperty('kmsKeyName')
+
+// Pipeline options to be used for pipeline invocation.
+String pipelineOptions = System.getProperty('pipelineOptions', '')
+ }
+
// A class defining the set of configurable properties accepted by
containerImageName.
class ContainerImageNameConfiguration {
String root = null // Sets the docker repository root (optional).
@@ -1733,6 +1751,45 @@ class BeamModulePlugin implements Plugin {
outputs.files project.fileTree(dir:
"${pythonRootDir}/target/.tox/${tox_env}/log/")
}
}
+
+ // Run single or a set of integration tests with provided test options
and pipeline options.
+ project.ext.enablePythonPerformanceTest = {
+
+// Use the implicit it parameter of the closure to handle zero
argument or one argument map calls.
+// See: http://groovy-lang.org/closures.html#implicit-it
+def config = it ? it as PythonPerformanceTestConfiguration : new
PythonPerformanceTestConfiguration()
+
+project.task('integrationTest') {
+ dependsOn 'installGcpTest'
+ dependsOn 'sdist'
+
+ doLast {
+def argMap = [:]
+
+// Build test options that configures test environment and
framework
+def testOptions = []
+if (config.tests)
+ testOptions += "--tests=$config.tests"
+if (config.attribute)
+ testOptions += "--attr=$config.attribute"
+testOptions.addAll(config.extraTestOptions)
+argMap["test_opts"] = testOptions
+
+// Build pipeline options that configures pipeline job
+if (config.pipelineOptions)
+ argMap["pipeline_opts"] = config.pipelineOptions
+if (config.kmsKeyName)
+ argMap["kms_key_name"] = config.kmsKeyName
+
+def cmdArgs = project.mapToArgString(argMap)
+def runScriptsDir = "${pythonRootDir}/scripts"
+project.exec {
+ executable 'sh'
+ args '-c', ". ${project.ext.envdir}/bin/activate &&
${runScriptsDir}/run_integration_test.sh ${cmdArgs}"
+}
+ }
+}
+ }
}
}
}
diff --git a/sdks/python/apache_beam/io/gcp/gcsio_integration_test.py
b/sdks/python/apache_beam/io/gcp/gcsio_integration_test.py
index 18a367d..8312f07 100644
--- a/sdks/python/apache_beam/io/gcp/gcsio_integration_test.py
+++ b/sdks/python/apache_beam/io/gcp/gcsio_integration_test.py
@@ -32,8 +32,8 @@ permissions for the key specified in --kms_key_name.
To run these tests manually:
./gradlew beam-sdks-python:integrationTest \
--Ptests=apache_beam.io.gcp.gcsio_integration_test:GcsIOIntegrationTest \
--PkmsKeyName=KMS_KEY_NAME
+-Dtests=apache_beam.io.gcp.gcsio_integration_test:GcsIOIntegrationTest \
+-DkmsKeyName=KMS_KEY_NAME
"""
from __future__ import absolute_import
diff --git a/sdks/pytho
[beam] branch master updated: Fix Jenkins job virtualenv setup with specific py version
This is an automated email from the ASF dual-hosted git repository.
markliu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 48f5ca8 Fix Jenkins job virtualenv setup with specific py version
new 0dde4a0 Merge pull request #8336 from markflyhigh/perfkit-improve-2
48f5ca8 is described below
commit 48f5ca8416ad96123ac040538c11b7c58d27eeb7
Author: Mark Liu
AuthorDate: Wed Apr 17 10:18:15 2019 -0700
Fix Jenkins job virtualenv setup with specific py version
Need to pin python2.7 when setup virtualenv in some jobs that runs on new
Jenkins node. Otherwise, python3 interpreter will be used by default which can
break the job.
---
.test-infra/jenkins/CommonJobProperties.groovy| 8
.test-infra/jenkins/job_beam_PerformanceTests_Analysis.groovy | 5 +++--
2 files changed, 7 insertions(+), 6 deletions(-)
diff --git a/.test-infra/jenkins/CommonJobProperties.groovy
b/.test-infra/jenkins/CommonJobProperties.groovy
index d1048d0..cb4a4eb 100644
--- a/.test-infra/jenkins/CommonJobProperties.groovy
+++ b/.test-infra/jenkins/CommonJobProperties.groovy
@@ -238,7 +238,6 @@ class CommonJobProperties {
bigquery_table: 'beam_performance.pkb_results',
k8s_get_retry_count: 36, // wait up to 6 minutes for K8s LoadBalancer
k8s_get_wait_interval: 10,
- python_binary: '$WORKSPACE/.beam_env/bin/python',
temp_dir: '$WORKSPACE',
// Use source cloned by Jenkins and not clone it second time
(redundantly).
beam_location: '$WORKSPACE/src',
@@ -281,7 +280,7 @@ class CommonJobProperties {
static def buildPerformanceTest(def context, def argMap) {
def pkbArgs = genPerformanceArgs(argMap)
-// Absolute path of project root and virtualenv path of Beam and Perfkit.
+// Absolute path of project root and virtualenv path of Perfkit.
def perfkit_root = makePathAbsolute("PerfKitBenchmarker")
def perfkit_env = makePathAbsolute("env/.perfkit_env")
@@ -290,8 +289,9 @@ class CommonJobProperties {
shell("rm -rf ${perfkit_root}")
shell("rm -rf ${perfkit_env}")
-// create new VirtualEnv
-shell("virtualenv ${perfkit_env}")
+// create new VirtualEnv for Perfkit framework. Explicitly pin to
python2.7
+// here otherwise python3 is used by default.
+shell("virtualenv ${perfkit_env} --python=python2.7")
// update setuptools and pip
shell("${perfkit_env}/bin/pip install --upgrade setuptools pip")
diff --git a/.test-infra/jenkins/job_beam_PerformanceTests_Analysis.groovy
b/.test-infra/jenkins/job_beam_PerformanceTests_Analysis.groovy
index efa8353..f137868 100644
--- a/.test-infra/jenkins/job_beam_PerformanceTests_Analysis.groovy
+++ b/.test-infra/jenkins/job_beam_PerformanceTests_Analysis.groovy
@@ -66,8 +66,9 @@ job(testConfiguration.jobName) {
shell('rm -rf PerfKitBenchmarker')
shell('rm -rf .env')
-// create new VirtualEnv, inherit already existing packages
-shell('virtualenv .env --system-site-packages')
+// create new VirtualEnv, inherit already existing packages. Explicitly
+// pin to python2.7 here otherwise python3 is used by default.
+shell('virtualenv .env --python=python2.7 --system-site-packages')
// update setuptools and pip
shell('.env/bin/pip install --upgrade setuptools pip')
[beam] branch master updated: Disable Py37 tox suite to reduce test flakiness.
This is an automated email from the ASF dual-hosted git repository.
markliu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 282df6e Disable Py37 tox suite to reduce test flakiness.
new 07320c3 Merge pull request #8292 from tvalentyn/disable_py37
282df6e is described below
commit 282df6e47ae74bed55bd0d59359a77132bd364f6
Author: Valentyn Tymofieiev
AuthorDate: Fri Apr 12 10:08:18 2019 -0700
Disable Py37 tox suite to reduce test flakiness.
---
build.gradle | 3 ++-
sdks/python/test-suites/tox/py36/build.gradle | 2 +-
sdks/python/test-suites/tox/py37/build.gradle | 2 +-
3 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/build.gradle b/build.gradle
index ce1eaed..df3b28c 100644
--- a/build.gradle
+++ b/build.gradle
@@ -188,7 +188,8 @@ task pythonPreCommit() {
dependsOn ":beam-sdks-python:preCommitPy2"
dependsOn ":beam-sdks-python-test-suites-tox-py35:preCommitPy35"
dependsOn ":beam-sdks-python-test-suites-tox-py36:preCommitPy36"
- dependsOn ":beam-sdks-python-test-suites-tox-py37:preCommitPy37"
+ // TODO(BEAM-7066): Enable once Python3.7-dev package is installed on all
Jenkins nodes.
+ // dependsOn ":beam-sdks-python-test-suites-tox-py37:preCommitPy37"
dependsOn ":beam-sdks-python-test-suites-dataflow:preCommitIT"
}
diff --git a/sdks/python/test-suites/tox/py36/build.gradle
b/sdks/python/test-suites/tox/py36/build.gradle
index 09a8764..1f07ae3 100644
--- a/sdks/python/test-suites/tox/py36/build.gradle
+++ b/sdks/python/test-suites/tox/py36/build.gradle
@@ -42,6 +42,6 @@ testPy36Cython.mustRunAfter testPython36, testPy36Gcp
task preCommitPy36() {
dependsOn "testPython36"
dependsOn "testPy36Gcp"
-// TODO: Activate when Jenkins workers are updated with python3.6-dev
+// TODO(BEAM-7066): Activate when Jenkins workers are updated with
python3.6-dev
// dependsOn "testPy36Cython"
}
diff --git a/sdks/python/test-suites/tox/py37/build.gradle
b/sdks/python/test-suites/tox/py37/build.gradle
index 7f19dc2..c63eb66 100644
--- a/sdks/python/test-suites/tox/py37/build.gradle
+++ b/sdks/python/test-suites/tox/py37/build.gradle
@@ -42,6 +42,6 @@ testPy37Cython.mustRunAfter testPython37, testPy37Gcp
task preCommitPy37() {
dependsOn "testPython37"
dependsOn "testPy37Gcp"
-// TODO: Activate when Jenkins workers are updated with python3.7-dev
+// TODO(BEAM-7066): Activate when Jenkins workers are updated with
python3.7-dev
// dependsOn "testPy37Cython"
}
[beam] branch master updated (8334eb5 -> e4cf63b)
This is an automated email from the ASF dual-hosted git repository.
markliu pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 8334eb5 Merge pull request #8283 from ibzib/options-exception
new 9cb9175 Add Python 3.6 and 3.7 test suites
new 92af51a Skip tests failing on Python 3.7
new e9d50a0 Deactivate Python 3.6 and 3.7 cython test suites.
new e4cf63b Merge pull request #8036 from RobbeSneyders/python-3.6-3.7
The 20947 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
build.gradle | 1 +
.../apache_beam/transforms/ptransform_test.py | 4 ++
.../typehints/native_type_compatibility_test.py| 6 +++
.../apache_beam/typehints/typed_pipeline_test.py | 10
.../python/apache_beam/typehints/typehints_test.py | 6 +++
sdks/python/test-suites/tox/py36/build.gradle | 15 +-
.../test-suites/tox/{py35 => py37}/build.gradle| 36 ++
sdks/python/tox.ini| 56 +-
settings.gradle| 2 +
9 files changed, 113 insertions(+), 23 deletions(-)
copy sdks/python/test-suites/tox/{py35 => py37}/build.gradle (63%)
[beam] branch master updated: [BEAM-6977] Python version is Configurable in Gradle
This is an automated email from the ASF dual-hosted git repository.
markliu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 1c02056 [BEAM-6977] Python version is Configurable in Gradle
new 0f99468 Merge pull request #8202 from
markflyhigh/gradle-config-py-version
1c02056 is described below
commit 1c0205672e473fee9e51c3d4c56a1f50a7667b8a
Author: Mark Liu
AuthorDate: Tue Apr 2 12:47:15 2019 -0700
[BEAM-6977] Python version is Configurable in Gradle
---
.../groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 13 +++--
sdks/python/test-suites/dataflow/py3/build.gradle | 2 +-
sdks/python/test-suites/direct/py3/build.gradle | 2 +-
sdks/python/test-suites/tox/py35/build.gradle | 2 +-
sdks/python/test-suites/tox/py36/build.gradle | 3 +++
5 files changed, 13 insertions(+), 9 deletions(-)
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 2d4a01c..d9f2546 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -1594,19 +1594,20 @@ class BeamModulePlugin implements Plugin {
project.ext.envdir =
"${project.rootProject.buildDir}/gradleenv/${project.name.hashCode()}"
def pythonRootDir = "${project.rootDir}/sdks/python"
- // This is current supported Python3 version. It should match the one in
- // sdks/python/container/py3/Dockerfile
- final PYTHON3_VERSION = '3.5'
+ // Python interpreter version for virtualenv setup and test run. This
value can be
+ // set from commandline with -PpythonVersion, or in build script of
certain project.
+ // If none of them applied, version set here will be used as default
value.
+ if(!project.hasProperty('pythonVersion')) {
+project.ext.pythonVersion = '2.7'
+ }
project.task('setupVirtualenv') {
doLast {
def virtualenvCmd = [
'virtualenv',
"${project.ext.envdir}",
+"--python=python${project.ext.pythonVersion}",
]
- if (project.hasProperty('python3')) {
-virtualenvCmd += '--python=python' + PYTHON3_VERSION
- }
project.exec { commandLine virtualenvCmd }
project.exec {
executable 'sh'
diff --git a/sdks/python/test-suites/dataflow/py3/build.gradle
b/sdks/python/test-suites/dataflow/py3/build.gradle
index 15e2a49..d83b01c 100644
--- a/sdks/python/test-suites/dataflow/py3/build.gradle
+++ b/sdks/python/test-suites/dataflow/py3/build.gradle
@@ -20,7 +20,7 @@ apply plugin: org.apache.beam.gradle.BeamModulePlugin
applyPythonNature()
// Required to setup a Python 3 virtualenv.
-project.ext.python3 = true
+pythonVersion = '3.5'
def runScriptsDir = "${project.rootDir}/sdks/python/scripts"
diff --git a/sdks/python/test-suites/direct/py3/build.gradle
b/sdks/python/test-suites/direct/py3/build.gradle
index a1321ee..c9e3ed8 100644
--- a/sdks/python/test-suites/direct/py3/build.gradle
+++ b/sdks/python/test-suites/direct/py3/build.gradle
@@ -20,7 +20,7 @@ plugins { id 'org.apache.beam.module' }
applyPythonNature()
// Required to setup a Python 3 virtualenv.
-project.ext.python3 = true
+pythonVersion = '3.5'
def runScriptsDir = "${project.rootDir}/sdks/python/scripts"
diff --git a/sdks/python/test-suites/tox/py35/build.gradle
b/sdks/python/test-suites/tox/py35/build.gradle
index 5f86fe9..ca3d37c 100644
--- a/sdks/python/test-suites/tox/py35/build.gradle
+++ b/sdks/python/test-suites/tox/py35/build.gradle
@@ -24,7 +24,7 @@ plugins { id 'org.apache.beam.module' }
applyPythonNature()
// Required to setup a Python 3 virtualenv.
-project.ext.python3 = true
+pythonVersion = '3.5'
task lint {}
check.dependsOn lint
diff --git a/sdks/python/test-suites/tox/py36/build.gradle
b/sdks/python/test-suites/tox/py36/build.gradle
index 8dc497d..c1615ef 100644
--- a/sdks/python/test-suites/tox/py36/build.gradle
+++ b/sdks/python/test-suites/tox/py36/build.gradle
@@ -23,6 +23,9 @@
plugins { id 'org.apache.beam.module' }
applyPythonNature()
+// Required to setup a Python 3 virtualenv.
+pythonVersion = '3.6'
+
toxTask "testPython36", "py36"
test.dependsOn testPython36
[beam] branch master updated: [BEAM-6619] [BEAM-6593] update gradle to include all py3 it tests
This is an automated email from the ASF dual-hosted git repository.
markliu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 4db0f94 [BEAM-6619] [BEAM-6593] update gradle to include all py3 it
tests
new 5521415 Merge pull request #8098 from Juta/it-tests
4db0f94 is described below
commit 4db0f94b7a65c796420ed6160ae0e2bd9c415d98
Author: Juta
AuthorDate: Wed Mar 20 14:06:40 2019 +0100
[BEAM-6619] [BEAM-6593] update gradle to include all py3 it tests
---
.../examples/complete/game/game_stats_it_test.py | 6 ++
.../complete/game/hourly_team_score_it_test.py | 6 ++
.../examples/complete/game/leader_board_it_test.py | 6 ++
sdks/python/apache_beam/examples/fastavro_it_test.py | 6 ++
.../io/gcp/big_query_query_to_table_it_test.py | 6 ++
.../apache_beam/io/gcp/bigquery_file_loads_test.py | 5 +
sdks/python/test-suites/dataflow/py3/build.gradle | 18 +-
sdks/python/test-suites/direct/py3/build.gradle| 16 +++-
8 files changed, 39 insertions(+), 30 deletions(-)
diff --git
a/sdks/python/apache_beam/examples/complete/game/game_stats_it_test.py
b/sdks/python/apache_beam/examples/complete/game/game_stats_it_test.py
index 67b80c0..e7b89aa 100644
--- a/sdks/python/apache_beam/examples/complete/game/game_stats_it_test.py
+++ b/sdks/python/apache_beam/examples/complete/game/game_stats_it_test.py
@@ -33,6 +33,8 @@ Usage:
from __future__ import absolute_import
import logging
+import os
+import sys
import time
import unittest
import uuid
@@ -49,6 +51,10 @@ from apache_beam.testing.pipeline_verifiers import
PipelineStateMatcher
from apache_beam.testing.test_pipeline import TestPipeline
+@unittest.skipIf(sys.version_info[0] == 3 and
+ os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
+ 'This test still needs to be fixed on Python 3'
+ 'TODO: BEAM-6711')
class GameStatsIT(unittest.TestCase):
# Input events containing user, team, score, processing time, window start.
diff --git
a/sdks/python/apache_beam/examples/complete/game/hourly_team_score_it_test.py
b/sdks/python/apache_beam/examples/complete/game/hourly_team_score_it_test.py
index 5685132..2fce1fc 100644
---
a/sdks/python/apache_beam/examples/complete/game/hourly_team_score_it_test.py
+++
b/sdks/python/apache_beam/examples/complete/game/hourly_team_score_it_test.py
@@ -33,6 +33,8 @@ Usage:
from __future__ import absolute_import
import logging
+import os
+import sys
import unittest
from hamcrest.core.core.allof import all_of
@@ -46,6 +48,10 @@ from apache_beam.testing.pipeline_verifiers import
PipelineStateMatcher
from apache_beam.testing.test_pipeline import TestPipeline
+@unittest.skipIf(sys.version_info[0] == 3 and
+ os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
+ 'This test still needs to be fixed on Python 3'
+ 'TODO: BEAM-6870')
class HourlyTeamScoreIT(unittest.TestCase):
DEFAULT_INPUT_FILE = 'gs://dataflow-samples/game/gaming_data*'
diff --git
a/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py
b/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py
index 27a170f..b86e49e 100644
--- a/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py
+++ b/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py
@@ -33,6 +33,8 @@ Usage:
from __future__ import absolute_import
import logging
+import os
+import sys
import time
import unittest
import uuid
@@ -50,6 +52,10 @@ from apache_beam.testing.pipeline_verifiers import
PipelineStateMatcher
from apache_beam.testing.test_pipeline import TestPipeline
+@unittest.skipIf(sys.version_info[0] == 3 and
+ os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
+ 'This test still needs to be fixed on Python 3'
+ 'TODO: BEAM-6711')
class LeaderBoardIT(unittest.TestCase):
# Input event containing user, team, score, processing time, window start.
diff --git a/sdks/python/apache_beam/examples/fastavro_it_test.py
b/sdks/python/apache_beam/examples/fastavro_it_test.py
index 327b10f..61833eb 100644
--- a/sdks/python/apache_beam/examples/fastavro_it_test.py
+++ b/sdks/python/apache_beam/examples/fastavro_it_test.py
@@ -46,6 +46,8 @@ from __future__ import absolute_import
from __future__ import division
import logging
+import os
+import sys
import unittest
import uuid
@@ -82,6 +84,10 @@ def record(i):
}
+@unittest.skipIf(sys.version_info[0] == 3 and
+ os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
+ 'This test still needs to be fixed on Python 3'
+ 'TODO: BEAM-6522')
class FastavroIT(unittest.TestCase):
SCHEMA = Parse('''
diff --git a/sdks/python/apache_beam/io/gcp
[beam] branch master updated: Update default test in run_integration_test.sh
This is an automated email from the ASF dual-hosted git repository. markliu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 9756d68 Update default test in run_integration_test.sh new 23d1e3a Merge pull request #7899 from markflyhigh/patch-4 9756d68 is described below commit 9756d687f55c855f56826e277cce2e361a8d4d69 Author: Mark Liu AuthorDate: Tue Feb 19 17:51:18 2019 -0800 Update default test in run_integration_test.sh Change default test to `apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it` if people run this script without other arguments or `gradlew :beam-sdks-python:integrationTest`. Use `--attr=IT` will result in executing all integration tests which may not be necessary for the default test set especially for people who are not familiar with this script and run first time. --- sdks/python/scripts/run_integration_test.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/sdks/python/scripts/run_integration_test.sh b/sdks/python/scripts/run_integration_test.sh index 33e8199..03e12b9 100755 --- a/sdks/python/scripts/run_integration_test.sh +++ b/sdks/python/scripts/run_integration_test.sh @@ -74,8 +74,9 @@ WORKER_JAR="" KMS_KEY_NAME="projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test" # Default test (nose) options. -# Default test sets are full integration tests. -TEST_OPTS="--attr=IT --nocapture" +# Run WordCountIT.test_wordcount_it by default if no test options are +# provided. +TEST_OPTS="--tests=apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it --nocapture" while [[ $# -gt 0 ]] do
