[
https://issues.apache.org/jira/browse/BEAM-5957?focusedWorklogId=162814&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-162814
]
ASF GitHub Bot logged work on BEAM-5957:
----------------------------------------
Author: ASF GitHub Bot
Created on: 06/Nov/18 00:40
Start Date: 06/Nov/18 00:40
Worklog Time Spent: 10m
Work Description: aaltay closed pull request #6938: [BEAM-5957] Fix
beam_PerformanceTests_Python Jenkins and Gradle configs
URL: https://github.com/apache/beam/pull/6938
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/.test-infra/jenkins/job_PerformanceTests_Python.groovy
b/.test-infra/jenkins/job_PerformanceTests_Python.groovy
index 7a856004190..40944dcb378 100644
--- a/.test-infra/jenkins/job_PerformanceTests_Python.groovy
+++ b/.test-infra/jenkins/job_PerformanceTests_Python.groovy
@@ -53,6 +53,8 @@ job('beam_PerformanceTests_Python'){
beam_it_class :
'apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it',
beam_prebuilt : 'true', // skip beam prebuild
beam_python_sdk_location : 'build/apache-beam.tar.gz',
+ beam_runner : 'TestDataflowRunner',
+ beam_it_timeout : '1200',
beam_it_args : pipelineArgsJoined,
]
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index b43edf66e28..147a3646604 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -217,11 +217,16 @@ def basicTestOpts = [
"--process-timeout=4500", // timeout of whole command execution
]
-def mapToArgString(argMap) {
+static def mapToArgString(argMap) {
def argList = []
argMap.each { k, v ->
if (v in List) {
v = "\"${v.join(' ')}\""
+ } else if (v in String && v.contains(' ')) {
+ // We should use double quote around the arg value if it contains series
+ // of flags joined with space. Otherwise, commandline parsing of the
+ // shell script will be broken.
+ v = "\"${v.replace('"', '')}\""
}
argList.add("--$k $v")
}
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 162814)
Time Spent: 0.5h (was: 20m)
> [beam_PerformanceTests_Python] Consistently failing due to failing
> integration test.
> ------------------------------------------------------------------------------------
>
> Key: BEAM-5957
> URL: https://issues.apache.org/jira/browse/BEAM-5957
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
> Reporter: Daniel Oliveira
> Assignee: Mark Liu
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Most recent failure:
> [https://builds.apache.org/job/beam_PerformanceTests_Python/1634/]
> I haven't been able to debug the root cause of the failure with what I know.
> From what I can tell the performance test fails because one of the
> integration tests it's benchmarking has failed. But I can't tell what
> integration test is causing the failure.
> Relevant parts of the logs:
> {noformat}
> 11:19:01 BUILD FAILED in 36s
> 11:19:01
> 11:19:01 2018-11-02 18:19:01,413 3c18295b MainThread
> beam_integration_benchmark(1/1) ERROR Error during benchmark
> beam_integration_benchmark
> 11:19:01 Traceback (most recent call last):
> 11:19:01 File
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
> line 719, in RunBenchmark
> 11:19:01 DoRunPhase(spec, collector, detailed_timer)
> 11:19:01 File
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
> line 580, in DoRunPhase
> 11:19:01 samples = spec.BenchmarkRun(spec)
> 11:19:01 File
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py",
> line 160, in Run
> 11:19:01 job_type=job_type)
> 11:19:01 File
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",
> line 90, in SubmitJob
> 11:19:01 assert retcode == 0, "Integration Test Failed."
> 11:19:01 AssertionError: Integration Test Failed.
> 11:19:01 2018-11-02 18:19:01,414 3c18295b MainThread
> beam_integration_benchmark(1/1) INFO Cleaning up benchmark
> beam_integration_benchmark
> 11:19:01 2018-11-02 18:19:01,416 3c18295b MainThread
> beam_integration_benchmark(1/1) ERROR Exception running benchmark
> 11:19:01 Traceback (most recent call last):
> 11:19:01 File
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
> line 860, in RunBenchmarkTask
> 11:19:01 RunBenchmark(spec, collector)
> 11:19:01 File
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
> line 719, in RunBenchmark
> 11:19:01 DoRunPhase(spec, collector, detailed_timer)
> 11:19:01 File
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
> line 580, in DoRunPhase
> 11:19:01 samples = spec.BenchmarkRun(spec)
> 11:19:01 File
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py",
> line 160, in Run
> 11:19:01 job_type=job_type)
> 11:19:01 File
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",
> line 90, in SubmitJob
> 11:19:01 assert retcode == 0, "Integration Test Failed."
> 11:19:01 AssertionError: Integration Test Failed.
> 11:19:01 2018-11-02 18:19:01,416 3c18295b MainThread
> beam_integration_benchmark(1/1) ERROR Benchmark 1/1
> beam_integration_benchmark (UID: beam_integration_benchmark0) failed.
> Execution will continue.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
