See
<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/3979/display/redirect?page=changes>
Changes:
[github] [BEAM-7709] Re-use node for explicit flattens
------------------------------------------
[...truncated 173.54 KB...]
return fun(*args, **kwargs)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py";,>
line 523, in create_job
self.create_job_description(job)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py";,>
line 553, in create_job_description
resources = self._stage_resources(job.options)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py";,>
line 483, in _stage_resources
staging_location=google_cloud_options.staging_location)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/runners/portability/stager.py";,>
line 168, in stage_job_resources
requirements_cache_path)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/utils/retry.py";,>
line 197, in wrapper
return fun(*args, **kwargs)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/runners/portability/stager.py";,>
line 487, in _populate_requirements_cache
processes.check_output(cmd_args, stderr=processes.STDOUT)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/utils/processes.py";,>
line 91, in check_output
.format(traceback.format_exc(), args[0][6], error.output))
RuntimeError: Full traceback: Traceback (most recent call last):
File
"<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/utils/processes.py";,>
line 83, in check_output
out = subprocess.check_output(*args, **kwargs)
File "/usr/lib/python3.7/subprocess.py", line 395, in check_output
**kwargs).stdout
File "/usr/lib/python3.7/subprocess.py", line 487, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command
'['<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967052/bin/python',>
'-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r',
'postcommit_requirements.txt', '--exists-action', 'i', '--no-binary', ':all:']'
returned non-zero exit status 1.
Pip install failed for package: -r
Output from execution of subprocess: b'Collecting pyhamcrest (from -r
postcommit_requirements.txt (line 1))\n File was already downloaded
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz\nCollecting mock (from
-r postcommit_requirements.txt (line 2))\n File was already downloaded
/tmp/dataflow-requirements-cache/mock-3.0.5.tar.gz\nCollecting setuptools (from
pyhamcrest->-r postcommit_requirements.txt (line 1))\n ERROR: Could not find a
version that satisfies the requirement setuptools (from pyhamcrest->-r
postcommit_requirements.txt (line 1)) (from versions: none)\nERROR: No matching
distribution found for setuptools (from pyhamcrest->-r
postcommit_requirements.txt (line 1))\n'
-------------------- >> begin captured logging << --------------------
root: DEBUG: Connecting using Google Application Default Credentials.
root: INFO: Starting GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0709205902-171178.1562705942.171333/pipeline.pb...
oauth2client.transport: INFO: Attempting refresh to obtain initial access_token
oauth2client.transport: INFO: Attempting refresh to obtain initial access_token
root: INFO: Completed GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0709205902-171178.1562705942.171333/pipeline.pb
in 0 seconds.
root: INFO: Starting GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0709205902-171178.1562705942.171333/requirements.txt...
root: INFO: Completed GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0709205902-171178.1562705942.171333/requirements.txt
in 0 seconds.
root: INFO: Executing command:
['<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967052/bin/python',>
'-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r',
'postcommit_requirements.txt', '--exists-action', 'i', '--no-binary', ':all:']
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML:
<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 15 tests in 1111.316s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:validatesRunnerStreamingTests
> FAILED
> Task :sdks:python:test-suites:dataflow:py35:validatesRunnerStreamingTests
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_13_59_33-5688856248601095108?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_07_30-12252046211350546304?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_13_59_34-17119004528939680008?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_09_00-4367009987649609586?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_13_59_34-10539548120002526508?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_09_46-6953962972397068802?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_13_59_34-13485218533522351322?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_13_59_34-4102840813890004389?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_08_00-6745519923552796559?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_13_59_34-15941775433069659424?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_08_34-11447756251399398580?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_13_59_34-7816759105828246635?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_08_05-16451101476217827005?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_13_59_34-5274644221282262753?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_08_05-11913624862564145500?project=apache-beam-testing.
test_dofn_lifecycle
(apache_beam.transforms.dofn_lifecycle_test.DoFnLifecycleTest) ... ok
test_par_do_with_multiple_outputs_and_using_yield
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
test_as_list_and_as_dict_side_inputs
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_as_dict_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ...
ok
test_undeclared_outputs (apache_beam.transforms.ptransform_test.PTransformTest)
... ok
test_flatten_multiple_pcollections_having_multiple_consumers
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
test_multiple_empty_outputs
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
test_par_do_with_multiple_outputs_and_using_return
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
test_as_singleton_with_different_defaults
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_default_value_singleton_side_input
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_as_list_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ...
ok
test_as_singleton_without_unique_labels
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_empty_singleton_side_input
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_flattened_side_input
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_iterable_side_input
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
----------------------------------------------------------------------
XML:
<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 15 tests in 1205.779s
OK
> Task :sdks:python:validatesRunnerBatchTests
test_multi_valued_singleton_side_input
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
----------------------------------------------------------------------
XML:
<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 17 tests in 1380.584s
OK
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_13_59_13-9484206276798489044?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_07_18-12573998385982198058?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_14_27-16260769045003761364?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_13_59_15-3006894188571267446?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_06_30-7319575328516724655?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_13_59_16-5226349842562382?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_07_15-10890105053408106024?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_13_59_16-2611837814756053367?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_07_36-10477795205920692683?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_13_59_16-3221714449324857448?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_08_21-4660611596149865668?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_13_59_16-10002385724646639694?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_07_12-2107643914816726757?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_13_59_17-12323249168017421543?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_07_12-7651512908825541786?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_13_59_16-10302200446605308512?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_07_21-7217322208260091593?project=apache-beam-testing.
> Task :sdks:python:validatesRunnerStreamingTests
>>> RUNNING integration tests with pipeline options:
>>> --runner=TestDataflowRunner --project=apache-beam-testing
>>> --staging_location=gs://temp-storage-for-end-to-end-tests/staging-it
>>> --temp_location=gs://temp-storage-for-end-to-end-tests/temp-it
>>> --output=gs://temp-storage-for-end-to-end-tests/py-it-cloud/output
>>> --sdk_location=build/apache-beam.tar.gz
>>> --requirements_file=postcommit_requirements.txt --num_workers=1
>>> --sleep_secs=20 --streaming
>>> --dataflow_worker_jar=<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/build/libs/beam-runners-google-cloud-dataflow-java-fn-api-worker-2.15.0-SNAPSHOT.jar>
>>>
>>> --kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test
>>>
>>> --dataflow_kms_key=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test
>>> test options: --nocapture --processes=8 --process-timeout=4500
>>> --attr=ValidatesRunner,!sickbay-streaming
<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/1922375555/local/lib/python2.7/site-packages/setuptools/dist.py>:472:
UserWarning: Normalizing '2.15.0.dev' to '2.15.0.dev0'
normalized_version,
running nosetests
running egg_info
writing requirements to apache_beam.egg-info/requires.txt
writing apache_beam.egg-info/PKG-INFO
writing top-level names to apache_beam.egg-info/top_level.txt
writing dependency_links to apache_beam.egg-info/dependency_links.txt
writing entry points to apache_beam.egg-info/entry_points.txt
reading manifest file 'apache_beam.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
warning: no files found matching 'LICENSE'
writing manifest file 'apache_beam.egg-info/SOURCES.txt'
test_par_do_with_multiple_outputs_and_using_return
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
test_multiple_empty_outputs
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
test_dofn_lifecycle
(apache_beam.transforms.dofn_lifecycle_test.DoFnLifecycleTest) ... ok
test_as_list_and_as_dict_side_inputs
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_as_dict_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ...
ok
test_flatten_multiple_pcollections_having_multiple_consumers
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
test_par_do_with_multiple_outputs_and_using_yield
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
test_undeclared_outputs (apache_beam.transforms.ptransform_test.PTransformTest)
... ok
test_as_list_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ...
ok
test_iterable_side_input
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_flattened_side_input
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_as_singleton_with_different_defaults
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_default_value_singleton_side_input
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_as_singleton_without_unique_labels
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_empty_singleton_side_input
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
----------------------------------------------------------------------
XML:
<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 15 tests in 1129.137s
OK
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_22_11-15237717511525928591?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_31_22-2723812818893878990?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_22_15-18372693749980355042?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_30_10-10918286342131334928?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_22_11-15425661954617864926?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_30_06-3436268688759575597?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_22_11-11144122487525762252?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_29_47-1151115643320769489?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_22_11-18312242731570182730?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_31_26-16058145869899658775?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_22_11-9079434441380865689?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_31_30-1867904262499379727?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_22_11-16093270703897871164?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_31_22-10398814733288179179?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-09_14_22_12-265821250337514338?project=apache-beam-testing.
FAILURE: Build completed with 5 failures.
1: Task failed with an exception.
-----------
* Where:
Build file
'<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/test-suites/dataflow/py36/build.gradle'>
line: 67
* What went wrong:
Execution failed for task
':sdks:python:test-suites:dataflow:py36:validatesRunnerBatchTests'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
2: Task failed with an exception.
-----------
* Where:
Build file
'<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/test-suites/dataflow/py35/build.gradle'>
line: 67
* What went wrong:
Execution failed for task
':sdks:python:test-suites:dataflow:py35:validatesRunnerBatchTests'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
3: Task failed with an exception.
-----------
* Where:
Build file
'<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/test-suites/dataflow/py36/build.gradle'>
line: 87
* What went wrong:
Execution failed for task
':sdks:python:test-suites:dataflow:py36:validatesRunnerStreamingTests'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
4: Task failed with an exception.
-----------
* Where:
Build file
'<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/test-suites/dataflow/py37/build.gradle'>
line: 97
* What went wrong:
Execution failed for task
':sdks:python:test-suites:dataflow:py37:validatesRunnerBatchTests'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
5: Task failed with an exception.
-----------
* Where:
Build file
'<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/test-suites/dataflow/py37/build.gradle'>
line: 117
* What went wrong:
Execution failed for task
':sdks:python:test-suites:dataflow:py37:validatesRunnerStreamingTests'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 42m 54s
76 actionable tasks: 59 executed, 17 from cache
Publishing build scan...
https://gradle.com/s/rxigmaz4ep65y
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
