See
<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/6273/display/redirect?page=changes>
Changes:
[robertwb] Conditionally cleanup tempdir.
[mxm] [BEAM-6661] Properly close channel for external environment
[mxm] [BEAM-6661] Avoid FileNotFoundException when no files have been staged
[mxm] [BEAM-6661] Avoid printing empty lines in subprocess job server
[robertwb] Use logging to reduce verbosity of license checking when all goes
well.
[github] Speedup on FnApiRunner start up time (#11543)
[github] All Python DirectRunners call setup/teardown on DoFns (#11547)
[github] [BEAM-9815] Correct go integration test if clauses. (#11524)
[relax] Support schemas in state API
[github] [BEAM-7923] Support docker screendiff integration tests (#11507)
------------------------------------------
[...truncated 737.40 KB...]
copying apache_beam/utils/retry.py -> apache-beam-2.22.0.dev0/apache_beam/utils
copying apache_beam/utils/retry_test.py ->
apache-beam-2.22.0.dev0/apache_beam/utils
copying apache_beam/utils/subprocess_server.py ->
apache-beam-2.22.0.dev0/apache_beam/utils
copying apache_beam/utils/subprocess_server_test.py ->
apache-beam-2.22.0.dev0/apache_beam/utils
copying apache_beam/utils/thread_pool_executor.py ->
apache-beam-2.22.0.dev0/apache_beam/utils
copying apache_beam/utils/thread_pool_executor_test.py ->
apache-beam-2.22.0.dev0/apache_beam/utils
copying apache_beam/utils/timestamp.py ->
apache-beam-2.22.0.dev0/apache_beam/utils
copying apache_beam/utils/timestamp_test.py ->
apache-beam-2.22.0.dev0/apache_beam/utils
copying apache_beam/utils/urns.py -> apache-beam-2.22.0.dev0/apache_beam/utils
copying apache_beam/utils/windowed_value.pxd ->
apache-beam-2.22.0.dev0/apache_beam/utils
copying apache_beam/utils/windowed_value.py ->
apache-beam-2.22.0.dev0/apache_beam/utils
copying apache_beam/utils/windowed_value_test.py ->
apache-beam-2.22.0.dev0/apache_beam/utils
Writing apache-beam-2.22.0.dev0/setup.cfg
creating dist
Creating tar archive
removing 'apache-beam-2.22.0.dev0' (and everything under it)
SDK_LOCATION=$(find dist/apache-beam-*.tar.gz)
find dist/apache-beam-*.tar.gz
# Run ValidatesRunner tests on Google Cloud Dataflow service
echo ">>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST"
>>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST
python setup.py nosetests \
--attr ValidatesContainer \
--nologcapture \
--processes=1 \
--process-timeout=900 \
--with-xunitmp \
--xunitmp-file=$XUNIT_FILE \
--ignore-files '.*py3\d?\.py$' \
--test-pipeline-options=" \
--runner=TestDataflowRunner \
--project=$PROJECT \
--region=$REGION \
--worker_harness_container_image=$CONTAINER:$TAG \
--staging_location=$GCS_LOCATION/staging-validatesrunner-test \
--temp_location=$GCS_LOCATION/temp-validatesrunner-test \
--output=$GCS_LOCATION/output \
--sdk_location=$SDK_LOCATION \
--num_workers=1"
setup.py:262: UserWarning: You are using Apache Beam with Python 2. New
releases of Apache Beam will soon support Python 3 only.
'You are using Apache Beam with Python 2. '
<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/setuptools/dist.py>:476:
UserWarning: Normalizing '2.22.0.dev' to '2.22.0.dev0'
normalized_version,
running nosetests
running egg_info
<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/gen_protos.py>:241:
UserWarning: Installing grpcio-tools is recommended for development.
warnings.warn('Installing grpcio-tools is recommended for development.')
INFO:gen_protos:Skipping proto regeneration: all files up to date
writing requirements to apache_beam.egg-info/requires.txt
writing apache_beam.egg-info/PKG-INFO
writing top-level names to apache_beam.egg-info/top_level.txt
writing dependency_links to apache_beam.egg-info/dependency_links.txt
writing entry points to apache_beam.egg-info/entry_points.txt
reading manifest file 'apache_beam.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
warning: no files found matching 'LICENSE'
writing manifest file 'apache_beam.egg-info/SOURCES.txt'
<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/__init__.py>:82:
UserWarning: You are using Apache Beam with Python 2. New releases of Apache
Beam will soon support Python 3 only.
'You are using Apache Beam with Python 2. '
WARNING:root:python-snappy is not installed; some tests will be skipped.
WARNING:root:Tensorflow is not installed, so skipping some tests.
WARNING:apache_beam.runners.interactive.interactive_environment:Interactive
Beam requires Python 3.5.3+.
WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies
required for Interactive Beam PCollection visualization are not available,
please use: `pip install apache-beam[interactive]` to install necessary
dependencies to enable all data visualization features.
WARNING:apache_beam.runners.interactive.interactive_environment:You cannot use
Interactive Beam features when you are not in an interactive environment such
as a Jupyter notebook or ipython terminal.
WARNING:root:Make sure that locally built Python SDK docker image has Python
2.7 interpreter.
<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/transforms/trigger_test.py>:575:
YAMLLoadWarning: calling yaml.load_all() without Loader=... is deprecated, as
the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full
details.
for spec in yaml.load_all(open(transcript_filename)):
WARNING:root:Make sure that locally built Python SDK docker image has Python
2.7 interpreter.
test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT)
... ERROR
test_metrics_fnapi_it
(apache_beam.runners.dataflow.dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest)
... ERROR
======================================================================
ERROR: test_wordcount_fnapi_it
(apache_beam.examples.wordcount_it_test.WordCountIT)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py";,>
line 812, in run
test(orig)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/nose/case.py";,>
line 45, in __call__
return self.run(*arg, **kwarg)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/nose/case.py";,>
line 133, in run
self.runTest(result)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/nose/case.py";,>
line 151, in runTest
test(result)
File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__
return self.run(*args, **kwds)
File "/usr/lib/python2.7/unittest/case.py", line 329, in run
testMethod()
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/examples/wordcount_it_test.py";,>
line 54, in test_wordcount_fnapi_it
self._run_wordcount_it(wordcount.run, experiment='beam_fn_api')
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/examples/wordcount_it_test.py";,>
line 91, in _run_wordcount_it
save_main_session=False)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/examples/wordcount.py";,>
line 121, in run
result = p.run()
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pipeline.py";,>
line 510, in run
self._options).run(False)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pipeline.py";,>
line 523, in run
return self.runner.run_pipeline(self, self._options)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py";,>
line 75, in run_pipeline
self.result.cancel()
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py";,>
line 1662, in cancel
self.job_id(), 'JOB_STATE_CANCELLED'):
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/utils/retry.py";,>
line 236, in wrapper
return fun(*args, **kwargs)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py";,>
line 777, in modify_job_state
self._client.projects_locations_jobs.Update(request)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py";,>
line 796, in Update
config, request, global_params=global_params)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/apitools/base/py/base_api.py";,>
line 729, in _RunMethod
http, http_request, **opts)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py";,>
line 356, in MakeRequest
max_retry_wait, total_wait_sec))
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py";,>
line 310, in HandleExceptionsAndRebuildHttpConnections
retry_args.num_retries, max_wait=retry_args.max_retry_wait))
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py";,>
line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_wordcount_fnapi_it
(apache_beam.examples.wordcount_it_test.WordCountIT)'
======================================================================
ERROR: test_metrics_fnapi_it
(apache_beam.runners.dataflow.dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py";,>
line 812, in run
test(orig)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/nose/case.py";,>
line 45, in __call__
return self.run(*arg, **kwarg)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/nose/case.py";,>
line 133, in run
self.runTest(result)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/nose/case.py";,>
line 151, in runTest
test(result)
File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__
return self.run(*args, **kwds)
File "/usr/lib/python2.7/unittest/case.py", line 329, in run
testMethod()
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py";,>
line 57, in test_metrics_fnapi_it
result = self.run_pipeline(experiment='beam_fn_api')
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py";,>
line 45, in run_pipeline
return dataflow_exercise_metrics_pipeline.apply_and_run(p)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline.py";,>
line 167, in apply_and_run
| 'm_out' >> beam.FlatMap(
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/transforms/ptransform.py";,>
line 562, in __ror__
result = p.apply(self, pvalueish, label)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pipeline.py";,>
line 643, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py";,>
line 151, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/runner.py";,>
line 198, in apply
return m(transform, input, options)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/runner.py";,>
line 228, in apply_PTransform
return transform.expand(input)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/transforms/core.py";,>
line 2716, in expand
| Map(coder.decode).with_output_types(self.get_output_type()))
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pvalue.py";,>
line 140, in __or__
return self.pipeline.apply(ptransform, self)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pipeline.py";,>
line 643, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py";,>
line 151, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/runner.py";,>
line 198, in apply
return m(transform, input, options)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/runner.py";,>
line 228, in apply_PTransform
return transform.expand(input)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/transforms/core.py";,>
line 2707, in expand
return pcoll | Reshuffle()
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pvalue.py";,>
line 140, in __or__
return self.pipeline.apply(ptransform, self)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pipeline.py";,>
line 643, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py";,>
line 151, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/runner.py";,>
line 198, in apply
return m(transform, input, options)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/runner.py";,>
line 228, in apply_PTransform
return transform.expand(input)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/transforms/util.py";,>
line 717, in expand
KeyedT).with_output_types(T) # type: ignore[misc]
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pvalue.py";,>
line 140, in __or__
return self.pipeline.apply(ptransform, self)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pipeline.py";,>
line 590, in apply
transform.transform, pvalueish, label or transform.label)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pipeline.py";,>
line 600, in apply
return self.apply(transform, pvalueish)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pipeline.py";,>
line 639, in apply
type_options = self._options.view_as(TypeOptions)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/options/pipeline_options.py";,>
line 355, in view_as
view = cls(self._flags)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/options/pipeline_options.py";,>
line 197, in __init__
parser = _BeamArgumentParser()
File "/usr/lib/python2.7/argparse.py", line 1613, in __init__
help=_('show this help message and exit'))
File "/usr/lib/python2.7/argparse.py", line 1304, in add_argument
self._get_formatter()._format_args(action, None)
File "/usr/lib/python2.7/argparse.py", line 574, in _format_args
get_metavar = self._metavar_formatter(action, default_metavar)
File "/usr/lib/python2.7/argparse.py", line 557, in _metavar_formatter
def _metavar_formatter(self, action, default_metavar):
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py";,>
line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_metrics_fnapi_it
(apache_beam.runners.dataflow.dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest)'
----------------------------------------------------------------------
XML: nosetests-beam_python2.7_sdk.xml
----------------------------------------------------------------------
XML:
<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 2 tests in 904.377s
FAILED (errors=2)
cleanup_container
Untagged:
us.gcr.io/apache-beam-testing/jenkins/beam_python2.7_sdk:20200429-000821
Untagged:
us.gcr.io/apache-beam-testing/jenkins/beam_python2.7_sdk@sha256:bafd98f6fbb2376523a5ff4df0d521209136fc0071ebef3cb1dd77dac708c28d
Deleted: sha256:e79ddaef993f69bc7be196223fc7e5626629a3cefe2291fc1f66f5380786f504
Deleted: sha256:9e39eab86d063e9c5342eacdf8762d22cd30f580acc6c46bc3e0977becfcd75f
Deleted: sha256:3d9d676246bdb852ea84630ad159f9411925104ba0fd218bf0f945d3e58eae67
Deleted: sha256:04665bb42191e9dec0c2ad308eeeb3e782b337bff546f50d5315ae5be64b5235
Deleted: sha256:9dbfd1733787a6cdd3f9d41965cf270fd5b9c061e541f21d85213d11a88edd7a
Deleted: sha256:0f7346d069b65cdf0a9173e225b119ccf26956b6b2e252e1321a67f094209709
Deleted: sha256:9cccb86b99cabd95f2ce1a65a76bd577a0eee79ea55d7460b38d41cb37958797
Deleted: sha256:c9356091eda7a0c51869ebfd168e9e32348e60fe986ccb34d34cb7218162ed95
Deleted: sha256:cf90397724134387bef7b4b9208e7ad88d2cad78d12309e8782f4e80e6431cff
Deleted: sha256:e961e49680191093fae0516394e1738af342c27e9797009bf7f9eb15484610c0
Deleted: sha256:ef5a81b768fbea53b6a61d3a7d22dad7bdc91585c58c2a566a2d5c7d4be04a31
Deleted: sha256:dca49224421df7c99cbf515c0d4514176cc0c64e86ac5ce88d8fc807e89f6c94
Deleted: sha256:05fb897213028630d548325b81e023dfc523b6cea174bfc31527ef749a602ec5
Deleted: sha256:a56dccf862fa60813578d57950239a79b7928582b13947b70743722b7ca72c38
Deleted: sha256:52132966a290f47f2d3c877e78bd8e813583ceb6890ca2b3b1631ef48a32b65b
Deleted: sha256:8948859c731c5a4bba43558693d8e243bfb12035255feaaf40c056b784ba79de
Deleted: sha256:15afa1c8a46e3576ca8b6090e02d63e51f04d3c6059754f6419d0f15e3bd1ef4
Deleted: sha256:68e74dc0675ba1c2f60de62c03865f1bce232d5ca101454f812171cf7804be9e
Deleted: sha256:587a6091bb18b9d2beef41f4adfd03380722777560fc21f5689c1146f1147f2a
Deleted: sha256:2ffd646cfdfdcde0e26f04e9e782d1b0e75aea802e10f569e3bbc8bdc24c729e
Deleted: sha256:56448c50547f65dab38e49bf10a1abd9f09c2e258319b64fef64ba7c909dd6af
Deleted: sha256:b786696d71219e5edf7bbfe4c90e100f78ebc1a9281884f48aec1b91ddbd8edd
Deleted: sha256:c30fb1b751787e317e94803d4a8f909c431ffdb388c7cd368e1b4e96905f47a8
Deleted: sha256:1a4dc1b66a131d95b0d71f5b4f5ff48bbcba350d7b1b7aa0996e29cf62cf64b0
Deleted: sha256:194fdae67631424c7ce67bb432ed308ca13d7967cbe5c56a67d9088f91b8d5c1
Deleted: sha256:4cd4e14b479909098ff514a94490a7619a3d773301d72c4211dd13d143277684
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python2.7_sdk@sha256:bafd98f6fbb2376523a5ff4df0d521209136fc0071ebef3cb1dd77dac708c28d
Associated tags:
- 20200429-000821
Tags:
- us.gcr.io/apache-beam-testing/jenkins/beam_python2.7_sdk:20200429-000821
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python2.7_sdk:20200429-000821].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python2.7_sdk@sha256:bafd98f6fbb2376523a5ff4df0d521209136fc0071ebef3cb1dd77dac708c28d].
Removed the container
Build step 'Execute shell' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
