See
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/6870/display/redirect?page=changes>
Changes:
[milan.cermak] Adds unit tests for the parse_table_reference function
[noreply] Merge pull request #12918 from [BEAM-10910]: Validate the BigQuery
table
[noreply] [Minor] Typo/grammatical changes (#12849)
------------------------------------------
[...truncated 912.02 KB...]
copying apache_beam/transforms/cy_dataflow_distribution_counter.pxd ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/cy_dataflow_distribution_counter.pyx ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/dataflow_distribution_counter_test.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/deduplicate.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/deduplicate_test.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/display.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/display_test.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/dofn_lifecycle_test.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/environments.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/environments_test.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/external.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/external_it_test.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/external_java.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/external_test.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/external_test_py3.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/external_test_py37.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/periodicsequence.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/periodicsequence_test.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/ptransform.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/ptransform_test.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/py_dataflow_distribution_counter.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/sideinputs.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/sideinputs_test.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/sql.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/sql_test.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/stats.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/stats_test.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/timeutil.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/transforms_keyword_only_args_test_py3.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/trigger.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/trigger_test.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/userstate.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/userstate_test.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/util.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/util_test.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/validate_runner_xlang_test.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/window.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/window_test.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/transforms/write_ptransform_test.py ->
apache-beam-2.26.0.dev0/apache_beam/transforms
copying apache_beam/typehints/__init__.py ->
apache-beam-2.26.0.dev0/apache_beam/typehints
copying apache_beam/typehints/decorators.py ->
apache-beam-2.26.0.dev0/apache_beam/typehints
copying apache_beam/typehints/decorators_test.py ->
apache-beam-2.26.0.dev0/apache_beam/typehints
copying apache_beam/typehints/decorators_test_py3.py ->
apache-beam-2.26.0.dev0/apache_beam/typehints
copying apache_beam/typehints/native_type_compatibility.py ->
apache-beam-2.26.0.dev0/apache_beam/typehints
copying apache_beam/typehints/native_type_compatibility_test.py ->
apache-beam-2.26.0.dev0/apache_beam/typehints
copying apache_beam/typehints/opcodes.py ->
apache-beam-2.26.0.dev0/apache_beam/typehints
copying apache_beam/typehints/row_type.py ->
apache-beam-2.26.0.dev0/apache_beam/typehints
copying apache_beam/typehints/schemas.py ->
apache-beam-2.26.0.dev0/apache_beam/typehints
copying apache_beam/typehints/schemas_test.py ->
apache-beam-2.26.0.dev0/apache_beam/typehints
copying apache_beam/typehints/trivial_inference.py ->
apache-beam-2.26.0.dev0/apache_beam/typehints
copying apache_beam/typehints/trivial_inference_test.py ->
apache-beam-2.26.0.dev0/apache_beam/typehints
copying apache_beam/typehints/trivial_inference_test_py3.py ->
apache-beam-2.26.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typecheck.py ->
apache-beam-2.26.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typecheck_test_py3.py ->
apache-beam-2.26.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typed_pipeline_test.py ->
apache-beam-2.26.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typed_pipeline_test_py3.py ->
apache-beam-2.26.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typehints.py ->
apache-beam-2.26.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typehints_test.py ->
apache-beam-2.26.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typehints_test_py3.py ->
apache-beam-2.26.0.dev0/apache_beam/typehints
copying apache_beam/utils/__init__.py ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/annotations.py ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/annotations_test.py ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/counters.pxd ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/counters.py ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/counters_test.py ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/histogram.py ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/histogram_test.py ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/interactive_utils.py ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/interactive_utils_test.py ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/plugin.py -> apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/processes.py ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/processes_test.py ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/profiler.py ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/proto_utils.py ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/retry.py -> apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/retry_test.py ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/shared.py -> apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/shared_test.py ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/subprocess_server.py ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/subprocess_server_test.py ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/thread_pool_executor.py ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/thread_pool_executor_test.py ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/timestamp.py ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/timestamp_test.py ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/urns.py -> apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/windowed_value.pxd ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/windowed_value.py ->
apache-beam-2.26.0.dev0/apache_beam/utils
copying apache_beam/utils/windowed_value_test.py ->
apache-beam-2.26.0.dev0/apache_beam/utils
Writing apache-beam-2.26.0.dev0/setup.cfg
creating dist
Creating tar archive
removing 'apache-beam-2.26.0.dev0' (and everything under it)
SDK_LOCATION=$(find dist/apache-beam-*.tar.gz)
find dist/apache-beam-*.tar.gz
# Run ValidatesRunner tests on Google Cloud Dataflow service
echo ">>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST"
>>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST
python setup.py nosetests \
--attr ValidatesContainer \
--nologcapture \
--processes=1 \
--process-timeout=900 \
--with-xunitmp \
--xunitmp-file=$XUNIT_FILE \
--ignore-files '.*py3\d?\.py$' \
--test-pipeline-options=" \
--runner=TestDataflowRunner \
--project=$PROJECT \
--region=$REGION \
--worker_harness_container_image=$CONTAINER:$TAG \
--staging_location=$GCS_LOCATION/staging-validatesrunner-test \
--temp_location=$GCS_LOCATION/temp-validatesrunner-test \
--output=$GCS_LOCATION/output \
--sdk_location=$SDK_LOCATION \
--num_workers=1 \
--prebuild_sdk_container_base_image=$CONTAINER:$TAG \
--docker_registry_push_url=us.gcr.io/$PROJECT/$USER"
running nosetests
running egg_info
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python3.6/lib/python3.6/site-packages/setuptools/dist.py>:452:
UserWarning: Normalizing '2.26.0.dev' to '2.26.0.dev0'
warnings.warn(tmpl.format(**locals()))
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/gen_protos.py>:243:
UserWarning: Installing grpcio-tools is recommended for development.
warnings.warn('Installing grpcio-tools is recommended for development.')
INFO:gen_protos:Skipping proto regeneration: all files up to date
writing apache_beam.egg-info/PKG-INFO
writing dependency_links to apache_beam.egg-info/dependency_links.txt
writing entry points to apache_beam.egg-info/entry_points.txt
writing requirements to apache_beam.egg-info/requires.txt
writing top-level names to apache_beam.egg-info/top_level.txt
reading manifest file 'apache_beam.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
warning: no files found matching 'LICENSE'
writing manifest file 'apache_beam.egg-info/SOURCES.txt'
WARNING:root:python-snappy is not installed; some tests will be skipped.
WARNING:root:Tensorflow is not installed, so skipping some tests.
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/io/vcfio.py>:50:
UserWarning: VCF IO will support Python 3 after migration to Nucleus, see:
BEAM-5628.
"VCF IO will support Python 3 after migration to Nucleus, "
WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies
required for Interactive Beam PCollection visualization are not available,
please use: `pip install apache-beam[interactive]` to install necessary
dependencies to enable all data visualization features.
WARNING:apache_beam.runners.interactive.interactive_environment:You cannot use
Interactive Beam features when you are not in an interactive environment such
as a Jupyter notebook or ipython terminal.
WARNING:root:Make sure that locally built Python SDK docker image has Python
3.6 interpreter.
WARNING:root:Make sure that locally built Python SDK docker image has Python
3.6 interpreter.
WARNING:root:Make sure that locally built Python SDK docker image has Python
3.6 interpreter.
test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT)
... ok
test_wordcount_it_with_prebuilt_sdk_container
(apache_beam.examples.wordcount_it_test.WordCountIT) ... ERROR
test_metrics_fnapi_it
(apache_beam.runners.dataflow.dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest)
... ok
======================================================================
ERROR: test_wordcount_it_with_prebuilt_sdk_container
(apache_beam.examples.wordcount_it_test.WordCountIT)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/examples/wordcount_it_test.py";,>
line 64, in test_wordcount_it_with_prebuilt_sdk_container
prebuild_sdk_container_engine='local_docker')
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/examples/wordcount_it_test.py";,>
line 107, in _run_wordcount_it
save_main_session=False,
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/examples/wordcount.py";,>
line 94, in run
output | 'Write' >> WriteToText(known_args.output)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pipeline.py";,>
line 568, in __exit__
self.result = self.run()
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pipeline.py";,>
line 518, in run
allow_proto_holders=True).run(False)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pipeline.py";,>
line 547, in run
return self.runner.run_pipeline(self, self._options)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py";,>
line 57, in run_pipeline
self).run_pipeline(pipeline, options)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py";,>
line 486, in run_pipeline
environments.DockerEnvironment.from_options(options))
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/transforms/environments.py";,>
line 259, in from_options
options)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/portability/sdk_container_builder.py";,>
line 141, in build_container_image
return builder.build()
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/portability/sdk_container_builder.py";,>
line 90, in build
self.invoke_docker_build_and_push(container_image_name)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/portability/sdk_container_builder.py";,>
line 154, in invoke_docker_build_and_push
cwd=self._temp_src_dir)
File "/usr/lib/python3.6/subprocess.py", line 423, in run
with Popen(*popenargs, **kwargs) as process:
TypeError: __init__() got an unexpected keyword argument 'capture_output'
----------------------------------------------------------------------
XML: nosetests-beam_python3.6_sdk.xml
----------------------------------------------------------------------
XML:
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 3 tests in 583.086s
FAILED (errors=1)
cleanup_container
Untagged:
us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20200925-000027
Untagged:
us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk@sha256:c3286541068710b963b25b1f2802d30567b17c764f1f33cb59ecf05fc1d2b36f
Deleted: sha256:c349adc848a714714cf8d0e7c13433c329665d4967fc1102e334adf2d352ac72
Deleted: sha256:95d4fd2f255708658fa672447527692e1bb4220c37a1e92ea4fff25b42759992
Deleted: sha256:dbda88cd943167b83f33a771957e4319647375007a9950fc0cb033753f941f34
Deleted: sha256:e9360e629d4345a3ffb44eab25e85296f0943036c0ead5845a32bc4d5a7a745e
Deleted: sha256:30513a8553fee8eaa72af5db6a981272688d184ea5c4c7a29741478895ad9cba
Deleted: sha256:d5f562e716ed754261cf71d7a00e51dd944801662d16650b320b6710e8bb05a1
Deleted: sha256:c860415d9ce41bf1930730f931c27f6337d9830d99c86ec6b4d084abd8fd344d
Deleted: sha256:b3b2bc4f2706d46bea92dc5aaf0a9bf76eb37ebde968941423f8fad3e8ee98e1
Deleted: sha256:2b648b7db87d54c6d37167ddb0b684d6d7d181365c2180e691e63782b52067b1
Deleted: sha256:0e315bd7a6d52434dc680f14c7516f91ff66538bb1a716d344a91ea7c52f6ea6
Deleted: sha256:094a94b0de8d28a1ba356da2342b60416ccb8778e4f809e2baa79936ee12ac45
Deleted: sha256:42193da90a33c9be5743495082b6097045709aadd95f0a14040a129520f0f791
Deleted: sha256:ccda88c587480a5c127de2362cb953726964e530d7adc071f4295bbb031636f6
Deleted: sha256:acdf73eb4e4476f0184642a36975b7117349ee3ab23d2756c6424175fa9f16b2
Deleted: sha256:805adfb0ead27ea35ec304c02a4ddfb29d4f1303eededb7f653586ad12e97232
Deleted: sha256:78e822cbe1a88388ba83909c9d37466e8c2dff2275c082f51e8427d713d1bd10
Deleted: sha256:dab2f5a780523443f2cdf41d505345a2307abdd0e7606c16a250f8ca49ee70d6
Deleted: sha256:db33d95400da8c7356a82f3c42c6d84dfce4d1dcdb223c5cb35706dd3ec6fc20
Deleted: sha256:010e20d526672d4596974062e6618ce62c40b6c4ade9e7ef6ab6fb8960d078e5
Deleted: sha256:fd2992ffde7b26dfd24bfeecdb86ef7d712f05445813109e61eda3c139eea7c3
Deleted: sha256:1533a87fd1a75513e4bd8e4b53e7ee58ee00797fb905841dc0405db83a6fffde
Deleted: sha256:d2341e6d01e587b1b4a447f4f9cba8c3c59e6e118349069dba6ec1bfff728286
Deleted: sha256:1f9f9e7efc771b93b2cf37303a7d68fd44645aaa53a96eb3a8b31b8d222a7e5d
Deleted: sha256:106c1854c5ba2c7219e8fad427794338564187d8cc138aeeb636ef07c1aaa591
Deleted: sha256:80e669bf5ba72cb6a243a6d974106819075d362cee078283174a07e0e3cb12ed
Deleted: sha256:ea0b17ab22e01e3953ad79d5a161a401ee52930b4732a195f5e5bea97e184c47
Deleted: sha256:baa8e460108e50dd72875926841f1764d57c3f4d2c1f63e934bba3f79084dc73
Deleted: sha256:71b20aa3ab6cafcce9a85525164c276bb2d64cf494076542e34c4bdf49454674
Deleted: sha256:0ae0632d282a272013c3b64513edbf8d953cf9d6bd0f23a0e6f5c202f2dc2a6b
Deleted: sha256:8212908ae0c011f37549c0ac8e44ca9c5c965541cdfb30366782177af152c38f
Deleted: sha256:9ce9c832b51d3ee2a075dec4f2a6cf137a94445fe60331235178d69c4be98a21
Deleted: sha256:08629a696e19b14ce6f5bae2b06fe02657c715f25cf5f8de67eeac192bbca6bb
Deleted: sha256:e4ed0541c724ed2f475a724cbb94673211e61e1b4e4e920a8a8ad090831c4241
Deleted: sha256:b7a2fe4b4011e4c33f8aca577d515b328c5221171eaf88e2cd0aefe06ddc86c3
Deleted: sha256:93ebef446344b6b1446beab88686aef6f24ac9167e7b93b10f5c648586f41d65
Deleted: sha256:f2fd13c6b16f26461295c94e25d114b5929e4b450921d37b5973f83cac84a28a
Deleted: sha256:d94948e6453b61b32d3ef209fc507ee1fc50351d2f965cf01cceb0d58b0c5c12
Deleted: sha256:80779690a4c69f49b55faf3e96b219ebfc65d2b33d5749bb21fc62e0924d78a9
Deleted: sha256:da04de0d24a7f215507c518346d7b0898669e7b808afcc455c311808a5d2dd84
docker images --format '{{.Repository}}:{{.Tag}}' | grep 'prebuilt_sdk'
"docker rmi" requires at least 1 argument.
See 'docker rmi --help'.
Usage: docker rmi [OPTIONS] IMAGE [IMAGE...]
Remove one or more images
Failed to remove prebuilt sdk container
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk@sha256:c3286541068710b963b25b1f2802d30567b17c764f1f33cb59ecf05fc1d2b36f
Associated tags:
- 20200925-000027
Tags:
- us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20200925-000027
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20200925-000027].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk@sha256:c3286541068710b963b25b1f2802d30567b17c764f1f33cb59ecf05fc1d2b36f].
Removed the container
Build step 'Execute shell' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
