See
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/7466/display/redirect>
Changes:
------------------------------------------
[...truncated 1.09 MB...]
copying apache_beam/tools/utils.py -> apache-beam-2.29.0.dev0/apache_beam/tools
copying apache_beam/transforms/__init__.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/combinefn_lifecycle_pipeline.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/combinefn_lifecycle_test.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/combiners.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/combiners_test.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/core.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/create_source.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/create_test.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/cy_combiners.pxd ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/cy_combiners.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/cy_dataflow_distribution_counter.pxd ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/cy_dataflow_distribution_counter.pyx ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/dataflow_distribution_counter_test.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/deduplicate.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/deduplicate_test.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/display.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/display_test.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/dofn_lifecycle_test.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/environments.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/environments_test.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/external.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/external_it_test.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/external_java.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/external_test.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/external_test_py3.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/external_test_py37.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/periodicsequence.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/periodicsequence_test.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/ptransform.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/ptransform_test.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/py_dataflow_distribution_counter.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/sideinputs.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/sideinputs_test.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/sql.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/sql_test.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/stats.pxd ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/stats.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/stats_test.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/timeutil.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/transforms_keyword_only_args_test_py3.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/trigger.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/trigger_test.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/userstate.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/userstate_test.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/util.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/util_test.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/validate_runner_xlang_test.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/window.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/window_test.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/transforms/write_ptransform_test.py ->
apache-beam-2.29.0.dev0/apache_beam/transforms
copying apache_beam/typehints/__init__.py ->
apache-beam-2.29.0.dev0/apache_beam/typehints
copying apache_beam/typehints/decorators.py ->
apache-beam-2.29.0.dev0/apache_beam/typehints
copying apache_beam/typehints/decorators_test.py ->
apache-beam-2.29.0.dev0/apache_beam/typehints
copying apache_beam/typehints/decorators_test_py3.py ->
apache-beam-2.29.0.dev0/apache_beam/typehints
copying apache_beam/typehints/native_type_compatibility.py ->
apache-beam-2.29.0.dev0/apache_beam/typehints
copying apache_beam/typehints/native_type_compatibility_test.py ->
apache-beam-2.29.0.dev0/apache_beam/typehints
copying apache_beam/typehints/opcodes.py ->
apache-beam-2.29.0.dev0/apache_beam/typehints
copying apache_beam/typehints/row_type.py ->
apache-beam-2.29.0.dev0/apache_beam/typehints
copying apache_beam/typehints/schemas.py ->
apache-beam-2.29.0.dev0/apache_beam/typehints
copying apache_beam/typehints/schemas_test.py ->
apache-beam-2.29.0.dev0/apache_beam/typehints
copying apache_beam/typehints/sharded_key_type.py ->
apache-beam-2.29.0.dev0/apache_beam/typehints
copying apache_beam/typehints/sharded_key_type_test.py ->
apache-beam-2.29.0.dev0/apache_beam/typehints
copying apache_beam/typehints/trivial_inference.py ->
apache-beam-2.29.0.dev0/apache_beam/typehints
copying apache_beam/typehints/trivial_inference_test.py ->
apache-beam-2.29.0.dev0/apache_beam/typehints
copying apache_beam/typehints/trivial_inference_test_py3.py ->
apache-beam-2.29.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typecheck.py ->
apache-beam-2.29.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typecheck_test_py3.py ->
apache-beam-2.29.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typed_pipeline_test.py ->
apache-beam-2.29.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typed_pipeline_test_py3.py ->
apache-beam-2.29.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typehints.py ->
apache-beam-2.29.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typehints_test.py ->
apache-beam-2.29.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typehints_test_py3.py ->
apache-beam-2.29.0.dev0/apache_beam/typehints
copying apache_beam/utils/__init__.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/annotations.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/annotations_test.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/counters.pxd ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/counters.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/counters_test.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/histogram.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/histogram_test.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/interactive_utils.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/interactive_utils_test.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/plugin.py -> apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/processes.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/processes_test.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/profiler.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/profiler_test.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/proto_utils.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/retry.py -> apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/retry_test.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/sentinel.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/sharded_key.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/shared.py -> apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/shared_test.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/subprocess_server.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/subprocess_server_test.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/thread_pool_executor.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/thread_pool_executor_test.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/timestamp.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/timestamp_test.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/urns.py -> apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/windowed_value.pxd ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/windowed_value.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
copying apache_beam/utils/windowed_value_test.py ->
apache-beam-2.29.0.dev0/apache_beam/utils
Writing apache-beam-2.29.0.dev0/setup.cfg
creating dist
Creating tar archive
removing 'apache-beam-2.29.0.dev0' (and everything under it)
SDK_LOCATION=$(find dist/apache-beam-*.tar.gz)
find dist/apache-beam-*.tar.gz
# Run ValidatesRunner tests on Google Cloud Dataflow service
echo ">>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST"
>>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST
python setup.py nosetests \
--attr ValidatesContainer \
--nologcapture \
--processes=1 \
--process-timeout=900 \
--with-xunitmp \
--xunitmp-file=$XUNIT_FILE \
--ignore-files '.*py3\d?\.py$' \
--test-pipeline-options=" \
--runner=TestDataflowRunner \
--project=$PROJECT \
--region=$REGION \
--worker_harness_container_image=$CONTAINER:$TAG \
--staging_location=$GCS_LOCATION/staging-validatesrunner-test \
--temp_location=$GCS_LOCATION/temp-validatesrunner-test \
--output=$GCS_LOCATION/output \
--sdk_location=$SDK_LOCATION \
--num_workers=1 \
--prebuild_sdk_container_base_image=$CONTAINER:$TAG \
--docker_registry_push_url=us.gcr.io/$PROJECT/$USER"
running nosetests
running egg_info
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python3.6/lib/python3.6/site-packages/setuptools/dist.py>:461:
UserWarning: Normalizing '2.29.0.dev' to '2.29.0.dev0'
warnings.warn(tmpl.format(**locals()))
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/gen_protos.py>:243:
UserWarning: Installing grpcio-tools is recommended for development.
warnings.warn('Installing grpcio-tools is recommended for development.')
INFO:gen_protos:Skipping proto regeneration: all files up to date
writing apache_beam.egg-info/PKG-INFO
writing dependency_links to apache_beam.egg-info/dependency_links.txt
writing entry points to apache_beam.egg-info/entry_points.txt
writing requirements to apache_beam.egg-info/requires.txt
writing top-level names to apache_beam.egg-info/top_level.txt
reading manifest file 'apache_beam.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
warning: no files found matching 'LICENSE'
writing manifest file 'apache_beam.egg-info/SOURCES.txt'
WARNING:root:python-snappy is not installed; some tests will be skipped.
WARNING:root:Tensorflow is not installed, so skipping some tests.
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/io/vcfio.py>:50:
UserWarning: VCF IO will support Python 3 after migration to Nucleus, see:
BEAM-5628.
"VCF IO will support Python 3 after migration to Nucleus, "
WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies
required for Interactive Beam PCollection visualization are not available,
please use: `pip install apache-beam[interactive]` to install necessary
dependencies to enable all data visualization features.
WARNING:apache_beam.runners.interactive.interactive_environment:You cannot use
Interactive Beam features when you are not in an interactive environment such
as a Jupyter notebook or ipython terminal.
WARNING:root:Make sure that locally built Python SDK docker image has Python
3.6 interpreter.
WARNING:root:Make sure that locally built Python SDK docker image has Python
3.6 interpreter.
Sending build context to Docker daemon 1.738MB
Step 1/4 : FROM
us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20210221-060023
---> c09cde5d8b97
Step 2/4 : RUN mkdir -p /opt/apache/beam/artifacts
---> Running in 4ac3f87f7c86
Removing intermediate container 4ac3f87f7c86
---> a00afb786f67
Step 3/4 : COPY ./* /opt/apache/beam/artifacts/
---> 4a5d31dbcc71
Step 4/4 : RUN /opt/apache/beam/boot --setup_only --artifacts
/opt/apache/beam/artifacts/artifacts_info.json
---> Running in d5d1e01d4748
[91m2021/02/21 06:20:11 Installing setup packages ...
[0m[91m2021/02/21 06:20:11 Failed to install required packages: failed to
install SDK: package 'dataflow_python_sdk.tar' not found
[0mThe command '/bin/sh -c /opt/apache/beam/boot --setup_only --artifacts
/opt/apache/beam/artifacts/artifacts_info.json' returned a non-zero code: 1
WARNING:root:Make sure that locally built Python SDK docker image has Python
3.6 interpreter.
test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT)
... ok
test_wordcount_it_with_prebuilt_sdk_container
(apache_beam.examples.wordcount_it_test.WordCountIT) ... ERROR
test_metrics_fnapi_it
(apache_beam.runners.dataflow.dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest)
... ok
======================================================================
ERROR: test_wordcount_it_with_prebuilt_sdk_container
(apache_beam.examples.wordcount_it_test.WordCountIT)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/portability/sdk_container_builder.py";,>
line 173, in _invoke_docker_build_and_push
cwd=self._temp_src_dir)
File "/usr/lib/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['docker', 'build', '.', '-t',
'us.gcr.io/apache-beam-testing/jenkins/beam_python_prebuilt_sdk:e089dad8-561a-449d-8ce7-7210a1edd37d']'
returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/examples/wordcount_it_test.py";,>
line 64, in test_wordcount_it_with_prebuilt_sdk_container
prebuild_sdk_container_engine='local_docker')
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/examples/wordcount_it_test.py";,>
line 107, in _run_wordcount_it
save_main_session=False,
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/examples/wordcount.py";,>
line 94, in run
output | 'Write' >> WriteToText(known_args.output)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pipeline.py";,>
line 580, in __exit__
self.result = self.run()
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pipeline.py";,>
line 530, in run
self._options).run(False)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pipeline.py";,>
line 559, in run
return self.runner.run_pipeline(self, self._options)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py";,>
line 57, in run_pipeline
self).run_pipeline(pipeline, options)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py";,>
line 497, in run_pipeline
environments.DockerEnvironment.from_options(options))
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/transforms/environments.py";,>
line 285, in from_options
options)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/portability/sdk_container_builder.py";,>
line 140, in build_container_image
return builder._build()
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/portability/sdk_container_builder.py";,>
line 93, in _build
self._invoke_docker_build_and_push(container_image_name)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/portability/sdk_container_builder.py";,>
line 177, in _invoke_docker_build_and_push
'stderr:\n %s.' % err.stderr)
RuntimeError: Failed to build sdk container with local docker, stderr:
None.
----------------------------------------------------------------------
XML: nosetests-beam_python3.6_sdk.xml
----------------------------------------------------------------------
XML:
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 3 tests in 570.684s
FAILED (errors=1)
cleanup_container
Untagged:
us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20210221-060023
Untagged:
us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk@sha256:be6ccf05acafe34d6fe61432e7d7c732f99c9f8ce45d2f58661938c21bea0781
docker images --format '{{.Repository}}:{{.Tag}}' | grep 'prebuilt_sdk'
"docker rmi" requires at least 1 argument.
See 'docker rmi --help'.
Usage: docker rmi [OPTIONS] IMAGE [IMAGE...]
Remove one or more images
Failed to remove prebuilt sdk container
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk@sha256:be6ccf05acafe34d6fe61432e7d7c732f99c9f8ce45d2f58661938c21bea0781
Associated tags:
- 20210221-060023
Tags:
- us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20210221-060023
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20210221-060023].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk@sha256:be6ccf05acafe34d6fe61432e7d7c732f99c9f8ce45d2f58661938c21bea0781].
Removed the container
Build step 'Execute shell' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
