See
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/8396/display/redirect?page=changes>
Changes:
[noreply] [BEAM-13096] Double test timeout. (#15774)
[noreply] [BEAM-13019] Add `containsInAnyOrder` with matchers to the
[Daniel Oliveira] Avoiding read-only Go module cache in Gradle config.
[noreply] [BEAM-11758] Update basics page: Pipeline, PCollection, PTransform
[noreply] Test SetState addIfAbsent with no read (#15776)
[noreply] lazy creation of source splits for export-based ReadFromBigQuery
[noreply] [BEAM-11275] Support remote package download from remote filesystems
in
------------------------------------------
[...truncated 111.18 KB...]
--docker_registry_push_url=$PREBUILD_SDK_CONTAINER_REGISTRY_PATH"
> Task :sdks:python:test-suites:dataflow:py36:validatesContainer
cleanup_container
docker images --format '{{.Repository}}:{{.Tag}}' | grep
$PREBUILD_SDK_CONTAINER_REGISTRY_PATH
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk@sha256:900aa976126936510ce7c6bfc888d43628d215f1e230b36a2d927251638736bf
Associated tags:
- 20211022-001329737056142
Tags:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20211022-001329737056142
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20211022-001329737056142].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk@sha256:900aa976126936510ce7c6bfc888d43628d215f1e230b36a2d927251638736bf].
gcloud container images list-tags
$PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk
--format="get(digest)"
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:87f332574b60f03ff16587aec332d8bc66dcbf17f3852f12969a9bf78f8a56e9
Associated tags:
- ea6780a3-a614-4d89-9cef-2227f5399fd0
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk:ea6780a3-a614-4d89-9cef-2227f5399fd0].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:87f332574b60f03ff16587aec332d8bc66dcbf17f3852f12969a9bf78f8a56e9].
> Task :sdks:python:test-suites:dataflow:py36:validatesContainer FAILED
> Task :sdks:python:test-suites:dataflow:py37:validatesContainer
[gw0] [100%] PASSED
apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py::ExerciseMetricsPipelineTest::test_metrics_fnapi_it
=================================== FAILURES ===================================
____ WordCountIT.test_wordcount_it_with_prebuilt_sdk_container_cloud_build _____
[gw0] linux -- Python 3.7.10
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/build/gradleenv/-1734967052/bin/python3.7>
self = <apache_beam.examples.wordcount_it_test.WordCountIT
testMethod=test_wordcount_it_with_prebuilt_sdk_container_cloud_build>
@pytest.mark.it_validatescontainer
def test_wordcount_it_with_prebuilt_sdk_container_cloud_build(self):
self._run_wordcount_it(
wordcount.run,
experiment='beam_fn_api',
> prebuild_sdk_container_engine='cloud_build')
apache_beam/examples/wordcount_it_test.py:67:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
apache_beam/examples/wordcount_it_test.py:109: in _run_wordcount_it
save_main_session=False,
apache_beam/examples/wordcount.py:89: in run
output | 'Write' >> WriteToText(known_args.output)
apache_beam/pipeline.py:596: in __exit__
self.result = self.run()
apache_beam/pipeline.py:549: in run
self._options).run(False)
apache_beam/pipeline.py:573: in run
return self.runner.run_pipeline(self, self._options)
apache_beam/runners/dataflow/test_dataflow_runner.py:53: in run_pipeline
self.result = super().run_pipeline(pipeline, options)
apache_beam/runners/dataflow/dataflow_runner.py:459: in run_pipeline
environments.DockerEnvironment.from_options(options))
apache_beam/transforms/environments.py:341: in from_options
options)
apache_beam/runners/portability/sdk_container_builder.py:139: in
build_container_image
return builder._build()
apache_beam/runners/portability/sdk_container_builder.py:90: in _build
self._invoke_docker_build_and_push(container_image_name)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self =
<apache_beam.runners.portability.sdk_container_builder._SdkContainerImageCloudBuilder
object at 0x7fbe638fe7d0>
container_image_name =
'us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk:44ad48a0-c041-4222-966a-88397b98b767'
def _invoke_docker_build_and_push(self, container_image_name):
project_id = self._google_cloud_options.project
temp_location = self._google_cloud_options.temp_location
# google cloud build service expects all the build source file to be
# compressed into a tarball.
tarball_path = os.path.join(self._temp_src_dir, '%s.tgz' % SOURCE_FOLDER)
self._make_tarfile(tarball_path, self._temp_src_dir)
_LOGGER.info(
"Compressed source files for building sdk container at %s" %
tarball_path)
container_image_tag = container_image_name.split(':')[-1]
gcs_location = os.path.join(
temp_location, '%s-%s.tgz' % (SOURCE_FOLDER, container_image_tag))
self._upload_to_gcs(tarball_path, gcs_location)
build = cloudbuild.Build()
build.steps = []
step = cloudbuild.BuildStep()
step.name = 'gcr.io/kaniko-project/executor:latest'
step.args = ['--destination=' + container_image_name, '--cache=true']
step.dir = SOURCE_FOLDER
build.steps.append(step)
source = cloudbuild.Source()
source.storageSource = cloudbuild.StorageSource()
gcs_bucket, gcs_object = self._get_gcs_bucket_and_name(gcs_location)
source.storageSource.bucket = os.path.join(gcs_bucket)
source.storageSource.object = gcs_object
build.source = source
# TODO(zyichi): make timeout configurable
build.timeout = '7200s'
now = time.time()
# operation = client.create_build(project_id=project_id, build=build)
request = cloudbuild.CloudbuildProjectsBuildsCreateRequest(
projectId=project_id, build=build)
build = self._cloudbuild_client.projects_builds.Create(request)
build_id, log_url = self._get_cloud_build_id_and_log_url(build.metadata)
_LOGGER.info(
'Building sdk container with Google Cloud Build, this may '
'take a few minutes, you may check build log at %s' % log_url)
# block until build finish, if build fails exception will be raised and
# stops the job submission.
response = self._cloudbuild_client.projects_builds.Get(
cloudbuild.CloudbuildProjectsBuildsGetRequest(
id=build_id, projectId=project_id))
while response.status in [cloudbuild.Build.StatusValueValuesEnum.QUEUED,
cloudbuild.Build.StatusValueValuesEnum.WORKING]:
time.sleep(10)
response = self._cloudbuild_client.projects_builds.Get(
cloudbuild.CloudbuildProjectsBuildsGetRequest(
id=build_id, projectId=project_id))
if response.status != cloudbuild.Build.StatusValueValuesEnum.SUCCESS:
raise RuntimeError(
> 'Failed to build python sdk container image on google cloud build, '
'please check build log for error.')
E RuntimeError: Failed to build python sdk container image on google
cloud build, please check build log for error.
apache_beam/runners/portability/sdk_container_builder.py:287: RuntimeError
=============================== warnings summary ===============================
apache_beam/io/filesystems_test.py:54
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:54:
DeprecationWarning: invalid escape sequence \c
self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf')) # pylint:
disable=anomalous-backslash-in-string
apache_beam/io/filesystems_test.py:62
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:62:
DeprecationWarning: invalid escape sequence \d
self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'), #
pylint: disable=anomalous-backslash-in-string
-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated xml file:
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/pytest-beam_python3.7_sdk.xml>
-
============== 1 failed, 3 passed, 2 warnings in 1341.78 seconds ===============
cleanup_container
Untagged:
us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk:20211022-001328753561021
Untagged:
us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk@sha256:99443c058d2f5c3df90083bb4d78f559a27a431c1cb5816e8536aa4d9beaa106
docker images --format '{{.Repository}}:{{.Tag}}' | grep
$PREBUILD_SDK_CONTAINER_REGISTRY_PATH
Untagged:
us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk:da0a9ad8-e33e-432d-a71f-2d92e0040909
Untagged:
us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk@sha256:0b46e58d94ed7cc623aacfddac061ea0cc30083a0da03596245e37312e26a9fa
Deleted: sha256:6d4faf24d42983c6e22c40d4db35ec68c8de1e9f54e0da677754953025249a92
Deleted: sha256:069cf5ff69366a3606975ee2f749cfda3bd92cc342978082a81e05f068e4b8b1
Deleted: sha256:a157d2fa3c89560d1c5b4ff982e24336a8ded5d9be78ff47f2438cc7af80b006
Deleted: sha256:72d13589db76dd2d30ab48c9f2bfe591bf492386a39eeb26d3e20eabb21e27ae
Deleted: sha256:9c76f7d037776d58f366831eb332de3995fab2bb372272d884163818c4697f32
Deleted: sha256:9c64ab95ac47b887b6e797a62b2a7ed5e83e6a96b2df68a0b72eae887e37b8a1
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk@sha256:99443c058d2f5c3df90083bb4d78f559a27a431c1cb5816e8536aa4d9beaa106
Associated tags:
- 20211022-001328753561021
Tags:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk:20211022-001328753561021
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk:20211022-001328753561021].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk@sha256:99443c058d2f5c3df90083bb4d78f559a27a431c1cb5816e8536aa4d9beaa106].
gcloud container images list-tags
$PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk
--format="get(digest)"
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk@sha256:0b46e58d94ed7cc623aacfddac061ea0cc30083a0da03596245e37312e26a9fa
Associated tags:
- da0a9ad8-e33e-432d-a71f-2d92e0040909
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk:da0a9ad8-e33e-432d-a71f-2d92e0040909].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk@sha256:0b46e58d94ed7cc623aacfddac061ea0cc30083a0da03596245e37312e26a9fa].
Removed the container
> Task :sdks:python:test-suites:dataflow:py37:validatesContainer FAILED
> Task :sdks:python:test-suites:dataflow:py38:validatesContainer
cleanup_container
docker images --format '{{.Repository}}:{{.Tag}}' | grep
$PREBUILD_SDK_CONTAINER_REGISTRY_PATH
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk@sha256:9f8c48d06ac80a8ca6d152d829f3aedb7fbd9571a36ff99262bae9612784ded9
Associated tags:
- 20211022-001341063789993
Tags:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk:20211022-001341063789993
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk:20211022-001341063789993].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk@sha256:9f8c48d06ac80a8ca6d152d829f3aedb7fbd9571a36ff99262bae9612784ded9].
gcloud container images list-tags
$PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk
--format="get(digest)"
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk@sha256:53421ed0c7ec4e9fead21765ac2e5d92d1e7283da27c1a11e6b55dc9e4183d19
Associated tags:
- 88ab6e6e-2396-4228-bf44-80d60f59ffd3
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk:88ab6e6e-2396-4228-bf44-80d60f59ffd3].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk@sha256:53421ed0c7ec4e9fead21765ac2e5d92d1e7283da27c1a11e6b55dc9e4183d19].
> Task :sdks:python:test-suites:dataflow:py38:validatesContainer FAILED
FAILURE: Build completed with 3 failures.
1: Task failed with an exception.
-----------
* Where:
Script
'<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/test-suites/dataflow/common.gradle'>
line: 276
* What went wrong:
Execution failed for task
':sdks:python:test-suites:dataflow:py36:validatesContainer'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
2: Task failed with an exception.
-----------
* Where:
Script
'<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/test-suites/dataflow/common.gradle'>
line: 276
* What went wrong:
Execution failed for task
':sdks:python:test-suites:dataflow:py37:validatesContainer'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
3: Task failed with an exception.
-----------
* Where:
Script
'<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/test-suites/dataflow/common.gradle'>
line: 276
* What went wrong:
Execution failed for task
':sdks:python:test-suites:dataflow:py38:validatesContainer'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 37m 59s
44 actionable tasks: 40 executed, 4 up-to-date
Publishing build scan...
https://gradle.com/s/ypy4untzz3euy
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
