See
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont_with_RC/754/display/redirect?page=changes>
Changes:
[noreply] Remove warning in dataframes io. (#29084)
------------------------------------------
[...truncated 2.71 MB...]
source.storageSource.bucket = os.path.join(gcs_bucket)
source.storageSource.object = gcs_object
build.source = source
# TODO(zyichi): make timeout configurable
build.timeout = '7200s'
now = time.time()
# operation = client.create_build(project_id=project_id, build=build)
request = cloudbuild.CloudbuildProjectsBuildsCreateRequest(
projectId=project_id, build=build)
build = self._cloudbuild_client.projects_builds.Create(request)
build_id, log_url = self._get_cloud_build_id_and_log_url(build.metadata)
_LOGGER.info(
'Building sdk container with Google Cloud Build, this may '
'take a few minutes, you may check build log at %s' % log_url)
# block until build finish, if build fails exception will be raised and
# stops the job submission.
response = self._cloudbuild_client.projects_builds.Get(
cloudbuild.CloudbuildProjectsBuildsGetRequest(
id=build_id, projectId=project_id))
while response.status in [cloudbuild.Build.StatusValueValuesEnum.QUEUED,
cloudbuild.Build.StatusValueValuesEnum.PENDING,
cloudbuild.Build.StatusValueValuesEnum.WORKING]:
time.sleep(10)
response = self._cloudbuild_client.projects_builds.Get(
cloudbuild.CloudbuildProjectsBuildsGetRequest(
id=build_id, projectId=project_id))
if response.status != cloudbuild.Build.StatusValueValuesEnum.SUCCESS:
> raise RuntimeError(
'Failed to build python sdk container image on google cloud build, '
'please check build log for error.')
E RuntimeError: Failed to build python sdk container image on google
cloud build, please check build log for error.
apache_beam/runners/portability/sdk_container_builder.py:299: RuntimeError
------------------------------ Captured log call -------------------------------
INFO apache_beam.runners.portability.stager:stager.py:322 Copying Beam SDK
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont_with_RC/ws/src/sdks/python/build/apache-beam.tar.gz";>
to staging location.
INFO
apache_beam.runners.portability.sdk_container_builder:sdk_container_builder.py:239
Compressed source files for building sdk container at
/tmp/tmpwck1t58z/source.tgz
INFO
apache_beam.runners.portability.sdk_container_builder:sdk_container_builder.py:313
Starting GCS upload to
gs://temp-storage-for-end-to-end-tests/temp-validatesrunner-test/source-9aeed439-47da-4e17-938e-965c838b128b.tgz...
INFO
apache_beam.runners.portability.sdk_container_builder:sdk_container_builder.py:332
Completed GCS upload to
gs://temp-storage-for-end-to-end-tests/temp-validatesrunner-test/source-9aeed439-47da-4e17-938e-965c838b128b.tgz.
INFO
apache_beam.runners.portability.sdk_container_builder:sdk_container_builder.py:281
Building sdk container with Google Cloud Build, this may take a few minutes,
you may check build log at
https://console.cloud.google.com/cloud-build/builds/709d4237-6ca7-4e31-a84e-091bea693cc2?project=844138762903
- generated xml file:
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont_with_RC/ws/src/sdks/python/pytest-beam_python3.8_sdk.xml>
-
=========================== short test summary info ============================
FAILED
apache_beam/examples/wordcount_it_test.py::WordCountIT::test_wordcount_it_with_prebuilt_sdk_container_cloud_build
- RuntimeError: Failed to build python sdk container image on google cloud
build, please check build log for error.
============= 1 failed, 2 passed, 17 skipped in 856.68s (0:14:16) ==============
cleanup_container
Untagged:
us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk:20231020-142858519027479
Untagged:
us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk@sha256:1623dced2bad1fd6490ee886cb98fc47e5a66df35319e1121bb0f08cad33479c
Untagged:
us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk:dc3a07bf-2370-4012-9e74-32442c6b28e3
Untagged:
us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk@sha256:4e7877f5dddac41451a439fd219f8a595c900d37de8f09937eb04d3b4d764881
Deleted: sha256:c8a789d0d87ebd8249a6f14f9812f9701a3d41cddbaffe8cb1e630afc07963b5
Deleted: sha256:06585e18718a1284ebe22399593fb647859d1c16c0272bfa27c7f3f96f094d20
Deleted: sha256:ff047a7f8569184a5a6cd751accb458c20613444fc5b79acd7d9a0c67e80dc0f
Deleted: sha256:9141480098f74b21707059c6215bc68783c0bc795207d67f8fbc1f9db0641eba
Deleted: sha256:f5cc2880b90e780317616b23798635d3c9b52c374ec8784f5af240a47843ed7d
Deleted: sha256:b16ed4dae2c8d8566b13ba0104c4b9b38744c6887926c631d08230d571d42a54
WARNING: Successfully resolved tag to sha256, but it is recommended to use
sha256 directly.
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk@sha256:1623dced2bad1fd6490ee886cb98fc47e5a66df35319e1121bb0f08cad33479c
Associated tags:
- 20231020-142858519027479
Tags:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk:20231020-142858519027479
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk:20231020-142858519027479].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk@sha256:1623dced2bad1fd6490ee886cb98fc47e5a66df35319e1121bb0f08cad33479c].
> Task :sdks:python:test-suites:dataflow:py39:validatesContainer
[gw0] [ 66%] PASSED
apache_beam/examples/wordcount_it_test.py::WordCountIT::test_wordcount_it_with_prebuilt_sdk_container_cloud_build
apache_beam/examples/wordcount_it_test.py::WordCountIT::test_wordcount_it_with_prebuilt_sdk_container_local_docker
> Task :sdks:python:test-suites:dataflow:py38:validatesContainer FAILED
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk@sha256:4e7877f5dddac41451a439fd219f8a595c900d37de8f09937eb04d3b4d764881
Associated tags:
- dc3a07bf-2370-4012-9e74-32442c6b28e3
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk:dc3a07bf-2370-4012-9e74-32442c6b28e3].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk@sha256:4e7877f5dddac41451a439fd219f8a595c900d37de8f09937eb04d3b4d764881].
Removed the container
> Task :sdks:python:test-suites:dataflow:py311:validatesContainer
[gw0] [100%] PASSED
apache_beam/examples/wordcount_it_test.py::WordCountIT::test_wordcount_it_with_prebuilt_sdk_container_local_docker
=============================== warnings summary ===============================
../../build/gradleenv/2050596099/lib/python3.11/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont_with_RC/ws/src/build/gradleenv/2050596099/lib/python3.11/site-packages/google/api_core/operations_v1/abstract_operations_client.py>:17:
DeprecationWarning: The distutils package is deprecated and slated for removal
in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils import util
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file:
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont_with_RC/ws/src/sdks/python/pytest-beam_python3.11_sdk.xml>
-
============ 3 passed, 17 skipped, 1 warning in 1265.61s (0:21:05) =============
echo ">>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST"
>>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST
cleanup_container
Untagged:
us.gcr.io/apache-beam-testing/jenkins/beam_python3.11_sdk:20231020-142752579316560
Untagged:
us.gcr.io/apache-beam-testing/jenkins/beam_python3.11_sdk@sha256:2214151bf1a905e4873eb94bc818399a7dd362da8570c063e96fdb2011ad7ed5
Untagged:
us.gcr.io/apache-beam-testing/jenkins/prebuild_python311_sdk/beam_python_prebuilt_sdk:5b951705-2c8b-47ac-937e-67f0ae4d9884
Untagged:
us.gcr.io/apache-beam-testing/jenkins/prebuild_python311_sdk/beam_python_prebuilt_sdk@sha256:30a2da6bc35caeed6d6fd378edbe0e75c9921e8ca851a60830db0298195bb426
Deleted: sha256:e2a8f6d8864fa23fd957eb559449b745528ebaf19853791557e84662945bcdc8
Deleted: sha256:9470377fc522df996dbb7519cbe5dca5d325585b3a88ca679d981e9978ace0c4
Deleted: sha256:78f80aeb268bca49fe4ea69b90ea032b43ac2626a25679fae471cd75994282f9
Deleted: sha256:55fb3fe1a5c4ec2762db56250a58d7a4649650fe44747b99bf3a4fb6eb2742d3
Deleted: sha256:d906e9ac10fa7c153e35a5c063a2ab63cf93f63a63fe804e55e48499c169694c
Deleted: sha256:9db927aebcdf9de81cc93e2f95e725e58dab034cb3070495182b1e6e54948651
WARNING: Successfully resolved tag to sha256, but it is recommended to use
sha256 directly.
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.11_sdk@sha256:2214151bf1a905e4873eb94bc818399a7dd362da8570c063e96fdb2011ad7ed5
Associated tags:
- 20231020-142752579316560
Tags:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.11_sdk:20231020-142752579316560
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.11_sdk:20231020-142752579316560].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.11_sdk@sha256:2214151bf1a905e4873eb94bc818399a7dd362da8570c063e96fdb2011ad7ed5].
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/prebuild_python311_sdk/beam_python_prebuilt_sdk@sha256:30a2da6bc35caeed6d6fd378edbe0e75c9921e8ca851a60830db0298195bb426
Associated tags:
- 5b951705-2c8b-47ac-937e-67f0ae4d9884
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python311_sdk/beam_python_prebuilt_sdk:5b951705-2c8b-47ac-937e-67f0ae4d9884].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python311_sdk/beam_python_prebuilt_sdk@sha256:30a2da6bc35caeed6d6fd378edbe0e75c9921e8ca851a60830db0298195bb426].
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/prebuild_python311_sdk/beam_python_prebuilt_sdk@sha256:fbaa01ab0b0cbd83a885eaa5f262ac7bab2521b9b3082e77d7f257ab36857756
Associated tags:
- 938dbdb7-4d30-4a6d-bf1b-d6a8e61fae10
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python311_sdk/beam_python_prebuilt_sdk:938dbdb7-4d30-4a6d-bf1b-d6a8e61fae10].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python311_sdk/beam_python_prebuilt_sdk@sha256:fbaa01ab0b0cbd83a885eaa5f262ac7bab2521b9b3082e77d7f257ab36857756].
Removed the container
> Task :sdks:python:test-suites:dataflow:py310:validatesContainer
[gw0] [100%] PASSED
apache_beam/examples/wordcount_it_test.py::WordCountIT::test_wordcount_it_with_prebuilt_sdk_container_local_docker
=============================== warnings summary ===============================
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont_with_RC/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py>:17:
DeprecationWarning: The distutils package is deprecated and slated for removal
in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils import util
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file:
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont_with_RC/ws/src/sdks/python/pytest-beam_python3.10_sdk.xml>
-
============ 3 passed, 17 skipped, 1 warning in 1265.35s (0:21:05) =============
>>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST
echo ">>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST"
cleanup_container
Untagged:
us.gcr.io/apache-beam-testing/jenkins/beam_python3.10_sdk:20231020-142819755684202
Untagged:
us.gcr.io/apache-beam-testing/jenkins/beam_python3.10_sdk@sha256:fdf1a91b9cea6f120697fc403bb837c5e031a5523f30c3aa01f056c38a61e1c8
Untagged:
us.gcr.io/apache-beam-testing/jenkins/prebuild_python310_sdk/beam_python_prebuilt_sdk:df3caa77-e0bc-4732-944f-c8c2ebea88d6
Untagged:
us.gcr.io/apache-beam-testing/jenkins/prebuild_python310_sdk/beam_python_prebuilt_sdk@sha256:a00514f0666c478b9194071d03f5e5931ff7dd19e649cdcd79b9fcc42a925bbc
Deleted: sha256:230e9f347e50df203713a406260a42cdfe634998f2fa12ddd4be89eaff7bc570
Deleted: sha256:9f21b66492a3f79139fdb1ba58525a224e998e5138560ac7065cf63b2bbeac37
Deleted: sha256:9a312dd7a46c0ca3a26356a723f07d45fc0f306145bd135d02c16d88073992e0
Deleted: sha256:c216c0d2870cf45c8f5f971225d0257cdcd1066a9d54a153b9b4dbf48be79324
Deleted: sha256:ff112303dfbfa7fd98072fbee18dc1eea72066a69e9d269a53aa1a02fccedd45
Deleted: sha256:5568a77446de40e38752d346b5f3691268057049c49f74d33d9eef83031f415d
WARNING: Successfully resolved tag to sha256, but it is recommended to use
sha256 directly.
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.10_sdk@sha256:fdf1a91b9cea6f120697fc403bb837c5e031a5523f30c3aa01f056c38a61e1c8
Associated tags:
- 20231020-142819755684202
Tags:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.10_sdk:20231020-142819755684202
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.10_sdk:20231020-142819755684202].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.10_sdk@sha256:fdf1a91b9cea6f120697fc403bb837c5e031a5523f30c3aa01f056c38a61e1c8].
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/prebuild_python310_sdk/beam_python_prebuilt_sdk@sha256:a00514f0666c478b9194071d03f5e5931ff7dd19e649cdcd79b9fcc42a925bbc
Associated tags:
- df3caa77-e0bc-4732-944f-c8c2ebea88d6
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python310_sdk/beam_python_prebuilt_sdk:df3caa77-e0bc-4732-944f-c8c2ebea88d6].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python310_sdk/beam_python_prebuilt_sdk@sha256:a00514f0666c478b9194071d03f5e5931ff7dd19e649cdcd79b9fcc42a925bbc].
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/prebuild_python310_sdk/beam_python_prebuilt_sdk@sha256:31d2ecdcf443b8f3942e982d36e352abe901a9d93ef58630a3a15f6f8e9de902
Associated tags:
- 5d1712ab-f2fb-4490-932b-9c84e5f2937c
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python310_sdk/beam_python_prebuilt_sdk:5d1712ab-f2fb-4490-932b-9c84e5f2937c].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python310_sdk/beam_python_prebuilt_sdk@sha256:31d2ecdcf443b8f3942e982d36e352abe901a9d93ef58630a3a15f6f8e9de902].
Removed the container
> Task :sdks:python:test-suites:dataflow:py39:validatesContainer
[gw0] [100%] PASSED
apache_beam/examples/wordcount_it_test.py::WordCountIT::test_wordcount_it_with_prebuilt_sdk_container_local_docker
- generated xml file:
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont_with_RC/ws/src/sdks/python/pytest-beam_python3.9_sdk.xml>
-
================== 3 passed, 17 skipped in 1273.90s (0:21:13) ==================
>>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST
echo ">>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST"
cleanup_container
Untagged:
us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk:20231020-142837669556450
Untagged:
us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk@sha256:029f0ed19e143de53f64395e06618270284b8238159b0c1dce327fd84ce0e377
Untagged:
us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk:bebc462d-1c39-42c1-9149-ebf2240d7e0d
Untagged:
us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk@sha256:c3d7fefaad31792884cc6387a6ba5709aba45cd7f2932ac575a84bb500aaf955
Deleted: sha256:ed7947f98f08b1ebf074a0ed06e597d35a0444556fc6499b6df679649445f503
Deleted: sha256:dfc8000a1bc4df23741233ad7f80f9f9a07dc31b23c833dc99791552c75d501b
Deleted: sha256:f754bec297e2f5f428468294cc79484e9ccd91eb550c5c423f6be03dd57dd625
Deleted: sha256:278e21c61e8495dd2d3d4102b85269505fcc75f71a3b860f9440ee9286cae2d5
Deleted: sha256:77cb787ae9dd0a9e517a1734bc2338fc8523cc5f9fc74a7f69eb6879a37317d4
Deleted: sha256:8989095aab929d1a46edf26e26b92b01d4546cba64cf82370185e7d750256eb1
WARNING: Successfully resolved tag to sha256, but it is recommended to use
sha256 directly.
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk@sha256:029f0ed19e143de53f64395e06618270284b8238159b0c1dce327fd84ce0e377
Associated tags:
- 20231020-142837669556450
Tags:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk:20231020-142837669556450
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk:20231020-142837669556450].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk@sha256:029f0ed19e143de53f64395e06618270284b8238159b0c1dce327fd84ce0e377].
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk@sha256:c3d7fefaad31792884cc6387a6ba5709aba45cd7f2932ac575a84bb500aaf955
Associated tags:
- bebc462d-1c39-42c1-9149-ebf2240d7e0d
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk:bebc462d-1c39-42c1-9149-ebf2240d7e0d].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk@sha256:c3d7fefaad31792884cc6387a6ba5709aba45cd7f2932ac575a84bb500aaf955].
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk@sha256:1eda5273b6930951c56883267337a3c3c02a1c73d2905e7b3812a5afb74ed09a
Associated tags:
- ac509c3f-56b8-4643-aaae-53011e49494d
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk:ac509c3f-56b8-4643-aaae-53011e49494d].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk@sha256:1eda5273b6930951c56883267337a3c3c02a1c73d2905e7b3812a5afb74ed09a].
Removed the container
FAILURE: Build failed with an exception.
* Where:
Script
'<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont_with_RC/ws/src/sdks/python/test-suites/dataflow/common.gradle'>
line: 374
* What went wrong:
Execution failed for task
':sdks:python:test-suites:dataflow:py38:validatesContainer'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings
and determine if they come from your own scripts or plugins.
For more on this, please refer to
https://docs.gradle.org/8.3/userguide/command_line_interface.html#sec:command_line_warnings
in the Gradle documentation.
BUILD FAILED in 38m 5s
60 actionable tasks: 54 executed, 1 from cache, 5 up-to-date
Publishing build scan...
https://ge.apache.org/s/f4cshkmwxe6ou
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
