See <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/9516/display/redirect>
Changes: ------------------------------------------ [...truncated 1.28 MB...] WARNING: `gcloud docker` will not be supported for Docker client versions above 18.03. As an alternative, use `gcloud auth configure-docker` to configure `docker` to use `gcloud` as a credential helper, then use `docker` as you would for non-GCR registries, e.g. `docker pull gcr.io/project-id/my-image`. Add `--verbosity=error` to silence this warning: `gcloud docker --verbosity=error -- pull gcr.io/project-id/my-image`. See: https://cloud.google.com/container-registry/docs/support/deprecation-notices#gcloud-docker > Task :sdks:python:test-suites:dataflow:py38:validatesContainer # Where to store integration test outputs. GCS_LOCATION=${GCS_LOCATION:-gs://temp-storage-for-end-to-end-tests} # Project for the container and integration test PROJECT=${PROJECT:-apache-beam-testing} REGION=${REGION:-us-central1} IMAGE_PREFIX="$(grep 'docker_image_default_repo_prefix' gradle.properties | cut -d'=' -f2)" SDK_VERSION="$(grep 'sdk_version' gradle.properties | cut -d'=' -f2)" PY_VERSION=$1 IMAGE_NAME="${IMAGE_PREFIX}python${PY_VERSION}_sdk" CONTAINER_PROJECT="sdks:python:container:py${PY_VERSION//.}" # Note: we substitute away the dot in the version. PY_INTERPRETER="python${PY_VERSION}" XUNIT_FILE="pytest-$IMAGE_NAME.xml" # Verify in the root of the repository test -d sdks/python/container # Verify docker and gcloud commands exist command -v docker command -v gcloud docker -v gcloud -v # Verify docker image has been built. docker images | grep "apache/$IMAGE_NAME" | grep "$SDK_VERSION" TAG=$(date +%Y%m%d-%H%M%S%N) CONTAINER=us.gcr.io/$PROJECT/$USER/$IMAGE_NAME PREBUILD_SDK_CONTAINER_REGISTRY_PATH=us.gcr.io/$PROJECT/$USER/prebuild_python${PY_VERSION//.}_sdk echo "Using container $CONTAINER" # Tag the docker container. docker tag "apache/$IMAGE_NAME:$SDK_VERSION" "$CONTAINER:$TAG" # Push the container. gcloud docker -- push $CONTAINER:$TAG WARNING: `gcloud docker` will not be supported for Docker client versions above 18.03. As an alternative, use `gcloud auth configure-docker` to configure `docker` to use `gcloud` as a credential helper, then use `docker` as you would for non-GCR registries, e.g. `docker pull gcr.io/project-id/my-image`. Add `--verbosity=error` to silence this warning: `gcloud docker --verbosity=error -- pull gcr.io/project-id/my-image`. See: https://cloud.google.com/container-registry/docs/support/deprecation-notices#gcloud-docker unknown blob > Task :sdks:python:test-suites:dataflow:py38:validatesContainer FAILED > Task :sdks:python:test-suites:dataflow:py37:validatesContainer function cleanup_container { # Delete the container locally and remotely docker rmi $CONTAINER:$TAG || echo "Failed to remove container image" for image in $(docker images --format '{{.Repository}}:{{.Tag}}' | grep $PREBUILD_SDK_CONTAINER_REGISTRY_PATH) do docker rmi $image || echo "Failed to remove prebuilt sdk container image" done gcloud --quiet container images delete $CONTAINER:$TAG || echo "Failed to delete container" for digest in $(gcloud container images list-tags $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk --format="get(digest)") do gcloud container images delete $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk@$digest --force-delete-tags --quiet || echo "Failed to remove prebuilt sdk container image" done echo "Removed the container" } trap cleanup_container EXIT echo ">>> Successfully built and push container $CONTAINER" cd sdks/python SDK_LOCATION=$2 # Run ValidatesRunner tests on Google Cloud Dataflow service echo ">>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST" pytest -o junit_suite_name=$IMAGE_NAME \ -m="it_validatescontainer" \ --show-capture=no \ --numprocesses=1 \ --timeout=900 \ --junitxml=$XUNIT_FILE \ --ignore-glob '.*py3\d?\.py$' \ --log-cli-level=INFO \ --test-pipeline-options=" \ --runner=TestDataflowRunner \ --project=$PROJECT \ --region=$REGION \ --sdk_container_image=$CONTAINER:$TAG \ --staging_location=$GCS_LOCATION/staging-validatesrunner-test \ --temp_location=$GCS_LOCATION/temp-validatesrunner-test \ --output=$GCS_LOCATION/output \ --sdk_location=$SDK_LOCATION \ --num_workers=1 \ --docker_registry_push_url=$PREBUILD_SDK_CONTAINER_REGISTRY_PATH" > Task :sdks:python:test-suites:dataflow:py39:validatesContainer function cleanup_container { # Delete the container locally and remotely docker rmi $CONTAINER:$TAG || echo "Failed to remove container image" for image in $(docker images --format '{{.Repository}}:{{.Tag}}' | grep $PREBUILD_SDK_CONTAINER_REGISTRY_PATH) do docker rmi $image || echo "Failed to remove prebuilt sdk container image" done gcloud --quiet container images delete $CONTAINER:$TAG || echo "Failed to delete container" for digest in $(gcloud container images list-tags $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk --format="get(digest)") do gcloud container images delete $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk@$digest --force-delete-tags --quiet || echo "Failed to remove prebuilt sdk container image" done echo "Removed the container" } trap cleanup_container EXIT echo ">>> Successfully built and push container $CONTAINER" cd sdks/python SDK_LOCATION=$2 # Run ValidatesRunner tests on Google Cloud Dataflow service echo ">>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST" pytest -o junit_suite_name=$IMAGE_NAME \ -m="it_validatescontainer" \ --show-capture=no \ --numprocesses=1 \ --timeout=900 \ --junitxml=$XUNIT_FILE \ --ignore-glob '.*py3\d?\.py$' \ --log-cli-level=INFO \ --test-pipeline-options=" \ --runner=TestDataflowRunner \ --project=$PROJECT \ --region=$REGION \ --sdk_container_image=$CONTAINER:$TAG \ --staging_location=$GCS_LOCATION/staging-validatesrunner-test \ --temp_location=$GCS_LOCATION/temp-validatesrunner-test \ --output=$GCS_LOCATION/output \ --sdk_location=$SDK_LOCATION \ --num_workers=1 \ --docker_registry_push_url=$PREBUILD_SDK_CONTAINER_REGISTRY_PATH" > Task :sdks:python:test-suites:dataflow:py37:validatesContainer echo ">>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST" cleanup_container Digests: - us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk@sha256:35365c9ae0fb02cd392708c7705ce5dba8d16d440ebc0acc431fce085a04c713 Associated tags: - 20220801-045124257283676 Tags: - us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk:20220801-045124257283676 Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk:20220801-045124257283676]. Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk@sha256:35365c9ae0fb02cd392708c7705ce5dba8d16d440ebc0acc431fce085a04c713]. Digests: - us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk@sha256:03db5625a255668bcf0a34e718f761321a2ad2d70463d91e1fd7b7001276fa28 Associated tags: - 6b6198d8-8344-4323-95d3-96cea8dd3a25 Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk:6b6198d8-8344-4323-95d3-96cea8dd3a25]. Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk@sha256:03db5625a255668bcf0a34e718f761321a2ad2d70463d91e1fd7b7001276fa28]. Digests: - us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk@sha256:b8d1149ef0e4074c97c584fb9c46d6e85a52281815dda88c2f6d3d7f4876ddeb Associated tags: - 46fd8db8-d808-4aa0-aa2d-6bed9a765759 Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk:46fd8db8-d808-4aa0-aa2d-6bed9a765759]. Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk@sha256:b8d1149ef0e4074c97c584fb9c46d6e85a52281815dda88c2f6d3d7f4876ddeb]. > Task :sdks:python:test-suites:dataflow:py39:validatesContainer [gw0] [100%] PASSED apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py::ExerciseMetricsPipelineTest::test_metrics_fnapi_it =============================== warnings summary =============================== <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead def call(self, fn, *args, **kwargs): -- Docs: https://docs.pytest.org/en/latest/warnings.html - generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/pytest-beam_python3.9_sdk.xml> - ============== 4 passed, 2 skipped, 1 warnings in 1683.21 seconds ============== echo ">>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST" cleanup_container >>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST Untagged: us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk:20220801-045124451402925 Untagged: us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk@sha256:97c63bf7952fce761dc830b834a05c127f75b1a237dd4feb91ea19ad53ed0e60 Untagged: us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk:35dda9e9-5d4e-443b-8e15-276fd610ff89 Untagged: us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk@sha256:b13ffa6089709180ee3083c284f4175b200afe51f06722a79b9e8cdd17395767 Deleted: sha256:e8c3e177d3ed17159ed2c1de8355bbb0271f2aef4b96a7ccb5e14ab15b7a4ace Deleted: sha256:7a49eabbd8cc9a1420015d145428fb75bc390b1bfd75dff19957e3cf50d9d4df Deleted: sha256:1f726577d50348332d787a4e0e923b1e7034cf918934a0659f7500d8d53e0039 Deleted: sha256:a23caa9752b475f3bba193dc50a43530ca168df43ea43036087da231ee809ff2 Deleted: sha256:f30f8f46f98a1aea0248cb5ce8e10f253a17c1125ea0a4e481203ea2a61b560d Deleted: sha256:aaa7f1b715faffb4d51c1f3d2a0b1b874b4aa7f1961ec9abd01f3cb4cc49d6fb Digests: - us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk@sha256:97c63bf7952fce761dc830b834a05c127f75b1a237dd4feb91ea19ad53ed0e60 Associated tags: - 20220801-045124451402925 Tags: - us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk:20220801-045124451402925 Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk:20220801-045124451402925]. Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.9_sdk@sha256:97c63bf7952fce761dc830b834a05c127f75b1a237dd4feb91ea19ad53ed0e60]. Digests: - us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk@sha256:b13ffa6089709180ee3083c284f4175b200afe51f06722a79b9e8cdd17395767 Associated tags: - 35dda9e9-5d4e-443b-8e15-276fd610ff89 Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk:35dda9e9-5d4e-443b-8e15-276fd610ff89]. Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk@sha256:b13ffa6089709180ee3083c284f4175b200afe51f06722a79b9e8cdd17395767]. Digests: - us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk@sha256:8d524cb2b0a5a68ffb413570e95c8fe66279271f0bc90d553e6003b73c5804d5 Associated tags: - 98266427-9b98-4300-9d26-622e0627b83a Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk:98266427-9b98-4300-9d26-622e0627b83a]. Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python39_sdk/beam_python_prebuilt_sdk@sha256:8d524cb2b0a5a68ffb413570e95c8fe66279271f0bc90d553e6003b73c5804d5]. Removed the container FAILURE: Build failed with an exception. * Where: Script '<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 359 * What went wrong: Execution failed for task ':sdks:python:test-suites:dataflow:py38:validatesContainer'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. * Get more help at https://help.gradle.org BUILD FAILED in 42m 3s 51 actionable tasks: 41 executed, 4 from cache, 6 up-to-date Publishing build scan... https://gradle.com/s/kidilrxg4aomc Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
