See <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/8139/display/redirect?page=changes>
Changes: [samuelw] [BEAM-12740] [BEAM-8268] Improve error handling and retries for GCS [aromanenko.dev] [BEAM-12270] Add Parquet source support for TPD-DS benchmark [noreply] [BEAM-12742] Samza Runner does not properly delete modified timer ------------------------------------------ [...truncated 745.83 KB...] # Tag the docker container. docker tag "apache/$IMAGE_NAME:$SDK_VERSION" "$CONTAINER:$TAG" # Push the container. gcloud docker -- push $CONTAINER:$TAG WARNING: `gcloud docker` will not be supported for Docker client versions above 18.03. As an alternative, use `gcloud auth configure-docker` to configure `docker` to use `gcloud` as a credential helper, then use `docker` as you would for non-GCR registries, e.g. `docker pull gcr.io/project-id/my-image`. Add `--verbosity=error` to silence this warning: `gcloud docker --verbosity=error -- pull gcr.io/project-id/my-image`. See: https://cloud.google.com/container-registry/docs/support/deprecation-notices#gcloud-docker > Task :sdks:python:test-suites:dataflow:py38:validatesContainer # Where to store integration test outputs. GCS_LOCATION=${GCS_LOCATION:-gs://temp-storage-for-end-to-end-tests} # Project for the container and integration test PROJECT=${PROJECT:-apache-beam-testing} REGION=${REGION:-us-central1} IMAGE_PREFIX="$(grep 'docker_image_default_repo_prefix' gradle.properties | cut -d'=' -f2)" grep 'docker_image_default_repo_prefix' gradle.properties | cut -d'=' -f2 SDK_VERSION="$(grep 'sdk_version' gradle.properties | cut -d'=' -f2)" grep 'sdk_version' gradle.properties | cut -d'=' -f2 # Other variables branched by Python version. if [[ $1 == "python36" ]]; then IMAGE_NAME="${IMAGE_PREFIX}python3.6_sdk" # Use this to create CONTAINER_IMAGE variable. CONTAINER_PROJECT="sdks:python:container:py36" # Use this to build container by Gradle. PY_INTERPRETER="python3.6" # Use this in virtualenv command. elif [[ $1 == "python37" ]]; then IMAGE_NAME="${IMAGE_PREFIX}python3.7_sdk" # Use this to create CONTAINER_IMAGE variable. CONTAINER_PROJECT="sdks:python:container:py37" # Use this to build container by Gradle. PY_INTERPRETER="python3.7" # Use this in virtualenv command. elif [[ $1 == "python38" ]]; then IMAGE_NAME="${IMAGE_PREFIX}python3.8_sdk" # Use this to create CONTAINER_IMAGE variable. CONTAINER_PROJECT="sdks:python:container:py38" # Use this to build container by Gradle. PY_INTERPRETER="python3.8" # Use this in virtualenv command. else echo "Must set Python version with one of 'python36', 'python37' and 'python38' from commandline." exit 1 fi XUNIT_FILE="pytest-$IMAGE_NAME.xml" # Verify in the root of the repository test -d sdks/python/container # Verify docker and gcloud commands exist command -v docker command -v gcloud docker -v gcloud -v # Verify docker image has been built. docker images | grep "apache/$IMAGE_NAME" | grep "$SDK_VERSION" TAG=$(date +%Y%m%d-%H%M%S%N) date +%Y%m%d-%H%M%S%N CONTAINER=us.gcr.io/$PROJECT/$USER/$IMAGE_NAME PREBUILD_SDK_CONTAINER_REGISTRY_PATH=us.gcr.io/$PROJECT/$USER/prebuild_$1_sdk echo "Using container $CONTAINER" # Tag the docker container. docker tag "apache/$IMAGE_NAME:$SDK_VERSION" "$CONTAINER:$TAG" # Push the container. gcloud docker -- push $CONTAINER:$TAG WARNING: `gcloud docker` will not be supported for Docker client versions above 18.03. As an alternative, use `gcloud auth configure-docker` to configure `docker` to use `gcloud` as a credential helper, then use `docker` as you would for non-GCR registries, e.g. `docker pull gcr.io/project-id/my-image`. Add `--verbosity=error` to silence this warning: `gcloud docker --verbosity=error -- pull gcr.io/project-id/my-image`. See: https://cloud.google.com/container-registry/docs/support/deprecation-notices#gcloud-docker > Task :sdks:python:test-suites:dataflow:py36:validatesContainer function cleanup_container { # Delete the container locally and remotely docker rmi $CONTAINER:$TAG || echo "Failed to remove container image" for image in $(docker images --format '{{.Repository}}:{{.Tag}}' | grep $PREBUILD_SDK_CONTAINER_REGISTRY_PATH) do docker rmi $image || echo "Failed to remove prebuilt sdk container image" done gcloud --quiet container images delete $CONTAINER:$TAG || echo "Failed to delete container" for digest in $(gcloud container images list-tags $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk --format="get(digest)") do gcloud container images delete $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk@$digest --force-delete-tags --quiet || echo "Failed to remove prebuilt sdk container image" done echo "Removed the container" } trap cleanup_container EXIT echo ">>> Successfully built and push container $CONTAINER" cd sdks/python SDK_LOCATION=$2 # Run ValidatesRunner tests on Google Cloud Dataflow service echo ">>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST" pytest -o junit_suite_name=$IMAGE_NAME \ -m="it_validatescontainer" \ --show-capture=no \ --numprocesses=1 \ --timeout=900 \ --junitxml=$XUNIT_FILE \ --ignore-glob '.*py3\d?\.py$' \ --log-cli-level=INFO \ --test-pipeline-options=" \ --runner=TestDataflowRunner \ --project=$PROJECT \ --region=$REGION \ --sdk_container_image=$CONTAINER:$TAG \ --staging_location=$GCS_LOCATION/staging-validatesrunner-test \ --temp_location=$GCS_LOCATION/temp-validatesrunner-test \ --output=$GCS_LOCATION/output \ --sdk_location=$SDK_LOCATION \ --num_workers=1 \ --prebuild_sdk_container_base_image=$CONTAINER:$TAG \ --docker_registry_push_url=$PREBUILD_SDK_CONTAINER_REGISTRY_PATH" > Task :sdks:python:test-suites:dataflow:py38:validatesContainer function cleanup_container { # Delete the container locally and remotely docker rmi $CONTAINER:$TAG || echo "Failed to remove container image" for image in $(docker images --format '{{.Repository}}:{{.Tag}}' | grep $PREBUILD_SDK_CONTAINER_REGISTRY_PATH) do docker rmi $image || echo "Failed to remove prebuilt sdk container image" done gcloud --quiet container images delete $CONTAINER:$TAG || echo "Failed to delete container" for digest in $(gcloud container images list-tags $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk --format="get(digest)") do gcloud container images delete $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk@$digest --force-delete-tags --quiet || echo "Failed to remove prebuilt sdk container image" done echo "Removed the container" } trap cleanup_container EXIT echo ">>> Successfully built and push container $CONTAINER" cd sdks/python SDK_LOCATION=$2 # Run ValidatesRunner tests on Google Cloud Dataflow service echo ">>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST" pytest -o junit_suite_name=$IMAGE_NAME \ -m="it_validatescontainer" \ --show-capture=no \ --numprocesses=1 \ --timeout=900 \ --junitxml=$XUNIT_FILE \ --ignore-glob '.*py3\d?\.py$' \ --log-cli-level=INFO \ --test-pipeline-options=" \ --runner=TestDataflowRunner \ --project=$PROJECT \ --region=$REGION \ --sdk_container_image=$CONTAINER:$TAG \ --staging_location=$GCS_LOCATION/staging-validatesrunner-test \ --temp_location=$GCS_LOCATION/temp-validatesrunner-test \ --output=$GCS_LOCATION/output \ --sdk_location=$SDK_LOCATION \ --num_workers=1 \ --prebuild_sdk_container_base_image=$CONTAINER:$TAG \ --docker_registry_push_url=$PREBUILD_SDK_CONTAINER_REGISTRY_PATH" > Task :sdks:python:test-suites:dataflow:py36:validatesContainer echo ">>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST" cleanup_container docker images --format '{{.Repository}}:{{.Tag}}' | grep $PREBUILD_SDK_CONTAINER_REGISTRY_PATH Digests: - us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk@sha256:33269c6d07f5e2f06557a4d8ec479e1c299ba09246525bc41defa163fdb4169d Associated tags: - 20210818-181315868126627 Tags: - us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20210818-181315868126627 Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20210818-181315868126627]. Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk@sha256:33269c6d07f5e2f06557a4d8ec479e1c299ba09246525bc41defa163fdb4169d]. gcloud container images list-tags $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk --format="get(digest)" Digests: - us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:d219c94cb5f51eaf9703c47f684331f4f64897b18365fc719b98e0945776fc1e Associated tags: - 1411b5a9-124f-467e-8e9b-c32571b266ff Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk:1411b5a9-124f-467e-8e9b-c32571b266ff]. Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:d219c94cb5f51eaf9703c47f684331f4f64897b18365fc719b98e0945776fc1e]. Digests: - us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:bcec6015f2c17073f6700f5897c674ab36032ed1281d1d2149ed30f3a854c8a8 Associated tags: - edfc2965-5806-470d-b31f-8fcde11e12fc Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk:edfc2965-5806-470d-b31f-8fcde11e12fc]. Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:bcec6015f2c17073f6700f5897c674ab36032ed1281d1d2149ed30f3a854c8a8]. > Task :sdks:python:test-suites:dataflow:py38:validatesContainer echo ">>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST" cleanup_container docker images --format '{{.Repository}}:{{.Tag}}' | grep $PREBUILD_SDK_CONTAINER_REGISTRY_PATH Digests: - us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk@sha256:eda8bda410c6073e924b8815c0e136f64a16ea77204329ed23bb3d2f48935cde Associated tags: - 20210818-181322618670162 Tags: - us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk:20210818-181322618670162 Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk:20210818-181322618670162]. Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk@sha256:eda8bda410c6073e924b8815c0e136f64a16ea77204329ed23bb3d2f48935cde]. gcloud container images list-tags $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk --format="get(digest)" Digests: - us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk@sha256:4d93580297f511d17029d7f3614b5de2842139ce4386eb9b1fd683e86135e282 Associated tags: - dd203735-eb2c-4feb-810d-00fc29083a02 Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk:dd203735-eb2c-4feb-810d-00fc29083a02]. Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk@sha256:4d93580297f511d17029d7f3614b5de2842139ce4386eb9b1fd683e86135e282]. Digests: - us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk@sha256:c583db8e5c50314e32b71499bd25c8f5bbdd1985cb491e9f9d58f0e9a9747ca2 Associated tags: - e0e39a79-6444-42b0-b569-53b8eba767d3 Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk:e0e39a79-6444-42b0-b569-53b8eba767d3]. Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk@sha256:c583db8e5c50314e32b71499bd25c8f5bbdd1985cb491e9f9d58f0e9a9747ca2]. FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':sdks:python:container:py37:docker'. > Process 'command 'docker'' finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 51m 37s 43 actionable tasks: 39 executed, 4 up-to-date Publishing build scan... https://gradle.com/s/nv7snqeekexsm Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
