See <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/8980/display/redirect>
Changes: ------------------------------------------ [...truncated 106.02 KB...] tensorflow==2.6.2 tensorflow-estimator==2.6.0 termcolor==1.1.0 testcontainers==3.4.2 threadpoolctl==3.1.0 tqdm==4.63.0 typing-extensions==3.7.4.3 typing-inspect==0.7.1 typing-utils==0.1.0 uritemplate==4.1.1 urllib3==1.26.8 wcwidth==0.2.5 websocket-client==1.3.1 Werkzeug==2.0.3 wheel==0.37.0 wrapt==1.12.1 zipp==3.6.0 Removing intermediate container f360ca6e834c ---> c93dcfbcc698 Step 16/31 : COPY target/LICENSE /opt/apache/beam/ ---> 414e795169bd Step 17/31 : COPY target/LICENSE.python /opt/apache/beam/ ---> 6a632763e2b5 Step 18/31 : COPY target/NOTICE /opt/apache/beam/ ---> b572f0dcb4ea Step 19/31 : COPY target/launcher/linux_amd64/boot /opt/apache/beam/ ---> db0595a5cb88 Step 20/31 : ENTRYPOINT ["/opt/apache/beam/boot"] ---> Running in 80affe4f06fa Removing intermediate container 80affe4f06fa ---> c27e24400590 Step 21/31 : FROM beam as third_party_licenses ---> c27e24400590 Step 22/31 : ARG pull_licenses ---> Running in a7e4d9bf3c4d Removing intermediate container a7e4d9bf3c4d ---> 0433d0532916 Step 23/31 : COPY target/license_scripts /tmp/license_scripts/ ---> c46e7b192563 Step 24/31 : COPY target/LICENSE target/go-licenses/* /opt/apache/beam/third_party_licenses/golang/ ---> c1000ed2f0ee Step 25/31 : RUN rm /opt/apache/beam/third_party_licenses/golang/LICENSE ---> Running in 58f131648bb7 Removing intermediate container 58f131648bb7 ---> 7cbee61ef10d Step 26/31 : COPY target/license_scripts /tmp/license_scripts/ ---> 51730b68fb44 Step 27/31 : RUN if [ "$pull_licenses" = "true" ] ; then pip install 'pip-licenses<3.0.0' pyyaml tenacity && python /tmp/license_scripts/pull_licenses_py.py ; fi ---> Running in 1f9629dc3693 Collecting pip-licenses<3.0.0 Downloading pip_licenses-2.3.0-py3-none-any.whl (14 kB) Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/site-packages (6.0) Requirement already satisfied: tenacity in /usr/local/lib/python3.6/site-packages (5.1.5) Collecting PTable Downloading PTable-0.9.2.tar.gz (31 kB) Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/site-packages (from tenacity) (1.15.0) Building wheels for collected packages: PTable Building wheel for PTable (setup.py): started Building wheel for PTable (setup.py): finished with status 'done' Created wheel for PTable: filename=PTable-0.9.2-py3-none-any.whl size=22924 sha256=c8fdd8082353211f345d5833871aafa3fbf9d721f8783ea3476797964e52fbb2 Stored in directory: /root/.cache/pip/wheels/f3/65/67/71f473ec87ea4554d1bdfb1b5128cfe1414c2b113b72a1ee8e Successfully built PTable Installing collected packages: PTable, pip-licenses Successfully installed PTable-0.9.2 pip-licenses-2.3.0 [91mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [0m[91mWARNING: You are using pip version 21.2.4; however, version 21.3.1 is available. You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command. [0m[91mINFO:root:Successfully pulled licenses for 139 dependencies [0mSkip pulling license for bs4 Removing intermediate container 1f9629dc3693 ---> 2b448aea1f84 Step 28/31 : FROM beam ---> c27e24400590 Step 29/31 : ARG pull_licenses ---> Running in 83acab188fe6 Removing intermediate container 83acab188fe6 ---> b57e793bd100 Step 30/31 : COPY --from=third_party_licenses /opt/apache/beam/third_party_licenses /opt/apache/beam/third_party_licenses ---> a91c959d81e3 Step 31/31 : RUN if [ "$pull_licenses" != "true" ] ; then rm -rf /opt/apache/beam/third_party_licenses ; fi ---> Running in 61a3842ad4bf Removing intermediate container 61a3842ad4bf ---> ab72b4725527 Successfully built ab72b4725527 Successfully tagged apache/beam_python3.6_sdk:2.38.0.dev > Task :sdks:python:test-suites:dataflow:py36:validatesContainer # Where to store integration test outputs. GCS_LOCATION=${GCS_LOCATION:-gs://temp-storage-for-end-to-end-tests} # Project for the container and integration test PROJECT=${PROJECT:-apache-beam-testing} REGION=${REGION:-us-central1} IMAGE_PREFIX="$(grep 'docker_image_default_repo_prefix' gradle.properties | cut -d'=' -f2)" SDK_VERSION="$(grep 'sdk_version' gradle.properties | cut -d'=' -f2)" PY_VERSION=$1 IMAGE_NAME="${IMAGE_PREFIX}python${PY_VERSION}_sdk" CONTAINER_PROJECT="sdks:python:container:py${PY_VERSION//.}" # Note: we substitute away the dot in the version. PY_INTERPRETER="python${PY_VERSION}" XUNIT_FILE="pytest-$IMAGE_NAME.xml" # Verify in the root of the repository test -d sdks/python/container # Verify docker and gcloud commands exist command -v docker command -v gcloud docker -v gcloud -v # Verify docker image has been built. docker images | grep "apache/$IMAGE_NAME" | grep "$SDK_VERSION" TAG=$(date +%Y%m%d-%H%M%S%N) CONTAINER=us.gcr.io/$PROJECT/$USER/$IMAGE_NAME PREBUILD_SDK_CONTAINER_REGISTRY_PATH=us.gcr.io/$PROJECT/$USER/prebuild_python${PY_VERSION//.}_sdk echo "Using container $CONTAINER" # Tag the docker container. docker tag "apache/$IMAGE_NAME:$SDK_VERSION" "$CONTAINER:$TAG" # Push the container. gcloud docker -- push $CONTAINER:$TAG WARNING: `gcloud docker` will not be supported for Docker client versions above 18.03. As an alternative, use `gcloud auth configure-docker` to configure `docker` to use `gcloud` as a credential helper, then use `docker` as you would for non-GCR registries, e.g. `docker pull gcr.io/project-id/my-image`. Add `--verbosity=error` to silence this warning: `gcloud docker --verbosity=error -- pull gcr.io/project-id/my-image`. See: https://cloud.google.com/container-registry/docs/support/deprecation-notices#gcloud-docker function cleanup_container { # Delete the container locally and remotely docker rmi $CONTAINER:$TAG || echo "Failed to remove container image" for image in $(docker images --format '{{.Repository}}:{{.Tag}}' | grep $PREBUILD_SDK_CONTAINER_REGISTRY_PATH) do docker rmi $image || echo "Failed to remove prebuilt sdk container image" done gcloud --quiet container images delete $CONTAINER:$TAG || echo "Failed to delete container" for digest in $(gcloud container images list-tags $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk --format="get(digest)") do gcloud container images delete $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk@$digest --force-delete-tags --quiet || echo "Failed to remove prebuilt sdk container image" done echo "Removed the container" } trap cleanup_container EXIT echo ">>> Successfully built and push container $CONTAINER" cd sdks/python SDK_LOCATION=$2 # Run ValidatesRunner tests on Google Cloud Dataflow service echo ">>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST" pytest -o junit_suite_name=$IMAGE_NAME \ -m="it_validatescontainer" \ --show-capture=no \ --numprocesses=1 \ --timeout=900 \ --junitxml=$XUNIT_FILE \ --ignore-glob '.*py3\d?\.py$' \ --log-cli-level=INFO \ --test-pipeline-options=" \ --runner=TestDataflowRunner \ --project=$PROJECT \ --region=$REGION \ --sdk_container_image=$CONTAINER:$TAG \ --staging_location=$GCS_LOCATION/staging-validatesrunner-test \ --temp_location=$GCS_LOCATION/temp-validatesrunner-test \ --output=$GCS_LOCATION/output \ --sdk_location=$SDK_LOCATION \ --num_workers=1 \ --docker_registry_push_url=$PREBUILD_SDK_CONTAINER_REGISTRY_PATH" echo ">>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST" cleanup_container Digests: - us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk@sha256:aad882efe0d2d1cf1aa5cc8c6280de0f579c3238bddf03bddb6d5d84c28de179 Associated tags: - 20220319-181529483132772 Tags: - us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20220319-181529483132772 Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20220319-181529483132772]. Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk@sha256:aad882efe0d2d1cf1aa5cc8c6280de0f579c3238bddf03bddb6d5d84c28de179]. Digests: - us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:15d0373ff4dfc2dc668efefe8aa3d782ad4a854160e273f04e4ed1e04b751656 Associated tags: - 2a398d0e-25f5-465c-b444-e96a4e4246e5 Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk:2a398d0e-25f5-465c-b444-e96a4e4246e5]. Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:15d0373ff4dfc2dc668efefe8aa3d782ad4a854160e273f04e4ed1e04b751656]. Digests: - us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:0f437cb1531a53cc0106d8e4efcc2c48571b59f5b230de45240b3381d6926170 Associated tags: - 8f2d725e-06e5-41fa-b785-48e708c9a1a7 Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk:8f2d725e-06e5-41fa-b785-48e708c9a1a7]. Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:0f437cb1531a53cc0106d8e4efcc2c48571b59f5b230de45240b3381d6926170]. FAILURE: Build completed with 3 failures. 1: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':sdks:python:container:py39:docker'. > Process 'command 'docker'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. ============================================================================== 2: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':sdks:python:container:py38:docker'. > Process 'command 'docker'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. ============================================================================== 3: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':sdks:python:container:py37:docker'. > Process 'command 'docker'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. ============================================================================== * Get more help at https://help.gradle.org BUILD FAILED in 50m 59 actionable tasks: 48 executed, 4 from cache, 7 up-to-date Publishing build scan... https://gradle.com/s/yswwmflnknrkq Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
