See <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/8977/display/redirect?page=changes>
Changes: [Kiley Sok] Add Java 17 Nexmark metrics to Grafana [mmack] [BEAM-14125] Update website IO matrix to recommend aws2 IOs [noreply] [BEAM-14128] Eliminating quadratic behavior of [noreply] [BEAM-13972] Add RunInference interface (#16917) [noreply] Merge pull request #17116 from [BEAM-12164] Remove change_stream in [noreply] Deprecate tags.go (#17025) [noreply] [BEAM-12753] and [BEAM-12815] Fix Flink Integration Tests (#17067) [noreply] Merge pull request #16895 from [BEAM-13882][Playground] More tests for [noreply] [BEAM-13925] Add weekly automation to update our reviewer config [noreply] Merge pull request #17076 from Beam 14082 update payground for mobile ------------------------------------------ [...truncated 95.53 KB...] > Task :release:go-licenses:py:dockerRun + go-licenses save github.com/apache/beam/sdks/python/container --save_path=/output/licenses + go-licenses csv github.com/apache/beam/sdks/python/container + tee /output/licenses/list.csv golang.org/x/net,https://go.googlesource.com/net/+/refs/heads/master/LICENSE,BSD-3-Clause google.golang.org/protobuf,https://go.googlesource.com/protobuf/+/refs/heads/master/LICENSE,BSD-3-Clause google.golang.org/genproto/googleapis/rpc/status,https://github.com/googleapis/go-genproto/blob/master/LICENSE,Apache-2.0 golang.org/x/text,https://go.googlesource.com/text/+/refs/heads/master/LICENSE,BSD-3-Clause github.com/nightlyone/lockfile,https://github.com/nightlyone/lockfile/blob/master/LICENSE,MIT github.com/apache/beam/sdks/python/container,https://github.com/apache/beam/blob/master/LICENSE,Apache-2.0 google.golang.org/grpc,https://github.com/grpc/grpc-go/blob/master/LICENSE,Apache-2.0 github.com/golang/protobuf,https://github.com/golang/protobuf/blob/master/LICENSE,BSD-3-Clause github.com/apache/beam/sdks/go/pkg/beam,https://github.com/apache/beam/blob/master/sdks/go/README.md,Apache-2.0 golang.org/x/sys,https://go.googlesource.com/sys/+/refs/heads/master/LICENSE,BSD-3-Clause + chmod -R a+w /output/licenses > Task :release:go-licenses:py:createLicenses > Task :sdks:python:container:py37:copyGolangLicenses > Task :sdks:python:container:py39:copyGolangLicenses > Task :sdks:python:container:py36:copyGolangLicenses > Task :sdks:python:container:py38:copyGolangLicenses > Task :sdks:python:container:py37:dockerPrepare > Task :sdks:python:container:py39:dockerPrepare > Task :sdks:python:container:py38:dockerPrepare > Task :sdks:python:container:py36:dockerPrepare > Task :sdks:python:container:py37:docker > Task :sdks:python:container:py39:docker > Task :sdks:python:container:py38:docker > Task :sdks:python:container:py36:docker > Task :sdks:python:container:py39:docker The command '/bin/sh -c if [ "$pull_licenses" = "true" ] ; then pip install 'pip-licenses<3.0.0' pyyaml tenacity && python /tmp/license_scripts/pull_licenses_py.py ; fi' returned a non-zero code: 1 > Task :sdks:python:container:py38:docker The command '/bin/sh -c if [ "$pull_licenses" = "true" ] ; then pip install 'pip-licenses<3.0.0' pyyaml tenacity && python /tmp/license_scripts/pull_licenses_py.py ; fi' returned a non-zero code: 1 > Task :sdks:python:container:py38:docker FAILED > Task :sdks:python:container:py39:docker FAILED > Task :sdks:python:container:py37:docker FAILED The command '/bin/sh -c if [ "$pull_licenses" = "true" ] ; then pip install 'pip-licenses<3.0.0' pyyaml tenacity && python /tmp/license_scripts/pull_licenses_py.py ; fi' returned a non-zero code: 1 > Task :sdks:python:container:py36:docker Removing intermediate container 813b6b227a86 ---> 5e3197360c09 Step 26/31 : COPY target/license_scripts /tmp/license_scripts/ ---> 75ae32eee8af Step 27/31 : RUN if [ "$pull_licenses" = "true" ] ; then pip install 'pip-licenses<3.0.0' pyyaml tenacity && python /tmp/license_scripts/pull_licenses_py.py ; fi ---> Running in c4e7aea1b70b Collecting pip-licenses<3.0.0 Downloading pip_licenses-2.3.0-py3-none-any.whl (14 kB) Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/site-packages (6.0) Requirement already satisfied: tenacity in /usr/local/lib/python3.6/site-packages (5.1.5) Collecting PTable Downloading PTable-0.9.2.tar.gz (31 kB) Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/site-packages (from tenacity) (1.15.0) Building wheels for collected packages: PTable Building wheel for PTable (setup.py): started Building wheel for PTable (setup.py): finished with status 'done' Created wheel for PTable: filename=PTable-0.9.2-py3-none-any.whl size=22924 sha256=91a41cb980f61a40d0db571f2f8dbddfbfe704dd65358a18a8b74343775d163a Stored in directory: /root/.cache/pip/wheels/f3/65/67/71f473ec87ea4554d1bdfb1b5128cfe1414c2b113b72a1ee8e Successfully built PTable Installing collected packages: PTable, pip-licenses Successfully installed PTable-0.9.2 pip-licenses-2.3.0 [91mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [0m[91mWARNING: You are using pip version 21.2.4; however, version 21.3.1 is available. You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command. [0m[91mINFO:root:Successfully pulled licenses for 139 dependencies [0mSkip pulling license for bs4 Removing intermediate container c4e7aea1b70b ---> 0f62aff6cb4f Step 28/31 : FROM beam ---> 213360388a80 Step 29/31 : ARG pull_licenses ---> Running in 059ac150f91d Removing intermediate container 059ac150f91d ---> 382b84db03dc Step 30/31 : COPY --from=third_party_licenses /opt/apache/beam/third_party_licenses /opt/apache/beam/third_party_licenses ---> 5d23d460059b Step 31/31 : RUN if [ "$pull_licenses" != "true" ] ; then rm -rf /opt/apache/beam/third_party_licenses ; fi ---> Running in d0871446085e Removing intermediate container d0871446085e ---> 535c4d3da219 Successfully built 535c4d3da219 Successfully tagged apache/beam_python3.6_sdk:2.38.0.dev > Task :sdks:python:test-suites:dataflow:py36:validatesContainer # Where to store integration test outputs. GCS_LOCATION=${GCS_LOCATION:-gs://temp-storage-for-end-to-end-tests} # Project for the container and integration test PROJECT=${PROJECT:-apache-beam-testing} REGION=${REGION:-us-central1} IMAGE_PREFIX="$(grep 'docker_image_default_repo_prefix' gradle.properties | cut -d'=' -f2)" SDK_VERSION="$(grep 'sdk_version' gradle.properties | cut -d'=' -f2)" PY_VERSION=$1 IMAGE_NAME="${IMAGE_PREFIX}python${PY_VERSION}_sdk" CONTAINER_PROJECT="sdks:python:container:py${PY_VERSION//.}" # Note: we substitute away the dot in the version. PY_INTERPRETER="python${PY_VERSION}" XUNIT_FILE="pytest-$IMAGE_NAME.xml" # Verify in the root of the repository test -d sdks/python/container # Verify docker and gcloud commands exist command -v docker command -v gcloud docker -v gcloud -v # Verify docker image has been built. docker images | grep "apache/$IMAGE_NAME" | grep "$SDK_VERSION" TAG=$(date +%Y%m%d-%H%M%S%N) CONTAINER=us.gcr.io/$PROJECT/$USER/$IMAGE_NAME PREBUILD_SDK_CONTAINER_REGISTRY_PATH=us.gcr.io/$PROJECT/$USER/prebuild_python${PY_VERSION//.}_sdk echo "Using container $CONTAINER" # Tag the docker container. docker tag "apache/$IMAGE_NAME:$SDK_VERSION" "$CONTAINER:$TAG" # Push the container. gcloud docker -- push $CONTAINER:$TAG WARNING: `gcloud docker` will not be supported for Docker client versions above 18.03. As an alternative, use `gcloud auth configure-docker` to configure `docker` to use `gcloud` as a credential helper, then use `docker` as you would for non-GCR registries, e.g. `docker pull gcr.io/project-id/my-image`. Add `--verbosity=error` to silence this warning: `gcloud docker --verbosity=error -- pull gcr.io/project-id/my-image`. See: https://cloud.google.com/container-registry/docs/support/deprecation-notices#gcloud-docker function cleanup_container { # Delete the container locally and remotely docker rmi $CONTAINER:$TAG || echo "Failed to remove container image" for image in $(docker images --format '{{.Repository}}:{{.Tag}}' | grep $PREBUILD_SDK_CONTAINER_REGISTRY_PATH) do docker rmi $image || echo "Failed to remove prebuilt sdk container image" done gcloud --quiet container images delete $CONTAINER:$TAG || echo "Failed to delete container" for digest in $(gcloud container images list-tags $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk --format="get(digest)") do gcloud container images delete $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk@$digest --force-delete-tags --quiet || echo "Failed to remove prebuilt sdk container image" done echo "Removed the container" } trap cleanup_container EXIT echo ">>> Successfully built and push container $CONTAINER" cd sdks/python SDK_LOCATION=$2 # Run ValidatesRunner tests on Google Cloud Dataflow service echo ">>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST" pytest -o junit_suite_name=$IMAGE_NAME \ -m="it_validatescontainer" \ --show-capture=no \ --numprocesses=1 \ --timeout=900 \ --junitxml=$XUNIT_FILE \ --ignore-glob '.*py3\d?\.py$' \ --log-cli-level=INFO \ --test-pipeline-options=" \ --runner=TestDataflowRunner \ --project=$PROJECT \ --region=$REGION \ --sdk_container_image=$CONTAINER:$TAG \ --staging_location=$GCS_LOCATION/staging-validatesrunner-test \ --temp_location=$GCS_LOCATION/temp-validatesrunner-test \ --output=$GCS_LOCATION/output \ --sdk_location=$SDK_LOCATION \ --num_workers=1 \ --docker_registry_push_url=$PREBUILD_SDK_CONTAINER_REGISTRY_PATH" echo ">>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST" cleanup_container Digests: - us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk@sha256:c525d2bcd7e279f1b7594107012e7cc3295240cb2f099985d5a94271d01e4e81 Associated tags: - 20220319-002159811037800 Tags: - us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20220319-002159811037800 Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20220319-002159811037800]. Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk@sha256:c525d2bcd7e279f1b7594107012e7cc3295240cb2f099985d5a94271d01e4e81]. Digests: - us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:f01d003b73518ab7187b9f990c8cf4771cc1d7e1efd52feab27a4ebe439e6e85 Associated tags: - afd94ffe-1087-4316-a1c7-8a5f625753df Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk:afd94ffe-1087-4316-a1c7-8a5f625753df]. Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:f01d003b73518ab7187b9f990c8cf4771cc1d7e1efd52feab27a4ebe439e6e85]. Digests: - us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:9f79540fd79532b87896250c775faf93c57020eec7b92189fdf1c4d68e32b506 Associated tags: - 617c5f50-75aa-476f-97e8-292146235a80 Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk:617c5f50-75aa-476f-97e8-292146235a80]. Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:9f79540fd79532b87896250c775faf93c57020eec7b92189fdf1c4d68e32b506]. FAILURE: Build completed with 3 failures. 1: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':sdks:python:container:py38:docker'. > Process 'command 'docker'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. ============================================================================== 2: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':sdks:python:container:py39:docker'. > Process 'command 'docker'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. ============================================================================== 3: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':sdks:python:container:py37:docker'. > Process 'command 'docker'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. ============================================================================== * Get more help at https://help.gradle.org BUILD FAILED in 50m 1s 59 actionable tasks: 48 executed, 4 from cache, 7 up-to-date Publishing build scan... https://gradle.com/s/43aetottx3foc Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
