See <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/8973/display/redirect?page=changes>
Changes: [noreply] Populate environment capabilities in v1beta3 protos. (#17042) [noreply] [BEAM-14038] Auto-startup for Python expansion service. (#17035) ------------------------------------------ [...truncated 85.14 KB...] + go-licenses save github.com/apache/beam/sdks/python/container --save_path=/output/licenses + go-licenses csv github.com/apache/beam/sdks/python/container + tee /output/licenses/list.csv golang.org/x/net,https://go.googlesource.com/net/+/refs/heads/master/LICENSE,BSD-3-Clause golang.org/x/sys,https://go.googlesource.com/sys/+/refs/heads/master/LICENSE,BSD-3-Clause github.com/golang/protobuf,https://github.com/golang/protobuf/blob/master/LICENSE,BSD-3-Clause google.golang.org/genproto/googleapis/rpc/status,https://github.com/googleapis/go-genproto/blob/master/LICENSE,Apache-2.0 golang.org/x/text,https://go.googlesource.com/text/+/refs/heads/master/LICENSE,BSD-3-Clause github.com/nightlyone/lockfile,https://github.com/nightlyone/lockfile/blob/master/LICENSE,MIT github.com/apache/beam/sdks/python/container,https://github.com/apache/beam/blob/master/LICENSE,Apache-2.0 github.com/apache/beam/sdks/go/pkg/beam,https://github.com/apache/beam/blob/master/sdks/go/README.md,Apache-2.0 google.golang.org/grpc,https://github.com/grpc/grpc-go/blob/master/LICENSE,Apache-2.0 google.golang.org/protobuf,https://go.googlesource.com/protobuf/+/refs/heads/master/LICENSE,BSD-3-Clause + chmod -R a+w /output/licenses > Task :release:go-licenses:py:createLicenses > Task :sdks:python:container:py37:copyGolangLicenses > Task :sdks:python:container:py39:copyGolangLicenses > Task :sdks:python:container:py38:copyGolangLicenses > Task :sdks:python:container:py36:copyGolangLicenses > Task :sdks:python:container:py39:dockerPrepare > Task :sdks:python:container:py38:dockerPrepare > Task :sdks:python:container:py37:dockerPrepare > Task :sdks:python:container:py36:dockerPrepare > Task :sdks:python:container:py39:docker > Task :sdks:python:container:py38:docker > Task :sdks:python:container:py37:docker > Task :sdks:python:container:py36:docker > Task :sdks:python:container:py39:docker The command '/bin/sh -c if [ "$pull_licenses" = "true" ] ; then pip install 'pip-licenses<3.0.0' pyyaml tenacity && python /tmp/license_scripts/pull_licenses_py.py ; fi' returned a non-zero code: 1 > Task :sdks:python:container:py39:docker FAILED > Task :sdks:python:container:py38:docker [91mTraceback (most recent call last): File "/tmp/license_scripts/pull_licenses_py.py", line 166, in <module> dependencies = run_pip_licenses() File "/tmp/license_scripts/pull_licenses_py.py", line 49, in run_pip_licenses dependencies = run_bash_command(command) File "/tmp/license_scripts/pull_licenses_py.py", line 44, in run_bash_command return subprocess.check_output(command.split()).decode('utf-8') File "/usr/local/lib/python3.8/subprocess.py", line 415, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "/usr/local/lib/python3.8/subprocess.py", line 516, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['pip-licenses', '--with-license-file', '--with-urls', '--from=mixed', '--ignore', 'apache-beam', '--format=json']' returned non-zero exit status 1. [0m > Task :sdks:python:container:py37:docker [91mTraceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/piplicenses.py", line 40, in <module> from pip._internal.utils.misc import get_installed_distributions ImportError: cannot import name 'get_installed_distributions' from 'pip._internal.utils.misc' (/usr/local/lib/python3.7/site-packages/pip/_internal/utils/misc.py) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/bin/pip-licenses", line 5, in <module> from piplicenses import main File "/usr/local/lib/python3.7/site-packages/piplicenses.py", line 42, in <module> from pip import get_installed_distributions ImportError: cannot import name 'get_installed_distributions' from 'pip' (/usr/local/lib/python3.7/site-packages/pip/__init__.py) [0m[91mTraceback (most recent call last): File "/tmp/license_scripts/pull_licenses_py.py", line 166, in <module> dependencies = run_pip_licenses() File "/tmp/license_scripts/pull_licenses_py.py", line 49, in run_pip_licenses dependencies = run_bash_command(command) File "/tmp/license_scripts/pull_licenses_py.py", line 44, in run_bash_command return subprocess.check_output(command.split()).decode('utf-8') File "/usr/local/lib/python3.7/subprocess.py", line 411, in check_output **kwargs).stdout File "/usr/local/lib/python3.7/subprocess.py", line 512, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['pip-licenses', '--with-license-file', '--with-urls', '--from=mixed', '--ignore', 'apache-beam', '--format=json']' returned non-zero exit status 1. [0m > Task :sdks:python:container:py36:docker Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/site-packages (from tenacity) (1.15.0) Building wheels for collected packages: PTable Building wheel for PTable (setup.py): started > Task :sdks:python:container:py38:docker FAILED The command '/bin/sh -c if [ "$pull_licenses" = "true" ] ; then pip install 'pip-licenses<3.0.0' pyyaml tenacity && python /tmp/license_scripts/pull_licenses_py.py ; fi' returned a non-zero code: 1 > Task :sdks:python:container:py37:docker FAILED The command '/bin/sh -c if [ "$pull_licenses" = "true" ] ; then pip install 'pip-licenses<3.0.0' pyyaml tenacity && python /tmp/license_scripts/pull_licenses_py.py ; fi' returned a non-zero code: 1 > Task :sdks:python:test-suites:dataflow:py36:validatesContainer # Where to store integration test outputs. GCS_LOCATION=${GCS_LOCATION:-gs://temp-storage-for-end-to-end-tests} # Project for the container and integration test PROJECT=${PROJECT:-apache-beam-testing} REGION=${REGION:-us-central1} IMAGE_PREFIX="$(grep 'docker_image_default_repo_prefix' gradle.properties | cut -d'=' -f2)" SDK_VERSION="$(grep 'sdk_version' gradle.properties | cut -d'=' -f2)" PY_VERSION=$1 IMAGE_NAME="${IMAGE_PREFIX}python${PY_VERSION}_sdk" CONTAINER_PROJECT="sdks:python:container:py${PY_VERSION//.}" # Note: we substitute away the dot in the version. PY_INTERPRETER="python${PY_VERSION}" XUNIT_FILE="pytest-$IMAGE_NAME.xml" # Verify in the root of the repository test -d sdks/python/container # Verify docker and gcloud commands exist command -v docker command -v gcloud docker -v gcloud -v # Verify docker image has been built. docker images | grep "apache/$IMAGE_NAME" | grep "$SDK_VERSION" TAG=$(date +%Y%m%d-%H%M%S%N) CONTAINER=us.gcr.io/$PROJECT/$USER/$IMAGE_NAME PREBUILD_SDK_CONTAINER_REGISTRY_PATH=us.gcr.io/$PROJECT/$USER/prebuild_python${PY_VERSION//.}_sdk echo "Using container $CONTAINER" # Tag the docker container. docker tag "apache/$IMAGE_NAME:$SDK_VERSION" "$CONTAINER:$TAG" # Push the container. gcloud docker -- push $CONTAINER:$TAG WARNING: `gcloud docker` will not be supported for Docker client versions above 18.03. As an alternative, use `gcloud auth configure-docker` to configure `docker` to use `gcloud` as a credential helper, then use `docker` as you would for non-GCR registries, e.g. `docker pull gcr.io/project-id/my-image`. Add `--verbosity=error` to silence this warning: `gcloud docker --verbosity=error -- pull gcr.io/project-id/my-image`. See: https://cloud.google.com/container-registry/docs/support/deprecation-notices#gcloud-docker function cleanup_container { # Delete the container locally and remotely docker rmi $CONTAINER:$TAG || echo "Failed to remove container image" for image in $(docker images --format '{{.Repository}}:{{.Tag}}' | grep $PREBUILD_SDK_CONTAINER_REGISTRY_PATH) do docker rmi $image || echo "Failed to remove prebuilt sdk container image" done gcloud --quiet container images delete $CONTAINER:$TAG || echo "Failed to delete container" for digest in $(gcloud container images list-tags $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk --format="get(digest)") do gcloud container images delete $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk@$digest --force-delete-tags --quiet || echo "Failed to remove prebuilt sdk container image" done echo "Removed the container" } trap cleanup_container EXIT echo ">>> Successfully built and push container $CONTAINER" cd sdks/python SDK_LOCATION=$2 # Run ValidatesRunner tests on Google Cloud Dataflow service echo ">>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST" pytest -o junit_suite_name=$IMAGE_NAME \ -m="it_validatescontainer" \ --show-capture=no \ --numprocesses=1 \ --timeout=900 \ --junitxml=$XUNIT_FILE \ --ignore-glob '.*py3\d?\.py$' \ --log-cli-level=INFO \ --test-pipeline-options=" \ --runner=TestDataflowRunner \ --project=$PROJECT \ --region=$REGION \ --sdk_container_image=$CONTAINER:$TAG \ --staging_location=$GCS_LOCATION/staging-validatesrunner-test \ --temp_location=$GCS_LOCATION/temp-validatesrunner-test \ --output=$GCS_LOCATION/output \ --sdk_location=$SDK_LOCATION \ --num_workers=1 \ --docker_registry_push_url=$PREBUILD_SDK_CONTAINER_REGISTRY_PATH" echo ">>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST" cleanup_container Digests: - us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk@sha256:df3bbe462b5f42534d74c4c4c312fccf6f806c654d08dd639b7c7f91dad0516f Associated tags: - 20220318-001635787377576 Tags: - us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20220318-001635787377576 Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20220318-001635787377576]. Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk@sha256:df3bbe462b5f42534d74c4c4c312fccf6f806c654d08dd639b7c7f91dad0516f]. Digests: - us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:efd2b53173e84e01712aba1cb5cf4d863b1a0a524db24e21ece1ac73f56450a9 Associated tags: - 44306aef-85f9-4b21-8c97-a68934025ba2 Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk:44306aef-85f9-4b21-8c97-a68934025ba2]. Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:efd2b53173e84e01712aba1cb5cf4d863b1a0a524db24e21ece1ac73f56450a9]. Digests: - us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:979e117a03c7a0a6c0f9658d389c1d1febac58ba5ab84795e076a1498de5acd6 Associated tags: - 49786744-442c-43c6-9f56-e3c31f9cc939 Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk:49786744-442c-43c6-9f56-e3c31f9cc939]. Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:979e117a03c7a0a6c0f9658d389c1d1febac58ba5ab84795e076a1498de5acd6]. FAILURE: Build completed with 3 failures. 1: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':sdks:python:container:py39:docker'. > Process 'command 'docker'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. ============================================================================== 2: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':sdks:python:container:py38:docker'. > Process 'command 'docker'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. ============================================================================== 3: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':sdks:python:container:py37:docker'. > Process 'command 'docker'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. ============================================================================== * Get more help at https://help.gradle.org BUILD FAILED in 52m 45s 59 actionable tasks: 48 executed, 4 from cache, 7 up-to-date Publishing build scan... https://gradle.com/s/r4ctidge4e6y6 Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
