See
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/8975/display/redirect>
Changes:
------------------------------------------
[...truncated 94.40 KB...]
> Task :sdks:python:test-suites:dataflow:py37:installGcpTest
Successfully installed apache-beam-2.38.0.dev0 atomicwrites-1.4.0 attrs-21.4.0
azure-core-1.23.0 azure-storage-blob-12.10.0 boto3-1.21.21 botocore-1.24.21
cachetools-4.2.4 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.12
cloudpickle-2.0.0 crcmod-1.7 cryptography-36.0.2 deprecation-2.1.0 dill-0.3.1.1
docker-5.0.3 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.10 fasteners-0.17.3
freezegun-1.2.1 google-api-core-1.31.5 google-apitools-0.5.31
google-auth-1.35.0 google-cloud-bigquery-2.34.2
google-cloud-bigquery-storage-2.13.0 google-cloud-bigtable-1.7.0
google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-3.6.2
google-cloud-language-1.3.0 google-cloud-pubsub-2.11.0
google-cloud-pubsublite-1.4.1 google-cloud-recommendations-ai-0.2.0
google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1
google-cloud-vision-1.0.0 google-crc32c-1.3.0 google-resumable-media-2.3.2
googleapis-common-protos-1.56.0 greenlet-1.1.2 grpc-google-iam-v1-0.12.3
grpcio-gcp-0.2.2 grpcio-status-1.44.0 hdfs-2.6.0 httplib2-0.19.1 idna-3.3
isodate-0.6.1 jmespath-1.0.0 mock-2.0.0 more-itertools-8.12.0 msrest-0.6.21
numpy-1.21.5 oauth2client-4.1.3 oauthlib-3.2.0 orjson-3.6.7 overrides-6.1.0
pandas-1.3.5 parameterized-0.7.5 pbr-5.8.1 pluggy-0.13.1 proto-plus-1.20.3
psycopg2-binary-2.9.3 pyarrow-6.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8
pycparser-2.21 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.3 pymysql-1.0.2
pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.4.0 pytest-timeout-1.4.2
pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.3 pyyaml-6.0
requests-2.27.1 requests-oauthlib-1.3.1 requests_mock-1.9.3 rsa-4.8
s3transfer-0.5.2 sqlalchemy-1.4.32 tenacity-5.1.5 testcontainers-3.4.2
typing-extensions-4.1.1 typing-utils-0.1.0 urllib3-1.26.9 wcwidth-0.2.5
websocket-client-1.3.1 wrapt-1.14.0
> Task :release:go-licenses:py:createLicenses
> Task :sdks:python:container:py38:copyGolangLicenses
> Task :sdks:python:container:py39:copyGolangLicenses
> Task :sdks:python:container:py36:copyGolangLicenses
> Task :sdks:python:container:py37:copyGolangLicenses
> Task :sdks:python:container:py39:dockerPrepare
> Task :sdks:python:container:py38:dockerPrepare
> Task :sdks:python:container:py37:dockerPrepare
> Task :sdks:python:container:py36:dockerPrepare
> Task :sdks:python:container:py39:docker
> Task :sdks:python:container:py38:docker
> Task :sdks:python:container:py37:docker
> Task :sdks:python:container:py36:docker
> Task :sdks:python:container:py37:docker
The command '/bin/sh -c if [ "$pull_licenses" = "true" ] ; then pip
install 'pip-licenses<3.0.0' pyyaml tenacity && python
/tmp/license_scripts/pull_licenses_py.py ; fi' returned a non-zero code: 1
> Task :sdks:python:container:py37:docker FAILED
> Task :sdks:python:test-suites:dataflow:py36:validatesContainer
# Where to store integration test outputs.
GCS_LOCATION=${GCS_LOCATION:-gs://temp-storage-for-end-to-end-tests}
# Project for the container and integration test
PROJECT=${PROJECT:-apache-beam-testing}
REGION=${REGION:-us-central1}
IMAGE_PREFIX="$(grep 'docker_image_default_repo_prefix' gradle.properties | cut
-d'=' -f2)"
SDK_VERSION="$(grep 'sdk_version' gradle.properties | cut -d'=' -f2)"
PY_VERSION=$1
IMAGE_NAME="${IMAGE_PREFIX}python${PY_VERSION}_sdk"
CONTAINER_PROJECT="sdks:python:container:py${PY_VERSION//.}" # Note: we
substitute away the dot in the version.
PY_INTERPRETER="python${PY_VERSION}"
XUNIT_FILE="pytest-$IMAGE_NAME.xml"
# Verify in the root of the repository
test -d sdks/python/container
# Verify docker and gcloud commands exist
command -v docker
command -v gcloud
docker -v
gcloud -v
# Verify docker image has been built.
docker images | grep "apache/$IMAGE_NAME" | grep "$SDK_VERSION"
TAG=$(date +%Y%m%d-%H%M%S%N)
CONTAINER=us.gcr.io/$PROJECT/$USER/$IMAGE_NAME
PREBUILD_SDK_CONTAINER_REGISTRY_PATH=us.gcr.io/$PROJECT/$USER/prebuild_python${PY_VERSION//.}_sdk
echo "Using container $CONTAINER"
# Tag the docker container.
docker tag "apache/$IMAGE_NAME:$SDK_VERSION" "$CONTAINER:$TAG"
# Push the container.
gcloud docker -- push $CONTAINER:$TAG
WARNING: `gcloud docker` will not be supported for Docker client versions above
18.03.
As an alternative, use `gcloud auth configure-docker` to configure `docker` to
use `gcloud` as a credential helper, then use `docker` as you would for non-GCR
registries, e.g. `docker pull gcr.io/project-id/my-image`. Add
`--verbosity=error` to silence this warning: `gcloud docker
--verbosity=error -- pull gcr.io/project-id/my-image`.
See:
https://cloud.google.com/container-registry/docs/support/deprecation-notices#gcloud-docker
> Task :sdks:python:container:py38:docker FAILED
The command '/bin/sh -c if [ "$pull_licenses" = "true" ] ; then pip
install 'pip-licenses<3.0.0' pyyaml tenacity && python
/tmp/license_scripts/pull_licenses_py.py ; fi' returned a non-zero code: 1
> Task :sdks:python:container:py39:docker FAILED
The command '/bin/sh -c if [ "$pull_licenses" = "true" ] ; then pip
install 'pip-licenses<3.0.0' pyyaml tenacity && python
/tmp/license_scripts/pull_licenses_py.py ; fi' returned a non-zero code: 1
> Task :sdks:python:test-suites:dataflow:py36:validatesContainer
f021252d0ffd: Pushed
b50503944708: Pushed
20220318-123355167814946: digest:
sha256:45bdf2a2fbde8fb7b9fb11ac149f67eb51dd8c14e686c9c531d6249b6f888412 size:
5361
function cleanup_container {
# Delete the container locally and remotely
docker rmi $CONTAINER:$TAG || echo "Failed to remove container image"
for image in $(docker images --format '{{.Repository}}:{{.Tag}}' | grep
$PREBUILD_SDK_CONTAINER_REGISTRY_PATH)
do docker rmi $image || echo "Failed to remove prebuilt sdk container image"
done
gcloud --quiet container images delete $CONTAINER:$TAG || echo "Failed to
delete container"
for digest in $(gcloud container images list-tags
$PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk
--format="get(digest)")
do gcloud container images delete
$PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk@$digest
--force-delete-tags --quiet || echo "Failed to remove prebuilt sdk container
image"
done
echo "Removed the container"
}
trap cleanup_container EXIT
echo ">>> Successfully built and push container $CONTAINER"
cd sdks/python
>>> Successfully built and push container
>>> us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk
SDK_LOCATION=$2
>>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST
# Run ValidatesRunner tests on Google Cloud Dataflow service
echo ">>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST"
pytest -o junit_suite_name=$IMAGE_NAME \
-m="it_validatescontainer" \
--show-capture=no \
--numprocesses=1 \
--timeout=900 \
--junitxml=$XUNIT_FILE \
--ignore-glob '.*py3\d?\.py$' \
--log-cli-level=INFO \
--test-pipeline-options=" \
--runner=TestDataflowRunner \
--project=$PROJECT \
--region=$REGION \
--sdk_container_image=$CONTAINER:$TAG \
--staging_location=$GCS_LOCATION/staging-validatesrunner-test \
--temp_location=$GCS_LOCATION/temp-validatesrunner-test \
--output=$GCS_LOCATION/output \
--sdk_location=$SDK_LOCATION \
--num_workers=1 \
--docker_registry_push_url=$PREBUILD_SDK_CONTAINER_REGISTRY_PATH"
============================= test session starts ==============================
platform linux -- Python 3.6.15, pytest-4.6.11, py-1.11.0, pluggy-0.13.1
rootdir:
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python,>
inifile: pytest.ini
plugins: xdist-1.34.0, timeout-1.4.2, forked-1.4.0, requests-mock-1.9.3
timeout: 900.0s
timeout method: signal
timeout func_only: False
gw0 I
[gw0] Python 3.6.15 (default, Jan 15 2022, 19:47:13) -- [GCC 9.3.0]
gw0 [4]
scheduling tests via LoadScheduling
apache_beam/examples/wordcount_it_test.py::WordCountIT::test_wordcount_fnapi_it
[gw0] [ 25%] PASSED
apache_beam/examples/wordcount_it_test.py::WordCountIT::test_wordcount_fnapi_it
apache_beam/examples/wordcount_it_test.py::WordCountIT::test_wordcount_it_with_prebuilt_sdk_container_cloud_build
[gw0] [ 50%] PASSED
apache_beam/examples/wordcount_it_test.py::WordCountIT::test_wordcount_it_with_prebuilt_sdk_container_cloud_build
apache_beam/examples/wordcount_it_test.py::WordCountIT::test_wordcount_it_with_prebuilt_sdk_container_local_docker
[gw0] [ 75%] PASSED
apache_beam/examples/wordcount_it_test.py::WordCountIT::test_wordcount_it_with_prebuilt_sdk_container_local_docker
apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py::ExerciseMetricsPipelineTest::test_metrics_fnapi_it
[gw0] [100%] PASSED
apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py::ExerciseMetricsPipelineTest::test_metrics_fnapi_it
=============================== warnings summary ===============================
apache_beam/io/filesystems_test.py:54
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:54:
DeprecationWarning: invalid escape sequence \c
self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf')) # pylint:
disable=anomalous-backslash-in-string
apache_beam/io/filesystems_test.py:62
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:62:
DeprecationWarning: invalid escape sequence \d
self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'), #
pylint: disable=anomalous-backslash-in-string
-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated xml file:
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/pytest-beam_python3.6_sdk.xml>
-
=================== 4 passed, 2 warnings in 1890.15 seconds ====================
echo ">>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST"
>>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST
cleanup_container
Untagged:
us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20220318-123355167814946
Untagged:
us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk@sha256:45bdf2a2fbde8fb7b9fb11ac149f67eb51dd8c14e686c9c531d6249b6f888412
Untagged:
us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk:9f8f6519-5400-42c9-ae7b-2b50ea7f64c0
Untagged:
us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:799daf9a0e94e3e1643d60c48c2e8829edb809b824871c527c3163e8c2ba428d
Deleted: sha256:8113633e81678f71d5756d75dc04d1a57366e75449ab6e3683c8143a23f8e342
Deleted: sha256:979c20fa3e0caa5ab2c639598af39513fde64402ba6a68c9c99a2063cc602801
Deleted: sha256:92df61eadd0e02ae58e7b925bb199f08940ce7809d83fc0f0cf28d41b13a06fb
Deleted: sha256:3ce9fe8529cf21dd346073feec001f9a9f0501cf454e7c9414a34837bb7e39c2
Deleted: sha256:a3d5f56c5a6bc111ff9e81eb6cbe134e717704b8ebd8d3918fc3190e9b9b60a8
Deleted: sha256:3db2f10eaa9fa2b5436dc5959e2ae35524d32ab119134fb33339d246d0a76e55
Deleted: sha256:3ab5b677d8763df8f981b57074ec280fecb6228efa6d936b46422f0a27d3de73
Deleted: sha256:f3556744637d470c8918e89d420704968cc53ac54b86840e62573d57897418e3
Deleted: sha256:cd0506f3ca733177dbf9ef66f16118946b6e7df262d35eb41a25b39b6ae26c72
Deleted: sha256:280e188f8cf8a4f32fc30a9fe234c1ebea89b52a6c1e70771ff636fc931d6b7c
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk@sha256:45bdf2a2fbde8fb7b9fb11ac149f67eb51dd8c14e686c9c531d6249b6f888412
Associated tags:
- 20220318-123355167814946
Tags:
-
us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20220318-123355167814946
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20220318-123355167814946].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk@sha256:45bdf2a2fbde8fb7b9fb11ac149f67eb51dd8c14e686c9c531d6249b6f888412].
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:799daf9a0e94e3e1643d60c48c2e8829edb809b824871c527c3163e8c2ba428d
Associated tags:
- 9f8f6519-5400-42c9-ae7b-2b50ea7f64c0
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk:9f8f6519-5400-42c9-ae7b-2b50ea7f64c0].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:799daf9a0e94e3e1643d60c48c2e8829edb809b824871c527c3163e8c2ba428d].
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:062bc42670c99c1036fff5d90933528f5151530c257ae445a81201ed1b4630d9
Associated tags:
- 3c2a5cf7-e3ca-4bda-a321-6d4abffe8e43
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk:3c2a5cf7-e3ca-4bda-a321-6d4abffe8e43].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:062bc42670c99c1036fff5d90933528f5151530c257ae445a81201ed1b4630d9].
Removed the container
FAILURE: Build completed with 3 failures.
1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':sdks:python:container:py37:docker'.
> Process 'command 'docker'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================
2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':sdks:python:container:py38:docker'.
> Process 'command 'docker'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================
3: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':sdks:python:container:py39:docker'.
> Process 'command 'docker'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================
* Get more help at https://help.gradle.org
BUILD FAILED in 1h 7m 38s
59 actionable tasks: 48 executed, 4 from cache, 7 up-to-date
Publishing build scan...
https://gradle.com/s/f2yk6hatk6wpm
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
