See
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/62/display/redirect?page=changes>
Changes:
[Valentyn Tymofieiev] Add a container for Python 3.9.
[Valentyn Tymofieiev] Allow job submission with Python 3.9 on Dataflow runner
[Valentyn Tymofieiev] Add Python 3.9 test suites. Keep Dataflow V1 suites
unchanged for now.
[Valentyn Tymofieiev] Add py3.9 Github actions suites.
[Valentyn Tymofieiev] Py39 Doc updates.
[Valentyn Tymofieiev] [BEAM-9980] Simplify run_validates_container.sh to avoid
branching.
[Valentyn Tymofieiev] Update Cython to a new version that has py39 wheels.
[Valentyn Tymofieiev] [BEAM-13845] Fix comparison with potentially incomparable
default
[Valentyn Tymofieiev] [BEAM-12920] Assume that bare generators types define
simple generators.
[Valentyn Tymofieiev] Mark Python 3.9 as supported version.
[Valentyn Tymofieiev] [BEAM-13868] Remove gsutil dep from hdfs IT test.
[noreply] [BEAM-13867] Drop NaNs returned by nlargest in flight_delays example
[noreply] Announce Python 3.9 in CHANGES.md (#16802)
[Brian Hulette] Moving to 2.38.0-SNAPSHOT on master branch.
[noreply] [BEAM-11095] Better error handling for iter/reiter/multimap (#16794)
------------------------------------------
[...truncated 40.48 KB...]
Collecting importlib-resources>=1.0
Using cached importlib_resources-5.4.0-py3-none-any.whl (28 kB)
Collecting platformdirs<3,>=2
Using cached platformdirs-2.4.0-py3-none-any.whl (14 kB)
Installing collected packages: zipp, six, pyparsing, platformdirs,
importlib-resources, importlib-metadata, filelock, distlib, virtualenv, toml,
setuptools, py, protobuf, pluggy, packaging, grpcio, wheel, tox, mypy-protobuf,
grpcio-tools
Attempting uninstall: setuptools
Found existing installation: setuptools 40.6.2
Uninstalling setuptools-40.6.2:
Successfully uninstalled setuptools-40.6.2
Successfully installed distlib-0.3.1 filelock-3.4.1 grpcio-1.43.0
grpcio-tools-1.37.0 importlib-metadata-2.1.3 importlib-resources-5.4.0
mypy-protobuf-1.18 packaging-21.3 platformdirs-2.4.0 pluggy-1.0.0
protobuf-3.19.4 py-1.11.0 pyparsing-3.0.7 setuptools-59.6.0 six-1.16.0
toml-0.10.2 tox-3.20.1 virtualenv-20.13.1 wheel-0.37.1 zipp-3.6.0
> Task :sdks:java:container:pullLicenses
Collecting pip
Using cached
https://files.pythonhosted.org/packages/a4/6d/6463d49a933f547439d6b5b98b46af8742cc03ae83543e4d7688c2420f8b/pip-21.3.1-py3-none-any.whl
Collecting setuptools
Using cached
https://files.pythonhosted.org/packages/b0/3a/88b210db68e56854d0bcf4b38e165e03be377e13907746f825790f3df5bf/setuptools-59.6.0-py3-none-any.whl
Collecting wheel
Using cached
https://files.pythonhosted.org/packages/27/d6/003e593296a85fd6ed616ed962795b2f87709c3eee2bca4f6d0fe55c6d00/wheel-0.37.1-py2.py3-none-any.whl
Installing collected packages: pip, setuptools, wheel
Found existing installation: pip 18.1
Uninstalling pip-18.1:
Successfully uninstalled pip-18.1
Found existing installation: setuptools 40.6.2
Uninstalling setuptools-40.6.2:
Successfully uninstalled setuptools-40.6.2
Successfully installed pip-21.3.1 setuptools-59.6.0 wheel-0.37.1
Collecting beautifulsoup4<5.0,>=4.9.0
Using cached beautifulsoup4-4.10.0-py3-none-any.whl (97 kB)
Collecting pyyaml<6.0.0,>=3.12
Using cached PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl (640 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting soupsieve>1.2
Using cached soupsieve-2.3.1-py3-none-any.whl (37 kB)
Collecting six>=1.9.0
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: soupsieve, six, tenacity, pyyaml, beautifulsoup4
Successfully installed beautifulsoup4-4.10.0 pyyaml-5.4.1 six-1.16.0
soupsieve-2.3.1 tenacity-5.1.5
Executing
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/ws/src/sdks/java/container/build/virtualenv/bin/python>
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/ws/src/sdks/java/container/license_scripts/pull_licenses_java.py>
--license_index=<https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/ws/src/sdks/java/container/build/reports/dependency-license/index.json>
--output_dir=<https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/ws/src/sdks/java/container/build/target/java_third_party_licenses>
--dep_url_yaml=<https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/ws/src/sdks/java/container/license_scripts/dep_urls_java.yaml>
--manual_license_path=<https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/ws/src/sdks/java/container/license_scripts/manual_licenses>
INFO:root:Pulling license for 220 dependencies using 16 threads.
> Task :sdks:python:sdist
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/ws/src/build/gradleenv/1922375555/lib/python3.6/site-packages/setuptools/dist.py>:493:
UserWarning: Normalizing '2.38.0.dev' to '2.38.0.dev0'
warnings.warn(tmpl.format(**locals()))
INFO:gen_protos:Regenerating Python proto definitions (no output files).
INFO:gen_protos:Found protoc_gen_mypy at
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/ws/src/build/gradleenv/1922375555/bin/protoc-gen-mypy>
beam_interactive_api.proto:36:1: warning: Import
google/protobuf/timestamp.proto is unused.
Writing mypy to external_transforms_pb2.pyi
Writing mypy to beam_runner_api_pb2.pyi
Writing mypy to standard_window_fns_pb2.pyi
Writing mypy to metrics_pb2.pyi
Writing mypy to endpoints_pb2.pyi
Writing mypy to schema_pb2.pyi
Writing mypy to beam_artifact_api_pb2.pyi
Writing mypy to beam_expansion_api_pb2.pyi
Writing mypy to beam_job_api_pb2.pyi
Writing mypy to beam_provision_api_pb2.pyi
Writing mypy to beam_fn_api_pb2.pyi
Writing mypy to beam_interactive_api_pb2.pyi
INFO:gen_protos:Writing urn stubs:
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/ws/src/sdks/python/apache_beam/portability/api/metrics_pb2_urns.py>
INFO:gen_protos:Writing urn stubs:
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/ws/src/sdks/python/apache_beam/portability/api/beam_artifact_api_pb2_urns.py>
INFO:gen_protos:Writing urn stubs:
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/ws/src/sdks/python/apache_beam/portability/api/standard_window_fns_pb2_urns.py>
INFO:gen_protos:Writing urn stubs:
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/ws/src/sdks/python/apache_beam/portability/api/external_transforms_pb2_urns.py>
INFO:gen_protos:Writing urn stubs:
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/ws/src/sdks/python/apache_beam/portability/api/beam_runner_api_pb2_urns.py>
warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
warning: no files found matching 'LICENSE'
warning: no files found matching 'LICENSE.python'
warning: sdist: standard file not found: should have one of README, README.rst,
README.txt, README.md
> Task :sdks:python:container:py36:copyDockerfileDependencies
> Task :sdks:java:container:pullLicenses
INFO:root:Replaced local file URL with
file://<https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/ws/src/sdks/java/container/license_scripts/manual_licenses/xz/COPYING>
for xz-1.5
INFO:root:pull_licenses_java.py succeed. It took 9.341037 seconds with 16
threads.
Copying licenses from
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/ws/src/sdks/java/container/build/target/java_third_party_licenses>
to
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/ws/src/sdks/java/container/build/target/third_party_licenses.>
Finished license_scripts.sh
> Task :sdks:python:installGcpTest
Processing ./build/apache-beam.tar.gz
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Collecting crcmod<2.0,>=1.7
Using cached crcmod-1.7-cp36-cp36m-linux_x86_64.whl
Collecting orjson<4.0
Using cached orjson-3.6.1-cp36-cp36m-manylinux_2_24_x86_64.whl (233 kB)
Collecting dill<0.3.2,>=0.3.1.1
Using cached dill-0.3.1.1-py3-none-any.whl
Collecting cloudpickle<3,>=2.0.0
Using cached cloudpickle-2.0.0-py3-none-any.whl (25 kB)
Collecting fastavro<2,>=0.23.6
Using cached
fastavro-1.4.7-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3
MB)
Requirement already satisfied: grpcio<2,>=1.29.0 in
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/ws/src/build/gradleenv/1922375555/lib/python3.6/site-packages>
(from apache-beam==2.38.0.dev0) (1.43.0)
Collecting hdfs<3.0.0,>=2.1.0
Using cached hdfs-2.6.0-py3-none-any.whl (33 kB)
Collecting httplib2<0.20.0,>=0.8
Using cached httplib2-0.19.1-py3-none-any.whl (95 kB)
Collecting numpy<1.22.0,>=1.14.3
Using cached numpy-1.19.5-cp36-cp36m-manylinux2010_x86_64.whl (14.8 MB)
> Task :sdks:java:container:java8:copyJavaThirdPartyLicenses
> Task :release:go-licenses:java:dockerRun
+ go-licenses save github.com/apache/beam/sdks/java/container
--save_path=/output/licenses
> Task :release:go-licenses:py:dockerRun
+ go-licenses save github.com/apache/beam/sdks/python/container
--save_path=/output/licenses
+ go-licenses csv github.com/apache/beam/sdks/python/container
+ tee /output/licenses/list.csv
golang.org/x/text,https://go.googlesource.com/text/+/refs/heads/master/LICENSE,BSD-3-Clause
+ chmod -R a+w /output/licenses
github.com/nightlyone/lockfile,https://github.com/nightlyone/lockfile/blob/master/LICENSE,MIT
golang.org/x/net,https://go.googlesource.com/net/+/refs/heads/master/LICENSE,BSD-3-Clause
github.com/golang/protobuf,https://github.com/golang/protobuf/blob/master/LICENSE,BSD-3-Clause
google.golang.org/protobuf,https://go.googlesource.com/protobuf/+/refs/heads/master/LICENSE,BSD-3-Clause
google.golang.org/genproto/googleapis/rpc/status,https://github.com/googleapis/go-genproto/blob/master/LICENSE,Apache-2.0
github.com/apache/beam/sdks/python/container,https://github.com/apache/beam/blob/master/LICENSE,Apache-2.0
github.com/apache/beam/sdks/go/pkg/beam,https://github.com/apache/beam/blob/master/sdks/go/README.md,Apache-2.0
google.golang.org/grpc,https://github.com/grpc/grpc-go/blob/master/LICENSE,Apache-2.0
golang.org/x/sys,https://go.googlesource.com/sys/+/refs/heads/master/LICENSE,BSD-3-Clause
> Task :release:go-licenses:java:dockerRun
+ go-licenses csv github.com/apache/beam/sdks/java/container
+ tee /output/licenses/list.csv
github.com/apache/beam/sdks/go/pkg/beam,https://github.com/apache/beam/blob/master/sdks/go/README.md,Apache-2.0
google.golang.org/grpc,https://github.com/grpc/grpc-go/blob/master/LICENSE,Apache-2.0
github.com/golang/protobuf,https://github.com/golang/protobuf/blob/master/LICENSE,BSD-3-Clause
golang.org/x/text,https://go.googlesource.com/text/+/refs/heads/master/LICENSE,BSD-3-Clause
github.com/apache/beam/sdks/java/container,https://github.com/apache/beam/blob/master/LICENSE,Apache-2.0
golang.org/x/net,https://go.googlesource.com/net/+/refs/heads/master/LICENSE,BSD-3-Clause
google.golang.org/protobuf,https://go.googlesource.com/protobuf/+/refs/heads/master/LICENSE,BSD-3-Clause
google.golang.org/genproto/googleapis/rpc/status,https://github.com/googleapis/go-genproto/blob/master/LICENSE,Apache-2.0
golang.org/x/sys,https://go.googlesource.com/sys/+/refs/heads/master/LICENSE,BSD-3-Clause
+ chmod -R a+w /output/licenses
> Task :release:go-licenses:java:createLicenses
> Task :sdks:java:container:java8:copyGolangLicenses
> Task :release:go-licenses:py:createLicenses
> Task :sdks:java:container:java8:dockerPrepare
> Task :sdks:python:container:py36:copyGolangLicenses
> Task :sdks:python:container:py36:dockerPrepare
> Task :sdks:java:container:java8:docker
> Task :runners:google-cloud-dataflow-java:buildAndPushDockerJavaContainer
WARNING: `gcloud docker` will not be supported for Docker client versions above
18.03.
As an alternative, use `gcloud auth configure-docker` to configure `docker` to
use `gcloud` as a credential helper, then use `docker` as you would for non-GCR
registries, e.g. `docker pull gcr.io/project-id/my-image`. Add
`--verbosity=error` to silence this warning: `gcloud docker
--verbosity=error -- pull gcr.io/project-id/my-image`.
See:
https://cloud.google.com/container-registry/docs/support/deprecation-notices#gcloud-docker
> Task :sdks:python:container:py36:docker
> Task :runners:google-cloud-dataflow-java:buildAndPushDockerPythonContainer
WARNING: `gcloud docker` will not be supported for Docker client versions above
18.03.
As an alternative, use `gcloud auth configure-docker` to configure `docker` to
use `gcloud` as a credential helper, then use `docker` as you would for non-GCR
registries, e.g. `docker pull gcr.io/project-id/my-image`. Add
`--verbosity=error` to silence this warning: `gcloud docker
--verbosity=error -- pull gcr.io/project-id/my-image`.
See:
https://cloud.google.com/container-registry/docs/support/deprecation-notices#gcloud-docker
> Task :runners:google-cloud-dataflow-java:setupXVR
> Task :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerSetup
> Task
> :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava
> Task
> :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava
> FAILED
> Task :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerCleanup
> Task :runners:google-cloud-dataflow-java:cleanupXVR UP-TO-DATE
> Task :runners:google-cloud-dataflow-java:cleanUpDockerPythonImages
Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/python:20220210061208]
- referencing digest:
[us.gcr.io/apache-beam-testing/java-postcommit-it/python@sha256:9897896d284c0d05898a62c9fd8bb6e2357a7f04eda994c32ddb4984a654422a]
Deleted
[[us.gcr.io/apache-beam-testing/java-postcommit-it/python:20220210061208]
(referencing
[us.gcr.io/apache-beam-testing/java-postcommit-it/python@sha256:9897896d284c0d05898a62c9fd8bb6e2357a7f04eda994c32ddb4984a654422a])].
Digests:
-
us.gcr.io/apache-beam-testing/java-postcommit-it/python@sha256:5a8e2784174855116ed89222e61f5fcecb0d18850a3743f739f6d0955c3cecef
ERROR: (gcloud.container.images.delete) Not found: response:
{'docker-distribution-api-version': 'registry/2.0', 'content-type':
'application/json', 'date': 'Thu, 10 Feb 2022 06:26:34 GMT', 'server': 'Docker
Registry', 'cache-control': 'private', 'x-xss-protection': '0',
'x-frame-options': 'SAMEORIGIN', 'transfer-encoding': 'chunked', 'status':
'404', 'content-length': '168', '-content-encoding': 'gzip'}
Failed to compute blob liveness for manifest:
'sha256:5a8e2784174855116ed89222e61f5fcecb0d18850a3743f739f6d0955c3cecef': None
> Task :runners:google-cloud-dataflow-java:cleanUpDockerPythonImages FAILED
> Task :runners:google-cloud-dataflow-java:cleanUpDockerJavaImages
Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220210061208]
- referencing digest:
[us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:a973447abcbdfd44479e3b09b75e00953e7b4dc03def50d634171857c5847f01]
Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220210061208]
(referencing
[us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:a973447abcbdfd44479e3b09b75e00953e7b4dc03def50d634171857c5847f01])].
Digests:
-
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:a973447abcbdfd44479e3b09b75e00953e7b4dc03def50d634171857c5847f01
Deleted
[us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:a973447abcbdfd44479e3b09b75e00953e7b4dc03def50d634171857c5847f01].
Digests:
-
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:6b88deaf66be3441dbc337eedbb33dabe7617db8ba2e39f493a6c2019071f19d
Deleted
[us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:6b88deaf66be3441dbc337eedbb33dabe7617db8ba2e39f493a6c2019071f19d].
FAILURE: Build completed with 2 failures.
1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task
':runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================
2: Task failed with an exception.
-----------
* Where:
Build file
'<https://ci-beam.apache.org/job/beam_PostCommit_XVR_PythonUsingJava_Dataflow/ws/src/runners/google-cloud-dataflow-java/build.gradle'>
line: 338
* What went wrong:
Execution failed for task
':runners:google-cloud-dataflow-java:cleanUpDockerPythonImages'.
> Process 'command './scripts/cleanup_untagged_gcr_images.sh'' finished with
> non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings
and determine if they come from your own scripts or plugins.
See
https://docs.gradle.org/7.3.2/userguide/command_line_interface.html#sec:command_line_warnings
Execution optimizations have been disabled for 1 invalid unit(s) of work during
this build to ensure correctness.
Please consult deprecation warnings for more details.
BUILD FAILED in 15m 3s
113 actionable tasks: 78 executed, 29 from cache, 6 up-to-date
Publishing build scan...
https://gradle.com/s/ft3kxdkktoo7y
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
