See <https://ci-beam.apache.org/job/beam_Dependency_Check/347/display/redirect?page=changes>
Changes: [baeminbo] [BEAM-6721] Set numShards dynamically for TextIO.write() [Luke Cwik] [BEAM-12522] Enable side inputs on all splittable DoFn execution time [heejong] [BEAM-12978] Customizable dependency for Java external transform [melissapa] [BEAM-11758] Update basics page: Splittable DoFn [heejong] add ConfigT param to getDependencies method [aydar.zaynutdinov] [BEAM-13062][Playground] [Alexey Romanenko] [BEAM-13104] ParquetIO: SplitReadFn must read the whole block [ilya.kozyrev] Add environment_service.go and structures for beam sdk, network envs, [aydar.zaynutdinov] [BEAM-13062][Playground] [noreply] Merge pull request #15772 from [BEAM-13032] [Playground] Implement the [noreply] Merge pull request #15714 from [BEAM-13005] [Playground] Implement local [noreply] Merge pull request #15770 from [BEAM-13095][Playground] Using working [noreply] [BEAM-11758] Update basics page: Aggregation, Runner, UDF, Schema [noreply] Merge pull request #15783 from [BEAM-13048] [Playground] Add shortcuts [Luke Cwik] [BEAM-8543] Fix test filtering for Dataflow Runner V2 to exclude [relax] support json [noreply] Merge pull request #15744 from [BEAM-13072][Playground] Executor builder [aydar.zaynutdinov] [BEAM-12970][Playground] [noreply] Change sql.Options to an interface under sqlx. (#15790) [melissapa] Address review feedback [Luke Cwik] [BEAM-13015] Migrate bundle processing in the SDK harness to using [noreply] [BEAM-4149] Ensure that we always provide and require the worker id. [Andrew Pilloud] [BEAM-13118] Don't force push to master [noreply] [BEAM-13098] Fix translation of repeated TableRow fields (#15779) [aydar.zaynutdinov] [BEAM-12970][Playground] [eugene.nikolayev] Fix Python docs build pre-commit failures. [mmack] [BEAM-8958] Use AWS credentials provider with BasicKinesisProvider (AWS [aydar.zaynutdinov] [BEAM-12970][Playground] [noreply] Merge pull request #15731: [BEAM-13067] Mark GroupIntoBatches output as [noreply] [BEAM-13066, BEAM-13087] Workaround for coder type combiner packing [noreply] [BEAM-13066] Disable using abstract iterable by default. (#15805) [aydar.zaynutdinov] [BEAM-12970][Playground] [artur.khanin] Implemented GetCompileOutput and covered with unit tests [artur.khanin] Replaced logging library regarding PR comment [daria.malkova] fix executor tests [mmack] [BEAM-13138] Update code to retrieve container endpoint [aydar.zaynutdinov] [BEAM-13120][Playground] [noreply] Merge pull request #15709 from [BEAM-12967] [Playground] Create Example [noreply] Merge pull request #15294 from [BEAM-11986] Spanner write metric [noreply] [BEAM-13130] Remove persistent references to stateKeyReaders (#15815) [Brian Hulette] [BEAM-13099] Replace call to RelNode.metadata with BeamRelMetadataQuery [Brian Hulette] [BEAM-13099] Update all rels to use BeamRelMetadataQuery [Kyle Weaver] [BEAM-13143] Fix python doc generator error. [Pablo Estrada] Handle runner-provided shards for TextIO [noreply] Merge pull request #15721 from [BEAM-13023][Playground] Implement Redis [noreply] Merge pull request #15802 from [BEAM-12970][Playground] Implement gRPC [noreply] Merge pull request #15800 from [BEAM-12970][Playground] Implement gRPC [noreply] [BEAM-8958] Use AWS credentials provider with BasicKinesisProvider [Brian Hulette] [BEAM-13099] Copy vendored calcite build for 1.28.0 [Brian Hulette] [BEAM-13099] Modifications for vendor/calcite-1_28_0 build [noreply] [BEAM-13099] Use BlockBuilder.add(..) rather than [noreply] Minor: Remove broken python compatibility checks (#15828) [noreply] [BEAM-12047] Updates CHANGES.md to mention the URN convention (#15845) [noreply] [BEAM-13001] DoFn metrics for Go SDK (#15657) [noreply] Merge pull request #15753 from [BEAM-13080] Add option in Reshuffle to ------------------------------------------ [...truncated 72.84 KB...] 5 actionable tasks: 5 executed Publishing build scan... https://gradle.com/s/qtwbydx2buuek Build step 'Invoke Gradle script' changed build result to SUCCESS [beam_Dependency_Check] $ /bin/bash -xe /tmp/jenkins8459596704972431746.sh + cd src + bash .test-infra/jenkins/dependency_check/generate_report.sh REPORT_DESCRIPTION=" <h4> A dependency update is high priority if it satisfies one of following criteria: </h4> <ul> <li> It has major versions update available, e.g. org.assertj:assertj-core 2.5.0 -> 3.10.0; </li> </ul> <ul> <li> It is over 3 minor versions behind the latest version, e.g. org.tukaani:xz 1.5 -> 1.8; </li> </ul> <ul> <li> The current version is behind the later version for over 180 days, e.g. com.google.auto.service:auto-service 2014-10-24 -> 2017-12-11. </li> </ul> <h4> In Beam, we make a best-effort attempt at keeping all dependencies up-to-date. In the future, issues will be filed and tracked for these automatically, but in the meantime you can search for existing issues or open a new one. </h4> <h4> For more information: <a href=\"https://beam.apache.org/contribute/dependencies/\";> Beam Dependency Guide </a></h4>" # Virtualenv for the rest of the script to run setup virtualenv dependency/check Using base prefix '/usr' New python executable in <https://ci-beam.apache.org/job/beam_Dependency_Check/ws/src/dependency/check/bin/python3> Also creating executable in <https://ci-beam.apache.org/job/beam_Dependency_Check/ws/src/dependency/check/bin/python> Installing setuptools, pip, wheel... done. . dependency/check/bin/activate # This file must be used with "source bin/activate" *from bash* # you cannot run it directly deactivate () { unset -f pydoc >/dev/null 2>&1 # reset old environment variables # ! [ -z ${VAR+_} ] returns true if VAR is declared at all if ! [ -z "${_OLD_VIRTUAL_PATH+_}" ] ; then PATH="$_OLD_VIRTUAL_PATH" export PATH unset _OLD_VIRTUAL_PATH fi if ! [ -z "${_OLD_VIRTUAL_PYTHONHOME+_}" ] ; then PYTHONHOME="$_OLD_VIRTUAL_PYTHONHOME" export PYTHONHOME unset _OLD_VIRTUAL_PYTHONHOME fi # This should detect bash and zsh, which have a hash command that must # be called to get it to forget past commands. Without forgetting # past commands the $PATH changes we made may not be respected if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then hash -r 2>/dev/null fi if ! [ -z "${_OLD_VIRTUAL_PS1+_}" ] ; then PS1="$_OLD_VIRTUAL_PS1" export PS1 unset _OLD_VIRTUAL_PS1 fi unset VIRTUAL_ENV if [ ! "${1-}" = "nondestructive" ] ; then # Self destruct! unset -f deactivate fi } # unset irrelevant variables deactivate nondestructive VIRTUAL_ENV="<https://ci-beam.apache.org/job/beam_Dependency_Check/ws/src/dependency/check";> export VIRTUAL_ENV _OLD_VIRTUAL_PATH="$PATH" PATH="$VIRTUAL_ENV/bin:$PATH" export PATH # unset PYTHONHOME if set if ! [ -z "${PYTHONHOME+_}" ] ; then _OLD_VIRTUAL_PYTHONHOME="$PYTHONHOME" unset PYTHONHOME fi if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT-}" ] ; then _OLD_VIRTUAL_PS1="${PS1-}" if [ "x" != x ] ; then PS1="${PS1-}" else PS1="(`basename \"$VIRTUAL_ENV\"`) ${PS1-}" fi export PS1 fi basename "$VIRTUAL_ENV" # Make sure to unalias pydoc if it's already there alias pydoc 2>/dev/null >/dev/null && unalias pydoc || true pydoc () { python -m pydoc "$@" } # This should detect bash and zsh, which have a hash command that must # be called to get it to forget past commands. Without forgetting # past commands the $PATH changes we made may not be respected if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then hash -r 2>/dev/null fi pip install --upgrade google-cloud-bigquery DEPRECATION: Python 3.5 reached the end of its life on September 13th, 2020. Please upgrade your Python as Python 3.5 is no longer maintained. pip 21.0 will drop support for Python 3.5 in January 2021. pip 21.0 will remove support for this functionality. Collecting google-cloud-bigquery Downloading google_cloud_bigquery-1.28.0-py2.py3-none-any.whl (187 kB) Collecting google-api-core<2.0dev,>=1.21.0 Downloading google_api_core-1.23.0-py2.py3-none-any.whl (91 kB) Collecting google-cloud-core<2.0dev,>=1.4.1 Downloading google_cloud_core-1.4.4-py2.py3-none-any.whl (27 kB) Collecting google-resumable-media<2.0dev,>=0.6.0 Downloading google_resumable_media-1.1.0-py2.py3-none-any.whl (75 kB) Collecting six<2.0.0dev,>=1.13.0 Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) Collecting googleapis-common-protos<2.0dev,>=1.6.0 Downloading googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB) Collecting protobuf>=3.12.0 Using cached protobuf-3.19.1-py2.py3-none-any.whl (162 kB) Collecting google-auth<2.0dev,>=1.21.1 Downloading google_auth-1.23.0-py2.py3-none-any.whl (114 kB) Collecting requests<3.0.0dev,>=2.18.0 Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB) Collecting pytz Using cached pytz-2021.3-py2.py3-none-any.whl (503 kB) Requirement already satisfied: setuptools>=34.0.0 in ./dependency/check/lib/python3.5/site-packages (from google-api-core<2.0dev,>=1.21.0->google-cloud-bigquery) (50.3.2) Collecting pyasn1-modules>=0.2.1 Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB) Collecting cachetools<5.0,>=2.0.0 Using cached cachetools-4.2.4-py3-none-any.whl (10 kB) Collecting rsa<5,>=3.1.4 Using cached rsa-4.7.2-py3-none-any.whl (34 kB) Collecting google-crc32c<2.0dev,>=1.0 Downloading google_crc32c-1.0.0-cp35-cp35m-manylinux2010_x86_64.whl (39 kB) Collecting cffi>=1.0.0 Downloading cffi-1.15.0.tar.gz (484 kB) Collecting pycparser Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB) Collecting pyasn1<0.5.0,>=0.4.6 Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB) Collecting certifi>=2017.4.17 Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB) Collecting chardet<5,>=3.0.2 Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB) Collecting urllib3<1.27,>=1.21.1 Using cached urllib3-1.26.7-py2.py3-none-any.whl (138 kB) Collecting idna<3,>=2.5 Downloading idna-2.10-py2.py3-none-any.whl (58 kB) Building wheels for collected packages: cffi Building wheel for cffi (setup.py): started Building wheel for cffi (setup.py): finished with status 'error' ERROR: Command errored out with exit status 1: command: <https://ci-beam.apache.org/job/beam_Dependency_Check/ws/src/dependency/check/bin/python3> -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-i49nunjp/cffi_35e7fd075b1a4f50bc7a3b833cc6bb6d/setup.py'"'"'; __file__='"'"'/tmp/pip-install-i49nunjp/cffi_35e7fd075b1a4f50bc7a3b833cc6bb6d/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-_ljxx7au cwd: /tmp/pip-install-i49nunjp/cffi_35e7fd075b1a4f50bc7a3b833cc6bb6d/ Complete output (34 lines): running bdist_wheel running build running build_py creating build creating build/lib.linux-x86_64-3.5 creating build/lib.linux-x86_64-3.5/cffi copying cffi/verifier.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/model.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/api.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/cparser.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/recompiler.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/commontypes.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/__init__.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/error.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/lock.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.5/cffi copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.5/cffi copying cffi/_embedding.h -> build/lib.linux-x86_64-3.5/cffi copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.5/cffi running build_ext building '_cffi_backend' extension creating build/temp.linux-x86_64-3.5 creating build/temp.linux-x86_64-3.5/c x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python3.5m -I<https://ci-beam.apache.org/job/beam_Dependency_Check/ws/src/dependency/check/include/python3.5m> -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.5/c/_cffi_backend.o c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory compilation terminated. error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 ---------------------------------------- ERROR: Failed building wheel for cffi Running setup.py clean for cffi Failed to build cffi Installing collected packages: pyasn1, urllib3, six, rsa, pycparser, pyasn1-modules, protobuf, idna, chardet, certifi, cachetools, requests, pytz, googleapis-common-protos, google-auth, cffi, google-crc32c, google-api-core, google-resumable-media, google-cloud-core, google-cloud-bigquery Running setup.py install for cffi: started Running setup.py install for cffi: finished with status 'error' ERROR: Command errored out with exit status 1: command: <https://ci-beam.apache.org/job/beam_Dependency_Check/ws/src/dependency/check/bin/python3> -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-i49nunjp/cffi_35e7fd075b1a4f50bc7a3b833cc6bb6d/setup.py'"'"'; __file__='"'"'/tmp/pip-install-i49nunjp/cffi_35e7fd075b1a4f50bc7a3b833cc6bb6d/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-0ua5bq1k/install-record.txt --single-version-externally-managed --compile --install-headers <https://ci-beam.apache.org/job/beam_Dependency_Check/ws/src/dependency/check/include/site/python3.5/cffi> cwd: /tmp/pip-install-i49nunjp/cffi_35e7fd075b1a4f50bc7a3b833cc6bb6d/ Complete output (34 lines): running install running build running build_py creating build creating build/lib.linux-x86_64-3.5 creating build/lib.linux-x86_64-3.5/cffi copying cffi/verifier.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/model.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/api.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/cparser.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/recompiler.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/commontypes.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/__init__.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/error.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/lock.py -> build/lib.linux-x86_64-3.5/cffi copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.5/cffi copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.5/cffi copying cffi/_embedding.h -> build/lib.linux-x86_64-3.5/cffi copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.5/cffi running build_ext building '_cffi_backend' extension creating build/temp.linux-x86_64-3.5 creating build/temp.linux-x86_64-3.5/c x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python3.5m -I<https://ci-beam.apache.org/job/beam_Dependency_Check/ws/src/dependency/check/include/python3.5m> -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.5/c/_cffi_backend.o c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory compilation terminated. error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 ---------------------------------------- ERROR: Command errored out with exit status 1: <https://ci-beam.apache.org/job/beam_Dependency_Check/ws/src/dependency/check/bin/python3> -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-i49nunjp/cffi_35e7fd075b1a4f50bc7a3b833cc6bb6d/setup.py'"'"'; __file__='"'"'/tmp/pip-install-i49nunjp/cffi_35e7fd075b1a4f50bc7a3b833cc6bb6d/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-0ua5bq1k/install-record.txt --single-version-externally-managed --compile --install-headers <https://ci-beam.apache.org/job/beam_Dependency_Check/ws/src/dependency/check/include/site/python3.5/cffi> Check the logs for full command output. Build step 'Execute shell' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
