See
<https://builds.apache.org/job/beam_Dependency_Check/244/display/redirect?page=changes>
Changes:
[kirillkozlov] Updated the cost model to favor IO with push-down
[kirillkozlov] BigQueryFilter numSupported method
[dcavazos] [BEAM-7390] Add code snippet for Mean
[nielm] Add limit on number of mutated rows to batching/sorting stages.
[kirillkozlov] Update SQL BigQuery doc
[echauchot] [BEAM-8830] flatten: better detect empty collections
[dcavazos] [BEAM-7390] Add code snippet for GroupByKey
[kcweaver] Version Flink job server container images
[kcweaver] [BEAM-8337] publish Flink job server container images
[kcweaver] [BEAM-8870] Skip test_sdf_with_watermark_tracking for Spark
[kcweaver] [BEAM-8870] Cache environment in spark_runner_test.
[kamil.wasilewski] Strict equality comparision for the version of tensorflow
dependency
[kamil.wasilewski] Extract installChicagoTaxiExampleRequirements step
[kamil.wasilewski] Look up log level values using `getattr`
[kcweaver] [BEAM-8139] Test Spark portable job jar.
[kcweaver] [BEAM-8139] Add main method to SparkPipelineRunner.
[kcweaver] [BEAM-8139] Add beam_PostCommit_PortableJar_Spark.
[echauchot] [BEAM-8830] Fix empty dataset creation: use the beam coder wrapper
that
[heejong] [BEAM-8903] handling --jar_packages experimental flag in
PortableRunner
[github] Changing _BatchSizeEstimator defaults to match BatchElements defaults
[suztomo] [BEAM-8911] New Guava version: 25.1-jre
[ningk] [BEAM-7926] Data-centric Interactive Part1
[kcweaver] Don't override default for environment_cache_millis
[zyichi] [BEAM-8886] Add a mongodb io dataflow integration test
[ehudm] [BEAM-3713] pytest migration: py27-gcp-pytest
[michal.walenia] [BEAM-8895] Add BigQuery table name sanitization to
BigQueryIOIT
[michal.walenia] [BEAM-8918] Split batch BQIOIT into avro and json using tests
[sunjincheng121] [BEAM-8929] Remove unnecessary exception handling in
[suztomo] BEAM-8693 com.google.cloud.datastore:datastore-v1-proto-client 1.6.3
[suztomo] google_cloud_bigquery is not declaring google_http_client_jackson
[github] Changing RowAsDictJsonCoder implementation for efficiency (#10300)
[github] Merge pull request #10151: [BEAM-7116] Remove use of KV in Schema
[chadrik] Pass artifact and provision endpoints to external workers from python
[suztomo] Commons-codec dependency in extensions/sql
[kcweaver] Get Flink version numbers from subdirectories
[suztomo] Adding joda_time dependency
[kcweaver] Warn if Flink versions can't be listed.
[suztomo] Commons_lang3 and jackson_databind dependency
[pabloem] [BEAM-8335] Adds support for multi-output TestStream (#9953)
[apilloud] [BEAM-8362] Don't use toString() for accessing Enum Types
[heejong] [BEAM-8905] matching Java PCollectionTuple translation naming
convention
[echauchot] [BEAM-8894] exclude FlattenWithHeterogeneousCoders category because
[chamikara] [BEAM-8575] Test DoFn context params (#10130)
[github] Add an ML section to python SDK overview (#10233)
[michal.walenia] [BEAM-8947] Change Flink docker image names in Jenkins jobs to
match
[echauchot] [BEAM-8830] Exclude failing SDF tests
[lostluck] [BEAM-8920] Go SDK: faster transforms/filter.Distinct with
CombinePerKey
[echauchot] [BEAM-8025] Under load we get a NoHostAvailable exception at cluster
[ehudm] [BEAM-3713] pytest migration: py27-cython-pytest
[aaltay] [BEAM-8811] Upgrade Beam pipeline diagrams in docs (#10200)
[bhulette] [BEAM-8933] Update BigQuery proto dependency (#10334)
[pabloem] [BEAM-7746] Add python type hints (part 1) (#9915)
[heejong] [BEAM-8943] SDK harness servers don't shut down properly when SDK
[github] [BEAM-8955] Exclude AvroSchemaTest for Spark Runner (#10358)
[relax] attach values so we don't iterate over everything to verify types
[github] [BEAM-8786] Fixes a link in readme
[echauchot] [BEAM-8025] Remove temporary folder rule because it suppresses
files on
[echauchot] [BEAM-8025] Disable auto compaction on Cassandra node to avoid race
[pawel.pasterz] [BEAM-8946] Publish collection size of data written during
MongoDBIOIT
[chamikara] Merge pull request #10347: [BEAM-8885] PubsubGrpcClient doesn't
respect
[kenn] [BEAM-8917] jsr305 dependency declaration for Nullable class (#10324)
[xinyuliu.us] [BEAM-8342]: upgrade to samza 1.3.0 (#10357)
[aaltay] Make model_pcollection snippet self-contained (#10343)
[pabloem] Merge pull request #10050 from [BEAM-8575] Add streaming test case for
[github] Run beam_CancelStaleDataflowJobs every 4 hours.
[tvalentyn] [BEAM-8575] Added a unit test to test Combine works with sessions.
[github] [GoSDK] Improve StateChannel resilience. (#10363)
[ehudm] [BEAM-3713] pytest migration: py3x-{gcp,cython}
[tweise] [BEAM-8273] Expand portability environment documentation (#10116)
[younghoono] [BEAM-7970] Improved error help in Go PROTOBUF.md
[suztomo] Declare JSR305 dependency as 'shadow'
[robertwb] Fixed instructions and the example of running Interactive Beam on
Flink.
------------------------------------------
[...truncated 32.51 KB...]
- org.scala-lang:scala-library [2.12.7 -> 2.13.1]
https://www.scala-lang.org/
- org.slf4j:jcl-over-slf4j [1.7.25 -> 2.0.0-alpha1]
http://www.slf4j.org
- org.slf4j:slf4j-api [1.7.25 -> 2.0.0-alpha1]
http://www.slf4j.org
- org.slf4j:slf4j-jdk14 [1.7.25 -> 2.0.0-alpha1]
http://www.slf4j.org
- org.slf4j:slf4j-simple [1.7.25 -> 2.0.0-alpha1]
http://www.slf4j.org
- org.sonarqube:org.sonarqube.gradle.plugin [2.7 -> 2.8]
- org.springframework:spring-expression [4.3.18.RELEASE -> 5.2.2.RELEASE]
https://github.com/spring-projects/spring-framework
- org.testcontainers:clickhouse [1.10.5 -> 1.12.4]
https://testcontainers.org
- org.testcontainers:localstack [1.11.2 -> 1.12.4]
https://testcontainers.org
- org.testcontainers:testcontainers [1.11.3 -> 1.12.4]
https://testcontainers.org
- org.xerial.snappy:snappy-java [1.1.4 -> 1.1.7.3]
https://github.com/xerial/snappy-java
- redis.clients:jedis [3.0.1 -> 3.2.0]
https://github.com/xetorthio/jedis
- ru.yandex.clickhouse:clickhouse-jdbc [0.1.47 -> 0.2.2]
https://github.com/ClickHouse/clickhouse-jdbc
- software.amazon.awssdk:apache-client [2.5.71 -> 2.10.35]
https://aws.amazon.com/sdkforjava
- software.amazon.awssdk:auth [2.5.71 -> 2.10.35]
https://aws.amazon.com/sdkforjava
- software.amazon.awssdk:cloudwatch [2.5.71 -> 2.10.35]
https://aws.amazon.com/sdkforjava
- software.amazon.awssdk:dynamodb [2.5.71 -> 2.10.35]
https://aws.amazon.com/sdkforjava
- software.amazon.awssdk:sdk-core [2.5.71 -> 2.10.35]
https://aws.amazon.com/sdkforjava
- software.amazon.awssdk:sns [2.5.71 -> 2.10.35]
https://aws.amazon.com/sdkforjava
- software.amazon.awssdk:sqs [2.5.71 -> 2.10.35]
https://aws.amazon.com/sdkforjava
- sqlline:sqlline [1.4.0 -> 1.9.0]
https://github.com/julianhyde/sqlline
Gradle updates:
- Gradle: [5.2.1 -> 6.0.1]
Generated report file build/dependencyUpdates/report.txt
> Task :sdks:python:dependencyUpdates
# Virtualenv for the rest of the script to run setup
rm -rf sdks/
virtualenv sdks/python
Using base prefix '/usr'
New python executable in
<https://builds.apache.org/job/beam_Dependency_Check/ws/src/sdks/python/sdks/python/bin/python3>
Also creating executable in
<https://builds.apache.org/job/beam_Dependency_Check/ws/src/sdks/python/sdks/python/bin/python>
Installing setuptools, pip, wheel...
done.
. sdks/python/bin/activate
# This file must be used with "source bin/activate" *from bash*
# you cannot run it directly
deactivate () {
unset -f pydoc >/dev/null 2>&1
# reset old environment variables
# ! [ -z ${VAR+_} ] returns true if VAR is declared at all
if ! [ -z "${_OLD_VIRTUAL_PATH+_}" ] ; then
PATH="$_OLD_VIRTUAL_PATH"
export PATH
unset _OLD_VIRTUAL_PATH
fi
if ! [ -z "${_OLD_VIRTUAL_PYTHONHOME+_}" ] ; then
PYTHONHOME="$_OLD_VIRTUAL_PYTHONHOME"
export PYTHONHOME
unset _OLD_VIRTUAL_PYTHONHOME
fi
# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands. Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi
if ! [ -z "${_OLD_VIRTUAL_PS1+_}" ] ; then
PS1="$_OLD_VIRTUAL_PS1"
export PS1
unset _OLD_VIRTUAL_PS1
fi
unset VIRTUAL_ENV
if [ ! "${1-}" = "nondestructive" ] ; then
# Self destruct!
unset -f deactivate
fi
}
# unset irrelevant variables
deactivate nondestructive
VIRTUAL_ENV="<https://builds.apache.org/job/beam_Dependency_Check/ws/src/sdks/python/sdks/python";>
export VIRTUAL_ENV
_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH
# unset PYTHONHOME if set
if ! [ -z "${PYTHONHOME+_}" ] ; then
_OLD_VIRTUAL_PYTHONHOME="$PYTHONHOME"
unset PYTHONHOME
fi
if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT-}" ] ; then
_OLD_VIRTUAL_PS1="${PS1-}"
if [ "x" != x ] ; then
PS1="${PS1-}"
else
PS1="(`basename \"$VIRTUAL_ENV\"`) ${PS1-}"
fi
export PS1
fi
basename "$VIRTUAL_ENV"
# Make sure to unalias pydoc if it's already there
alias pydoc 2>/dev/null >/dev/null && unalias pydoc || true
pydoc () {
python -m pydoc "$@"
}
# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands. Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi
pip install -e .[gcp,test,docs]
Obtaining
file://<https://builds.apache.org/job/beam_Dependency_Check/ws/src/sdks/python>
ERROR: Command errored out with exit status 1:
command:
<https://builds.apache.org/job/beam_Dependency_Check/ws/src/sdks/python/sdks/python/bin/python3>
-c 'import sys, setuptools, tokenize; sys.argv[0] =
'"'"'<https://builds.apache.org/job/beam_Dependency_Check/ws/src/sdks/python/setup.py'"'"';>
__file__='"'"'<https://builds.apache.org/job/beam_Dependency_Check/ws/src/sdks/python/setup.py'"'"';f=getattr(tokenize,>
'"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info
cwd:
<https://builds.apache.org/job/beam_Dependency_Check/ws/src/sdks/python/>
Complete output (75 lines):
running egg_info
<https://builds.apache.org/job/beam_Dependency_Check/ws/src/sdks/python/sdks/python/lib/python3.5/site-packages/setuptools/dist.py>:476:
UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
normalized_version,
<https://builds.apache.org/job/beam_Dependency_Check/ws/src/sdks/python/gen_protos.py>:55:
UserWarning: Installing grpcio-tools is recommended for development.
warnings.warn('Installing grpcio-tools is recommended for development.')
WARNING:root:Installing grpcio-tools into
<https://builds.apache.org/job/beam_Dependency_Check/ws/src/sdks/python/.eggs/grpcio-wheels>
Collecting grpcio-tools<=1.14.2,>=1.3.5
Using cached
https://files.pythonhosted.org/packages/a7/85/9e4f4bbe54349ff2e0e80f6e5468d703b2ee4de0950f16c9cfb693f56970/grpcio_tools-1.14.2-cp35-cp35m-manylinux1_x86_64.whl
Collecting mypy-protobuf==1.12
Using cached
https://files.pythonhosted.org/packages/b6/28/041dea47c93564bfc0ece050362894292ec4f173caa92fa82994a6d061d1/mypy_protobuf-1.12-py3-none-any.whl
Processing
/home/jenkins/.cache/pip/wheels/bf/c9/a3/c538d90ef17cf7823fa51fc701a7a7a910a80f6a405bf15b1a/future-0.16.0-cp35-none-any.whl
Collecting grpcio>=1.14.2
Using cached
https://files.pythonhosted.org/packages/2b/dd/ad8ae8e63b4ec93058dec07c3fdd2b702a34c43744dd84060abe20213fed/grpcio-1.25.0-cp35-cp35m-manylinux2010_x86_64.whl
Collecting protobuf>=3.5.0.post1
Using cached
https://files.pythonhosted.org/packages/13/89/00dbb9f5f73be0d8101719d79fdd4b916aad441177a17f899aa7366854dd/protobuf-3.11.1-cp35-cp35m-manylinux1_x86_64.whl
Collecting six>=1.5.2
Using cached
https://files.pythonhosted.org/packages/65/26/32b8464df2a97e6dd1b656ed26b2c194606c16fe163c695a992b36c11cdf/six-1.13.0-py2.py3-none-any.whl
Collecting setuptools
Using cached
https://files.pythonhosted.org/packages/54/28/c45d8b54c1339f9644b87663945e54a8503cfef59cf0f65b3ff5dd17cf64/setuptools-42.0.2-py2.py3-none-any.whl
Installing collected packages: six, grpcio, setuptools, protobuf,
grpcio-tools, mypy-protobuf, future
Successfully installed future-0.16.0 grpcio-1.25.0 grpcio-tools-1.14.2
mypy-protobuf-1.12 protobuf-3.11.1 setuptools-42.0.2 six-1.13.0
WARNING:root:Installing grpcio-tools took 5.71 seconds.
Collecting mypy-protobuf==1.12
Using cached
https://files.pythonhosted.org/packages/b6/28/041dea47c93564bfc0ece050362894292ec4f173caa92fa82994a6d061d1/mypy_protobuf-1.12-py3-none-any.whl
Installing collected packages: mypy-protobuf
Successfully installed mypy-protobuf-1.12
beam_fn_api.proto: warning: Import google/protobuf/descriptor.proto but not
used.
beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto but not
used.
Traceback (most recent call last):
File
"<https://builds.apache.org/job/beam_Dependency_Check/ws/src/sdks/python/sdks/python/bin/protoc-gen-mypy";,>
line 13, in <module>
import google.protobuf.descriptor_pb2 as d
ImportError: No module named 'google'
--mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
Process Process-1:
Traceback (most recent call last):
File
"<https://builds.apache.org/job/beam_Dependency_Check/ws/src/sdks/python/gen_protos.py";,>
line 104, in generate_proto_files
from grpc_tools import protoc
ImportError: No module named 'grpc_tools'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in
_bootstrap
self.run()
File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File
"<https://builds.apache.org/job/beam_Dependency_Check/ws/src/sdks/python/gen_protos.py";,>
line 189, in _install_grpcio_tools_and_generate_proto_files
generate_proto_files()
File
"<https://builds.apache.org/job/beam_Dependency_Check/ws/src/sdks/python/gen_protos.py";,>
line 144, in generate_proto_files
'%s' % ret_code)
RuntimeError: Protoc returned non-zero status (see logs for details): 1
Traceback (most recent call last):
File
"<https://builds.apache.org/job/beam_Dependency_Check/ws/src/sdks/python/gen_protos.py";,>
line 104, in generate_proto_files
from grpc_tools import protoc
ImportError: No module named 'grpc_tools'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File
"<https://builds.apache.org/job/beam_Dependency_Check/ws/src/sdks/python/setup.py";,>
line 295, in <module>
'mypy': generate_protos_first(mypy),
File
"<https://builds.apache.org/job/beam_Dependency_Check/ws/src/sdks/python/sdks/python/lib/python3.5/site-packages/setuptools/__init__.py";,>
line 145, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python3.5/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File
"<https://builds.apache.org/job/beam_Dependency_Check/ws/src/sdks/python/setup.py";,>
line 220, in run
gen_protos.generate_proto_files(log=log)
File
"<https://builds.apache.org/job/beam_Dependency_Check/ws/src/sdks/python/gen_protos.py";,>
line 121, in generate_proto_files
raise ValueError("Proto generation failed (see log for details).")
ValueError: Proto generation failed (see log for details).
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check
the logs for full command output.
> Task :sdks:python:dependencyUpdates FAILED
FAILURE: Build failed with an exception.
* Where:
Build file
'<https://builds.apache.org/job/beam_Dependency_Check/ws/src/sdks/python/build.gradle'>
line: 94
* What went wrong:
Execution failed for task ':sdks:python:dependencyUpdates'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 2m 24s
2 actionable tasks: 2 executed
Publishing build scan...
https://gradle.com/s/ugr3vdedppito
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
