See <https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/190/display/redirect?page=changes>
Changes: [rezarokni] initial commit for BEAM-2871. WIP Todo: Add in the unit tests to work [relax] Update python version. [relax] Update container version. [kirpichov] Cherrypick of #3995 to 2.2.0: Add custom tempLocation support to [jbonofre] [BEAM-1017] Add RedisIO [kenn] Use 2.2.0 container tag for Dataflow [chamikara] Sets user agent in BigTableIO.Read.getBigTableService(). [valentyn] Pin runner harness container image for Dataflow distributions of BEAM [valentyn] Make linter happy. [valentyn] [BEAM-2600] Add minimal python SDK harness container [valentyn] Use 2.2.0 version instead of 2.3.0 for container Maven project on the [valentyn] Use tox instead of detox. Looks like we reverted to tox on the master [valentyn] Pin runner harness also for official BEAM releases. [relax] [maven-release-plugin] prepare release [relax] [maven-release-plugin] prepare for next development iteration [relax] Revert "[maven-release-plugin] prepare for next development iteration" [relax] Revert "[maven-release-plugin] prepare release [relax] [maven-release-plugin] prepare release v2.2.0-RC1 [relax] [maven-release-plugin] rollback changes from release preparation of [altay] Add an element batching transform. [altay] Align names with those produced by the dataflow runner harness. [altay] Fix from any -> bytes transition. [altay] Add an option for dataflow job labels. [altay] Unit test for label pipeline option [altay] Update container version for python DataflowRunner for the 2.2.0 release [relax] [maven-release-plugin] prepare release v2.2.0-RC2 [relax] [maven-release-plugin] rollback changes from release preparation of [relax] [maven-release-plugin] prepare release v2.2.0-RC3 [relax] [maven-release-plugin] rollback changes from release preparation of [valentyn] Set save_main_session to True for csv module [valentyn] Set OnTimeBehavior in windowing strategy to FIRE_ALWAYS. [valentyn] Update urns.py [ccy] CP #4112, #4122: Properly handle side input exception when all reader [relax] Cherrypick WriteFiles fix. [relax] [maven-release-plugin] prepare release v2.2.0-RC4 [relax] [maven-release-plugin] rollback changes from release preparation of [rezarokni] initial commit for BEAM-2871. WIP Todo: Add in the unit tests to work [rezarokni] [maven-release-plugin] prepare for next development iteration [rezarokni] Bump Python version for next release. [rezarokni] Only fail nightly release at end, for more detailed signal [rezarokni] Only fail Java postcommit at end [rezarokni] Move the Job Server into a Submodule [rezarokni] Add Artifact Staging Endpoint to PrepareJobResponse [rezarokni] Add model/ and submodules to Maven structure and gen_protos.py [rezarokni] Move sdk/common/runner-api protos into model/pipeline and [rezarokni] Move sdks/common/fn-api protos to model/fn-execution [rezarokni] Remove sdks/common [rezarokni] Re-namespace model/pipeline to model.pipeline [rezarokni] Re-namespace model/job-management to model.job_management [rezarokni] Re-namespace model/fn-execution to model.fn_execution [rezarokni] Improve error message when standard_coders.yaml not found [rezarokni] Dataflow runner: shade protobuf and beam model [rezarokni] PipelineOptionsFactory should prevent non PipelineOptions interfaces [rezarokni] [BEAM-3016] Fix blocking issue within run() when channel terminates [rezarokni] [BEAM-2576] Reflect portability proto changes in Go [rezarokni] Add licenses to generated code [rezarokni] Improve the documentation for CombineFn [rezarokni] Remove coders from DoFn's Remove extra space in hello.cc file Remove [rezarokni] Fix Bug if more than one external lib DoFn is used in same pipeline the [rezarokni] Removed more complex examples for later PR Added semaphore logic to [7542791+rezarokni] Added Assert for test pipeline. Added comments and new file names for [7542791+rezarokni] Added ability to read Binary output from files for use with Protobuf [7542791+rezarokni] Moved files from java8 to java [7542791+rezarokni] Correct checkstyle issues [swegner] Publish a build scan from Jenkins build [amyrvold] [BEAM-4035] Fix spark validatesrunner [amyrvold] Change ver of zookeeper instead of removing mavenLocal [7542791+rezarokni] Fix findBugs issues [lcwik] [BEAM-3249, BEAM-4014] Automatically propagate shading configuration to [lcwik] [BEAM-3249] Add support for translating 'provided' scope dependencies to [kenn] Basic run task for nexmark with runner selection [swegner] Migrate a few more references to Maven from our README.md [amyrvold] [BEAM-4053] Add a Go postcommit [lcwik] [BEAM-3249] Only generate all artifacts when publishing. [ehudm] Clean up docker networks generated by hdfs integration tests. [swegner] Add comment about build scans only on Jenkins [chamikara] Updates gcsio to use a API specific batch endpoint. ------------------------------------------ [...truncated 2.87 MB...] [INFO] Downloading from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/2.5/commons-io-2.5.jar [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-incremental/1.1/maven-shared-incremental-1.1.jar [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-java/0.9.2/plexus-java-0.9.2.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-java/0.9.2/plexus-java-0.9.2.jar (31 kB at 843 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/ow2/asm/asm/6.0_BETA/asm-6.0_BETA.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-incremental/1.1/maven-shared-incremental-1.1.jar (14 kB at 322 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/com/thoughtworks/qdox/qdox/2.0-M7/qdox-2.0-M7.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/2.5/commons-io-2.5.jar (209 kB at 4.5 MB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-compiler-api/2.8.2/plexus-compiler-api-2.8.2.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-utils/3.1.0/maven-shared-utils-3.1.0.jar (164 kB at 3.3 MB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-compiler-manager/2.8.2/plexus-compiler-manager-2.8.2.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/2.0.4/plexus-utils-2.0.4.jar (222 kB at 4.1 MB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-compiler-javac/2.8.2/plexus-compiler-javac-2.8.2.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/ow2/asm/asm/6.0_BETA/asm-6.0_BETA.jar (56 kB at 868 kB/s) [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-compiler-manager/2.8.2/plexus-compiler-manager-2.8.2.jar (4.7 kB at 65 kB/s) [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-compiler-api/2.8.2/plexus-compiler-api-2.8.2.jar (26 kB at 357 kB/s) [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-compiler-javac/2.8.2/plexus-compiler-javac-2.8.2.jar (20 kB at 248 kB/s) [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/com/thoughtworks/qdox/qdox/2.0-M7/qdox-2.0-M7.jar (315 kB at 3.6 MB/s) [INFO] Changes detected - recompiling the module! [WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent! [INFO] Compiling 31 source files to /tmp/groovy-generated-4783117475079429785-tmpdir/word-count-beam/target/classes [INFO] /tmp/groovy-generated-4783117475079429785-tmpdir/word-count-beam/src/main/java/org/apache/beam/examples/complete/game/utils/WriteToText.java: /tmp/groovy-generated-4783117475079429785-tmpdir/word-count-beam/src/main/java/org/apache/beam/examples/complete/game/utils/WriteToText.java uses unchecked or unsafe operations. [INFO] /tmp/groovy-generated-4783117475079429785-tmpdir/word-count-beam/src/main/java/org/apache/beam/examples/complete/game/utils/WriteToText.java: Recompile with -Xlint:unchecked for details. [INFO] [INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ word-count-beam --- [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-toolchain/2.2.1/maven-toolchain-2.2.1.pom [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-toolchain/2.2.1/maven-toolchain-2.2.1.pom (3.3 kB at 134 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-api/2.2.1/maven-reporting-api-2.2.1.pom [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-api/2.2.1/maven-reporting-api-2.2.1.pom (1.9 kB at 77 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting/2.2.1/maven-reporting-2.2.1.pom [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting/2.2.1/maven-reporting-2.2.1.pom (1.4 kB at 60 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-sink-api/1.1/doxia-sink-api-1.1.pom [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-sink-api/1.1/doxia-sink-api-1.1.pom (2.0 kB at 85 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia/1.1/doxia-1.1.pom [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia/1.1/doxia-1.1.pom (15 kB at 632 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-logging-api/1.1/doxia-logging-api-1.1.pom [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-logging-api/1.1/doxia-logging-api-1.1.pom (1.6 kB at 68 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-container-default/1.0-alpha-30/plexus-container-default-1.0-alpha-30.pom [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-container-default/1.0-alpha-30/plexus-container-default-1.0-alpha-30.pom (3.5 kB at 145 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-containers/1.0-alpha-30/plexus-containers-1.0-alpha-30.pom [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-containers/1.0-alpha-30/plexus-containers-1.0-alpha-30.pom (1.9 kB at 82 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/1.4.5/plexus-utils-1.4.5.pom [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/1.4.5/plexus-utils-1.4.5.pom (2.3 kB at 98 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/1.2-alpha-9/plexus-classworlds-1.2-alpha-9.pom [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/1.2-alpha-9/plexus-classworlds-1.2-alpha-9.pom (3.2 kB at 134 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-component-annotations/1.5.4/plexus-component-annotations-1.5.4.pom [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-component-annotations/1.5.4/plexus-component-annotations-1.5.4.pom (815 B at 34 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-containers/1.5.4/plexus-containers-1.5.4.pom [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-containers/1.5.4/plexus-containers-1.5.4.pom (4.2 kB at 184 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/2.0.5/plexus-2.0.5.pom [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/2.0.5/plexus-2.0.5.pom (17 kB at 694 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.20/plexus-utils-3.0.20.pom [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.20/plexus-utils-3.0.20.pom (3.8 kB at 166 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/commons/commons-exec/1.3/commons-exec-1.3.pom [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/commons/commons-exec/1.3/commons-exec-1.3.pom (11 kB at 458 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-toolchain/2.2.1/maven-toolchain-2.2.1.jar [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-project/2.2.1/maven-project-2.2.1.jar [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-profile/2.2.1/maven-profile-2.2.1.jar [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-registry/2.2.1/maven-plugin-registry-2.2.1.jar [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-settings/2.2.1/maven-settings-2.2.1.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-toolchain/2.2.1/maven-toolchain-2.2.1.jar (38 kB at 1.1 MB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.11/plexus-interpolation-1.11.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-profile/2.2.1/maven-profile-2.2.1.jar (35 kB at 982 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/2.2.1/maven-model-2.2.1.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-registry/2.2.1/maven-plugin-registry-2.2.1.jar (30 kB at 825 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact/2.2.1/maven-artifact-2.2.1.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-settings/2.2.1/maven-settings-2.2.1.jar (49 kB at 1.2 MB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact-manager/2.2.1/maven-artifact-manager-2.2.1.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-project/2.2.1/maven-project-2.2.1.jar (156 kB at 3.5 MB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/2.2.1/maven-repository-metadata-2.2.1.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.11/plexus-interpolation-1.11.jar (51 kB at 894 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/backport-util-concurrent/backport-util-concurrent/3.1/backport-util-concurrent-3.1.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/2.2.1/maven-model-2.2.1.jar (88 kB at 1.5 MB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-core/2.2.1/maven-core-2.2.1.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact/2.2.1/maven-artifact-2.2.1.jar (80 kB at 1.3 MB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-parameter-documenter/2.2.1/maven-plugin-parameter-documenter-2.2.1.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact-manager/2.2.1/maven-artifact-manager-2.2.1.jar (68 kB at 1.1 MB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-jdk14/1.5.6/slf4j-jdk14-1.5.6.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/2.2.1/maven-repository-metadata-2.2.1.jar (26 kB at 389 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.5.6/slf4j-api-1.5.6.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-jdk14/1.5.6/slf4j-jdk14-1.5.6.jar (8.8 kB at 100 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/slf4j/jcl-over-slf4j/1.5.6/jcl-over-slf4j-1.5.6.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-core/2.2.1/maven-core-2.2.1.jar (178 kB at 2.0 MB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-api/2.2.1/maven-reporting-api-2.2.1.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-parameter-documenter/2.2.1/maven-plugin-parameter-documenter-2.2.1.jar (22 kB at 249 kB/s) [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.5.6/slf4j-api-1.5.6.jar (22 kB at 251 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-sink-api/1.1/doxia-sink-api-1.1.jar [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-logging-api/1.1/doxia-logging-api-1.1.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/backport-util-concurrent/backport-util-concurrent/3.1/backport-util-concurrent-3.1.jar (332 kB at 3.3 MB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-error-diagnostics/2.2.1/maven-error-diagnostics-2.2.1.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-logging-api/1.1/doxia-logging-api-1.1.jar (11 kB at 100 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-descriptor/2.2.1/maven-plugin-descriptor-2.2.1.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-api/2.2.1/maven-reporting-api-2.2.1.jar (9.8 kB at 86 kB/s) [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-sink-api/1.1/doxia-sink-api-1.1.jar (13 kB at 112 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-monitor/2.2.1/maven-monitor-2.2.1.jar [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/2.2.1/maven-plugin-api-2.2.1.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/slf4j/jcl-over-slf4j/1.5.6/jcl-over-slf4j-1.5.6.jar (17 kB at 145 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.20/plexus-utils-3.0.20.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-error-diagnostics/2.2.1/maven-error-diagnostics-2.2.1.jar (13 kB at 103 kB/s) [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/commons/commons-exec/1.3/commons-exec-1.3.jar [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-monitor/2.2.1/maven-monitor-2.2.1.jar (10 kB at 76 kB/s) [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/2.2.1/maven-plugin-api-2.2.1.jar (12 kB at 89 kB/s) [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-descriptor/2.2.1/maven-plugin-descriptor-2.2.1.jar (39 kB at 264 kB/s) [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/commons/commons-exec/1.3/commons-exec-1.3.jar (54 kB at 360 kB/s) [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.20/plexus-utils-3.0.20.jar (243 kB at 1.6 MB/s) SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/tmp/groovy-generated-4783117475079429785-tmpdir/.m2/repository/org/slf4j/slf4j-jdk14/1.7.25/slf4j-jdk14-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/tmp/groovy-generated-4783117475079429785-tmpdir/.m2/repository/org/slf4j/slf4j-log4j12/1.7.16/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.JDK14LoggerFactory] Apr 12, 2018 11:04:16 AM org.apache.beam.runners.spark.SparkRunner fromOptions INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 213 files. Enable logging at DEBUG level to see which files will be staged. Apr 12, 2018 11:04:16 AM org.apache.beam.runners.spark.SparkRunner run INFO: Executing pipeline using the SparkRunner. Apr 12, 2018 11:04:16 AM org.apache.beam.runners.spark.translation.SparkContextFactory createSparkContext INFO: Creating a brand new Spark Context. Apr 12, 2018 11:04:16 AM org.apache.spark.internal.Logging$class logInfo INFO: Running Spark version 2.3.0 Apr 12, 2018 11:04:17 AM org.apache.hadoop.util.NativeCodeLoader <clinit> WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Apr 12, 2018 11:04:17 AM org.apache.spark.internal.Logging$class logInfo INFO: Submitted application: WordCount Apr 12, 2018 11:04:17 AM org.apache.spark.internal.Logging$class logInfo INFO: Changing view acls to: jenkins Apr 12, 2018 11:04:17 AM org.apache.spark.internal.Logging$class logInfo INFO: Changing modify acls to: jenkins Apr 12, 2018 11:04:17 AM org.apache.spark.internal.Logging$class logInfo INFO: Changing view acls groups to: Apr 12, 2018 11:04:17 AM org.apache.spark.internal.Logging$class logInfo INFO: Changing modify acls groups to: Apr 12, 2018 11:04:17 AM org.apache.spark.internal.Logging$class logInfo INFO: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); groups with view permissions: Set(); users with modify permissions: Set(jenkins); groups with modify permissions: Set() [WARNING] java.lang.NoSuchMethodError: io.netty.buffer.PooledByteBufAllocator.metric()Lio/netty/buffer/PooledByteBufAllocatorMetric; at org.apache.spark.network.util.NettyMemoryMetrics.registerMetrics (NettyMemoryMetrics.java:80) at org.apache.spark.network.util.NettyMemoryMetrics.<init> (NettyMemoryMetrics.java:76) at org.apache.spark.network.client.TransportClientFactory.<init> (TransportClientFactory.java:109) at org.apache.spark.network.TransportContext.createClientFactory (TransportContext.java:99) at org.apache.spark.rpc.netty.NettyRpcEnv.<init> (NettyRpcEnv.scala:71) at org.apache.spark.rpc.netty.NettyRpcEnvFactory.create (NettyRpcEnv.scala:461) at org.apache.spark.rpc.RpcEnv$.create (RpcEnv.scala:57) at org.apache.spark.SparkEnv$.create (SparkEnv.scala:249) at org.apache.spark.SparkEnv$.createDriverEnv (SparkEnv.scala:175) at org.apache.spark.SparkContext.createSparkEnv (SparkContext.scala:256) at org.apache.spark.SparkContext.<init> (SparkContext.scala:423) at org.apache.spark.api.java.JavaSparkContext.<init> (JavaSparkContext.scala:58) at org.apache.beam.runners.spark.translation.SparkContextFactory.createSparkContext (SparkContextFactory.java:103) at org.apache.beam.runners.spark.translation.SparkContextFactory.getSparkContext (SparkContextFactory.java:68) at org.apache.beam.runners.spark.SparkRunner.run (SparkRunner.java:207) at org.apache.beam.runners.spark.SparkRunner.run (SparkRunner.java:88) at org.apache.beam.sdk.Pipeline.run (Pipeline.java:311) at org.apache.beam.sdk.Pipeline.run (Pipeline.java:297) at org.apache.beam.examples.WordCount.runWordCount (WordCount.java:185) at org.apache.beam.examples.WordCount.main (WordCount.java:192) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282) at java.lang.Thread.run (Thread.java:748) [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:46 min [INFO] Finished at: 2018-04-12T11:04:17Z [INFO] Final Memory: 67M/989M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project word-count-beam: An exception occured while executing the Java class. io.netty.buffer.PooledByteBufAllocator.metric()Lio/netty/buffer/PooledByteBufAllocatorMetric; -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] Failed command :beam-runners-spark:runQuickstartJavaSpark FAILED :beam-runners-spark:runQuickstartJavaSpark (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 2 mins 10.665 secs. Apr 12, 2018 11:04:25 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T11:04:23.350Z: Executing operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Close Apr 12, 2018 11:04:25 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T11:04:23.443Z: Executing operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum+WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) Apr 12, 2018 11:04:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T11:04:26.367Z: Executing operation s12-u31 Apr 12, 2018 11:04:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T11:04:26.764Z: Executing operation WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView Apr 12, 2018 11:04:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T11:04:26.950Z: Executing operation WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)+WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)+WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map+WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize Apr 12, 2018 11:04:29 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T11:04:28.977Z: Cleaning up. Apr 12, 2018 11:04:29 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T11:04:29.121Z: Stopping worker pool... Apr 12, 2018 11:06:14 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T11:06:13.380Z: Autoscaling: Resized worker pool from 2 to 0. Apr 12, 2018 11:06:14 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T11:06:13.403Z: Autoscaling: Would further reduce the number of workers but reached the minimum number allowed for the job. Apr 12, 2018 11:06:14 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T11:06:13.441Z: Worker pool stopped. Apr 12, 2018 11:06:22 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish INFO: Job 2018-04-12_04_02_23-16013280443117059930 finished with status DONE. [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 05:23 min [INFO] Finished at: 2018-04-12T11:06:22Z [INFO] Final Memory: 62M/662M [INFO] ------------------------------------------------------------------------ gsutil cat gs://temp-storage-for-release-validation-tests/quickstart/count* | grep Montague: Montague: 47 Verified Montague: 47 gsutil rm gs://temp-storage-for-release-validation-tests/quickstart/count* Removing gs://temp-storage-for-release-validation-tests/quickstart/counts-00000-of-00003... / [1 objects] Removing gs://temp-storage-for-release-validation-tests/quickstart/counts-00001-of-00003... / [2 objects] Removing gs://temp-storage-for-release-validation-tests/quickstart/counts-00002-of-00003... / [3 objects] Operation completed over 3 objects. [SUCCESS] :beam-runners-google-cloud-dataflow-java:runQuickstartJavaDataflow (Thread[Task worker for ':',5,main]) completed. Took 5 mins 58.829 secs. FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':beam-runners-spark:runQuickstartJavaSpark'. > Process 'command '/usr/local/asfpackages/java/jdk1.8.0_152/bin/java'' > finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org BUILD FAILED in 6m 10s 6 actionable tasks: 6 executed Publishing build scan... https://gradle.com/s/g6swy3iaarugi Stopped 1 worker daemon(s). Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure Not sending mail to unregistered user 7542791+rezaro...@users.noreply.github.com Not sending mail to unregistered user sweg...@google.com Not sending mail to unregistered user eh...@google.com Not sending mail to unregistered user valen...@google.com Not sending mail to unregistered user re...@relax-macbookpro2.roam.corp.google.com Not sending mail to unregistered user c...@google.com Not sending mail to unregistered user amyrv...@google.com Not sending mail to unregistered user rezaro...@google.com Not sending mail to unregistered user kirpic...@google.com