See <https://builds.apache.org/job/beam_LoadTests_Java_Combine_Portable_Flink_Batch/3/display/redirect?page=changes>
Changes: [ehudm] [BEAM-8525] Support Const base in binary_subscr [ehudm] Do not perform test on Py2.7 [echauchot] [BEAM-9065] Reset MetricsContainerStepMapAccumulator upon initialization [kamil.wasilewski] [BEAM-8108] Allow run_chicago.sh to take variable argument number [kamil.wasilewski] [BEAM-8108] Use _ReadFromBigQuery transform [kamil.wasilewski] [BEAM-8108] Create Jenkins job that runs Chicago Taxi Example on Flink [kamil.wasilewski] [BEAM-8108] Rewrite Python print statement in dataproc init action [github] Update release guide with extended information. [github] Add link to mass comment script [github] Fix typo [ankurgoenka] [BEAM-9002] Add test_flatten_same_pcollections to fnapi runner [github] Address comments [hannahjiang] [BEAM-9084] cleaning up docker image tag [iemejia] [BEAM-5086] Update kudu-client to version 1.11.1 [valentyn] Add version guards to requirements file for integration tests. [robertwb] Automatically convert to with pipeline syntax. [robertwb] Quick pass through failed auto-conversions. [robertwb] Automatic conversion of more pipelines. [robertwb] Fix lint and tests due to autoconversion. [robertwb] A couple more conversions. [robertwb] Fix lint and tests due to autoconversion. [robertwb] Return non-None result for Dataflow dry run. [robertwb] Fix lint and tests due to autoconversion. [hannahjiang] [BEAM-9084] fix Java spotless [kamil.wasilewski] [BEAM-8939] A bash script that cancels stale dataflow jobs [chadrik] [BEAM-7746] Fix a typing issue where SourceBase was assumed to have a [marek.simunek] [BEAM-9123] HadoopResourceId returns wrong directoryName bugfix [iemejia] [website] Added security page [iemejia] [website] Update the 2.17.0 release blog post to include security issues [boyuanz] Moving to 2.20.0-SNAPSHOT on master branch. [github] Update twine install details. [crites] Changes watermark advance from 1001 to 1000 since Dataflow TestStream [lukecwik] [BEAM-8676] sdks/java: gax and grpc upgrades (#10554) [lukecwik] [BEAM-9030] Migrate Beam to use beam-vendor-grpc-1_26_0 (#10578) [lcwik] [BEAM-9030] Align version of protoc/protoc-gen-grpc-java to vendored [kirillkozlov] Create initial DataStoreV1Table [kirillkozlov] Added tests [kirillkozlov] Implement getTableStatistics [kirillkozlov] buildIOWriter [kirillkozlov] Read other types [kirillkozlov] Better conversion, support complex types [kirillkozlov] Store DataStore key as VARBINARY [kirillkozlov] Wrap DoFns in PTransforms [kirillkozlov] Table property for specifying key field [kirillkozlov] JavaDoc [kirillkozlov] Infer schema for RowToEntity [kirillkozlov] Better conversion performance [kirillkozlov] Mark Table as `Internal` and `Experimental` [kirillkozlov] Review changes [boyuanz] Exclude testOutputTimestamp from flink streaming tests. [lukecwik] [BEAM-7951] Supports multiple inputs/outputs for wire coder settings. [mxm] [BEAM-9116] Limit the number of past invocations stored in JobService [kirillkozlov] Add IT that does not rely on SQL [lukecwik] [BEAM-9124] Linkage Checker 1.1.2 to use Maven Central HTTPS URL [robertwb] lint, reviewer comments [relax] Merge pull request #10316: [BEAM-6857] Support Dynamic Timers [robertwb] lint [github] Merge pull request #10577 from Adding Python test for ReadFromBigQuery [kirillkozlov] fix style [github] [BEAM-9127] Fix output type declaration in xlang wordcount. (#10605) [robertwb] fix merge [apilloud] [BEAM-9140] Upgrade to ZetaSQL 2020.01.1 [angoenka] [BEAM-8625] Implement servlet for exposing sdk harness statuses in Da… [iemejia] [BEAM-9143] Make RedisIO code follow standard Beam conventions [iemejia] [BEAM-9143] Add withOutputParallelization to RedisIO.Read/ReadAll [suztomo] Beam's own TimestampConversion [suztomo] Link in Javadoc [ehudm] Migrate HDFS IT to use tox env. [ehudm] Fix for py2 [lukecwik] [BEAM-8695] Upgrade google-http-client to 1.34.0 (#10614) [rohde.samuel] Make apply_TestStream not a test b/c Nose thinks it is a test [chamikara] [BEAM-7246] Add Google Spanner IO Read on Python SDK (#9606) [boyuanz] [BEAM-9151] Fix misconfigured legacy dataflow tests. [hannahjiang] [BEAM-7861] update documentation [github] Merge pull request #10622: [BEAM=6857] Fix timermap test to not use [sunjincheng121] [BEAM-9153] Fix release guide heading level [kamil.wasilewski] Report status code 0 when no stale jobs are found [hsuryawirawan] Update to Python 3.7 and upgrade to Beam v2.17.0 [hsuryawirawan] Change "Built-in IOs" task type to "theory" [hsuryawirawan] Remove "run_common_tests()" in all tests.py to make it work with Python [hsuryawirawan] Update the course on Stepik [hsuryawirawan] Add back the license header [github] Fix typo in 2017-02-13-stateful-processing.md [github] cleanup typo [github] [BEAM-9120] Make WordCountsMatcher extend [github] Revert "Merge pull request #10582 for [INFRA-19670] Add .asf.yaml for [angoenka] [BEAM-9122] Add uses_keyed_state step property in python dataflow run… [github] [BEAM-9120] Make BigqueryMatcher extend TypeSafeMatcher<TableAndQuery> [github] [BEAM-9120] Make FileChecksumMatcher extend TypeSafeMatcher<ShardedFile> [pabloem] Merge pull request #10459 from [BEAM-9029]Fix two bugs in Python SDK S3 [lcwik] [BEAM-3419] Support iterable on Dataflow runner when using the unified [ehudm] [BEAM-9168] Temporary fix for RunnerAPIPTransformHolder usage [keijiy] Fix comments on bigquery.py * beam.io.gcp.WriteToBigQuery -> [github] Merge pull request #10534: Beam-2535: Support timer output timestamps in [github] [BEAM-8992] ignore Go Vet failures. (#10661) [github] [BEAM-4725] Preallocate CoGBK value buffers (#10660) [kirillkozlov] Use custom escape method [github] typehint fixes to DoOutputsTuple (#10494) [github] [BEAM-3419, BEAM-9173] Add TODO comment [github] [BEAM-9167] Reduce Go SDK metric overhead (#10654) [iemejia] [BEAM-9172] Add missing parameter to Nexmark CI execution for Flink [github] Chnage dataflow container version [hannahjiang] [BEAM-9084] cleaning up docker image tag [iemejia] Make Spark use of batch based invocation in Nexmark CI jobs cleaner [kirillkozlov] Inline, link JIRA [ehudm] Update beam website for 2.18.0 [apilloud] [BEAM-9027] Unparse LIKE as binary operator [github] Blog post for release 2.18.0 (#10575) [kcweaver] upgrade auto-value to version 1.7 [kcweaver] [BEAM-9149] Add SQL query parameters to public API and enable positional [bhulette] [BEAM-9072] [SQL] Primitive types should fall through (#10681) [elias.djurfeldt] Added valueprovider support for Datastore query namespaces [iemejia] [BEAM-9170] Unify Jenkins job names related to Java 11 testing [github] Add a minor comment. [aaltay] Bump tensorflow from 1.14.0 to 1.15.0 in /sdks/python/container (#10392) [github] [BEAM-9093] Log invalid overwrites in pipeline options (#10613) [github] [BEAM-8492] Allow None, Optional return hints for DoFn.process and ------------------------------------------ [...truncated 39.81 KB...] 03ff63c55220: Preparing bee1e39d7c3a: Preparing 1f59a4b2e206: Preparing 0ca7f54856c0: Preparing ebb9ae013834: Preparing 0ca7f54856c0: Waiting 619b5df84312: Waiting e7fe5541de5f: Waiting 03ff63c55220: Waiting a6ded049566a: Waiting ebb9ae013834: Waiting 1f59a4b2e206: Waiting bee1e39d7c3a: Waiting 7c13b2965ae4: Pushed 468df377a9d9: Pushed 14421810ded7: Pushed a6ded049566a: Layer already exists e7fe5541de5f: Layer already exists 03ff63c55220: Layer already exists bee1e39d7c3a: Layer already exists 1f59a4b2e206: Layer already exists 0ca7f54856c0: Layer already exists ebb9ae013834: Layer already exists 619b5df84312: Pushed b9b02e4fbbee: Pushed 88f235ed14ee: Pushed latest: digest: sha256:f1fa2c52e57c24f525e08f7dc5fe377f06bb134fe20af909f11790e38f884ee4 size: 3056 [Gradle] - Launching build. [src] $ <https://builds.apache.org/job/beam_LoadTests_Java_Combine_Portable_Flink_Batch/ws/src/gradlew> --continue --max-workers=12 -Dorg.gradle.jvmargs=-Xms2g -Dorg.gradle.jvmargs=-Xmx4g -Pdocker-repository-root=gcr.io/apache-beam-testing/beam_portability -Pdocker-tag=latest :runners:flink:1.9:job-server-container:docker > Task :buildSrc:compileJava NO-SOURCE > Task :buildSrc:compileGroovy UP-TO-DATE > Task :buildSrc:pluginDescriptors UP-TO-DATE > Task :buildSrc:processResources UP-TO-DATE > Task :buildSrc:classes UP-TO-DATE > Task :buildSrc:jar UP-TO-DATE > Task :buildSrc:assemble UP-TO-DATE > Task :buildSrc:spotlessGroovy UP-TO-DATE > Task :buildSrc:spotlessGroovyCheck UP-TO-DATE > Task :buildSrc:spotlessGroovyGradle UP-TO-DATE > Task :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE > Task :buildSrc:spotlessCheck UP-TO-DATE > Task :buildSrc:pluginUnderTestMetadata UP-TO-DATE > Task :buildSrc:compileTestJava NO-SOURCE > Task :buildSrc:compileTestGroovy NO-SOURCE > Task :buildSrc:processTestResources NO-SOURCE > Task :buildSrc:testClasses UP-TO-DATE > Task :buildSrc:test NO-SOURCE > Task :buildSrc:validateTaskProperties UP-TO-DATE > Task :buildSrc:check UP-TO-DATE > Task :buildSrc:build UP-TO-DATE Configuration on demand is an incubating feature. > Task :sdks:java:fn-execution:processResources NO-SOURCE > Task :sdks:java:extensions:google-cloud-platform-core:processResources > NO-SOURCE > Task :model:job-management:extractProto UP-TO-DATE > Task :sdks:java:harness:processResources NO-SOURCE > Task :runners:core-construction-java:processResources NO-SOURCE > Task :runners:core-java:processResources NO-SOURCE > Task :sdks:java:core:generateAvroProtocol NO-SOURCE > Task :model:fn-execution:extractProto UP-TO-DATE > Task :vendor:sdks-java-extensions-protobuf:processResources NO-SOURCE > Task :runners:java-fn-execution:processResources NO-SOURCE > Task :runners:flink:1.9:copyResourcesOverrides NO-SOURCE > Task :runners:flink:1.9:job-server:processResources NO-SOURCE > Task :sdks:java:core:generateAvroJava NO-SOURCE > Task :sdks:java:io:kafka:processResources NO-SOURCE > Task :sdks:java:extensions:protobuf:extractProto > Task :runners:flink:1.9:job-server-container:dockerClean UP-TO-DATE > Task :sdks:java:io:google-cloud-platform:processResources NO-SOURCE > Task :sdks:java:extensions:protobuf:processResources NO-SOURCE > Task :model:fn-execution:processResources UP-TO-DATE > Task :model:job-management:processResources UP-TO-DATE > Task :sdks:java:core:generateGrammarSource UP-TO-DATE > Task :sdks:java:core:processResources UP-TO-DATE > Task :runners:flink:1.9:copySourceOverrides > Task :runners:flink:1.9:copyTestResourcesOverrides NO-SOURCE > Task :model:pipeline:extractIncludeProto UP-TO-DATE > Task :sdks:java:build-tools:compileJava FROM-CACHE > Task :model:pipeline:extractProto UP-TO-DATE > Task :runners:flink:1.9:processResources > Task :sdks:java:build-tools:processResources > Task :sdks:java:build-tools:classes > Task :model:pipeline:generateProto UP-TO-DATE > Task :sdks:java:build-tools:jar > Task :model:pipeline:compileJava UP-TO-DATE > Task :model:pipeline:processResources UP-TO-DATE > Task :model:pipeline:classes UP-TO-DATE > Task :model:pipeline:jar UP-TO-DATE > Task :model:pipeline:shadowJar UP-TO-DATE > Task :model:job-management:extractIncludeProto UP-TO-DATE > Task :model:fn-execution:extractIncludeProto UP-TO-DATE > Task :model:job-management:generateProto UP-TO-DATE > Task :model:fn-execution:generateProto UP-TO-DATE > Task :model:job-management:compileJava UP-TO-DATE > Task :model:job-management:classes UP-TO-DATE > Task :model:job-management:shadowJar UP-TO-DATE > Task :model:fn-execution:compileJava UP-TO-DATE > Task :model:fn-execution:classes UP-TO-DATE > Task :model:fn-execution:shadowJar UP-TO-DATE > Task :sdks:java:core:compileJava UP-TO-DATE > Task :sdks:java:core:classes UP-TO-DATE > Task :sdks:java:core:shadowJar UP-TO-DATE > Task :vendor:sdks-java-extensions-protobuf:compileJava FROM-CACHE > Task :vendor:sdks-java-extensions-protobuf:classes UP-TO-DATE > Task :sdks:java:fn-execution:compileJava UP-TO-DATE > Task :sdks:java:fn-execution:classes UP-TO-DATE > Task :sdks:java:io:kafka:compileJava UP-TO-DATE > Task :sdks:java:io:kafka:classes UP-TO-DATE > Task :sdks:java:extensions:google-cloud-platform-core:compileJava UP-TO-DATE > Task :sdks:java:extensions:google-cloud-platform-core:classes UP-TO-DATE > Task :sdks:java:fn-execution:jar UP-TO-DATE > Task :sdks:java:io:kafka:jar UP-TO-DATE > Task :sdks:java:extensions:google-cloud-platform-core:jar UP-TO-DATE > Task :runners:core-construction-java:compileJava UP-TO-DATE > Task :runners:core-construction-java:classes UP-TO-DATE > Task :runners:core-construction-java:jar UP-TO-DATE > Task :vendor:sdks-java-extensions-protobuf:shadowJar > Task :runners:core-java:compileJava UP-TO-DATE > Task :runners:core-java:classes UP-TO-DATE > Task :runners:core-java:jar UP-TO-DATE > Task :sdks:java:harness:compileJava UP-TO-DATE > Task :sdks:java:harness:classes UP-TO-DATE > Task :sdks:java:harness:jar > Task :sdks:java:harness:shadowJar UP-TO-DATE > Task :sdks:java:extensions:protobuf:extractIncludeProto > Task :sdks:java:extensions:protobuf:generateProto NO-SOURCE > Task :sdks:java:extensions:protobuf:compileJava FROM-CACHE > Task :sdks:java:extensions:protobuf:classes UP-TO-DATE > Task :sdks:java:extensions:protobuf:jar > Task :sdks:java:io:google-cloud-platform:compileJava FROM-CACHE > Task :sdks:java:io:google-cloud-platform:classes UP-TO-DATE > Task :sdks:java:io:google-cloud-platform:jar > Task :runners:java-fn-execution:compileJava FROM-CACHE > Task :runners:java-fn-execution:classes UP-TO-DATE > Task :runners:java-fn-execution:jar > Task :runners:flink:1.9:compileJava FROM-CACHE > Task :runners:flink:1.9:classes > Task :runners:flink:1.9:jar > Task :runners:flink:1.9:job-server:compileJava NO-SOURCE > Task :runners:flink:1.9:job-server:classes UP-TO-DATE > Task :runners:flink:1.9:job-server:shadowJar > Task :runners:flink:1.9:job-server-container:copyDockerfileDependencies > Task :runners:flink:1.9:job-server-container:dockerPrepare > Task :runners:flink:1.9:job-server-container:docker Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 58s 58 actionable tasks: 16 executed, 6 from cache, 36 up-to-date Publishing build scan... https://gradle.com/s/kqv7h3bws2wdq [beam_LoadTests_Java_Combine_Portable_Flink_Batch] $ /bin/bash -xe /tmp/jenkins3680403557783725952.sh + echo 'Tagging image...' Tagging image... [beam_LoadTests_Java_Combine_Portable_Flink_Batch] $ /bin/bash -xe /tmp/jenkins3095438528072116417.sh + docker tag gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest [beam_LoadTests_Java_Combine_Portable_Flink_Batch] $ /bin/bash -xe /tmp/jenkins4489883657229385944.sh + echo 'Pushing image...' Pushing image... [beam_LoadTests_Java_Combine_Portable_Flink_Batch] $ /bin/bash -xe /tmp/jenkins7217377338939886267.sh + docker push gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest The push refers to repository [gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server] a093f3591473: Preparing b191cd8defc8: Preparing e35c185253ca: Preparing a6ded049566a: Preparing e7fe5541de5f: Preparing 03ff63c55220: Preparing bee1e39d7c3a: Preparing 1f59a4b2e206: Preparing 0ca7f54856c0: Preparing ebb9ae013834: Preparing 03ff63c55220: Waiting bee1e39d7c3a: Waiting 1f59a4b2e206: Waiting 0ca7f54856c0: Waiting ebb9ae013834: Waiting a6ded049566a: Layer already exists e7fe5541de5f: Layer already exists 03ff63c55220: Layer already exists bee1e39d7c3a: Layer already exists 0ca7f54856c0: Layer already exists 1f59a4b2e206: Layer already exists ebb9ae013834: Layer already exists a093f3591473: Pushed e35c185253ca: Pushed b191cd8defc8: Pushed latest: digest: sha256:2b1253585340ccbaf1e0d86ef4cfe8afe5727bf1d5f5811014e0fbb0763fdb50 size: 2427 [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties content JOB_SERVER_IMAGE=gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest CLUSTER_NAME=streaming-3 DETACHED_MODE=true HARNESS_IMAGES_TO_PULL=gcr.io/apache-beam-testing/beam_portability/java_sdk:latest FLINK_NUM_WORKERS=16 FLINK_DOWNLOAD_URL=https://archive.apache.org/dist/flink/flink-1.9.1/flink-1.9.1-bin-scala_2.11.tgz GCS_BUCKET=gs://beam-flink-cluster HADOOP_DOWNLOAD_URL=https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-7.0/flink-shaded-hadoop-2-uber-2.8.3-7.0.jar FLINK_TASKMANAGER_SLOTS=1 ARTIFACTS_DIR=gs://beam-flink-cluster/streaming-3 GCLOUD_ZONE=us-central1-a [EnvInject] - Variables injected successfully. [beam_LoadTests_Java_Combine_Portable_Flink_Batch] $ /bin/bash -xe /tmp/jenkins5953650193304024910.sh + echo Setting up flink cluster Setting up flink cluster [beam_LoadTests_Java_Combine_Portable_Flink_Batch] $ /bin/bash -xe /tmp/jenkins319433620560313628.sh + cd <https://builds.apache.org/job/beam_LoadTests_Java_Combine_Portable_Flink_Batch/ws/src/.test-infra/dataproc> + ./flink_cluster.sh create + GCLOUD_ZONE=us-central1-a + DATAPROC_VERSION=1.2 + MASTER_NAME=streaming-3-m + INIT_ACTIONS_FOLDER_NAME=init-actions + FLINK_INIT=gs://beam-flink-cluster/init-actions/flink.sh + BEAM_INIT=gs://beam-flink-cluster/init-actions/beam.sh + DOCKER_INIT=gs://beam-flink-cluster/init-actions/docker.sh + FLINK_LOCAL_PORT=8081 + FLINK_TASKMANAGER_SLOTS=1 + TASK_MANAGER_MEM=10240 + YARN_APPLICATION_MASTER= + create + upload_init_actions + echo 'Uploading initialization actions to GCS bucket: gs://beam-flink-cluster' Uploading initialization actions to GCS bucket: gs://beam-flink-cluster + gsutil cp -r init-actions/beam.sh init-actions/docker.sh init-actions/flink.sh gs://beam-flink-cluster/init-actions Copying file://init-actions/beam.sh [Content-Type=text/x-sh]... / [0 files][ 0.0 B/ 2.3 KiB] / [1 files][ 2.3 KiB/ 2.3 KiB] Copying file://init-actions/docker.sh [Content-Type=text/x-sh]... / [1 files][ 2.3 KiB/ 6.0 KiB] / [2 files][ 6.0 KiB/ 6.0 KiB] Copying file://init-actions/flink.sh [Content-Type=text/x-sh]... / [2 files][ 6.0 KiB/ 13.4 KiB] / [3 files][ 13.4 KiB/ 13.4 KiB] Operation completed over 3 objects/13.4 KiB. + create_cluster + local metadata=flink-snapshot-url=https://archive.apache.org/dist/flink/flink-1.9.1/flink-1.9.1-bin-scala_2.11.tgz, + metadata+=flink-start-yarn-session=true, + metadata+=flink-taskmanager-slots=1, + metadata+=hadoop-jar-url=https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-7.0/flink-shaded-hadoop-2-uber-2.8.3-7.0.jar + [[ -n gcr.io/apache-beam-testing/beam_portability/java_sdk:latest ]] + metadata+=,beam-sdk-harness-images-to-pull=gcr.io/apache-beam-testing/beam_portability/java_sdk:latest + [[ -n gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest ]] + metadata+=,beam-job-server-image=gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest + local image_version=1.2 + echo 'Starting dataproc cluster. Dataproc version: 1.2' Starting dataproc cluster. Dataproc version: 1.2 + local num_dataproc_workers=17 + gcloud dataproc clusters create streaming-3 --region=global --num-workers=17 --initialization-actions gs://beam-flink-cluster/init-actions/docker.sh,gs://beam-flink-cluster/init-actions/beam.sh,gs://beam-flink-cluster/init-actions/flink.sh --metadata flink-snapshot-url=https://archive.apache.org/dist/flink/flink-1.9.1/flink-1.9.1-bin-scala_2.11.tgz,flink-start-yarn-session=true,flink-taskmanager-slots=1,hadoop-jar-url=https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-7.0/flink-shaded-hadoop-2-uber-2.8.3-7.0.jar,beam-sdk-harness-images-to-pull=gcr.io/apache-beam-testing/beam_portability/java_sdk:latest,beam-job-server-image=gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest, --image-version=1.2 --zone=us-central1-a --quiet ERROR: (gcloud.dataproc.clusters.create) ALREADY_EXISTS: Already exists: Failed to create cluster: Cluster projects/apache-beam-testing/regions/global/clusters/streaming-3 Build step 'Execute shell' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
