Ankur Goenka created BEAM-6102:
----------------------------------
Summary: Wordcount broken on master
Key: BEAM-6102
URL: https://issues.apache.org/jira/browse/BEAM-6102
Project: Beam
Issue Type: Bug
Components: sdk-java-core
Reporter: Ankur Goenka
Assignee: Kenneth Knowles
The wordcount is broken on the master.
Its failing with serialization error mentioned below
{noformat}
11:08:59 AM: Executing task 'WordCount.main()'... Parallel execution is an
incubating feature. > Task :buildSrc:compileJava NO-SOURCE > Task
:buildSrc:compileGroovy UP-TO-DATE > Task :buildSrc:processResources NO-SOURCE
> Task :buildSrc:classes UP-TO-DATE > Task :buildSrc:jar UP-TO-DATE > Task
:buildSrc:assemble UP-TO-DATE > Task :buildSrc:spotlessGroovy UP-TO-DATE > Task
:buildSrc:spotlessGroovyCheck UP-TO-DATE > Task :buildSrc:spotlessGroovyGradle
UP-TO-DATE > Task :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE > Task
:buildSrc:spotlessCheck UP-TO-DATE > Task :buildSrc:compileTestJava NO-SOURCE >
Task :buildSrc:compileTestGroovy NO-SOURCE > Task
:buildSrc:processTestResources NO-SOURCE > Task :buildSrc:testClasses
UP-TO-DATE > Task :buildSrc:test NO-SOURCE > Task :buildSrc:check UP-TO-DATE >
Task :buildSrc:build UP-TO-DATE Parallel execution with configuration on demand
is an incubating feature. > Configure project :beam-model-pipeline
applyPortabilityNature with default configuration for project
beam-model-pipeline > Configure project :beam-model-job-management
applyPortabilityNature with default configuration for project
beam-model-job-management > Configure project :beam-model-fn-execution
applyPortabilityNature with default configuration for project
beam-model-fn-execution > Task :beam-examples-java:processResources NO-SOURCE >
Task :beam-sdks-java-core:generateAvroProtocol NO-SOURCE > Task
:beam-sdks-java-extensions-google-cloud-platform-core:processResources
NO-SOURCE > Task :beam-sdks-java-io-google-cloud-platform:processResources
NO-SOURCE > Task :beam-vendor-grpc-v1_13_1:compileJava NO-SOURCE > Task
:beam-runners-core-construction-java:processResources NO-SOURCE > Task
:beam-sdks-java-extensions-protobuf:extractProto UP-TO-DATE > Task
:beam-model-pipeline:extractProto UP-TO-DATE > Task
:beam-model-job-management:processResources UP-TO-DATE > Task
:beam-runners-core-java:processResources NO-SOURCE > Task
:beam-sdks-java-fn-execution:processResources NO-SOURCE > Task
:beam-sdks-java-harness:processResources NO-SOURCE > Task
:beam-vendor-sdks-java-extensions-protobuf:processResources NO-SOURCE > Task
:beam-sdks-java-extensions-protobuf:processResources NO-SOURCE > Task
:beam-sdks-java-core:generateAvroJava NO-SOURCE > Task
:beam-vendor-grpc-v1_13_1:processResources NO-SOURCE > Task
:beam-model-fn-execution:processResources UP-TO-DATE > Task
:beam-model-pipeline:processResources UP-TO-DATE > Task
:beam-runners-local-java-core:processResources NO-SOURCE > Task
:beam-runners-java-fn-execution:processResources NO-SOURCE > Task
:beam-runners-direct-java:processResources NO-SOURCE > Task
:beam-vendor-grpc-v1_13_1:classes UP-TO-DATE > Task
:beam-sdks-java-core:processResources UP-TO-DATE > Task
:beam-runners-google-cloud-dataflow-java:processResources UP-TO-DATE > Task
:beam-vendor-grpc-v1_13_1:shadowJar UP-TO-DATE > Task
:beam-model-fn-execution:extractIncludeProto UP-TO-DATE > Task
:beam-model-pipeline:extractIncludeProto UP-TO-DATE > Task
:beam-model-job-management:extractIncludeProto UP-TO-DATE > Task
:beam-model-pipeline:generateProto UP-TO-DATE > Task
:beam-model-pipeline:compileJava UP-TO-DATE > Task :beam-model-pipeline:classes
UP-TO-DATE > Task :beam-model-pipeline:shadowJar UP-TO-DATE > Task
:beam-model-pipeline:jar UP-TO-DATE > Task
:beam-model-job-management:extractProto UP-TO-DATE > Task
:beam-model-fn-execution:extractProto UP-TO-DATE > Task
:beam-model-fn-execution:generateProto UP-TO-DATE > Task
:beam-model-job-management:generateProto UP-TO-DATE > Task
:beam-model-job-management:compileJava UP-TO-DATE > Task
:beam-model-job-management:classes UP-TO-DATE > Task
:beam-model-fn-execution:compileJava UP-TO-DATE > Task
:beam-model-fn-execution:classes UP-TO-DATE > Task
:beam-model-job-management:shadowJar UP-TO-DATE > Task
:beam-sdks-java-core:compileJava UP-TO-DATE > Task :beam-sdks-java-core:classes
UP-TO-DATE > Task :beam-model-fn-execution:shadowJar UP-TO-DATE > Task
:beam-sdks-java-core:shadowJar UP-TO-DATE > Task
:beam-sdks-java-extensions-protobuf:extractIncludeProto UP-TO-DATE > Task
:beam-sdks-java-extensions-protobuf:generateProto NO-SOURCE > Task
:beam-sdks-java-fn-execution:compileJava UP-TO-DATE > Task
:beam-sdks-java-fn-execution:classes UP-TO-DATE > Task
:beam-runners-local-java-core:compileJava UP-TO-DATE > Task
:beam-runners-local-java-core:classes UP-TO-DATE > Task
:beam-runners-core-construction-java:compileJava UP-TO-DATE > Task
:beam-sdks-java-extensions-google-cloud-platform-core:compileJava UP-TO-DATE >
Task :beam-sdks-java-extensions-google-cloud-platform-core:classes UP-TO-DATE >
Task :beam-sdks-java-extensions-protobuf:compileJava UP-TO-DATE > Task
:beam-sdks-java-extensions-protobuf:classes UP-TO-DATE > Task
:beam-vendor-sdks-java-extensions-protobuf:compileJava UP-TO-DATE > Task
:beam-vendor-sdks-java-extensions-protobuf:classes UP-TO-DATE > Task
:beam-runners-core-construction-java:classes UP-TO-DATE > Task
:beam-runners-local-java-core:shadowJar UP-TO-DATE > Task
:beam-sdks-java-fn-execution:shadowJar UP-TO-DATE > Task
:beam-sdks-java-extensions-protobuf:shadowJar UP-TO-DATE > Task
:beam-sdks-java-extensions-google-cloud-platform-core:shadowJar UP-TO-DATE >
Task :beam-vendor-sdks-java-extensions-protobuf:shadowJar UP-TO-DATE > Task
:beam-runners-core-construction-java:shadowJar UP-TO-DATE > Task
:beam-runners-core-java:compileJava UP-TO-DATE > Task
:beam-runners-core-java:classes UP-TO-DATE > Task
:beam-runners-core-java:shadowJar UP-TO-DATE > Task
:beam-sdks-java-io-google-cloud-platform:compileJava UP-TO-DATE > Task
:beam-sdks-java-io-google-cloud-platform:classes UP-TO-DATE > Task
:beam-sdks-java-harness:compileJava UP-TO-DATE > Task
:beam-sdks-java-harness:classes UP-TO-DATE > Task :beam-sdks-java-harness:jar
UP-TO-DATE > Task :beam-sdks-java-io-google-cloud-platform:shadowJar UP-TO-DATE
> Task :beam-sdks-java-harness:shadowJar UP-TO-DATE > Task
:beam-examples-java:compileJava UP-TO-DATE > Task :beam-examples-java:classes
UP-TO-DATE > Task :beam-runners-java-fn-execution:compileJava UP-TO-DATE > Task
:beam-runners-java-fn-execution:classes UP-TO-DATE > Task
:beam-runners-java-fn-execution:shadowJar UP-TO-DATE > Task
:beam-runners-google-cloud-dataflow-java:compileJava UP-TO-DATE > Task
:beam-runners-google-cloud-dataflow-java:classes UP-TO-DATE > Task
:beam-runners-direct-java:compileJava UP-TO-DATE > Task
:beam-runners-direct-java:classes UP-TO-DATE > Task
:beam-runners-google-cloud-dataflow-java:shadowJar UP-TO-DATE > Task
:beam-runners-direct-java:shadowJar UP-TO-DATE > Task
:beam-examples-java:WordCount.main() Nov 20, 2018 11:09:02 AM
org.apache.beam.sdk.extensions.gcp.options.GcpOptions$DefaultProjectFactory
create INFO: Inferred default GCP project 'google.com:clouddfe' from gcloud. If
this is the incorrect project, please cancel this Pipeline and specify the
command-line argument --project. Nov 20, 2018 11:09:02 AM
com.google.auth.oauth2.DefaultCredentialsProvider
warnAboutProblematicCredentials WARNING: Your application has authenticated
using end user credentials from Google Cloud SDK. We recommend that most server
applications use service accounts instead. If your application continues to use
end user credentials from Cloud SDK, you might receive a "quota exceeded" or
"API not enabled" error. For more information about service accounts, see
https://cloud.google.com/docs/authentication/. Nov 20, 2018 11:09:03 AM
org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory
tryCreateDefaultBucket INFO: No tempLocation specified, attempting to use
default bucket: dataflow-staging-us-central1-927334603519 Nov 20, 2018 11:09:03
AM
org.apache.beam.sdk.util.RetryHttpRequestInitializer$LoggingHttpBackOffHandler
handleResponse WARNING: Request failed with code 409, performed 0 retries due
to IOExceptions, performed 0 retries due to unsuccessful status codes, HTTP
framework says request can be retried, (caller responsible for retrying):
https://www.googleapis.com/storage/v1/b?predefinedAcl=projectPrivate&predefinedDefaultObjectAcl=projectPrivate&project=google.com:clouddfe
Nov 20, 2018 11:09:03 AM
org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory
create INFO: No stagingLocation provided, falling back to gcpTempLocation Nov
20, 2018 11:09:04 AM org.apache.beam.runners.dataflow.DataflowRunner
fromOptions INFO: PipelineOptions.filesToStage was not specified. Defaulting to
files from the classpath: will stage 120 files. Enable logging at DEBUG level
to see which files will be staged. Nov 20, 2018 11:09:04 AM
org.apache.beam.runners.dataflow.DataflowRunner run INFO: Executing pipeline on
the Dataflow Service, which will have billing implications related to Google
Compute Engine usage and other Google Cloud Services. Nov 20, 2018 11:09:04 AM
org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO:
Uploading 120 files from PipelineOptions.filesToStage to staging location to
prepare for execution. Nov 20, 2018 11:09:04 AM
org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
WARNING: Skipping non-existent file to stage
/usr/local/google/home/goenka/d/work/tmp/tmp_beam/beam/examples/java/build/resources/main.
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/.gradle/caches/modules-2/files-2.1/com.google.http-client/google-http-client/1.27.0/52d6076f58edfedb55526c0cac4ba155c6dc645b/google-http-client-1.27.0.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/google-http-client-1.27.0-XiDeJfgpNiiCZxI1a4Q27Q.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/.gradle/caches/modules-2/files-2.1/com.google.oauth-client/google-oauth-client-java6/1.27.0/90570eedf1ae6188ee5028c11ec423fe52336373/google-oauth-client-java6-1.27.0.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/google-oauth-client-java6-1.27.0-VWEWXu2J1auJGvb5eLfbhg.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/.gradle/caches/modules-2/files-2.1/com.google.apis/google-api-services-pubsub/v1-rev20181105-1.27.0/31f49e03ec44c72845a6a0572ccba74d7aee1f57/google-api-services-pubsub-v1-rev20181105-1.27.0.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/google-api-services-pubsub-v1-rev20181105-1.27.0-_3JmrTY281LX4AcOiP7LkQ.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/.gradle/caches/modules-2/files-2.1/com.google.apis/google-api-services-dataflow/v1b3-rev20181107-1.27.0/d0b3764765916d65fd75a52ae1a8cf13a150450a/google-api-services-dataflow-v1b3-rev20181107-1.27.0.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/google-api-services-dataflow-v1b3-rev20181107-1.27.0-ZCpCwXOULzPs_S2gBBH7TA.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/.gradle/caches/modules-2/files-2.1/com.google.oauth-client/google-oauth-client/1.27.0/1979e3b499f1ef49959005ca59101791361abca9/google-oauth-client-1.27.0.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/google-oauth-client-1.27.0-waNrxu589XHIvY73DhrxNA.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/.gradle/caches/modules-2/files-2.1/com.google.apis/google-api-services-clouddebugger/v2-rev20180801-1.27.0/2866c6cfca03ef7bbf687ba3b62fa843377e664b/google-api-services-clouddebugger-v2-rev20180801-1.27.0.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/google-api-services-clouddebugger-v2-rev20180801-1.27.0-xrIcl7yUGCktYEb9_xtvjQ.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/.gradle/caches/modules-2/files-2.1/com.google.api-client/google-api-client-jackson2/1.27.0/1e1a91b684fc2e6e97d2369c7d8d1d9efc115543/google-api-client-jackson2-1.27.0.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/google-api-client-jackson2-1.27.0-M222JGX1Wh3HoNtcvwCaEw.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/.gradle/caches/modules-2/files-2.1/com.google.api-client/google-api-client-java6/1.27.0/2c06247935819b429424797d9844aa33955f4fb0/google-api-client-java6-1.27.0.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/google-api-client-java6-1.27.0-zPlGOao_rbTkn6baqjfy2w.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/d/work/tmp/tmp_beam/beam/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.9.0-SNAPSHOT.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/beam-sdks-java-io-google-cloud-platform-2.9.0-SNAPSHOT-xFFgKvdT2GS2td3iSJaJqQ.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/.gradle/caches/modules-2/files-2.1/com.google.apis/google-api-services-bigquery/v2-rev20181104-1.27.0/d539fb4b7ac318cadc344579fa6f80ae6cb8070e/google-api-services-bigquery-v2-rev20181104-1.27.0.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/google-api-services-bigquery-v2-rev20181104-1.27.0-oUKA0dpY-zmhgM8DIAq8Zg.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/.gradle/caches/modules-2/files-2.1/com.google.http-client/google-http-client-protobuf/1.27.0/b81859b8a284ea416d74644a70524c4d6be3e9a/google-http-client-protobuf-1.27.0.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/google-http-client-protobuf-1.27.0-1E0HsNaSwDtqOduUs-NssA.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/.gradle/caches/modules-2/files-2.1/com.google.http-client/google-http-client-jackson/1.27.0/2b5eff5b6a343e716db21673743576db9a53e5ae/google-http-client-jackson-1.27.0.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/google-http-client-jackson-1.27.0-XQrBFTlCGWMqVu9vUPkMcw.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/.gradle/caches/modules-2/files-2.1/com.google.api-client/google-api-client/1.27.0/7d498c11db92fb72521784212c953612a42e50db/google-api-client-1.27.0.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/google-api-client-1.27.0-UjH-dlvVLqF5D_4Trh8foQ.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/.gradle/caches/modules-2/files-2.1/com.google.apis/google-api-services-storage/v1-rev20181013-1.27.0/10593f99e06d962017b0663348ee1fed0f8528/google-api-services-storage-v1-rev20181013-1.27.0.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/google-api-services-storage-v1-rev20181013-1.27.0-Y65RPabM4FA03rQX5lUvKg.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/d/work/tmp/tmp_beam/beam/model/job-management/build/libs/beam-model-job-management-2.9.0-SNAPSHOT.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/beam-model-job-management-2.9.0-SNAPSHOT-PRRkG0x941fKfrBa33f-Qg.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/.gradle/caches/modules-2/files-2.1/com.google.apis/google-api-services-cloudresourcemanager/v1-rev20181015-1.27.0/354796285db3b2fce9089147db3fe1f2a8cca444/google-api-services-cloudresourcemanager-v1-rev20181015-1.27.0.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/google-api-services-cloudresourcemanager-v1-rev20181015-1.27.0-SQfpsiUEwOlTbzfnWC4EzA.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/d/work/tmp/tmp_beam/beam/model/pipeline/build/libs/beam-model-pipeline-2.9.0-SNAPSHOT.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/beam-model-pipeline-2.9.0-SNAPSHOT-XD9dbaFWaiUJnjDcCAkrOg.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpcore/4.4.9/a86ce739e5a7175b4b234c290a00a5fdb80957a0/httpcore-4.4.9.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/httpcore-4.4.9-uJRVUHg5wJ1hGWYd79IWag.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/d/work/tmp/tmp_beam/beam/sdks/java/extensions/protobuf/build/libs/beam-sdks-java-extensions-protobuf-2.9.0-SNAPSHOT.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/beam-sdks-java-extensions-protobuf-2.9.0-SNAPSHOT-4q_iSCAMQwCGDowjT6QSKw.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/d/work/tmp/tmp_beam/beam/sdks/java/extensions/google-cloud-platform-core/build/libs/beam-sdks-java-extensions-google-cloud-platform-core-2.9.0-SNAPSHOT.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/beam-sdks-java-extensions-google-cloud-platform-core-2.9.0-SNAPSHOT-qVkigYLlAbU1ujEdaTQZ0Q.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpclient/4.5.5/1603dfd56ebcd583ccdf337b6c3984ac55d89e58/httpclient-4.5.5.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/httpclient-4.5.5-l-flsTVHa30lpasx4epJIg.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/d/work/tmp/tmp_beam/beam/runners/core-construction-java/build/libs/beam-runners-core-construction-java-2.9.0-SNAPSHOT.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/beam-runners-core-construction-java-2.9.0-SNAPSHOT-XbHTR4DoxxMKF5aSfjqWYQ.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/.gradle/caches/modules-2/files-2.1/com.google.http-client/google-http-client-jackson2/1.27.0/25ee9240648b5dfd01a3385937bcf332b6f222a7/google-http-client-jackson2-1.27.0.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/google-http-client-jackson2-1.27.0-s26dP1SAvqbYDq_HiJ9Bbw.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/d/work/tmp/tmp_beam/beam/runners/direct-java/build/libs/beam-runners-direct-java-2.9.0-SNAPSHOT.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/beam-runners-direct-java-2.9.0-SNAPSHOT-jn9fS-cJcB_-dTRLAU-meQ.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/d/work/tmp/tmp_beam/beam/runners/google-cloud-dataflow-java/build/libs/beam-runners-google-cloud-dataflow-java-2.9.0-SNAPSHOT.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/beam-runners-google-cloud-dataflow-java-2.9.0-SNAPSHOT-oQdbiEtnM6C2iUC5axtRjA.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/d/work/tmp/tmp_beam/beam/examples/java/build/classes/java/main
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/main-cxzukVaCwjdrlMI-NQ8Jew.jar
Nov 20, 2018 11:09:05 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/d/work/tmp/tmp_beam/beam/vendor/grpc-v1_13_1/build/libs/beam-vendor-grpc-v1_13_1-2.9.0-SNAPSHOT.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/beam-vendor-grpc-v1_13_1-2.9.0-SNAPSHOT-E8CXf40HkGDxZZU040A-bA.jar
Nov 20, 2018 11:09:06 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading
/usr/local/google/home/goenka/d/work/tmp/tmp_beam/beam/sdks/java/core/build/libs/beam-sdks-java-core-2.9.0-SNAPSHOT.jar
to
gs://dataflow-staging-us-central1-927334603519/temp/staging/beam-sdks-java-core-2.9.0-SNAPSHOT-Aoyvg0doZsYvr7jINSJAKA.jar
Nov 20, 2018 11:09:21 AM org.apache.beam.runners.dataflow.util.PackageUtil
stageClasspathElements INFO: Staging files complete: 91 files cached, 28 files
newly uploaded Nov 20, 2018 11:09:21 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding ReadLines/Read as step s1 Nov 20, 2018 11:09:21 AM
org.apache.beam.sdk.io.FileBasedSource getEstimatedSizeBytes INFO: Filepattern
gs://apache-beam-samples/shakespeare/kinglear.txt matched 1 files with total
size 157283 Nov 20, 2018 11:09:21 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WordCount.CountWords/ParDo(ExtractWords) as step s2 Nov 20, 2018
11:09:21 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WordCount.CountWords/Count.PerElement/Init/Map as step s3 Nov 20,
2018 11:09:21 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey as step
s4 Nov 20, 2018 11:09:21 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues
as step s5 Nov 20, 2018 11:09:21 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding MapElements/Map as step s6 Nov 20, 2018 11:09:21 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign as step s7
Nov 20, 2018 11:09:21 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles
as step s8 Nov 20, 2018 11:09:21 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten as step
s9 Nov 20, 2018 11:09:21 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten as step
s10 Nov 20, 2018 11:09:21 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum as step
s11 Nov 20, 2018 11:09:21 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections as
step s12 Nov 20, 2018 11:09:21 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)
as step s13 Nov 20, 2018 11:09:21 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView as
step s14 Nov 20, 2018 11:09:21 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)
as step s15 Nov 20, 2018 11:09:21 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)
as step s16 Nov 20, 2018 11:09:21 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map
as step s17 Nov 20, 2018 11:09:21 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize as step
s18 Nov 20, 2018 11:09:21 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with
random key as step s19 Nov 20, 2018 11:09:21 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
as step s20 Nov 20, 2018 11:09:21 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey
as step s21 Nov 20, 2018 11:09:21 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
as step s22 Nov 20, 2018 11:09:21 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map
as step s23 Nov 20, 2018 11:09:21 AM
org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging pipeline
description to gs://dataflow-staging-us-central1-927334603519/temp/staging/ Nov
20, 2018 11:09:22 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage INFO: Uploading <63064 bytes, hash 1eMyLHEGQDSnbL_rtt2mOQ> to
gs://dataflow-staging-us-central1-927334603519/temp/staging/pipeline-1eMyLHEGQDSnbL_rtt2mOQ.pb
Dataflow SDK version: 2.9.0-SNAPSHOT Nov 20, 2018 11:09:24 AM
org.apache.beam.runners.dataflow.DataflowRunner run Submitted job:
2018-11-20_11_09_23-3437480453113408186 INFO: To access the Dataflow monitoring
console, please navigate to
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-20_11_09_23-3437480453113408186?project=google.com%3Aclouddfe
Nov 20, 2018 11:09:24 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs
--project=google.com:clouddfe cancel --region=us-central1
2018-11-20_11_09_23-3437480453113408186 Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:23.285Z: Autoscaling is enabled for job
2018-11-20_11_09_23-3437480453113408186. The number of workers will be between
1 and 1000. Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:23.346Z: Autoscaling was automatically enabled for job
2018-11-20_11_09_23-3437480453113408186. Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:26.020Z: Checking permissions granted to controller
Service Account. Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:30.430Z: Worker configuration: n1-standard-1 in
us-central1-a. Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:31.335Z: Expanding CoGroupByKey operations into
optimizable parts. Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:31.555Z: Expanding GroupByKey operations into
optimizable parts. Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:31.614Z: Lifting ValueCombiningMappingFns into
MergeBucketsMappingFns Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:31.886Z: Fusing adjacent ParDo, Read, Write, and Flatten
operations Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:31.941Z: Fusing consumer
WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)
into
WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)
Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:31.994Z: Fusing consumer
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
into
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with
random key Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:32.044Z: Fusing consumer
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
into
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:32.099Z: Fusing consumer
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map
into
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:32.152Z: Fusing consumer
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
into
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:32.196Z: Fusing consumer
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
into
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:32.244Z: Fusing consumer
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
into
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:32.312Z: Fusing consumer
WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map
into
WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)
Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:32.364Z: Fusing consumer
WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize into
WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map
Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:32.410Z: Fusing consumer
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with
random key into WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize Nov 20,
2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:32.459Z: Unzipping flatten s12 for input
s11.org.apache.beam.sdk.values.PCollection.<init>:402#20ff67585e33a8f6 Nov 20,
2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:32.491Z: Fusing unzipped copy of
WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow),
through flatten
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections,
into producer
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum Nov 20,
2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:32.536Z: Fusing consumer
WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)
into
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles
Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:32.575Z: Fusing consumer
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum into
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten Nov 20,
2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:32.627Z: Fusing consumer
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues
into
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read Nov
20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:32.686Z: Fusing consumer
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write
into
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify
Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:32.726Z: Fusing consumer
WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign into MapElements/Map
Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:32.781Z: Fusing consumer
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract
into
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues
Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:32.838Z: Fusing consumer
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten into
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow
Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:32.884Z: Fusing consumer
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial
into WordCount.CountWords/Count.PerElement/Init/Map Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:32.928Z: Fusing consumer
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write
into
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify
Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:32.975Z: Fusing consumer
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles
into WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign Nov 20, 2018
11:09:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler
process INFO: 2018-11-20T19:09:33.029Z: Fusing consumer
WordCount.CountWords/Count.PerElement/Init/Map into
WordCount.CountWords/ParDo(ExtractWords) Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:33.074Z: Fusing consumer
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow
into
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read Nov
20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:33.112Z: Fusing consumer MapElements/Map into
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract
Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:33.152Z: Fusing consumer
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify
into
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial
Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:33.190Z: Fusing consumer
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify
into
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles
Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:33.245Z: Fusing consumer
WordCount.CountWords/ParDo(ExtractWords) into ReadLines/Read Nov 20, 2018
11:09:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler
process INFO: 2018-11-20T19:09:33.764Z: Executing operation
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create
Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:33.810Z: Executing operation
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Create
Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:33.850Z: Executing operation
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create
Nov 20, 2018 11:09:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:33.889Z: Starting 1 workers in us-central1-a... Nov 20,
2018 11:09:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:34.261Z: Executing operation
ReadLines/Read+WordCount.CountWords/ParDo(ExtractWords)+WordCount.CountWords/Count.PerElement/Init/Map+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write
Nov 20, 2018 11:09:44 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:09:43.287Z: Autoscaling: Raised the number of workers to 0
based on the rate of progress in the currently running step(s). Nov 20, 2018
11:10:16 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler
process INFO: 2018-11-20T19:10:15.798Z: Autoscaling: Raised the number of
workers to 1 based on the rate of progress in the currently running step(s).
Nov 20, 2018 11:10:16 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:10:15.859Z: Autoscaling: Would further reduce the number of
workers but reached the minimum number allowed for the job. Nov 20, 2018
11:10:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler
process INFO: 2018-11-20T19:10:21.327Z: Workers have started successfully. Nov
20, 2018 11:10:22 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:10:21.511Z: Workers have started successfully. Nov 20, 2018
11:10:35 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler
process SEVERE: 2018-11-20T19:10:32.138Z: java.lang.RuntimeException:
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.IllegalArgumentException: unable to deserialize Serialized DoFnInfo
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:193)
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:164)
at
org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:63)
at
org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:50)
at
org.apache.beam.runners.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:87)
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.create(IntrinsicMapTaskExecutorFactory.java:124)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:337)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:291)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:135)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:115)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:102)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) Caused by:
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.IllegalArgumentException: unable to deserialize Serialized DoFnInfo
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2214)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache.get(LocalCache.java:4053)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4899)
at
org.apache.beam.runners.dataflow.worker.UserParDoFnFactory.create(UserParDoFnFactory.java:91)
at
org.apache.beam.runners.dataflow.worker.DefaultParDoFnFactory.create(DefaultParDoFnFactory.java:75)
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.createParDoOperation(IntrinsicMapTaskExecutorFactory.java:263)
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.access$000(IntrinsicMapTaskExecutorFactory.java:85)
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:182)
... 14 more Caused by: java.lang.IllegalArgumentException: unable to
deserialize Serialized DoFnInfo at
org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:74)
at
org.apache.beam.runners.dataflow.worker.UserParDoFnFactory$UserDoFnExtractor.getDoFnInfo(UserParDoFnFactory.java:62)
at
org.apache.beam.runners.dataflow.worker.UserParDoFnFactory.lambda$create$0(UserParDoFnFactory.java:93)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4904)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3628)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2336)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2295)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2208)
... 21 more Caused by: java.io.InvalidClassException:
org.apache.beam.sdk.values.WindowingStrategy; local class incompatible: stream
classdesc serialVersionUID = -6607512772692666907, local class serialVersionUID
= -3616600070988263902 at
java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616) at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1630) at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521) at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018) at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942) at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:373) at
org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:71)
... 28 more Nov 20, 2018 11:10:39 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2018-11-20T19:10:36.934Z: java.lang.RuntimeException:
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.IllegalArgumentException: unable to deserialize Serialized DoFnInfo
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:193)
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:164)
at
org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:63)
at
org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:50)
at
org.apache.beam.runners.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:87)
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.create(IntrinsicMapTaskExecutorFactory.java:124)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:337)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:291)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:135)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:115)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:102)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) Caused by:
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.IllegalArgumentException: unable to deserialize Serialized DoFnInfo
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2214)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache.get(LocalCache.java:4053)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4899)
at
org.apache.beam.runners.dataflow.worker.UserParDoFnFactory.create(UserParDoFnFactory.java:91)
at
org.apache.beam.runners.dataflow.worker.DefaultParDoFnFactory.create(DefaultParDoFnFactory.java:75)
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.createParDoOperation(IntrinsicMapTaskExecutorFactory.java:263)
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.access$000(IntrinsicMapTaskExecutorFactory.java:85)
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:182)
... 14 more Caused by: java.lang.IllegalArgumentException: unable to
deserialize Serialized DoFnInfo at
org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:74)
at
org.apache.beam.runners.dataflow.worker.UserParDoFnFactory$UserDoFnExtractor.getDoFnInfo(UserParDoFnFactory.java:62)
at
org.apache.beam.runners.dataflow.worker.UserParDoFnFactory.lambda$create$0(UserParDoFnFactory.java:93)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4904)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3628)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2336)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2295)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2208)
... 21 more Caused by: java.io.InvalidClassException:
org.apache.beam.sdk.values.WindowingStrategy; local class incompatible: stream
classdesc serialVersionUID = -6607512772692666907, local class serialVersionUID
= -3616600070988263902 at
java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616) at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1630) at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521) at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018) at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942) at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:373) at
org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:71)
... 28 more Nov 20, 2018 11:10:45 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2018-11-20T19:10:44.927Z: java.lang.RuntimeException:
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.IllegalArgumentException: unable to deserialize Serialized DoFnInfo
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:193)
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:164)
at
org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:63)
at
org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:50)
at
org.apache.beam.runners.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:87)
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.create(IntrinsicMapTaskExecutorFactory.java:124)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:337)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:291)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:135)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:115)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:102)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) Caused by:
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.IllegalArgumentException: unable to deserialize Serialized DoFnInfo
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2214)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache.get(LocalCache.java:4053)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4899)
at
org.apache.beam.runners.dataflow.worker.UserParDoFnFactory.create(UserParDoFnFactory.java:91)
at
org.apache.beam.runners.dataflow.worker.DefaultParDoFnFactory.create(DefaultParDoFnFactory.java:75)
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.createParDoOperation(IntrinsicMapTaskExecutorFactory.java:263)
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.access$000(IntrinsicMapTaskExecutorFactory.java:85)
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:182)
... 14 more Caused by: java.lang.IllegalArgumentException: unable to
deserialize Serialized DoFnInfo at
org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:74)
at
org.apache.beam.runners.dataflow.worker.UserParDoFnFactory$UserDoFnExtractor.getDoFnInfo(UserParDoFnFactory.java:62)
at
org.apache.beam.runners.dataflow.worker.UserParDoFnFactory.lambda$create$0(UserParDoFnFactory.java:93)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4904)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3628)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2336)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2295)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2208)
... 21 more Caused by: java.io.InvalidClassException:
org.apache.beam.sdk.values.WindowingStrategy; local class incompatible: stream
classdesc serialVersionUID = -6607512772692666907, local class serialVersionUID
= -3616600070988263902 at
java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616) at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1630) at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521) at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018) at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942) at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:373) at
org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:71)
... 28 more Nov 20, 2018 11:11:01 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2018-11-20T19:11:00.729Z: java.lang.RuntimeException:
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.IllegalArgumentException: unable to deserialize Serialized DoFnInfo
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:193)
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:164)
at
org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:63)
at
org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:50)
at
org.apache.beam.runners.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:87)
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.create(IntrinsicMapTaskExecutorFactory.java:124)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:337)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:291)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:135)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:115)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:102)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) Caused by:
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.IllegalArgumentException: unable to deserialize Serialized DoFnInfo
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2214)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache.get(LocalCache.java:4053)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4899)
at
org.apache.beam.runners.dataflow.worker.UserParDoFnFactory.create(UserParDoFnFactory.java:91)
at
org.apache.beam.runners.dataflow.worker.DefaultParDoFnFactory.create(DefaultParDoFnFactory.java:75)
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.createParDoOperation(IntrinsicMapTaskExecutorFactory.java:263)
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.access$000(IntrinsicMapTaskExecutorFactory.java:85)
at
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:182)
... 14 more Caused by: java.lang.IllegalArgumentException: unable to
deserialize Serialized DoFnInfo at
org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:74)
at
org.apache.beam.runners.dataflow.worker.UserParDoFnFactory$UserDoFnExtractor.getDoFnInfo(UserParDoFnFactory.java:62)
at
org.apache.beam.runners.dataflow.worker.UserParDoFnFactory.lambda$create$0(UserParDoFnFactory.java:93)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4904)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3628)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2336)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2295)
at
org.apache.beam.runners.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2208)
... 21 more Caused by: java.io.InvalidClassException:
org.apache.beam.sdk.values.WindowingStrategy; local class incompatible: stream
classdesc serialVersionUID = -6607512772692666907, local class serialVersionUID
= -3616600070988263902 at
java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616) at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1630) at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521) at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018) at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942) at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:373) at
org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:71)
... 28 more Nov 20, 2018 11:11:01 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2018-11-20T19:11:00.885Z: Workflow failed. Causes:
S02:ReadLines/Read+WordCount.CountWords/ParDo(ExtractWords)+WordCount.CountWords/Count.PerElement/Init/Map+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write
failed., A work item was attempted 4 times without success. Each time the
worker eventually lost contact with the service. The work item was attempted
on: wordcount-goenka-11201909-11201109-n7q9-harness-6q3n,
wordcount-goenka-11201909-11201109-n7q9-harness-6q3n,
wordcount-goenka-11201909-11201109-n7q9-harness-6q3n,
wordcount-goenka-11201909-11201109-n7q9-harness-6q3n Nov 20, 2018 11:11:02 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:11:01.059Z: Cleaning up. Nov 20, 2018 11:11:02 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T19:11:01.473Z: Stopping worker pool...
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
