See <https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/954/display/redirect?page=changes>
Changes: [huangry] Include user distritribution into metric-dedicated validate runner test. [huangry] update inline comment. [huangry] To allow only assert on committed metrics, bypassing the attempted ones. [huangry] Use distribution matcher for testing. [kcweaver] [BEAM-9765] Exclude module-info.class from vendored Calcite. [pabloem] [BEAM-9769] Ensuring JSON is the default export format for BQ sink [echauchot] Add GBK load tests script for spark structured streaming runner [echauchot] Add CoGBK load tests script for spark structured streaming runner [echauchot] Add Combine load tests script for spark structured streaming runner [echauchot] Add Pardo load tests script for spark structured streaming runner [github] [BEAM-9678] Create Go SDK introduction kata (#11340) [github] Merge pull request #11436: Document Beam Schemas (forked PR) [github] [BEAM-9648]: DirectRunner should return null on timeout (#11275) [github] [BEAM-5379] Ignore go.sum files in RAT checks [github] [BEAM-5605, BEAM-2939] Add support for FnApiDoFnRunner to handle split ------------------------------------------ [...truncated 3.10 MB...] > Task :runners:google-cloud-dataflow-java:runQuickstartJavaDataflow Apr 17, 2020 11:16:40 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services. Apr 17, 2020 11:16:40 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Uploading 202 files from PipelineOptions.filesToStage to staging location to prepare for execution. Apr 17, 2020 11:16:41 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7941966508833165305-tmpdir/.m2/repository/org/apache/beam/beam-sdks-java-core/2.22.0-SNAPSHOT/beam-sdks-java-core-2.22.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-sdks-java-core-2.22.0-SNAPSHOT-u9TG1Le-8Ry7Tjp8VW13wA.jar Apr 17, 2020 11:16:41 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7941966508833165305-tmpdir/.m2/repository/org/apache/beam/beam-model-fn-execution/2.22.0-SNAPSHOT/beam-model-fn-execution-2.22.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-model-fn-execution-2.22.0-SNAPSHOT-zdmJbmv-R7VsuaHIgAYyKA.jar Apr 17, 2020 11:16:41 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7941966508833165305-tmpdir/.m2/repository/org/apache/beam/beam-model-job-management/2.22.0-SNAPSHOT/beam-model-job-management-2.22.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-model-job-management-2.22.0-SNAPSHOT-LDsVi0t-yoPUi_a16sgFfA.jar Apr 17, 2020 11:16:41 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7941966508833165305-tmpdir/.m2/repository/org/apache/beam/beam-model-pipeline/2.22.0-SNAPSHOT/beam-model-pipeline-2.22.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-model-pipeline-2.22.0-SNAPSHOT-Id_CKeYsqHOKkYRLRAlkhg.jar Apr 17, 2020 11:16:41 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7941966508833165305-tmpdir/.m2/repository/org/apache/beam/beam-sdks-java-io-google-cloud-platform/2.22.0-SNAPSHOT/beam-sdks-java-io-google-cloud-platform-2.22.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-sdks-java-io-google-cloud-platform-2.22.0-SNAPSHOT-fN6Vnn2bpOxXtwSc6vdSlA.jar Apr 17, 2020 11:16:41 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7941966508833165305-tmpdir/.m2/repository/org/apache/beam/beam-sdks-java-expansion-service/2.22.0-SNAPSHOT/beam-sdks-java-expansion-service-2.22.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-sdks-java-expansion-service-2.22.0-SNAPSHOT-v5uXj5Hy19rYCA_UFxUDog.jar Apr 17, 2020 11:16:41 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7941966508833165305-tmpdir/.m2/repository/org/apache/beam/beam-sdks-java-extensions-protobuf/2.22.0-SNAPSHOT/beam-sdks-java-extensions-protobuf-2.22.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-sdks-java-extensions-protobuf-2.22.0-SNAPSHOT-sKVZbcA3Ybz5ps7WcwCHfQ.jar Apr 17, 2020 11:16:41 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7941966508833165305-tmpdir/.m2/repository/org/apache/beam/beam-sdks-java-extensions-google-cloud-platform-core/2.22.0-SNAPSHOT/beam-sdks-java-extensions-google-cloud-platform-core-2.22.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-sdks-java-extensions-google-cloud-platform-core-2.22.0-SNAPSHOT-0Bwe7vHCmCUSVhJrgQFd0w.jar Apr 17, 2020 11:16:42 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7941966508833165305-tmpdir/.m2/repository/org/apache/beam/beam-runners-core-construction-java/2.22.0-SNAPSHOT/beam-runners-core-construction-java-2.22.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-runners-core-construction-java-2.22.0-SNAPSHOT-szvhtuZrB49IrwAxWAqkxw.jar Apr 17, 2020 11:16:42 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7941966508833165305-tmpdir/.m2/repository/org/apache/beam/beam-runners-google-cloud-dataflow-java/2.22.0-SNAPSHOT/beam-runners-google-cloud-dataflow-java-2.22.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-runners-google-cloud-dataflow-java-2.22.0-SNAPSHOT-Rl7phigmsa37s40et1xoTA.jar Apr 17, 2020 11:16:42 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7941966508833165305-tmpdir/word-count-beam/target/classes to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/classes-VS1de1hA6MNQK1IUsjbdQg.jar Apr 17, 2020 11:16:42 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Staging files complete: 191 files cached, 11 files newly uploaded in 1 seconds Apr 17, 2020 11:16:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding ReadLines/Read as step s1 Apr 17, 2020 11:16:43 AM org.apache.beam.sdk.io.FileBasedSource getEstimatedSizeBytes INFO: Filepattern gs://apache-beam-samples/shakespeare/* matched 44 files with total size 5443510 Apr 17, 2020 11:16:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WordCount.CountWords/ParDo(ExtractWords) as step s2 Apr 17, 2020 11:16:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WordCount.CountWords/Count.PerElement/Init/Map as step s3 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey as step s4 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues as step s5 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding MapElements/Map as step s6 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign as step s7 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles as step s8 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten as step s9 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten as step s10 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum as step s11 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections as step s12 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) as step s13 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView as step s14 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource) as step s15 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous) as step s16 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map as step s17 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize as step s18 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key as step s19 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign as step s20 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey as step s21 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable as step s22 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map as step s23 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging pipeline description to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/ Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <67452 bytes, hash Kcc_IT2bvltkTbKhruoSmw> to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/pipeline-Kcc_IT2bvltkTbKhruoSmw.pb Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Dataflow SDK version: 2.22.0-SNAPSHOT Apr 17, 2020 11:16:45 AM org.apache.beam.sdk.extensions.gcp.util.RetryHttpRequestInitializer$LoggingHttpBackOffHandler handleResponse WARNING: Request failed with code 403, performed 0 retries due to IOExceptions, performed 0 retries due to unsuccessful status codes, HTTP framework says request can be retried, (caller responsible for retrying): https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/null/jobs. [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project word-count-beam: An exception occured while executing the Java class. Failed to create a workflow job: Permission denied on 'locations/null' (or it may not exist). 403 Forbidden [ERROR] { [ERROR] "code" : 403, [ERROR] "errors" : [ { [ERROR] "domain" : "global", [ERROR] "message" : "Permission denied on 'locations/null' (or it may not exist).", [ERROR] "reason" : "forbidden" [ERROR] } ], [ERROR] "message" : "Permission denied on 'locations/null' (or it may not exist).", [ERROR] "status" : "PERMISSION_DENIED" [ERROR] } [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException Apr 17, 2020 11:16:39 AM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create INFO: No stagingLocation provided, falling back to gcpTempLocation Apr 17, 2020 11:16:39 AM org.apache.beam.runners.dataflow.DataflowRunner fromOptions INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 202 files. Enable logging at DEBUG level to see which files will be staged. Apr 17, 2020 11:16:40 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services. Apr 17, 2020 11:16:40 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Uploading 202 files from PipelineOptions.filesToStage to staging location to prepare for execution. Apr 17, 2020 11:16:41 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7941966508833165305-tmpdir/.m2/repository/org/apache/beam/beam-sdks-java-core/2.22.0-SNAPSHOT/beam-sdks-java-core-2.22.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-sdks-java-core-2.22.0-SNAPSHOT-u9TG1Le-8Ry7Tjp8VW13wA.jar Apr 17, 2020 11:16:41 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7941966508833165305-tmpdir/.m2/repository/org/apache/beam/beam-model-fn-execution/2.22.0-SNAPSHOT/beam-model-fn-execution-2.22.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-model-fn-execution-2.22.0-SNAPSHOT-zdmJbmv-R7VsuaHIgAYyKA.jar Apr 17, 2020 11:16:41 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7941966508833165305-tmpdir/.m2/repository/org/apache/beam/beam-model-job-management/2.22.0-SNAPSHOT/beam-model-job-management-2.22.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-model-job-management-2.22.0-SNAPSHOT-LDsVi0t-yoPUi_a16sgFfA.jar Apr 17, 2020 11:16:41 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7941966508833165305-tmpdir/.m2/repository/org/apache/beam/beam-model-pipeline/2.22.0-SNAPSHOT/beam-model-pipeline-2.22.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-model-pipeline-2.22.0-SNAPSHOT-Id_CKeYsqHOKkYRLRAlkhg.jar Apr 17, 2020 11:16:41 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7941966508833165305-tmpdir/.m2/repository/org/apache/beam/beam-sdks-java-io-google-cloud-platform/2.22.0-SNAPSHOT/beam-sdks-java-io-google-cloud-platform-2.22.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-sdks-java-io-google-cloud-platform-2.22.0-SNAPSHOT-fN6Vnn2bpOxXtwSc6vdSlA.jar Apr 17, 2020 11:16:41 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7941966508833165305-tmpdir/.m2/repository/org/apache/beam/beam-sdks-java-expansion-service/2.22.0-SNAPSHOT/beam-sdks-java-expansion-service-2.22.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-sdks-java-expansion-service-2.22.0-SNAPSHOT-v5uXj5Hy19rYCA_UFxUDog.jar Apr 17, 2020 11:16:41 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7941966508833165305-tmpdir/.m2/repository/org/apache/beam/beam-sdks-java-extensions-protobuf/2.22.0-SNAPSHOT/beam-sdks-java-extensions-protobuf-2.22.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-sdks-java-extensions-protobuf-2.22.0-SNAPSHOT-sKVZbcA3Ybz5ps7WcwCHfQ.jar Apr 17, 2020 11:16:41 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7941966508833165305-tmpdir/.m2/repository/org/apache/beam/beam-sdks-java-extensions-google-cloud-platform-core/2.22.0-SNAPSHOT/beam-sdks-java-extensions-google-cloud-platform-core-2.22.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-sdks-java-extensions-google-cloud-platform-core-2.22.0-SNAPSHOT-0Bwe7vHCmCUSVhJrgQFd0w.jar Apr 17, 2020 11:16:42 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7941966508833165305-tmpdir/.m2/repository/org/apache/beam/beam-runners-core-construction-java/2.22.0-SNAPSHOT/beam-runners-core-construction-java-2.22.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-runners-core-construction-java-2.22.0-SNAPSHOT-szvhtuZrB49IrwAxWAqkxw.jar Apr 17, 2020 11:16:42 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7941966508833165305-tmpdir/.m2/repository/org/apache/beam/beam-runners-google-cloud-dataflow-java/2.22.0-SNAPSHOT/beam-runners-google-cloud-dataflow-java-2.22.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-runners-google-cloud-dataflow-java-2.22.0-SNAPSHOT-Rl7phigmsa37s40et1xoTA.jar Apr 17, 2020 11:16:42 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7941966508833165305-tmpdir/word-count-beam/target/classes to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/classes-VS1de1hA6MNQK1IUsjbdQg.jar Apr 17, 2020 11:16:42 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Staging files complete: 191 files cached, 11 files newly uploaded in 1 seconds Apr 17, 2020 11:16:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding ReadLines/Read as step s1 Apr 17, 2020 11:16:43 AM org.apache.beam.sdk.io.FileBasedSource getEstimatedSizeBytes INFO: Filepattern gs://apache-beam-samples/shakespeare/* matched 44 files with total size 5443510 Apr 17, 2020 11:16:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WordCount.CountWords/ParDo(ExtractWords) as step s2 Apr 17, 2020 11:16:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WordCount.CountWords/Count.PerElement/Init/Map as step s3 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey as step s4 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues as step s5 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding MapElements/Map as step s6 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign as step s7 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles as step s8 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten as step s9 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten as step s10 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum as step s11 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections as step s12 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) as step s13 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView as step s14 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource) as step s15 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous) as step s16 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map as step s17 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize as step s18 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key as step s19 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign as step s20 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey as step s21 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable as step s22 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map as step s23 Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging pipeline description to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/ Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <67452 bytes, hash Kcc_IT2bvltkTbKhruoSmw> to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/pipeline-Kcc_IT2bvltkTbKhruoSmw.pb Apr 17, 2020 11:16:44 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Dataflow SDK version: 2.22.0-SNAPSHOT Apr 17, 2020 11:16:45 AM org.apache.beam.sdk.extensions.gcp.util.RetryHttpRequestInitializer$LoggingHttpBackOffHandler handleResponse WARNING: Request failed with code 403, performed 0 retries due to IOExceptions, performed 0 retries due to unsuccessful status codes, HTTP framework says request can be retried, (caller responsible for retrying): https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/null/jobs. [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project word-count-beam: An exception occured while executing the Java class. Failed to create a workflow job: Permission denied on 'locations/null' (or it may not exist). 403 Forbidden [ERROR] { [ERROR] "code" : 403, [ERROR] "errors" : [ { [ERROR] "domain" : "global", [ERROR] "message" : "Permission denied on 'locations/null' (or it may not exist).", [ERROR] "reason" : "forbidden" [ERROR] } ], [ERROR] "message" : "Permission denied on 'locations/null' (or it may not exist).", [ERROR] "status" : "PERMISSION_DENIED" [ERROR] } [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] Failed command > Task :runners:google-cloud-dataflow-java:runQuickstartJavaDataflow FAILED FAILURE: Build completed with 2 failures. 1: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 2: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':runners:google-cloud-dataflow-java:runQuickstartJavaDataflow'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 16m 26s 8 actionable tasks: 7 executed, 1 from cache Publishing build scan... https://gradle.com/s/zde6l27annqdw Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
