See <https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/944/display/redirect?page=changes>
Changes: [echauchot] [BEAM-5980] Change load-tests build to include spark-sql for spark [echauchot] [BEAM-9436] avoid one flatmap step and a KV creation per element by [kcweaver] [BEAM-9199] Require Dataflow --region in Python SDK. [kcweaver] Add --region to tests where needed. [kcweaver] [BEAM-9199] Require --region option for Dataflow in Java SDK. [kcweaver] Add --region to Java GCP tests. [kcweaver] Fix DataflowRunnerTest. [kcweaver] Fix more Java unit tests missing --region. [kcweaver] Add --region to DF streaming example tests. [kcweaver] Add unit tests for get_default_gcp_region [kcweaver] Add --region to Dataflow runner webpage. [kcweaver] lint [kcweaver] Add --region to more Java tests and examples. [kcweaver] Add --region to more Python tests and examples. [kcweaver] format [valentyn] Fixes platform-dependent assumptions in subprocess_server_test.py. [valentyn] Switches a test helper to a Py3-version thereof. [alex] Add Beam Schema Options to changelog [alex] [BEAM-9704] Deprecate FieldType metadata [eekkaaadrian] [BEAM-9705] Go sdk add value length validation checking on write to [kcweaver] Remove unrecognized --region option from non-DF tests. [robertwb] [BEAM-9618] Add protocol for requesting process bundle descriptors. [robertwb] [BEAM-9618] Update Python to support process bundle descriptor fetching. [robertwb] [BEAM-9618] Java FnApiClient support for process bundle descriptor [github] [BEAM-8019] Python SDK support for cross-langauge pipelines in Dataflow. [robertwb] Typo fix. [github] remove nose (#11307) [lcwik] [BEAM-4374, BEAM-6189] Delete and remove deprecated Metrics proto [ecapoccia] [BEAM-9434] Improve Spark runner reshuffle translation to maximize [github] [BEAM-9685] remove Go SDK container from release process (#11308) ------------------------------------------ [...truncated 3.39 MB...] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException Apr 07, 2020 11:17:24 AM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create INFO: No stagingLocation provided, falling back to gcpTempLocation Apr 07, 2020 11:17:25 AM org.apache.beam.runners.dataflow.DataflowRunner fromOptions INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 201 files. Enable logging at DEBUG level to see which files will be staged. Apr 07, 2020 11:17:26 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services. Apr 07, 2020 11:17:26 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Uploading 201 files from PipelineOptions.filesToStage to staging location to prepare for execution. Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/org/apache/beam/beam-sdks-java-core/2.21.0-SNAPSHOT/beam-sdks-java-core-2.21.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-sdks-java-core-2.21.0-SNAPSHOT-dy5kjoaG9n5HVRNSESEtwg.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/localedata.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/localedata-G4rFQgHFw0cQOGvVikCaJg.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/jaccess.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/jaccess-4gySnkY03eDhOEtoBmWqow.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/org/apache/beam/beam-sdks-java-extensions-google-cloud-platform-core/2.21.0-SNAPSHOT/beam-sdks-java-extensions-google-cloud-platform-core-2.21.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-sdks-java-extensions-google-cloud-platform-core-2.21.0-SNAPSHOT-nZOrSnquKfRha_-7ujUWvw.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/org/tukaani/xz/1.8/xz-1.8.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/xz-1.8-X5ghJ-DehbeFxLKrrSGqLg.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/org/apache/beam/beam-sdks-java-extensions-protobuf/2.21.0-SNAPSHOT/beam-sdks-java-extensions-protobuf-2.21.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-sdks-java-extensions-protobuf-2.21.0-SNAPSHOT-RUW0r5uRHFeSvJ2z5iA60g.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/apis/google-api-services-cloudresourcemanager/v1-rev20191206-1.30.3/google-api-services-cloudresourcemanager-v1-rev20191206-1.30.3.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/google-api-services-cloudresourcemanager-v1-rev20191206-1.30.3-e0eoAXp5NqLIiSz5VpMmcg.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/thoughtworks/paranamer/paranamer/2.7/paranamer-2.7.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/paranamer-2.7-VweilzYySf_-OOgYnNb5yw.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/cldrdata.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/cldrdata-qaM_JhdIwuqWPo5ejAdKeA.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/org/apache/beam/beam-model-job-management/2.21.0-SNAPSHOT/beam-model-job-management-2.21.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-model-job-management-2.21.0-SNAPSHOT-xVZWrsyE1MKYksxahNnrsA.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/icedtea-sound.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/icedtea-sound-b-rKYQnwdiezoZrbAnjK5A.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/org/apache/beam/beam-sdks-java-expansion-service/2.21.0-SNAPSHOT/beam-sdks-java-expansion-service-2.21.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-sdks-java-expansion-service-2.21.0-SNAPSHOT-EBsdCV05qTMw19K05ZmNBw.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/word-count-beam/target/classes to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/classes-yMzuCoD8iV5tZ5oqdYuWRQ.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/dnsns.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/dnsns-AzHyrfxoxb1n2WNpDUhIHQ.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/org/apache/beam/beam-model-pipeline/2.21.0-SNAPSHOT/beam-model-pipeline-2.21.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-model-pipeline-2.21.0-SNAPSHOT-k50g3h8Gk8sYAwh6GuVoPw.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/nashorn.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/nashorn-xv5j3ELDchuJ7w1XIL7s0Q.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/jackson-mapper-asl-1.9.13-F1D5wzk1L8S3KNYbVxcWEw.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/org/apache/beam/beam-sdks-java-io-google-cloud-platform/2.21.0-SNAPSHOT/beam-sdks-java-io-google-cloud-platform-2.21.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-sdks-java-io-google-cloud-platform-2.21.0-SNAPSHOT-Om34FUiER4J2jZJ5gJRlAg.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/org/apache/beam/beam-model-fn-execution/2.21.0-SNAPSHOT/beam-model-fn-execution-2.21.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-model-fn-execution-2.21.0-SNAPSHOT-ENy9Pt4lk4ENEDWFcCU0kA.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/api/grpc/grpc-google-common-protos/1.17.0/grpc-google-common-protos-1.17.0.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/grpc-google-common-protos-1.17.0-_cxSEr69pWXoIkY7hkpVVg.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/api/gax-httpjson/0.69.0/gax-httpjson-0.69.0.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/gax-httpjson-0.69.0-NlDjPwVLuzhig_M1kqSwxQ.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/api/grpc/proto-google-iam-v1/0.13.0/proto-google-iam-v1-0.13.0.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/proto-google-iam-v1-0.13.0-xzBogKliF_aHpC--6DwfhA.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/cloud/bigtable/bigtable-client-core/1.13.0/bigtable-client-core-1.13.0.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/bigtable-client-core-1.13.0-jy28vbjBp00oM7LpOk0OuQ.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/http-client/google-http-client-appengine/1.34.0/google-http-client-appengine-1.34.0.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/google-http-client-appengine-1.34.0-XFJT3Y0fOO3qlEyBpI-9CQ.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/api/grpc/grpc-google-cloud-bigtable-v2/1.9.1/grpc-google-cloud-bigtable-v2-1.9.1.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/grpc-google-cloud-bigtable-v2-1.9.1-4ZI7F4A19Ac1-7u02BvlDA.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/io/dropwizard/metrics/metrics-core/3.2.6/metrics-core-3.2.6.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/metrics-core-3.2.6-ZBcW7gga6qAWQVAZx0-0Rw.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/api/grpc/grpc-google-cloud-bigtable-admin-v2/1.9.1/grpc-google-cloud-bigtable-admin-v2-1.9.1.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/grpc-google-cloud-bigtable-admin-v2-1.9.1-486caPbziCFt1SkW8doV-w.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/cloud/google-cloud-core-http/1.92.1/google-cloud-core-http-1.92.1.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/google-cloud-core-http-1.92.1-uIQSryW3Wb82JcQ2mxzV-Q.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/cloud/google-cloud-bigtable/1.9.1/google-cloud-bigtable-1.9.1.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/google-cloud-bigtable-1.9.1-ZygYOF8z24IZJVcDdV8Qkg.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/io/opencensus/opencensus-contrib-grpc-util/0.24.0/opencensus-contrib-grpc-util-0.24.0.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/opencensus-contrib-grpc-util-0.24.0-oI65xuFtIanB3qCaDIRhYg.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/cloud/google-cloud-core/1.92.2/google-cloud-core-1.92.2.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/google-cloud-core-1.92.2-LpfPszayM4TSQDG2MrSBHg.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/cloud/google-cloud-core-grpc/1.92.2/google-cloud-core-grpc-1.92.2.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/google-cloud-core-grpc-1.92.2-_6Txcdr3cc3CJ-XtwjjqXw.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/protobuf/protobuf-java-util/3.11.1/protobuf-java-util-3.11.1.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/protobuf-java-util-3.11.1-Qm3uPl7YbU5Q3m14KN2SrA.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/commons-codec/commons-codec/1.13/commons-codec-1.13.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/commons-codec-1.13-UIXxhhVoIvo6AuVbzVWEqA.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/apis/google-api-services-bigquery/v2-rev20191211-1.30.3/google-api-services-bigquery-v2-rev20191211-1.30.3.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/google-api-services-bigquery-v2-rev20191211-1.30.3-A4Xk8vSx01NYgeeMSyzibQ.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/api-client/google-api-client/1.30.3/google-api-client-1.30.3.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/google-api-client-1.30.3-IMZSi0kOb_OQE6cfWivYVQ.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/api/grpc/proto-google-cloud-bigtable-v2/1.9.1/proto-google-cloud-bigtable-v2-1.9.1.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/proto-google-cloud-bigtable-v2-1.9.1-uQRjtoPkJEoUzbA7q2HddQ.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/api/grpc/proto-google-cloud-bigquerystorage-v1beta1/0.85.1/proto-google-cloud-bigquerystorage-v1beta1-0.85.1.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/proto-google-cloud-bigquerystorage-v1beta1-0.85.1-ow-dhZaOO7Wq-AvF5Lmzsw.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/api/grpc/proto-google-cloud-bigtable-admin-v2/1.9.1/proto-google-cloud-bigtable-admin-v2-1.9.1.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/proto-google-cloud-bigtable-admin-v2-1.9.1-mPQh8EDOps-lHPaklcStyA.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/api/grpc/proto-google-cloud-datastore-v1/0.85.0/proto-google-cloud-datastore-v1-0.85.0.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/proto-google-cloud-datastore-v1-0.85.0-2ISUqQIYcSZY6Q5p2RI7iw.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/http-client/google-http-client/1.34.0/google-http-client-1.34.0.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/google-http-client-1.34.0-zNPEEsJyZw5-MJGvh7HRUg.jar Apr 07, 2020 11:17:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/oauth-client/google-oauth-client/1.30.2/google-oauth-client-1.30.2.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/google-oauth-client-1.30.2-u_kMpa6sBSEEYcspLptwJw.jar Apr 07, 2020 11:17:28 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/j2objc/j2objc-annotations/1.3/j2objc-annotations-1.3.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/j2objc-annotations-1.3-X6TsTsDFqnCvin1JIt8ZMQ.jar Apr 07, 2020 11:17:28 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/org/hamcrest/hamcrest/2.1/hamcrest-2.1.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/hamcrest-2.1-oTm8x8sMLv9-n5czpY1b3Q.jar Apr 07, 2020 11:17:28 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/apis/google-api-services-clouddebugger/v2-rev20191003-1.30.3/google-api-services-clouddebugger-v2-rev20191003-1.30.3.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/google-api-services-clouddebugger-v2-rev20191003-1.30.3-kHDMTR6A6rF-wEZn0ylzww.jar Apr 07, 2020 11:17:28 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/org/apache/beam/beam-runners-core-construction-java/2.21.0-SNAPSHOT/beam-runners-core-construction-java-2.21.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-runners-core-construction-java-2.21.0-SNAPSHOT-WAqi-74aniwp9AA3-n0anA.jar Apr 07, 2020 11:17:28 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/apis/google-api-services-pubsub/v1-rev20191111-1.30.3/google-api-services-pubsub-v1-rev20191111-1.30.3.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/google-api-services-pubsub-v1-rev20191111-1.30.3-dkOpxR7rti-4BKDvqf7sgQ.jar Apr 07, 2020 11:17:28 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/org/apache/httpcomponents/httpclient/4.5.10/httpclient-4.5.10.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/httpclient-4.5.10-NnIh3eDvlOo1B5KO9Ay-dQ.jar Apr 07, 2020 11:17:28 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/org/apache/httpcomponents/httpcore/4.4.12/httpcore-4.4.12.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/httpcore-4.4.12-wVLyMb8lcOyjVMSe-HVrQQ.jar Apr 07, 2020 11:17:28 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/apis/google-api-services-dataflow/v1b3-rev20190927-1.30.3/google-api-services-dataflow-v1b3-rev20190927-1.30.3.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/google-api-services-dataflow-v1b3-rev20190927-1.30.3-Xlz0vYOfCxXyuXDomPU7aA.jar Apr 07, 2020 11:17:28 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/com/google/apis/google-api-services-storage/v1-rev20191011-1.30.3/google-api-services-storage-v1-rev20191011-1.30.3.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/google-api-services-storage-v1-rev20191011-1.30.3-inaJzH0erk9jPu9ywmI9AQ.jar Apr 07, 2020 11:17:28 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/junit/junit/4.13-beta-3/junit-4.13-beta-3.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/junit-4.13-beta-3-SR8V8LeBEDL0Hb3Yr3GZaA.jar Apr 07, 2020 11:17:28 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/org/apache/beam/beam-runners-google-cloud-dataflow-java/2.21.0-SNAPSHOT/beam-runners-google-cloud-dataflow-java-2.21.0-SNAPSHOT.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-runners-google-cloud-dataflow-java-2.21.0-SNAPSHOT-HDNwsb32hGYGLJ9au_9Scg.jar Apr 07, 2020 11:17:28 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/io/opencensus/opencensus-api/0.24.0/opencensus-api-0.24.0.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/opencensus-api-0.24.0-V-JtnC05R6CzcW7IuzLJvw.jar Apr 07, 2020 11:17:28 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/io/opencensus/opencensus-contrib-http-util/0.24.0/opencensus-contrib-http-util-0.24.0.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/opencensus-contrib-http-util-0.24.0-EtnfJf6yxv-BdGUQPdPhPw.jar Apr 07, 2020 11:17:28 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-7974912311336034238-tmpdir/.m2/repository/io/github/classgraph/classgraph/4.8.65/classgraph-4.8.65.jar to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/classgraph-4.8.65-W08dT45fGDTJjSfbojuwKA.jar Apr 07, 2020 11:17:28 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Staging files complete: 145 files cached, 56 files newly uploaded in 1 seconds Apr 07, 2020 11:17:28 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding ReadLines/Read as step s1 Apr 07, 2020 11:17:29 AM org.apache.beam.sdk.io.FileBasedSource getEstimatedSizeBytes INFO: Filepattern gs://apache-beam-samples/shakespeare/* matched 44 files with total size 5443510 Apr 07, 2020 11:17:29 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WordCount.CountWords/ParDo(ExtractWords) as step s2 Apr 07, 2020 11:17:29 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WordCount.CountWords/Count.PerElement/Init/Map as step s3 Apr 07, 2020 11:17:29 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey as step s4 Apr 07, 2020 11:17:29 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues as step s5 Apr 07, 2020 11:17:29 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding MapElements/Map as step s6 Apr 07, 2020 11:17:29 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign as step s7 Apr 07, 2020 11:17:29 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles as step s8 Apr 07, 2020 11:17:29 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten as step s9 Apr 07, 2020 11:17:30 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten as step s10 Apr 07, 2020 11:17:30 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum as step s11 Apr 07, 2020 11:17:30 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections as step s12 Apr 07, 2020 11:17:30 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/ToList/WithKeys/AddKeys/Map as step s13 Apr 07, 2020 11:17:30 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey as step s14 Apr 07, 2020 11:17:30 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/Combine.GroupedValues as step s15 Apr 07, 2020 11:17:30 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/ToList/Values/Values/Map as step s16 Apr 07, 2020 11:17:30 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/ToList/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow) as step s17 Apr 07, 2020 11:17:30 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/ToList/View.AsIterable/CreateDataflowView as step s18 Apr 07, 2020 11:17:30 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/ToList/CreateVoid/Read(CreateSource) as step s19 Apr 07, 2020 11:17:30 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/ToList/ProduceDefault as step s20 Apr 07, 2020 11:17:30 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/ToList/Flatten.PCollections as step s21 Apr 07, 2020 11:17:30 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize as step s22 Apr 07, 2020 11:17:30 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key as step s23 Apr 07, 2020 11:17:30 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign as step s24 Apr 07, 2020 11:17:30 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey as step s25 Apr 07, 2020 11:17:30 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable as step s26 Apr 07, 2020 11:17:30 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map as step s27 Apr 07, 2020 11:17:30 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging pipeline description to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/ Apr 07, 2020 11:17:30 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <74760 bytes, hash B4Cmju9WFPbdjAKQ9XT-hg> to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/pipeline-B4Cmju9WFPbdjAKQ9XT-hg.pb Apr 07, 2020 11:17:31 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Dataflow SDK version: 2.21.0-SNAPSHOT Apr 07, 2020 11:17:31 AM org.apache.beam.sdk.extensions.gcp.util.RetryHttpRequestInitializer$LoggingHttpBackOffHandler handleResponse WARNING: Request failed with code 403, performed 0 retries due to IOExceptions, performed 0 retries due to unsuccessful status codes, HTTP framework says request can be retried, (caller responsible for retrying): https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/null/jobs. [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project word-count-beam: An exception occured while executing the Java class. Failed to create a workflow job: Permission denied on 'locations/null' (or it may not exist). 403 Forbidden [ERROR] { [ERROR] "code" : 403, [ERROR] "errors" : [ { [ERROR] "domain" : "global", [ERROR] "message" : "Permission denied on 'locations/null' (or it may not exist).", [ERROR] "reason" : "forbidden" [ERROR] } ], [ERROR] "message" : "Permission denied on 'locations/null' (or it may not exist).", [ERROR] "status" : "PERMISSION_DENIED" [ERROR] } [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] Failed command > Task :runners:google-cloud-dataflow-java:runQuickstartJavaDataflow FAILED FAILURE: Build completed with 2 failures. 1: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 2: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':runners:google-cloud-dataflow-java:runQuickstartJavaDataflow'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 17m 11s 8 actionable tasks: 8 executed Publishing build scan... https://gradle.com/s/dqangrpgqcjvc Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
