See
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/2778/display/redirect>
Changes:
------------------------------------------
[...truncated 244.02 KB...]
INFO: Uploading 262 files from PipelineOptions.filesToStage to staging
location to prepare for execution.
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
stageClasspathElements
WARNING: Skipping non-existent file to stage ${dataflowWorkerJar}.
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
stageClasspathElements
WARNING: Skipping non-existent file to stage
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/java/io/file-based-io-tests/build/resources/test.>
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
stageClasspathElements
WARNING: Skipping non-existent file to stage
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/java/io/file-based-io-tests/build/classes/java/main.>
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
stageClasspathElements
WARNING: Skipping non-existent file to stage
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/java/io/file-based-io-tests/build/resources/main.>
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/java/io/xml/build/libs/beam-sdks-java-io-xml-2.19.0-SNAPSHOT-tests.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-sdks-java-io-xml-2.19.0-SNAPSHOT-tests-oMuKuXSIHP9lhzlaaV384A.jar
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/java/extensions/google-cloud-platform-core/build/libs/beam-sdks-java-extensions-google-cloud-platform-core-2.19.0-SNAPSHOT-tests.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-sdks-java-extensions-google-cloud-platform-core-2.19.0-SNAPSHOT-tests-iLUA54pWZh0ZsnMsKmo5fw.jar
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.19.0-SNAPSHOT-tests.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-sdks-java-io-google-cloud-platform-2.19.0-SNAPSHOT-tests-OcvyeSbvZZ-2x6Ju38ex8A.jar
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/runners/core-construction-java/build/libs/beam-runners-core-construction-java-2.19.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-runners-core-construction-java-2.19.0-SNAPSHOT-gtP-E2RdY8ftpUDCqaMRKQ.jar
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/runners/google-cloud-dataflow-java/build/libs/beam-runners-google-cloud-dataflow-java-2.19.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-runners-google-cloud-dataflow-java-2.19.0-SNAPSHOT-YUn4DVU0LwYvfp0dI3yflA.jar
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/java/io/common/build/libs/beam-sdks-java-io-common-2.19.0-SNAPSHOT-tests.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-sdks-java-io-common-2.19.0-SNAPSHOT-tests-PI0TMNGNe8Mf4b4EJ7eDRw.jar
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/java/io/xml/build/libs/beam-sdks-java-io-xml-2.19.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-sdks-java-io-xml-2.19.0-SNAPSHOT--nfHPyg8CIlj7Ro2fYTgpw.jar
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/java/extensions/google-cloud-platform-core/build/libs/beam-sdks-java-extensions-google-cloud-platform-core-2.19.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-sdks-java-extensions-google-cloud-platform-core-2.19.0-SNAPSHOT-XeNV3M5d6omEOTNSIKuU4g.jar
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/java/io/file-based-io-tests/build/classes/java/test>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/test-epfuHT42CVVzvuzu8DlUIA.jar
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/java/testing/test-utils/build/libs/beam-sdks-java-test-utils-2.19.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-sdks-java-test-utils-2.19.0-SNAPSHOT-rdaszrRFOAbOFD0exL8q3g.jar
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/java/io/parquet/build/libs/beam-sdks-java-io-parquet-2.19.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-sdks-java-io-parquet-2.19.0-SNAPSHOT-gX8U4Q5_W_2gEhQ6946lKw.jar
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/java/io/common/build/libs/beam-sdks-java-io-common-2.19.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-sdks-java-io-common-2.19.0-SNAPSHOT-VFKT3CHbg8gHCL5KLKh8-g.jar
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/java/testing/test-utils/build/libs/beam-sdks-java-test-utils-2.19.0-SNAPSHOT-tests.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-sdks-java-test-utils-2.19.0-SNAPSHOT-tests-D1942kTitJyhKITsaG8sdQ.jar
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/java/extensions/protobuf/build/libs/beam-sdks-java-extensions-protobuf-2.19.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-sdks-java-extensions-protobuf-2.19.0-SNAPSHOT-VvFhJuH9vSMvQQYLlErVAg.jar
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/java/io/parquet/build/libs/beam-sdks-java-io-parquet-2.19.0-SNAPSHOT-tests.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-sdks-java-io-parquet-2.19.0-SNAPSHOT-tests-46IudAlRofeyQsK-uweN3w.jar
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/runners/google-cloud-dataflow-java/build/libs/beam-runners-google-cloud-dataflow-java-2.19.0-SNAPSHOT-tests.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-runners-google-cloud-dataflow-java-2.19.0-SNAPSHOT-tests-oL1LkNwJc_1KaX4ZydWqUg.jar
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.19.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-sdks-java-io-google-cloud-platform-2.19.0-SNAPSHOT-6CoXIVOVes75kbfCf1zVVw.jar
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/model/job-management/build/libs/beam-model-job-management-2.19.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-model-job-management-2.19.0-SNAPSHOT-woqg1IRABkPOD973h7818Q.jar
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/model/pipeline/build/libs/beam-model-pipeline-2.19.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-model-pipeline-2.19.0-SNAPSHOT-XzzCmNsg47LHmZ6w-5kzgw.jar
Dec 09, 2019 6:25:17 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/runners/direct-java/build/libs/beam-runners-direct-java-2.19.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-runners-direct-java-2.19.0-SNAPSHOT-YirgRgrKiMbOh8szpqWXJg.jar
Dec 09, 2019 6:25:18 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/java/core/build/libs/beam-sdks-java-core-2.19.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-sdks-java-core-2.19.0-SNAPSHOT-BUBg6w2cDmILCBOZGg-JJg.jar
Dec 09, 2019 6:25:18 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/java/core/build/libs/beam-sdks-java-core-2.19.0-SNAPSHOT-tests.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-sdks-java-core-2.19.0-SNAPSHOT-tests-vkaR00Mwk3hGn7h0RAw3ZQ.jar
Dec 09, 2019 6:25:18 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.19.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.19.0-SNAPSHOT-jxww16G7UpgVET85xKjGsg.jar
Dec 09, 2019 6:25:19 AM org.apache.beam.runners.dataflow.util.PackageUtil
stageClasspathElements
INFO: Staging files complete: 235 files cached, 23 files newly uploaded in
2 seconds
Dec 09, 2019 6:25:20 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Generate sequence/Read(BoundedCountingSource) as step s1
Dec 09, 2019 6:25:20 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Produce text lines as step s2
Dec 09, 2019 6:25:20 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Transform strings to bytes/Map as step s3
Dec 09, 2019 6:25:20 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Record time before writing as step s4
Dec 09, 2019 6:25:20 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write content to files/Write/RewindowIntoGlobal/Window.Assign
as step s5
Dec 09, 2019 6:25:20 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write content to
files/Write/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles as step s6
Dec 09, 2019 6:25:20 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write content to
files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten as step s7
Dec 09, 2019 6:25:20 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write content to
files/Write/WriteUnshardedBundlesToTempFiles/WriteUnwritten as step s8
Dec 09, 2019 6:25:20 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write content to
files/Write/WriteUnshardedBundlesToTempFiles/DropShardNum as step s9
Dec 09, 2019 6:25:20 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write content to
files/Write/WriteUnshardedBundlesToTempFiles/Flatten.PCollections as step s10
Dec 09, 2019 6:25:20 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write content to
files/Write/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)
as step s11
Dec 09, 2019 6:25:20 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write content to
files/Write/GatherTempFileResults/View.AsList/CreateDataflowView as step s12
Dec 09, 2019 6:25:20 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write content to
files/Write/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)
as step s13
Dec 09, 2019 6:25:20 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write content to
files/Write/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)
as step s14
Dec 09, 2019 6:25:20 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write content to
files/Write/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map
as step s15
Dec 09, 2019 6:25:20 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write content to files/Write/FinalizeTempFileBundles/Finalize
as step s16
Dec 09, 2019 6:25:20 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write content to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key
as step s17
Dec 09, 2019 6:25:20 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write content to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
as step s18
Dec 09, 2019 6:25:20 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write content to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey
as step s19
Dec 09, 2019 6:25:20 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write content to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
as step s20
Dec 09, 2019 6:25:20 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write content to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map as
step s21
Dec 09, 2019 6:25:20 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to
gs://dataflow-staging-us-central1-844138762903/temp/staging/
Dec 09, 2019 6:25:20 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading <61710 bytes, hash jMd6lZGDg4tyjzwH0zMzSg> to
gs://dataflow-staging-us-central1-844138762903/temp/staging/pipeline-jMd6lZGDg4tyjzwH0zMzSg.pb
Dec 09, 2019 6:25:20 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.19.0-SNAPSHOT
Dec 09, 2019 6:25:21 AM
org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$DefaultGcpRegionFactory
create
WARNING: Region will default to us-central1. Future releases of Beam will
require the user to set the region explicitly.
https://cloud.google.com/compute/docs/regions-zones/regions-zones
Dec 09, 2019 6:25:22 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-08_22_25_21-15041567154797501647?project=apache-beam-testing
Dec 09, 2019 6:25:22 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2019-12-08_22_25_21-15041567154797501647
Dec 09, 2019 6:25:22 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel
--region=us-central1 2019-12-08_22_25_21-15041567154797501647
Dec 09, 2019 6:25:29 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:26.130Z: Checking permissions granted to controller
Service Account.
Dec 09, 2019 6:25:40 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:40.208Z: Worker configuration: n1-standard-1 in
us-central1-a.
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:40.933Z: Expanding CoGroupByKey operations into
optimizable parts.
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2019-12-09T06:25:40.964Z: Your project already contains 100
Dataflow-created metric descriptors and Stackdriver will not create new
Dataflow custom metrics for this job. Each unique user-defined metric name
(independent of the DoFn in which it is defined) produces a new metric
descriptor. To delete old / unused metric descriptors see
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list
and
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.040Z: Expanding GroupByKey operations into
optimizable parts.
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.077Z: Lifting ValueCombiningMappingFns into
MergeBucketsMappingFns
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.225Z: Fusing adjacent ParDo, Read, Write, and
Flatten operations
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.256Z: Unzipping flatten s10 for input
s6.writtenRecords
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.282Z: Fusing unzipped copy of Write content to
files/Write/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow),
through flatten Write content to
files/Write/WriteUnshardedBundlesToTempFiles/Flatten.PCollections, into
producer Write content to
files/Write/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.309Z: Fusing consumer Write content to
files/Write/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)
into Write content to files/Write/WriteUnshardedBundlesToTempFiles/DropShardNum
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.336Z: Fusing consumer Produce text lines into
Generate sequence/Read(BoundedCountingSource)
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.365Z: Fusing consumer Transform strings to
bytes/Map into Produce text lines
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.401Z: Fusing consumer Record time before writing
into Transform strings to bytes/Map
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.436Z: Fusing consumer Write content to
files/Write/RewindowIntoGlobal/Window.Assign into Record time before writing
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.465Z: Fusing consumer Write content to
files/Write/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles into Write
content to files/Write/RewindowIntoGlobal/Window.Assign
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.494Z: Fusing consumer Write content to
files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify into Write
content to files/Write/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.531Z: Fusing consumer Write content to
files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write into Write
content to files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.564Z: Fusing consumer Write content to
files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow into
Write content to
files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.599Z: Fusing consumer Write content to
files/Write/WriteUnshardedBundlesToTempFiles/WriteUnwritten into Write content
to files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.636Z: Fusing consumer Write content to
files/Write/WriteUnshardedBundlesToTempFiles/DropShardNum into Write content to
files/Write/WriteUnshardedBundlesToTempFiles/WriteUnwritten
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.670Z: Fusing consumer Write content to
files/Write/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)
into Write content to
files/Write/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.707Z: Fusing consumer Write content to
files/Write/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map
into Write content to
files/Write/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.740Z: Fusing consumer Write content to
files/Write/FinalizeTempFileBundles/Finalize into Write content to
files/Write/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.765Z: Fusing consumer Write content to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key
into Write content to files/Write/FinalizeTempFileBundles/Finalize
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.799Z: Fusing consumer Write content to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
into Write content to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.833Z: Fusing consumer Write content to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
into Write content to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.867Z: Fusing consumer Write content to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
into Write content to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.904Z: Fusing consumer Write content to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
into Write content to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.935Z: Fusing consumer Write content to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
into Write content to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:41.970Z: Fusing consumer Write content to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map
into Write content to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
Dec 09, 2019 6:25:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:42.552Z: Executing operation Write content to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create
Dec 09, 2019 6:25:45 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:42.587Z: Executing operation Write content to
files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create
Dec 09, 2019 6:25:45 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:42.635Z: Starting 5 workers in us-central1-a...
Dec 09, 2019 6:25:45 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:42.686Z: Finished operation Write content to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create
Dec 09, 2019 6:25:45 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:42.705Z: Finished operation Write content to
files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create
Dec 09, 2019 6:25:45 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:25:42.885Z: Executing operation Generate
sequence/Read(BoundedCountingSource)+Produce text lines+Transform strings to
bytes/Map+Record time before writing+Write content to
files/Write/RewindowIntoGlobal/Window.Assign+Write content to
files/Write/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+Write
content to
files/Write/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)+Write
content to
files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+Write content
to files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write
Dec 09, 2019 6:26:30 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2019-12-09T06:26:29.853Z: Autoscaling: Startup of the worker pool
in zone us-central1-a reached 1 workers, but the goal was 5 workers. The
service will retry. QUOTA_EXCEEDED: Quota 'CPUS' exceeded. Limit: 1250.0 in
region us-central1.
Dec 09, 2019 6:26:38 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2019-12-09T06:26:37.533Z: Startup of the worker pool in zone
us-central1-a failed to bring up any of the desired 5 workers. QUOTA_EXCEEDED:
Quota 'CPUS' exceeded. Limit: 1250.0 in region us-central1.
Dec 09, 2019 6:26:38 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2019-12-09T06:26:37.561Z: Workflow failed.
Dec 09, 2019 6:26:38 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2019-12-09T06:26:37.848Z: S02:Generate
sequence/Read(BoundedCountingSource)+Produce text lines+Transform strings to
bytes/Map+Record time before writing+Write content to
files/Write/RewindowIntoGlobal/Window.Assign+Write content to
files/Write/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+Write
content to
files/Write/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)+Write
content to
files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+Write content
to files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write failed.
Dec 09, 2019 6:26:38 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:26:37.881Z: Finished operation Generate
sequence/Read(BoundedCountingSource)+Produce text lines+Transform strings to
bytes/Map+Record time before writing+Write content to
files/Write/RewindowIntoGlobal/Window.Assign+Write content to
files/Write/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+Write
content to
files/Write/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)+Write
content to
files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+Write content
to files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write
Dec 09, 2019 6:26:38 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:26:38.007Z: Cleaning up.
Dec 09, 2019 6:26:38 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:26:38.096Z: Stopping worker pool...
Dec 09, 2019 6:26:54 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-12-09T06:26:54.220Z: Worker pool stopped.
Dec 09, 2019 6:26:59 AM
org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState
INFO: Job 2019-12-08_22_25_21-15041567154797501647 failed with status
FAILED.
Gradle Test Executor 1 finished executing tests.
> Task :sdks:java:io:file-based-io-tests:integrationTest FAILED
org.apache.beam.sdk.io.tfrecord.TFRecordIOIT > writeThenReadAll FAILED
java.lang.IllegalStateException: Unable to find any files matching
gs://temp-storage-for-perf-tests/beam_PerformanceTests_TFRecordIOIT/2778/_1575872714692*
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState(Preconditions.java:588)
at org.apache.beam.sdk.io.TFRecordIO$Read.expand(TFRecordIO.java:188)
at org.apache.beam.sdk.io.TFRecordIO$Read.expand(TFRecordIO.java:109)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:539)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:473)
at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:169)
at
org.apache.beam.sdk.io.tfrecord.TFRecordIOIT.writeThenReadAll(TFRecordIOIT.java:136)
1 test completed, 1 failed
Finished generating test XML results (0.018 secs) into:
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/java/io/file-based-io-tests/build/test-results/integrationTest>
Generating HTML test report...
Finished generating test html results (0.026 secs) into:
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/java/io/file-based-io-tests/build/reports/tests/integrationTest>
:sdks:java:io:file-based-io-tests:integrationTest (Thread[Execution worker for
':' Thread 11,5,main]) completed. Took 1 mins 48.268 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:java:io:file-based-io-tests:integrationTest'.
> There were failing tests. See the report at:
> file://<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/java/io/file-based-io-tests/build/reports/tests/integrationTest/index.html>
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to
get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 2m 28s
86 actionable tasks: 55 executed, 31 from cache
Publishing build scan...
https://scans.gradle.com/s/x3jfna55um2p2
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
