See
<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/1460/display/redirect>
Changes:
------------------------------------------
[...truncated 247.04 KB...]
INFO: Uploading
<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/sdks/java/testing/test-utils/build/libs/beam-sdks-java-test-utils-2.26.0-SNAPSHOT-tests.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-sdks-java-test-utils-2.26.0-SNAPSHOT-tests-B_BoWVbTsjdsWwqEzLL-p1WnsAoyaxXKNw3Gm-gj2B0.jar
Oct 30, 2020 10:40:04 PM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.26.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-sdks-java-io-google-cloud-platform-2.26.0-SNAPSHOT-0YAGEUgAKoRSqUeirZ1YwoKgcltvuy_q_BiQDONCqjU.jar
Oct 30, 2020 10:40:04 PM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/sdks/java/io/common/build/libs/beam-sdks-java-io-common-2.26.0-SNAPSHOT-tests.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-sdks-java-io-common-2.26.0-SNAPSHOT-tests-J5ViKZvp8hDx9nYFTUjS5e2cTiEgrDuHi7n640cvEm0.jar
Oct 30, 2020 10:40:04 PM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/runners/core-construction-java/build/libs/beam-runners-core-construction-java-2.26.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-runners-core-construction-java-2.26.0-SNAPSHOT-qcGjyMFsPfMlX5nU3xnhPwIn0kC2zPRrvXaN53Ko_h8.jar
Oct 30, 2020 10:40:04 PM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/vendor/sdks-java-extensions-protobuf/build/libs/beam-vendor-sdks-java-extensions-protobuf-2.26.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-vendor-sdks-java-extensions-protobuf-2.26.0-SNAPSHOT-OkC7FbN6RuSJU8ge0bIbk8GnfN1R26AfKOiNzu0fjFI.jar
Oct 30, 2020 10:40:04 PM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/sdks/java/core/build/libs/beam-sdks-java-core-2.26.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-sdks-java-core-2.26.0-SNAPSHOT-xoKDWU95WOw9_LWa0eeLgsncIHvog4wtSIsuNRP-A-U.jar
Oct 30, 2020 10:40:04 PM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/sdks/java/fn-execution/build/libs/beam-sdks-java-fn-execution-2.26.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-sdks-java-fn-execution-2.26.0-SNAPSHOT-iwMrYS-2_mj8S-AxvqkqMbIblr2MsoQy2WiJWFiAEGA.jar
Oct 30, 2020 10:40:04 PM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/sdks/java/io/synthetic/build/libs/beam-sdks-java-io-synthetic-2.26.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-sdks-java-io-synthetic-2.26.0-SNAPSHOT-Xtqg0HcRZSgJnTWcYXzlHdHSVvZIfKFhq1YtDAitWwQ.jar
Oct 30, 2020 10:40:04 PM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/model/job-management/build/libs/beam-model-job-management-2.26.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-model-job-management-2.26.0-SNAPSHOT-4R4H-wipu3YfN85p4JluENdnh-Cq4H53L9DprPTATd0.jar
Oct 30, 2020 10:40:04 PM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/runners/google-cloud-dataflow-java/****/legacy-****/build/libs/beam-runners-google-cloud-dataflow-java-legacy-****-2.26.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-runners-google-cloud-dataflow-java-legacy-****-2.26.0-SNAPSHOT-CjTdtTyEO8T6-GS1JQOXhKxi7XY9_NGyXcuBz_Cvh4c.jar
Oct 30, 2020 10:40:04 PM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/model/pipeline/build/libs/beam-model-pipeline-2.26.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-model-pipeline-2.26.0-SNAPSHOT-WSGLwuoOHhOi5an_JYetVDGgthBvdSBEH_jpnG7NmCE.jar
Oct 30, 2020 10:40:04 PM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading
<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/model/fn-execution/build/libs/beam-model-fn-execution-2.26.0-SNAPSHOT.jar>
to
gs://dataflow-staging-us-central1-844138762903/temp/staging/beam-model-fn-execution-2.26.0-SNAPSHOT-gBUD68WfdkUvDrt2muRxGdG28aI7z_qMdUbGVG9YRwU.jar
Oct 30, 2020 10:40:05 PM org.apache.beam.runners.dataflow.util.PackageUtil
stageClasspathElements
INFO: Staging files complete: 177 files cached, 26 files newly uploaded in
1 seconds
Oct 30, 2020 10:40:05 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Generate records as step s1
Oct 30, 2020 10:40:05 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Measure write time as step s2
Oct 30, 2020 10:40:05 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write to Kafka/Kafka ProducerRecord/Map as step s3
Oct 30, 2020 10:40:05 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write to Kafka/KafkaIO.WriteRecords/ParDo(KafkaWriter) as step
s4
Oct 30, 2020 10:40:05 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to
gs://dataflow-staging-us-central1-844138762903/temp/staging/
Oct 30, 2020 10:40:05 PM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading <88453 bytes, hash
c53df382df65b3bfc19f64c728ef9bd7280149211dac2443905951a974c88989> to
gs://dataflow-staging-us-central1-844138762903/temp/staging/pipeline-xT3zgt9ls7_Bn2THKO-b1ygBSSEdrCRDkFlRqXTIiYk.pb
Oct 30, 2020 10:40:05 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.26.0-SNAPSHOT
Oct 30, 2020 10:40:07 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-10-30_15_40_05-5346441651832381641?project=apache-beam-testing
Oct 30, 2020 10:40:07 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2020-10-30_15_40_05-5346441651832381641
Oct 30, 2020 10:40:07 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel
--region=us-central1 2020-10-30_15_40_05-5346441651832381641
Oct 30, 2020 10:40:22 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:40:20.803Z: Worker configuration: n1-standard-1 in
us-central1-f.
Oct 30, 2020 10:40:22 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:40:21.650Z: Expanding CoGroupByKey operations into
optimizable parts.
Oct 30, 2020 10:40:22 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:40:21.683Z: Expanding GroupByKey operations into
optimizable parts.
Oct 30, 2020 10:40:22 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:40:21.715Z: Lifting ValueCombiningMappingFns into
MergeBucketsMappingFns
Oct 30, 2020 10:40:22 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:40:21.799Z: Fusing adjacent ParDo, Read, Write, and
Flatten operations
Oct 30, 2020 10:40:22 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:40:21.835Z: Fusing consumer Measure write time into
Generate records
Oct 30, 2020 10:40:22 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:40:21.860Z: Fusing consumer Write to Kafka/Kafka
ProducerRecord/Map into Measure write time
Oct 30, 2020 10:40:22 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:40:21.892Z: Fusing consumer Write to
Kafka/KafkaIO.WriteRecords/ParDo(KafkaWriter) into Write to Kafka/Kafka
ProducerRecord/Map
Oct 30, 2020 10:40:22 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:40:22.366Z: Executing operation Generate
records+Measure write time+Write to Kafka/Kafka ProducerRecord/Map+Write to
Kafka/KafkaIO.WriteRecords/ParDo(KafkaWriter)
Oct 30, 2020 10:40:22 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:40:22.453Z: Starting 5 ****s in us-central1-f...
Oct 30, 2020 10:40:28 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:40:27.440Z: Your project already contains 100
Dataflow-created metric descriptors, so new user metrics of the form
custom.googleapis.com/* will not be created. However, all user metrics are also
available in the metric dataflow.googleapis.com/job/user_counter. If you rely
on the custom metrics, you can delete old / unused metric descriptors. See
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list
and
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
Oct 30, 2020 10:45:39 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:45:36.741Z: Autoscaling: Raised the number of ****s to
1 based on the rate of progress in the currently running stage(s).
Oct 30, 2020 10:45:39 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:45:36.954Z: Resized **** pool to 1, though goal was 5.
This could be a quota issue.
Oct 30, 2020 10:45:41 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:45:41.011Z: Workers have started successfully.
Oct 30, 2020 10:45:42 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:45:42.275Z: Autoscaling: Raised the number of ****s to
3 based on the rate of progress in the currently running stage(s).
Oct 30, 2020 10:45:42 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:45:42.307Z: Resized **** pool to 3, though goal was 5.
This could be a quota issue.
Oct 30, 2020 10:45:47 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:45:47.616Z: Autoscaling: Raised the number of ****s to
4 based on the rate of progress in the currently running stage(s).
Oct 30, 2020 10:45:47 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:45:47.649Z: Resized **** pool to 4, though goal was 5.
This could be a quota issue.
Oct 30, 2020 10:45:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:45:49.006Z: Workers have started successfully.
Oct 30, 2020 10:47:51 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:47:49.279Z: Autoscaling: Raised the number of ****s to
5 based on the rate of progress in the currently running stage(s).
Oct 30, 2020 10:49:01 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:49:00.395Z: Finished operation Generate records+Measure
write time+Write to Kafka/Kafka ProducerRecord/Map+Write to
Kafka/KafkaIO.WriteRecords/ParDo(KafkaWriter)
Oct 30, 2020 10:49:01 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:49:00.593Z: Cleaning up.
Oct 30, 2020 10:49:01 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:49:00.658Z: Stopping **** pool...
Oct 30, 2020 10:54:11 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:54:10.266Z: Autoscaling: Resized **** pool from 5 to 0.
Oct 30, 2020 10:54:11 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-30T22:54:10.306Z: Worker pool stopped.
Oct 30, 2020 10:54:16 PM
org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState
INFO: Job 2020-10-30_15_40_05-5346441651832381641 finished with status DONE.
Oct 30, 2020 10:54:16 PM
org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory
tryCreateDefaultBucket
INFO: No tempLocation specified, attempting to use default bucket:
dataflow-staging-us-central1-844138762903
Oct 30, 2020 10:54:17 PM
org.apache.beam.sdk.extensions.gcp.util.RetryHttpRequestInitializer$LoggingHttpBackOffHandler
handleResponse
WARNING: Request failed with code 409, performed 0 retries due to
IOExceptions, performed 0 retries due to unsuccessful status codes, HTTP
framework says request can be retried, (caller responsible for retrying):
https://storage.googleapis.com/storage/v1/b?predefinedAcl=projectPrivate&predefinedDefaultObjectAcl=projectPrivate&project=apache-beam-testing.
Oct 30, 2020 10:54:17 PM
org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory
create
INFO: No stagingLocation provided, falling back to gcpTempLocation
Oct 30, 2020 10:54:17 PM org.apache.beam.runners.dataflow.DataflowRunner
fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files
from the classpath: will stage 202 files. Enable logging at DEBUG level to see
which files will be staged.
Oct 30, 2020 10:54:17 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing
implications related to Google Compute Engine usage and other Google Cloud
Services.
Oct 30, 2020 10:54:18 PM org.apache.beam.runners.dataflow.util.PackageUtil
stageClasspathElements
INFO: Uploading 203 files from PipelineOptions.filesToStage to staging
location to prepare for execution.
Oct 30, 2020 10:54:18 PM
org.apache.beam.runners.dataflow.util.PackageUtil$PackageAttributes
forFileToStage
INFO: Staging custom dataflow-****.jar as
beam-runners-google-cloud-dataflow-java-legacy-****-2.26.0-SNAPSHOT-CjTdtTyEO8T6-GS1JQOXhKxi7XY9_NGyXcuBz_Cvh4c.jar
Oct 30, 2020 10:54:19 PM org.apache.beam.runners.dataflow.util.PackageUtil
stageClasspathElements
INFO: Staging files complete: 203 files cached, 0 files newly uploaded in 0
seconds
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from Kafka/Impulse as step s1
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from Kafka/ParDo(GenerateKafkaSourceDescriptor) as step s2
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
Kafka/KafkaIO.ReadSourceDescriptors/ParDo(ReadFromKafka) as step s3
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
Kafka/KafkaIO.ReadSourceDescriptors/Reshuffle.ViaRandomKey/Pair with random key
as step s4
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
Kafka/KafkaIO.ReadSourceDescriptors/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
as step s5
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
Kafka/KafkaIO.ReadSourceDescriptors/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey
as step s6
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
Kafka/KafkaIO.ReadSourceDescriptors/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
as step s7
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
Kafka/KafkaIO.ReadSourceDescriptors/Reshuffle.ViaRandomKey/Values/Values/Map as
step s8
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
Kafka/KafkaIO.ReadSourceDescriptors/KafkaCommitOffset/MapElements/Map as step s9
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
Kafka/KafkaIO.ReadSourceDescriptors/KafkaCommitOffset/Window.Into()/Window.Assign
as step s10
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
Kafka/KafkaIO.ReadSourceDescriptors/KafkaCommitOffset/Combine.perKey(MaxLong)/GroupByKey
as step s11
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
Kafka/KafkaIO.ReadSourceDescriptors/KafkaCommitOffset/Combine.perKey(MaxLong)/Combine.GroupedValues
as step s12
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
Kafka/KafkaIO.ReadSourceDescriptors/KafkaCommitOffset/ParDo(CommitOffset) as
step s13
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from Kafka/KafkaIO.ReadSourceDescriptors/MapElements/Map
as step s14
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Measure read time as step s15
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Map records to strings/Map as step s16
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/Reify.Window/ParDo(Anonymous) as step
s17
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/ParDo(ToSingletonIterables) as step s18
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/Create.Values/Impulse as step s19
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/Create.Values/ParDo(DecodeAndEmit) as
step s20
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/Flatten.PCollections as step s21
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/Window.Into()/Flatten.PCollections as
step s22
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/WithKeys/AddKeys/Map as step s23
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/GroupByKey as step s24
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/Values/Values/Map as step s25
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/ParDo(Concat) as step s26
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GetPane/Map as step s27
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/RunChecks as step s28
Oct 30, 2020 10:54:19 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/VerifyAssertions/ParDo(DefaultConclude) as step s29
Oct 30, 2020 10:54:19 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to
gs://dataflow-staging-us-central1-844138762903/temp/staging/
Oct 30, 2020 10:54:19 PM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading <148605 bytes, hash
3673e76d561c58d8da39b59245bcf26f81f446355faa11aa4572fdfa9e217755> to
gs://dataflow-staging-us-central1-844138762903/temp/staging/pipeline-NnPnbVYcWNjaObWSRbzyb4H0RjVfqhGqRXL9-p4hd1U.pb
Oct 30, 2020 10:54:19 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.26.0-SNAPSHOT
Oct 30, 2020 10:54:19 PM
org.apache.beam.sdk.extensions.gcp.util.RetryHttpRequestInitializer$LoggingHttpBackOffHandler
handleResponse
WARNING: Request failed with code 400, performed 0 retries due to
IOExceptions, performed 0 retries due to unsuccessful status codes, HTTP
framework says request can be retried, (caller responsible for retrying):
https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/us-central1/jobs.
Gradle Test Executor 2 finished executing tests.
> Task :sdks:java:io:kafka:integrationTest FAILED
org.apache.beam.sdk.io.kafka.KafkaIOIT > testSDFKafkaIORead FAILED
java.lang.RuntimeException: Failed to create a workflow job:
(fff20f41ab66c3db): The workflow could not be created. Causes:
(fff20f41ab66c998): An internal service error occurred. Please contact customer
support. Please include the message id if one is provided.
at
org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:1127)
at
org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:196)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:320)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:351)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:332)
at
org.apache.beam.sdk.io.kafka.KafkaIOIT.runKafkaTestPipeline(KafkaIOIT.java:139)
at
org.apache.beam.sdk.io.kafka.KafkaIOIT.testSDFKafkaIORead(KafkaIOIT.java:160)
Caused by:
com.google.api.client.googleapis.json.GoogleJsonResponseException: 400
Bad Request
{
"code" : 400,
"errors" : [ {
"domain" : "global",
"message" : "(fff20f41ab66c3db): The workflow could not be created.
Causes: (fff20f41ab66c998): An internal service error occurred. Please contact
customer support. Please include the message id if one is provided.",
"reason" : "badRequest"
} ],
"message" : "(fff20f41ab66c3db): The workflow could not be created.
Causes: (fff20f41ab66c998): An internal service error occurred. Please contact
customer support. Please include the message id if one is provided.",
"status" : "INVALID_ARGUMENT"
}
at
com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:149)
at
com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:112)
at
com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:39)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:443)
at
com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1108)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:541)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:474)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:591)
at
org.apache.beam.runners.dataflow.DataflowClient.createJob(DataflowClient.java:62)
at
org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:1113)
... 6 more
1 test completed, 1 failed
Finished generating test XML results (0.021 secs) into:
<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/sdks/java/io/kafka/build/test-results/integrationTest>
Generating HTML test report...
Finished generating test html results (0.027 secs) into:
<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/sdks/java/io/kafka/build/reports/tests/integrationTest>
:sdks:java:io:kafka:integrationTest (Thread[Execution **** for ':' Thread
2,5,main]) completed. Took 14 mins 25.209 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:java:io:kafka:integrationTest'.
> There were failing tests. See the report at:
> file://<https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/sdks/java/io/kafka/build/reports/tests/integrationTest/index.html>
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to
get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.6.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 15m 11s
85 actionable tasks: 52 executed, 33 from cache
Publishing build scan...
https://gradle.com/s/rk6jbemdegnza
Stopped 1 **** daemon(s).
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
