See
<https://builds.apache.org/job/beam_PerformanceTests_Kafka_IO/8/display/redirect>
Changes:
------------------------------------------
[...truncated 278.59 KB...]
INFO: Adding Calculate hashcode/WithKeys/AddKeys/Map as step s12
Nov 04, 2019 2:15:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Calculate hashcode/Combine.perKey(Hashing)/GroupByKey as step
s13
Nov 04, 2019 2:15:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues as step s14
Nov 04, 2019 2:15:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Calculate hashcode/Values/Values/Map as step s15
Nov 04, 2019 2:15:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/Window.Into()/Window.Assign as step s16
Nov 04, 2019 2:15:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
PAssert$0/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous) as step
s17
Nov 04, 2019 2:15:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map
as step s18
Nov 04, 2019 2:15:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
PAssert$0/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign as step s19
Nov 04, 2019 2:15:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey as step s20
Nov 04, 2019 2:15:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/GatherAllOutputs/Values/Values/Map as
step s21
Nov 04, 2019 2:15:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/RewindowActuals/Window.Assign as step
s22
Nov 04, 2019 2:15:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/KeyForDummy/AddKeys/Map as step s23
Nov 04, 2019 2:15:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
PAssert$0/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step s24
Nov 04, 2019 2:15:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/Create.Values/Read(CreateSource) as
step s25
Nov 04, 2019 2:15:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/WindowIntoDummy/Window.Assign as step
s26
Nov 04, 2019 2:15:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
PAssert$0/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s27
Nov 04, 2019 2:15:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/FlattenDummyAndContents as step s28
Nov 04, 2019 2:15:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/NeverTrigger/Flatten.PCollections as
step s29
Nov 04, 2019 2:15:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/GroupDummyAndContents as step s30
Nov 04, 2019 2:15:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/Values/Values/Map as step s31
Nov 04, 2019 2:15:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/ParDo(Concat) as step s32
Nov 04, 2019 2:15:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GetPane/Map as step s33
Nov 04, 2019 2:15:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/RunChecks as step s34
Nov 04, 2019 2:15:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/VerifyAssertions/ParDo(DefaultConclude) as step s35
Nov 04, 2019 2:15:30 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to
gs://dataflow-staging-us-central1-844138762903/temp/staging/
Nov 04, 2019 2:15:30 PM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading <78141 bytes, hash mVx1Td8sZtzvIa6eB2GXvg> to
gs://dataflow-staging-us-central1-844138762903/temp/staging/pipeline-mVx1Td8sZtzvIa6eB2GXvg.pb
Nov 04, 2019 2:15:30 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.18.0-SNAPSHOT
Nov 04, 2019 2:15:32 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-11-04_06_15_30-17038796269472273495?project=apache-beam-testing
Nov 04, 2019 2:15:32 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2019-11-04_06_15_30-17038796269472273495
Nov 04, 2019 2:15:32 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel
--region=us-central1 2019-11-04_06_15_30-17038796269472273495
Nov 04, 2019 2:15:35 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2019-11-04T14:15:34.727Z: The workflow name is not a valid Cloud
Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring
will be labeled with this modified job name:
kafkaioit0testkafkaioreadsandwritescorrectly-jenkins-11041-mj3i. For the best
monitoring experience, please name your job with a valid Cloud Label. For
details, see:
https://cloud.google.com/compute/docs/labeling-resources#restrictions
Nov 04, 2019 2:15:35 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:34.781Z: Checking permissions granted to controller
Service Account.
Nov 04, 2019 2:15:47 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:47.059Z: Worker configuration: n1-standard-1 in
us-central1-f.
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:47.698Z: Expanding CoGroupByKey operations into
optimizable parts.
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:47.882Z: Expanding GroupByKey operations into
optimizable parts.
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:47.914Z: Lifting ValueCombiningMappingFns into
MergeBucketsMappingFns
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.141Z: Fusing adjacent ParDo, Read, Write, and
Flatten operations
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.180Z: Elided trivial flatten
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.216Z: Elided trivial flatten
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.249Z: Elided trivial flatten
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.278Z: Unzipping flatten s28 for input
s23.org.apache.beam.sdk.values.PCollection.<init>:400#154393cf2f9d04a6
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.313Z: Fusing unzipped copy of
PAssert$0/GroupGlobally/GroupDummyAndContents/Reify, through flatten
PAssert$0/GroupGlobally/FlattenDummyAndContents, into producer
PAssert$0/GroupGlobally/KeyForDummy/AddKeys/Map
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.347Z: Fusing consumer
PAssert$0/GroupGlobally/GroupDummyAndContents/GroupByWindow into
PAssert$0/GroupGlobally/GroupDummyAndContents/Read
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.383Z: Fusing consumer
PAssert$0/GroupGlobally/Values/Values/Map into
PAssert$0/GroupGlobally/GroupDummyAndContents/GroupByWindow
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.415Z: Fusing consumer
PAssert$0/GroupGlobally/ParDo(Concat) into
PAssert$0/GroupGlobally/Values/Values/Map
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.443Z: Fusing consumer PAssert$0/GetPane/Map into
PAssert$0/GroupGlobally/ParDo(Concat)
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.467Z: Fusing consumer PAssert$0/RunChecks into
PAssert$0/GetPane/Map
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.500Z: Fusing consumer
PAssert$0/VerifyAssertions/ParDo(DefaultConclude) into PAssert$0/RunChecks
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.535Z: Unzipping flatten s28-u49 for input
s30-reify-value27-c47
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.563Z: Fusing unzipped copy of
PAssert$0/GroupGlobally/GroupDummyAndContents/Write, through flatten
PAssert$0/GroupGlobally/FlattenDummyAndContents/Unzipped-1, into producer
PAssert$0/GroupGlobally/GroupDummyAndContents/Reify
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.597Z: Fusing consumer Read from
Kafka/Read(KafkaUnboundedSource)/Split into Read from
Kafka/Read(KafkaUnboundedSource)/Create/Read(CreateSource)
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.623Z: Fusing consumer Read from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Pair with random key into Read from
Kafka/Read(KafkaUnboundedSource)/Split
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.652Z: Fusing consumer Read from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Reshuffle/Window.Into()/Window.Assign
into Read from Kafka/Read(KafkaUnboundedSource)/Reshuffle/Pair with random key
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.685Z: Fusing consumer Read from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Reshuffle/GroupByKey/Reify into Read
from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Reshuffle/Window.Into()/Window.Assign
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.721Z: Fusing consumer Read from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Reshuffle/GroupByKey/Write into Read
from Kafka/Read(KafkaUnboundedSource)/Reshuffle/Reshuffle/GroupByKey/Reify
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.757Z: Fusing consumer Read from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Reshuffle/GroupByKey/GroupByWindow
into Read from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Reshuffle/GroupByKey/Read
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.797Z: Fusing consumer Read from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Reshuffle/ExpandIterable into Read
from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Reshuffle/GroupByKey/GroupByWindow
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.832Z: Fusing consumer Read from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Values/Values/Map into Read from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Reshuffle/ExpandIterable
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.857Z: Fusing consumer Read from
Kafka/Read(KafkaUnboundedSource)/Read into Read from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Values/Values/Map
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.893Z: Fusing consumer Read from
Kafka/Read(KafkaUnboundedSource)/StripIds into Read from
Kafka/Read(KafkaUnboundedSource)/Read
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.931Z: Fusing consumer Measure read time into Read
from Kafka/Read(KafkaUnboundedSource)/StripIds
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.968Z: Fusing consumer Map records to strings/Map
into Measure read time
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:48.993Z: Fusing consumer Calculate
hashcode/WithKeys/AddKeys/Map into Map records to strings/Map
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:49.027Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial into Calculate
hashcode/WithKeys/AddKeys/Map
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:49.062Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify into Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:49.096Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Write into Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:49.125Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues into Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Read
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:49.161Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Extract into Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:49.195Z: Fusing consumer Calculate
hashcode/Values/Values/Map into Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Extract
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:49.222Z: Fusing consumer
PAssert$0/GroupGlobally/Window.Into()/Window.Assign into Calculate
hashcode/Values/Values/Map
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:49.257Z: Fusing consumer
PAssert$0/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous) into
PAssert$0/GroupGlobally/Window.Into()/Window.Assign
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:49.282Z: Fusing consumer
PAssert$0/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map into
PAssert$0/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:49.313Z: Fusing consumer
PAssert$0/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign into
PAssert$0/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:49.348Z: Fusing consumer
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Reify into
PAssert$0/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:49.382Z: Fusing consumer
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Write into
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Reify
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:49.414Z: Fusing consumer
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/GroupByWindow into
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Read
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:49.449Z: Fusing consumer
PAssert$0/GroupGlobally/GatherAllOutputs/Values/Values/Map into
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/GroupByWindow
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:49.489Z: Fusing consumer
PAssert$0/GroupGlobally/RewindowActuals/Window.Assign into
PAssert$0/GroupGlobally/GatherAllOutputs/Values/Values/Map
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:49.522Z: Fusing consumer
PAssert$0/GroupGlobally/KeyForDummy/AddKeys/Map into
PAssert$0/GroupGlobally/RewindowActuals/Window.Assign
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:49.554Z: Fusing consumer
PAssert$0/GroupGlobally/GroupDummyAndContents/Write into
PAssert$0/GroupGlobally/GroupDummyAndContents/Reify
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:49.586Z: Fusing consumer
PAssert$0/GroupGlobally/GroupDummyAndContents/Reify into
PAssert$0/GroupGlobally/WindowIntoDummy/Window.Assign
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:49.623Z: Fusing consumer
PAssert$0/GroupGlobally/WindowIntoDummy/Window.Assign into
PAssert$0/GroupGlobally/Create.Values/Read(CreateSource)
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:50.028Z: Executing operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Create
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:50.062Z: Executing operation
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Create
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:50.098Z: Executing operation
PAssert$0/GroupGlobally/GroupDummyAndContents/Create
Nov 04, 2019 2:15:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:50.110Z: Starting 5 workers in us-central1-f...
Nov 04, 2019 2:15:52 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:50.146Z: Executing operation Read from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Reshuffle/GroupByKey/Create
Nov 04, 2019 2:15:52 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:50.167Z: Finished operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Create
Nov 04, 2019 2:15:52 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:50.199Z: Finished operation
PAssert$0/GroupGlobally/GroupDummyAndContents/Create
Nov 04, 2019 2:15:52 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:50.199Z: Finished operation Read from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Reshuffle/GroupByKey/Create
Nov 04, 2019 2:15:52 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:50.199Z: Finished operation
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Create
Nov 04, 2019 2:15:52 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:50.472Z: Executing operation
PAssert$0/GroupGlobally/Create.Values/Read(CreateSource)+PAssert$0/GroupGlobally/WindowIntoDummy/Window.Assign+PAssert$0/GroupGlobally/GroupDummyAndContents/Reify+PAssert$0/GroupGlobally/GroupDummyAndContents/Write
Nov 04, 2019 2:15:52 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:15:50.506Z: Executing operation Read from
Kafka/Read(KafkaUnboundedSource)/Create/Read(CreateSource)+Read from
Kafka/Read(KafkaUnboundedSource)/Split+Read from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Pair with random key+Read from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Reshuffle/Window.Into()/Window.Assign+Read
from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Reshuffle/GroupByKey/Reify+Read from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Reshuffle/GroupByKey/Write
Nov 04, 2019 2:16:06 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2019-11-04T14:16:06.641Z: Your project already contains 100
Dataflow-created metric descriptors and Stackdriver will not create new
Dataflow custom metrics for this job. Each unique user-defined metric name
(independent of the DoFn in which it is defined) produces a new metric
descriptor. To delete old / unused metric descriptors see
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list
and
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
Nov 04, 2019 2:16:28 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:16:27.086Z: Autoscaling: Raised the number of workers
to 5 based on the rate of progress in the currently running step(s).
Nov 04, 2019 2:16:33 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:16:33.126Z: Workers have started successfully.
Nov 04, 2019 2:16:33 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:16:33.167Z: Workers have started successfully.
Nov 04, 2019 2:16:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:16:49.889Z: Finished operation
PAssert$0/GroupGlobally/Create.Values/Read(CreateSource)+PAssert$0/GroupGlobally/WindowIntoDummy/Window.Assign+PAssert$0/GroupGlobally/GroupDummyAndContents/Reify+PAssert$0/GroupGlobally/GroupDummyAndContents/Write
Nov 04, 2019 2:17:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:17:22.654Z: Finished operation Read from
Kafka/Read(KafkaUnboundedSource)/Create/Read(CreateSource)+Read from
Kafka/Read(KafkaUnboundedSource)/Split+Read from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Pair with random key+Read from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Reshuffle/Window.Into()/Window.Assign+Read
from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Reshuffle/GroupByKey/Reify+Read from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Reshuffle/GroupByKey/Write
Nov 04, 2019 2:17:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:17:22.728Z: Executing operation Read from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Reshuffle/GroupByKey/Close
Nov 04, 2019 2:17:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:17:22.784Z: Finished operation Read from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Reshuffle/GroupByKey/Close
Nov 04, 2019 2:17:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:17:22.852Z: Executing operation Read from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Reshuffle/GroupByKey/Read+Read from
Kafka/Read(KafkaUnboundedSource)/Reshuffle/Reshuffle/GroupByKey/GroupByWindow+Read
from Kafka/Read(KafkaUnboundedSource)/Reshuffle/Reshuffle/ExpandIterable+Read
from Kafka/Read(KafkaUnboundedSource)/Reshuffle/Values/Values/Map+Read from
Kafka/Read(KafkaUnboundedSource)/Read+Read from
Kafka/Read(KafkaUnboundedSource)/StripIds+Measure read time+Map records to
strings/Map+Calculate hashcode/WithKeys/AddKeys/Map+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Write
Nov 04, 2019 2:21:51 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:21:49.913Z: Checking permissions granted to controller
Service Account.
Nov 04, 2019 2:27:51 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-11-04T14:27:49.913Z: Checking permissions granted to controller
Service Account.
Nov 04, 2019 2:30:32 PM
org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
WARNING: No terminal state was returned within allotted timeout. State
value RUNNING
Gradle Test Executor 1 finished executing tests.
> Task :sdks:java:io:kafka:integrationTest FAILED
org.apache.beam.sdk.io.kafka.KafkaIOIT > testKafkaIOReadsAndWritesCorrectly
FAILED
java.lang.NullPointerException
at
org.apache.beam.sdk.io.kafka.KafkaIOIT.cancelReadAfterTimeout(KafkaIOIT.java:157)
at
org.apache.beam.sdk.io.kafka.KafkaIOIT.testKafkaIOReadsAndWritesCorrectly(KafkaIOIT.java:129)
1 test completed, 1 failed
Finished generating test XML results (0.024 secs) into:
<https://builds.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/sdks/java/io/kafka/build/test-results/integrationTest>
Generating HTML test report...
Finished generating test html results (0.035 secs) into:
<https://builds.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/sdks/java/io/kafka/build/reports/tests/integrationTest>
:sdks:java:io:kafka:integrationTest (Thread[Execution worker for ':' Thread
6,5,main]) completed. Took 22 mins 48.983 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:java:io:kafka:integrationTest'.
> There were failing tests. See the report at:
> file://<https://builds.apache.org/job/beam_PerformanceTests_Kafka_IO/ws/src/sdks/java/io/kafka/build/reports/tests/integrationTest/index.html>
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to
get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 23m 36s
82 actionable tasks: 54 executed, 28 from cache
Publishing build scan...
https://gradle.com/s/ltzzjlbo5jnjc
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
