lukecwik commented on issue #23562: URL: https://github.com/apache/beam/issues/23562#issuecomment-1275306500
The job logs show that the job succeeded successfully and then was cancelled during worker shutdown since it was taking too long: ``` insertId,"labels.""dataflow.googleapis.com/job_id""","labels.""dataflow.googleapis.com/job_name""","labels.""dataflow.googleapis.com/log_type""","labels.""dataflow.googleapis.com/region""",logName,receiveLocation,receiveTimestamp,receivedLocation,resource.labels.job_id,resource.labels.job_name,resource.labels.project_id,resource.labels.region,resource.labels.step_id,resource.type,severity,textPayload,timestamp vn250qb1v,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,system,us-central1,projects/apache-beam-testing/logs/dataflow.googleapis.com%2Fjob-message,,2022-09-15T20:19:48.324047054Z,,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,844138762903,us-central1,,dataflow_step,INFO,Worker pool stopped.,2022-09-15T20:19:48.138688500Z vn250qb1t,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,system,us-central1,projects/apache-beam-testing/logs/dataflow.googleapis.com%2Fjob-message,,2022-09-15T20:18:49.317472863Z,,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,844138762903,us-central1,,dataflow_step,INFO,Cancel request is committed for workflow job: 2022-09-15_13_08_48-251378908779169791.,2022-09-15T20:18:49.062956246Z vn250qb1s,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,system,us-central1,projects/apache-beam-testing/logs/dataflow.googleapis.com%2Fjob-message,,2022-09-15T20:17:36.308653242Z,,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,844138762903,us-central1,,dataflow_step,INFO,Stopping worker pool...,2022-09-15T20:17:35.403884853Z vn250qb1p,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,system,us-central1,projects/apache-beam-testing/logs/dataflow.googleapis.com%2Fjob-message,,2022-09-15T20:17:36.308653242Z,,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,844138762903,us-central1,,dataflow_step,INFO,Finished operation PAssert$19/GroupGlobally/GroupByKey/ReadStream+PAssert$19/GroupGlobally/GroupByKey/MergeBuckets+PAssert$19/GroupGlobally/Values/Values/Map+PAssert$19/GroupGlobally/ParDo(Concat)+PAssert$19/GetPane/Map+PAssert$19/RunChecks+PAssert$19/VerifyAssertions/ParDo(DefaultConclude),2022-09-15T20:17:35.250660974Z vn250qb1o,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,system,us-central1,projects/apache-beam-testing/logs/dataflow.googleapis.com%2Fjob-message,,2022-09-15T20:17:36.308653242Z,,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,844138762903,us-central1,,dataflow_step,INFO,Finished operation Create.Values/Read(CreateSource)/Read(CreateSource)/Read(BoundedToUnboundedSourceAdapter)/DataflowRunner.StreamingUnboundedRead.ReadWithIds+Create.Values/Read(CreateSource)/Read(CreateSource)/Read(BoundedToUnboundedSourceAdapter)/StripIds+ParMultiDo(Anonymous)+PAssert$19/GroupGlobally/Reify.Window/ParDo(Anonymous)+PAssert$19/GroupGlobally/ParDo(ToSingletonIterables)+PAssert$19/GroupGlobally/WithKeys/AddKeys/Map+PAssert$19/GroupGlobally/GroupByKey/WriteStream+PAssert$20/GroupGlobally/Reify.Window/ParDo(Anonymous)+PAssert $20/GroupGlobally/ParDo(ToSingletonIterables)+PAssert$20/GroupGlobally/WithKeys/AddKeys/Map+PAssert$20/GroupGlobally/GroupByKey/WriteStream,2022-09-15T20:17:35.250659474Z vn250qb1n,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,system,us-central1,projects/apache-beam-testing/logs/dataflow.googleapis.com%2Fjob-message,,2022-09-15T20:17:36.308653242Z,,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,844138762903,us-central1,,dataflow_step,INFO,Finished operation PAssert$20/GroupGlobally/GroupByKey/ReadStream+PAssert$20/GroupGlobally/GroupByKey/MergeBuckets+PAssert$20/GroupGlobally/Values/Values/Map+PAssert$20/GroupGlobally/ParDo(Concat)+PAssert$20/GetPane/Map+PAssert$20/RunChecks+PAssert$20/VerifyAssertions/ParDo(DefaultConclude),2022-09-15T20:17:35.250659444Z vn250qb1l,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,system,us-central1,projects/apache-beam-testing/logs/dataflow.googleapis.com%2Fjob-message,,2022-09-15T20:17:36.308653242Z,,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,844138762903,us-central1,,dataflow_step,INFO,Finished operation PAssert$20/GroupGlobally/Create.Values/Read(CreateSource)/Read(CreateSource)/Read(BoundedToUnboundedSourceAdapter)/DataflowRunner.StreamingUnboundedRead.ReadWithIds+PAssert$20/GroupGlobally/Create.Values/Read(CreateSource)/Read(CreateSource)/Read(BoundedToUnboundedSourceAdapter)/StripIds+PAssert$20/GroupGlobally/WithKeys/AddKeys/Map+PAssert$20/GroupGlobally/GroupByKey/WriteStream,2022-09-15T20:17:35.246025244Z vn250qb1m,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,system,us-central1,projects/apache-beam-testing/logs/dataflow.googleapis.com%2Fjob-message,,2022-09-15T20:17:36.308653242Z,,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,844138762903,us-central1,,dataflow_step,INFO,Finished operation PAssert$19/GroupGlobally/Create.Values/Read(CreateSource)/Read(CreateSource)/Read(BoundedToUnboundedSourceAdapter)/DataflowRunner.StreamingUnboundedRead.ReadWithIds+PAssert$19/GroupGlobally/Create.Values/Read(CreateSource)/Read(CreateSource)/Read(BoundedToUnboundedSourceAdapter)/StripIds+PAssert$19/GroupGlobally/WithKeys/AddKeys/Map+PAssert$19/GroupGlobally/GroupByKey/WriteStream,2022-09-15T20:17:35.246025204Z vn250qb1i,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,system,us-central1,projects/apache-beam-testing/logs/dataflow.googleapis.com%2Fjob-message,,2022-09-15T20:14:17.298514227Z,,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,844138762903,us-central1,,dataflow_step,INFO,"Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors. delete",2022-09-15T20:14:17.075300326Z vn250qb1h,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,system,us-central1,projects/apache-beam-testing/logs/dataflow.googleapis.com%2Fjob-message,,2022-09-15T20:14:05.292174946Z,,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,844138762903,us-central1,,dataflow_step,INFO,Executing operation Create.Values/Read(CreateSource)/Read(CreateSource)/Read(BoundedToUnboundedSourceAdapter)/DataflowRunner.StreamingUnboundedRead.ReadWithIds+Create.Values/Read(CreateSource)/Read(CreateSource)/Read(BoundedToUnboundedSourceAdapter)/StripIds+ParMultiDo(Anonymous)+PAssert$19/GroupGlobally/Reify.Window/ParDo(Anonymous)+PAssert$19/GroupGlobally/ParDo(ToSingletonIterables)+PAssert$19/GroupGlobally/WithKeys/AddKeys/Map+PAssert$19/GroupGlobally/GroupByKey/WriteStream+PAssert$20/GroupGlobally/Reify.Window/ParDo(Anonymous)+PAsser t$20/GroupGlobally/ParDo(ToSingletonIterables)+PAssert$20/GroupGlobally/WithKeys/AddKeys/Map+PAssert$20/GroupGlobally/GroupByKey/WriteStream,2022-09-15T20:14:04.298565100Z vn250qb1g,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,system,us-central1,projects/apache-beam-testing/logs/dataflow.googleapis.com%2Fjob-message,,2022-09-15T20:14:05.292174946Z,,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,844138762903,us-central1,,dataflow_step,INFO,Executing operation PAssert$19/GroupGlobally/Create.Values/Read(CreateSource)/Read(CreateSource)/Read(BoundedToUnboundedSourceAdapter)/DataflowRunner.StreamingUnboundedRead.ReadWithIds+PAssert$19/GroupGlobally/Create.Values/Read(CreateSource)/Read(CreateSource)/Read(BoundedToUnboundedSourceAdapter)/StripIds+PAssert$19/GroupGlobally/WithKeys/AddKeys/Map+PAssert$19/GroupGlobally/GroupByKey/WriteStream,2022-09-15T20:14:04.268195222Z vn250qb1f,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,system,us-central1,projects/apache-beam-testing/logs/dataflow.googleapis.com%2Fjob-message,,2022-09-15T20:14:05.292174946Z,,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,844138762903,us-central1,,dataflow_step,INFO,Executing operation PAssert$19/GroupGlobally/GroupByKey/ReadStream+PAssert$19/GroupGlobally/GroupByKey/MergeBuckets+PAssert$19/GroupGlobally/Values/Values/Map+PAssert$19/GroupGlobally/ParDo(Concat)+PAssert$19/GetPane/Map+PAssert$19/RunChecks+PAssert$19/VerifyAssertions/ParDo(DefaultConclude),2022-09-15T20:14:04.233708744Z vn250qb1e,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,system,us-central1,projects/apache-beam-testing/logs/dataflow.googleapis.com%2Fjob-message,,2022-09-15T20:14:04.284124359Z,,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,844138762903,us-central1,,dataflow_step,INFO,Starting 1 workers in us-central1-b...,2022-09-15T20:14:04.218472955Z vn250qb1c,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,system,us-central1,projects/apache-beam-testing/logs/dataflow.googleapis.com%2Fjob-message,,2022-09-15T20:14:04.284124359Z,,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,844138762903,us-central1,,dataflow_step,INFO,Executing operation PAssert$20/GroupGlobally/Create.Values/Read(CreateSource)/Read(CreateSource)/Read(BoundedToUnboundedSourceAdapter)/DataflowRunner.StreamingUnboundedRead.ReadWithIds+PAssert$20/GroupGlobally/Create.Values/Read(CreateSource)/Read(CreateSource)/Read(BoundedToUnboundedSourceAdapter)/StripIds+PAssert$20/GroupGlobally/WithKeys/AddKeys/Map+PAssert$20/GroupGlobally/GroupByKey/WriteStream,2022-09-15T20:14:04.187811217Z vn250qb1b,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,system,us-central1,projects/apache-beam-testing/logs/dataflow.googleapis.com%2Fjob-message,,2022-09-15T20:14:04.284124359Z,,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,844138762903,us-central1,,dataflow_step,INFO,Executing operation PAssert$20/GroupGlobally/GroupByKey/ReadStream+PAssert$20/GroupGlobally/GroupByKey/MergeBuckets+PAssert$20/GroupGlobally/Values/Values/Map+PAssert$20/GroupGlobally/ParDo(Concat)+PAssert$20/GetPane/Map+PAssert$20/RunChecks+PAssert$20/VerifyAssertions/ParDo(DefaultConclude),2022-09-15T20:14:04.155083290Z vn250qa1,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,system,us-central1,projects/apache-beam-testing/logs/dataflow.googleapis.com%2Fjob-message,,2022-09-15T20:14:01.283211679Z,,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,844138762903,us-central1,,dataflow_step,INFO,Worker configuration: e2-standard-4 in us-central1-b.,2022-09-15T20:14:01.140195894Z vn250qa0,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,system,us-central1,projects/apache-beam-testing/logs/dataflow.googleapis.com%2Fjob-message,,2022-09-15T20:13:51.313954336Z,,2022-09-15_13_08_48-251378908779169791,pardotest0multipleinputsandoutputtests0testpardowithonlytaggedoutput-jenkins-0915200839-123a1be2,844138762903,us-central1,,dataflow_step,WARNING,"The workflow name is not a valid Cloud Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring will be labeled with this modified job name: pardotest0multipleinputsandoutputtests0testpardowithonlyta-ec3z. For the best monitoring experience, please name your job with a valid Cloud Label. For details, see: https://cloud.google.com/compute/docs/labeling-resources#restrictions",2022-09-15T20:13:50.362284439Z ``` This failure will not block the release. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
