lnogueir opened a new issue, #22610: URL: https://github.com/apache/beam/issues/22610
### What happened? While performing load tests for the FhirIO [Import](https://github.com/lnogueir/beam/blob/a7abdc2771098576dacae7c71a8ebf32b77c2ed2/sdks/go/pkg/beam/io/fhirio/import.go#L220) transform I implemented in #22460 for the Go SDK, I noticed that after the resources were successfully imported, the pipeline would restart the pipeline from the beginning. It would do this 4 times until the pipeline fails with the error: `The job failed because a work item has failed 4 times. Root cause: The worker lost contact with the service.` My theory is that because the import request performed inside `FinishBundle` takes several minutes to complete (since it is importing millions of resources), the runner thinks it failed and then it retries. The reason for filing this issue is because a blocking call that takes the same amount of time does not cause this retry loop failure and completes successfully in the Java SDK. Also because using a long-running `FinishBundle` method is a Beam design pattern that is widespread and even considered a best practice as described [here](https://cloud.google.com/architecture/e-commerce/patterns/batching-external-calls). The issue can be reproduced with a dummy pipeline that just sleeps for 20 minutes inside `FinishBundle` or any other `DoFn` lifecycle method. I made these dummy pipelines available so others can easily reproduce the issue themselves: https://github.com/apache/beam/compare/master...lnogueir:beam:dataflowWorkerLiveness Instructions on how to execute them are commented on the files in the link above. I have validated that the Java SDK doesn't experience this issue on neither runner V1 nor V2. CC: @lostluck @jrmccluskey @msbukal ### Issue Priority Priority: 2 ### Issue Component Component: io-go-gcp -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
