johannaojeling commented on issue #29355: URL: https://github.com/apache/beam/issues/29355#issuecomment-1803329156
Hm.. I've been trying to troubleshoot it but the DoFn's ProcessElement method seems to return an error as expected. The DoFn behavior I could think of causing the flakiness is if the condition to check whether the path is a directory is not true for this CI environment. https://github.com/apache/beam/blob/6bf871ff424e106ac83cc56a6f42d393225289e6/sdks/go/pkg/beam/io/fileio/read.go#L135-L139 However I checked [another job](https://github.com/apache/beam/actions/runs/6806182771/job/18507021629) for the same type of runner and environment, and all fileio tests were successful. I can reproduce the failing test behavior by introducing some delay: https://github.com/apache/beam/blob/6bf871ff424e106ac83cc56a6f42d393225289e6/sdks/go/pkg/beam/core/runtime/harness/harness.go#L483-L485 ```diff if err != nil { + time.Sleep(100 * time.Millisecond) return fail(ctx, instID, "process bundle failed for instruction %v using plan %v : %v", instID, bdID, err) } ``` In this case the pipeline does not fail as it should. There might be some cancelled context having to do with it but I lack enough knowledge of this part of the code base to understand what is going on. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
