Abacn commented on PR #21877: URL: https://github.com/apache/beam/pull/21877#issuecomment-1156715224
Tests being flake mainly because there is no control of when the is pipeline started running. The writing file thread may start early and the watch transform misses to capture some changes. Test failure does not indicate the functionality issue of the methods being tested. To fix the flaky tests, logic of both unit test and integration test has been changed. For the unit test, file rewrite is triggered downstream of the pipeline. This eliminated a separate thread and guarantees the second write happens after the watch transform is running. For the integration test, it is observed that the downstream stage may be started after several seconds after the previous stage in dataflow runner. Putting rewrite in downstream then may have problems that it never gets triggered within the wait time of watch transform, causing another flake. Another approach is used for the integration test. A separate writing thread is set up before the pipeline running, and turned down after pipeline ending. And assertion logic is changed to be insensitive to when pipeline starts and stops. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
