damccorm commented on PR #24204: URL: https://github.com/apache/beam/pull/24204#issuecomment-1366007488
@Abacn I don't think that approach fully accomplishes what I'm aiming for here though. There are several advantages of sharding tests: [1] Faster execution time on individual Precommits so that we get a faster fail/succeed signal. [2] Lower rate of flakes per suite (makes retrying easier). [3] Better signal on which test suites are consistently flaky. [4] Gating precommits to tighter directories (e.g. don't run dataframes tests if only IOs have changed) I'm mostly going after item number [3] in this PR; splitting by language version doesn't really help with [3] since test suites generally fail together. This split would help with debuggability by providing a tighter signal, and hopefully shrinking our test surface enough that the test analyzer plugin works. It also comes with some tangential benefits to [1] and [2], but that isn't really the goal. Splitting by language version and/or tool version will help with items [1] and [2], but there's not a very clean way to logically represent that kind of matrix as Jenkins jobs so I'd probably prefer to defer that until after the GHA migration. Its also worth noting that this isn't an accurate final view of this PR and I'm planning on consolidating back down to several (4ish) jobs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
