mumrah commented on PR #17155: URL: https://github.com/apache/kafka/pull/17155#issuecomment-2341701891
@chia7712 I think I found the issue. Here's an example: green PR: https://github.com/apache/kafka/pull/17138 green CI: https://github.com/apache/kafka/actions/runs/10795610027/job/29944644706?pr=17138 red Build Scan: https://ge.apache.org/s/b4qg5njxvfnpe In order to prevent Gradle from caching flaky test results, we force the exit code to be 1 if running in github actions. See https://github.com/apache/kafka/blob/trunk/build.gradle#L526-L531. We ignore this exit code in the GH workflow and rely on "Parse JUnit Tests" to fail or succeed depending on the test failures. The result is that the build scan will show a task failure if _any_ tests failed (even if they succeeded on retry). PR with zero test failures: * CI workflow is green * All modules are cached * Build Scan shows all tasks passing * PR check is green for "JUnit tests" PR with only flaky tests: * CI workflow is green * Only modules with no failures are cached * Build Scan shows ":some-module:test" failed * PR check is green for "JUnit tests" PR with some failed tests: * CI workflow is red * Only modules with no failures are cached * Build Scan shows ":some-module:test" failed * PR check is red for "JUnit tests" I think this is probably as good as we can do for now. Maybe someday we can remove the test retrying. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
