[
https://issues.apache.org/jira/browse/KAFKA-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17906819#comment-17906819
]
Santhosh C T commented on KAFKA-18238:
--------------------------------------
Running this for every commit seems counter intuitive since we have the results
for all the commits already for that test.
I think we need more systematic approach as [~davidarthur] was mentioning. We
need to establish a solid baseline and then find the candidate for true first
commit through heuristics.
*Example:*
# *Fetch Test Results:*
** Fetch test results from Develocity APIs for commits {{C1}} through
{{{}C1000{}}}.
** Record each commit’s test result: pass, fail, or flaky.
# *Identify a Known Stable Point:*
** Suppose for commits {{C1}} through {{{}C500{}}}, the test always passed.
This is our stable baseline.
# *Check Subsequent Commits:*
** From {{C501}} onwards, look for the first appearance of flaky behavior.
** Say at {{C620}} you first see a single instance of flakiness, but then
{{C621}} and {{C622}} pass without issue. This might be noise—don’t jump to
conclusions yet.
** At {{C630}} through {{{}C640{}}}, you start seeing a pattern: 4 out of 10
runs are flaky. This looks like sustained flakiness.
# *Pinpoint the Transition:*
** Look backward from {{C630}} to find the first commit that, once introduced,
never yielded a stable streak of test runs afterward.
** If at {{C615}} everything was still stable, but starting at {{C620}} you
see intermittent issues that get worse over time (and never return to perfect
stability), {{C620}} is a strong candidate for the "true first commit" of
flakiness.
> Identify the first commit where a test became flaky
> ---------------------------------------------------
>
> Key: KAFKA-18238
> URL: https://issues.apache.org/jira/browse/KAFKA-18238
> Project: Kafka
> Issue Type: Sub-task
> Reporter: David Arthur
> Assignee: David Arthur
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
