FrankChen021 opened a new pull request, #20311: URL: https://github.com/apache/druid/pull/20311
### Description Fixes the recurring flake in `KafkaBoundedSupervisorTest` (previously #19542, #19543, #20124) by giving the bounded supervisors in that class a 60s `taskDuration` instead of the 500ms inherited from the shared Kafka fixture in `MoreResources.Supervisor.KAFKA_JSON`. #### What the failure looks like `test_boundedSupervisor_withMismatchedMetadata_is_unhealthy` times out waiting for 200 published rows, e.g. [job 102719409989](https://github.com/apache/druid/actions/runs/34428720842/job/102719409989): ``` ISE: Timed out waiting for event after [120,000]ms at StreamIndexTestBase.waitUntilPublishedRecordsAreIngested(StreamIndexTestBase.java:186) at KafkaBoundedSupervisorTest.test_boundedSupervisor_withMismatchedMetadata_is_unhealthy(KafkaBoundedSupervisorTest.java:221) ``` #### Root cause (from the captured test output of that job) 1. The bounded task for partition 1 had consumed 96 of its 100 records when the supervisor's task-duration check fired: `Stopping taskGroup[1] as it has already run for duration[PT0.898S], configured task duration[PT0.500S].` The task was paused and its end offset set to 96. Partition 0 finished its 100 records within the cycle and was fine. 2. The supervisor created a new bounded task for offsets 96 to 100. Kafka consumer startup takes longer than 500ms, so on the next cycle the new task had read nothing, was rolled over with `Checkpoint[96] is same as the start sequences`, published zero segments, and the cycle repeated: 116 rollovers and 115 zero-row tasks in 120 seconds, never reaching offset 100. 3. In passing runs both tasks finish their 100 records inside the first cycle, so no rollover happens. The test therefore fails whenever the first read takes longer than about one supervisor cycle, which is a matter of runner timing. The earlier fixes (#19543, #20124) raised the wait to 120s on the theory of a slow cold start. That cannot help here because no progress is made once the rollover loop starts. #### Why 500ms is in the fixture Unbounded Kafka tasks only publish when `taskDuration` elapses, so the shared fixture uses 500ms to make ingested rows queryable within seconds. Bounded tasks publish on their own at their end offset and gain nothing from a short duration, so this change overrides it only in `createBoundedKafkaSupervisor` and leaves the fixture untouched for the unbounded tests. #### Possible follow-up (not in this PR) The supervisor rolling over a bounded task that has made no progress is a product behavior that can livelock a bounded supervisor with a short `taskDuration`. A guard against rolling over a bounded task at its start offset would fix that for users too. <hr> This PR has: - [x] been self-reviewed. - [ ] added documentation for new or modified features or behaviors. - [ ] a release note entry in the PR description. - [x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md) - [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met. - [ ] added integration tests. - [x] been tested in a test Druid cluster. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
