kfaraz opened a new pull request, #20783: URL: https://github.com/apache/druid/pull/20783
## Description `KafkaConsumerMonitor` recently started reporting a running average of the Kafka `poll-idle-ratio-avg` metric for use by `CostBasedAutoScaler`. Before any sample has been collected for a task, the running average defaulted to `1.0`, which is indistinguishable from a consumer that is "always idle". This default value gets included in the cross-task average computed by the supervisor's autoscaler, which can pollute scaling decisions for tasks that simply haven't reported a sample yet (e.g. right after startup). This PR: - Changes `KafkaConsumerMonitor.RunningAverage`'s default/unknown sentinel from `1.0` to `-1.0`, consistent with the `RecordSupplier#getPollIdleRatioMetric()` / `getRunningAvgPollIdleRatio()` contract, which already documents negative values as "no valid metric available". - Updates `SeekableStreamIndexTaskRunner#doGetRowStats()` to only publish the `pollIdleRatio` / `pollIdleRatioRunningAvg` autoscaler metrics once the record supplier reports a valid (non-negative) value, instead of always publishing a possibly-unknown value. - `CostBasedAutoScaler` already filters out negative values when averaging across tasks and treats an overall negative idle ratio as "unknown" in its cost function, so no changes were needed there. ## Release note Fixed a bug where a Kafka indexing task would report a poll-idle-ratio of `1.0` (i.e. fully idle) before any sample was actually available, which could skew the supervisor autoscaler's averaged idle ratio across tasks. ## Test plan - [x] Added `testDoGetRowStatsOmitsPollIdleRatioUntilKnown` in `SeekableStreamIndexTaskRunnerTest`, verifying that `pollIdleRatio`/`pollIdleRatioRunningAvg` are omitted from `doGetRowStats()` until the record supplier reports a known (non-negative) value, and are published once available. - [x] Existing `SeekableStreamIndexTaskRunnerTest` (22 tests) and `CostBasedAutoScalerTest` (19 tests) pass unchanged. - [x] `kafka-indexing-service` module builds successfully with the updated `KafkaConsumerMonitor`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
