GWphua opened a new pull request, #18234:
URL: https://github.com/apache/druid/pull/18234
### Description
When logging a mismatch in kafka and task partitions, the cardinality count
of kafka partitions in the Set may be very high.
This may result in the logs appearing very cluttered, especially when we are
logging the entire `KafkaTopicPartition` object now, instead of integer.
Here's an example of a very long, and hard to understand logs, when Druid is
still logging the partition as an integer, in here, the partition 38 is
missing, but is not obvious from the logs:
`2025-02-27T22:56:15,406 WARN [KafkaSupervisor-XX-0]
org.apache.druid.indexing.kafka.supervisor.KafkaSupervisor - Lag metric: Kafka
partitions [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38,
39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58,
59, 60, 61, 62] do not match task partitions [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30,
31, 32, 33, 34, 35, 36, 37, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51,
52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62]`
Each kafka partition is changed from integer to a KafkaTopicPartition. e.g.
We will see logs outputtingKafkaTopicPartition{partition=1, topic='null',
multiTopicPartition=false} for each partition, increasing the log verbosity and
clutter.
I changed the logging logic to log the diff, instead of the entire task and
kafka partition set. Updated logs can be seen below:
`2025-07-11T07:14:05,807 WARN [Supervisor-XX-0]
org.apache.druid.indexing.kafka.supervisor.KafkaSupervisor - Mismatched kafka
and task partitions: Missing Task Partitions [38], Missing Kafka Partitions []`
#### Release note
<!-- Give your best effort to summarize your changes in a couple of
sentences aimed toward Druid users.
If your change doesn't have end user impact, you can skip this section.
For tips about how to write a good release note, see [Release
notes](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#release-notes).
-->
Reduce clutter of output logs when there is a mismatch in Kafka and Task
partition set.
<hr>
##### Key changed/added classes in this PR
* `KafkaSupervisor`
<hr>
<!-- Check the items by putting "x" in the brackets for the done things. Not
all of these items apply to every PR. Remove the items which are not done or
not relevant to the PR. None of the items from the checklist below are strictly
necessary, but it would be very helpful if you at least self-review the PR. -->
This PR has:
- [x] been self-reviewed.
- [x] a release note entry in the PR description.
- [x] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
