[jira] [Commented] (KAFKA-19424) FindCoordinator inconsistent across brokers when __consumer_offsets partition count is increased

2026-01-07 Thread sanghyeok An (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-19424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18050374#comment-18050374
 ] 

sanghyeok An commented on KAFKA-19424:
--

Hi [~squah-confluent] !

During my investigation of both __consumer_offsets and  {{__transaction_state}} 
alongside this work, 
I realized that this issue extends to all internal topics. 
Also, the documentation explicitly discourages modifying the partition count 
for internal topics post-deployment.

To improve this behavior, I have prepared a KIP. 
(https://cwiki.apache.org/confluence/display/KAFKA/KIP-1268%3A+Prevent+Dynamic+Partition+Expansion+of+Internal+Topics)
I would greatly appreciate your insights if you have the time to review it. 

Thanks for your time! 

> FindCoordinator inconsistent across brokers when __consumer_offsets partition 
> count is increased
> 
>
> Key: KAFKA-19424
> URL: https://issues.apache.org/jira/browse/KAFKA-19424
> Project: Kafka
>  Issue Type: Bug
>  Components: group-coordinator
>Reporter: Sean Quah
>Assignee: sanghyeok An
>Priority: Major
>
> {{GroupCoordinatorService}} captures the number of partitions of 
> {{\_\_consumer_offsets}} at startup. This is used to map group ids to 
> {{\_\_consumer_offsets}} partitions in 
> {{GroupCoordinatorService.partitionFor}}.
> When adding partitions to {{\_\_consumer_offsets}}, 
> {{GroupCoordinatorService}} doesn't update its cached partition count. This 
> means that newly started brokers will map group ids to partitions differently 
> to existing brokers and FindCoordinator requests can return different results 
> depending on which broker the client asks.
> It's proposed to make brokers consistent by updating the cached partition 
> count in {{GroupMetadataService.onNewMetadataImage}}.
> We should additionally log a warning when the partition count changes, since 
> adding partitions to {{\_\_consumer_offsets}} is not truly supported, as any 
> existing group info will remain on the old partition and become un-findable. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (KAFKA-19424) FindCoordinator inconsistent across brokers when __consumer_offsets partition count is increased

2025-11-10 Thread sanghyeok An (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-19424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18037285#comment-18037285
 ] 

sanghyeok An commented on KAFKA-19424:
--

[~squah-confluent] Thanks a lot! 

Self assigned! 

> FindCoordinator inconsistent across brokers when __consumer_offsets partition 
> count is increased
> 
>
> Key: KAFKA-19424
> URL: https://issues.apache.org/jira/browse/KAFKA-19424
> Project: Kafka
>  Issue Type: Bug
>  Components: group-coordinator
>Reporter: Sean Quah
>Priority: Major
>
> {{GroupCoordinatorService}} captures the number of partitions of 
> {{\_\_consumer_offsets}} at startup. This is used to map group ids to 
> {{\_\_consumer_offsets}} partitions in 
> {{GroupCoordinatorService.partitionFor}}.
> When adding partitions to {{\_\_consumer_offsets}}, 
> {{GroupCoordinatorService}} doesn't update its cached partition count. This 
> means that newly started brokers will map group ids to partitions differently 
> to existing brokers and FindCoordinator requests can return different results 
> depending on which broker the client asks.
> It's proposed to make brokers consistent by updating the cached partition 
> count in {{GroupMetadataService.onNewMetadataImage}}.
> We should additionally log a warning when the partition count changes, since 
> adding partitions to {{\_\_consumer_offsets}} is not truly supported, as any 
> existing group info will remain on the old partition and become un-findable. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (KAFKA-19424) FindCoordinator inconsistent across brokers when __consumer_offsets partition count is increased

2025-11-10 Thread Sean Quah (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-19424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18037263#comment-18037263
 ] 

Sean Quah commented on KAFKA-19424:
---

Hi, [~chickenchickenlove]. Sure, please feel free to reassign the task to 
yourself. I'm not currently working on it.

> FindCoordinator inconsistent across brokers when __consumer_offsets partition 
> count is increased
> 
>
> Key: KAFKA-19424
> URL: https://issues.apache.org/jira/browse/KAFKA-19424
> Project: Kafka
>  Issue Type: Bug
>  Components: group-coordinator
>Reporter: Sean Quah
>Priority: Major
>
> {{GroupCoordinatorService}} captures the number of partitions of 
> {{\_\_consumer_offsets}} at startup. This is used to map group ids to 
> {{\_\_consumer_offsets}} partitions in 
> {{GroupCoordinatorService.partitionFor}}.
> When adding partitions to {{\_\_consumer_offsets}}, 
> {{GroupCoordinatorService}} doesn't update its cached partition count. This 
> means that newly started brokers will map group ids to partitions differently 
> to existing brokers and FindCoordinator requests can return different results 
> depending on which broker the client asks.
> It's proposed to make brokers consistent by updating the cached partition 
> count in {{GroupMetadataService.onNewMetadataImage}}.
> We should additionally log a warning when the partition count changes, since 
> adding partitions to {{\_\_consumer_offsets}} is not truly supported, as any 
> existing group info will remain on the old partition and become un-findable. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (KAFKA-19424) FindCoordinator inconsistent across brokers when __consumer_offsets partition count is increased

2025-11-10 Thread sanghyeok An (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-19424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18036748#comment-18036748
 ] 

sanghyeok An commented on KAFKA-19424:
--

Hi, [~squah-confluent] .

If you are not working in this task, May I take a look?

> FindCoordinator inconsistent across brokers when __consumer_offsets partition 
> count is increased
> 
>
> Key: KAFKA-19424
> URL: https://issues.apache.org/jira/browse/KAFKA-19424
> Project: Kafka
>  Issue Type: Bug
>  Components: group-coordinator
>Reporter: Sean Quah
>Priority: Major
>
> {{GroupCoordinatorService}} captures the number of partitions of 
> {{\_\_consumer_offsets}} at startup. This is used to map group ids to 
> {{\_\_consumer_offsets}} partitions in 
> {{GroupCoordinatorService.partitionFor}}.
> When adding partitions to {{\_\_consumer_offsets}}, 
> {{GroupCoordinatorService}} doesn't update its cached partition count. This 
> means that newly started brokers will map group ids to partitions differently 
> to existing brokers and FindCoordinator requests can return different results 
> depending on which broker the client asks.
> It's proposed to make brokers consistent by updating the cached partition 
> count in {{GroupMetadataService.onNewMetadataImage}}.
> We should additionally log a warning when the partition count changes, since 
> adding partitions to {{\_\_consumer_offsets}} is not truly supported, as any 
> existing group info will remain on the old partition and become un-findable. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)