[
https://issues.apache.org/jira/browse/KAFKA-19431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Jacot resolved KAFKA-19431.
---------------------------------
Fix Version/s: 4.2.0
Resolution: Fixed
> Stronger assignment consistency with subscription for consumer groups
> ----------------------------------------------------------------------
>
> Key: KAFKA-19431
> URL: https://issues.apache.org/jira/browse/KAFKA-19431
> Project: Kafka
> Issue Type: Improvement
> Components: group-coordinator
> Reporter: Sean Quah
> Assignee: Sean Quah
> Priority: Major
> Fix For: 4.2.0
>
>
> Currently, consumer group assignments are eventually consistent with
> subscriptions: when a member has unrevoked partitions, it is not allowed to
> reconcile with the latest target assignment. If a member with unrevoked
> partitions shrinks its subscription, it may observe assignments from the
> broker containing topics it is no longer subscribed to.
> If we wanted to, we could tighten this up at the cost of extra CPU time. Note
> that it's not feasible to close the gap for regex subscriptions, since there
> will always be a window when the regex is not yet resolved and we cannot tell
> whether a topic is part of the subscription.
> One way to do this would be to update {{CurrentAssignmentBuilder}} and
> * Add a {{MetadataImage}} and map of resolved regexes
> * Define the set of subscribed topic uuids as the union of the topic name
> subscription and resolved regex topic names, like how
> {{TargetAssignmentBuilder}} does it.
> ** When the regex is unresolved, we can’t know which topics are part of the
> subscription. We treat unresolved regexes as matching no topics, to be
> conservative. This way, the assignment is always consistent with the
> subscription.
> * Update the loop over topics in {{computeNextAssignment}} to treat the
> assigned partitions as an empty set when the topic is not part of the
> subscription.
> * Do not advance the member epoch past the target assignment epoch when
> exiting the {{UNREVOKED_PARTITIONS}} state.
> * Define an {{updateCurrentAssignment}} method that drops any unsubscribed
> topics from the member’s current assignment and transitions the member to
> {{UNREVOKED_PARTITIONS}} if any topics were dropped.
> * Use {{updateCurrentAssignment}} on the other {{UNREVOKED_PARTITIONS}} path.
> Additionally, if we ever end up with asynchronous assignors (such as
> client-computed ones),
> * We add a new flag to {{maybeReconcile}} called
> {{{}hasSubscriptionChanged{}}}. When the flag is set, we run the
> {{CurrentAssignmentBuilder}} even when reconciled to the target assignment,
> since the target assignment can lag behind the group epoch.
> * Use {{updateCurrentAssignment}} on all {{CurrentAssignmentBuilder}} paths
> that do not use {{{}computeNextAssignment{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
