[
https://issues.apache.org/jira/browse/KAFKA-8951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-8951:
---------------------------------
Labels: new-rebalance-should-fix (was: )
> Avoid unnecessary rebalances and downtime for "safe" partitions
> ---------------------------------------------------------------
>
> Key: KAFKA-8951
> URL: https://issues.apache.org/jira/browse/KAFKA-8951
> Project: Kafka
> Issue Type: Improvement
> Components: clients, streams
> Affects Versions: 2.4.0
> Reporter: A. Sophie Blee-Goldman
> Priority: Major
> Labels: new-rebalance-should-fix
>
> With cooperative rebalancing, any partition that is encoded in one consumer's
> Subscription cannot be re-assigned to a different consumer during that
> rebalance. The partition must be removed from the assignment and revoked by
> its old owner before triggering a second rebalance during which it can be
> assigned. This is to enforce a synchronization barrier so that no two
> consumers can ever own the same partition at the same time
> This leads to down time for that partition plus a second rebalance, which may
> not always be necessary. In Streams for example, the consumer will pause all
> partitions of an active task until it is running (ie has been initialized and
> restored). It should be safe to give these partitions away, provided they are
> not resumed between sending the joinGroup request and receiving the syncGroup
> response.
> One proposal would be to modify two methods in the ConsumerPartitionAssignor
> interface. 1) ConsumerPartitionAssignor#subscriptionUserData would be passed
> in the set of `ownedPartitions` that will be included in the subscription,
> allowing it to remove any that it knows are safe to give away.
> 2) ConsumerPartitionAssignor#onAssignment would be passed the set of revoked
> partitions, allowing it to remove any that it knows were already reassigned
> and should not trigger another rebalance.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
