[
https://issues.apache.org/jira/browse/KAFKA-21111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18116312#comment-18116312
]
Lucas Brutschy commented on KAFKA-21111:
----------------------------------------
If we treat this conservatively after failover, wouldn't that mean that all
task movements are granted after failover, causing unavailability, even if all
members are on v1+?
> Ensure refiner backward compatibility
> -------------------------------------
>
> Key: KAFKA-21111
> URL: https://issues.apache.org/jira/browse/KAFKA-21111
> Project: Kafka
> Issue Type: Sub-task
> Components: group-coordinator, streams
> Reporter: Matthias J. Sax
> Assignee: Matthias J. Sax
> Priority: Critical
>
> We need to ensure that the refiner is only executed for new clients, RPC
> version 1+, but not for old RPC version 0 clients (4.2/4.3 release).
> Older clients don't report task-offsets, and thus we cannot use warmup tasks
> because w/o the ability to observe their restore progress, we could never
> promote them as active later.
> Note one corner case: after a GC fail-over, a member might not have send a
> heartbeat yet, and thus we don't know it's RPC version yet. Atm, the code
> treats this incorrectly, and would optimistically assume that it's a v1
> client and that the refiner can run. This must be change to being
> conservative and treating "unknown" as don't run the refiner.
> The high level idea is, to add new transient field to `StreamsGroup` to track
> the RPC version. The tricky part is, that the refiner is computing an
> intermediate assignment for the whole group, ie, if might be invoked on a v1
> heartbeat, that there could still be v0 members in the group. So we could
> give the individual member's RPC version into the refiner, and the refiner
> would avoid to put a warmup on older client. – A more conservative (maybe
> simpler?) approach would be, to only invoke the refiner at all, if _all_
> member in the group are on v1+.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
