[
https://issues.apache.org/jira/browse/KAFKA-20172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132848#comment-18132848
]
Ramu commented on KAFKA-20172:
------------------------------
h2. KAFKA-20172 - Implementation Update
*Commit:*
[947ad83e82b404aab5c3d7a21de2382ce162ac97|https://github.com/ramu11/kafka/commit/947ad83e82b404aab5c3d7a21de2382ce162ac97]
h3. Implemented
* Online migration from the classic Streams protocol to the Streams rebalance
protocol when a native member joins.
* Classic protocol bridging for JoinGroup, SyncGroup and heartbeat operations
while both client types coexist.
* Reverse migration to the classic Streams protocol when native members leave
and migrated classic members remain.
* Task ownership and assignment translation, including representing warm-up
tasks as standbys for classic clients.
* Subscription metadata version 12 support for probing rebalance intervals and
known task changelog end-offset sums.
* Handling of unknown end offsets without treating tasks as caught up.
* Regression tests for protocol bridging, probing metadata, assignment
translation and migration scenarios.
h3. Pending Work
*P1 - Probing lifecycle:* Validate repeated probing cycles end to end,
including configured interval enforcement, warm-up completion and unchanged
assignment epochs.
*P1 - Task ownership transfer:* Test cross-member revocation and ensure active
tasks are transferred only after the previous owner confirms revocation.
*P1 - Reverse migration and recovery:* Test last-member departure, fresh
coordinator replay, append failures, pending revocations and timer recovery.
*P2/P3 - Compatibility and broader validation:* Expand static membership,
LeaveGroup, timeout, mixed-version and Streams system-test coverage. Verify
end-offset telemetry influences assignment decisions correctly.
h3. Status and Scope
This patch represents an intermediate implementation checkpoint and does not
claim complete KIP-1071 compliance. Broader work on HA assignment and
rack-aware warm-up support, topology updates and regex topic subscriptions
remains tracked separately.
Focused Gradle tests were reported passing at this checkpoint. Comprehensive
recovery and system-test validation remains outstanding.
*References:*
[KIP-1071|https://cwiki.apache.org/confluence/display/KAFKA/KIP-1071%3A+Streams+Rebalance+Protocol]
[KAFKA-20172|https://issues.apache.org/jira/browse/KAFKA-20172]
[KAFKA-20116|https://issues.apache.org/jira/browse/KAFKA-20116]
[KAFKA-20170|https://issues.apache.org/jira/browse/KAFKA-20170]
[KAFKA-20171|https://issues.apache.org/jira/browse/KAFKA-20171]
> Support online migration between classic and streams rebalance protocols
> ------------------------------------------------------------------------
>
> Key: KAFKA-20172
> URL: https://issues.apache.org/jira/browse/KAFKA-20172
> Project: Kafka
> Issue Type: Task
> Components: group-coordinator, streams
> Reporter: Lucas Brutschy
> Priority: Major
> Labels: kip1071
> Attachments: KAFKA-20172-online-migration-947ad83-2026-10-27.zip
>
>
> The streams rebalance protocol \(KIP\-1071\) currently only supports offline
> migration between the classic and streams protocols. Users must shut down all
> application instances, wait for the session timeout to expire, update the
> configuration, and then restart. This requires a maintenance window and
> causes downtime for the streams application.
> As specified in KIP\-1071, online migration should allow converting classic
> consumer groups to streams groups during rolling client upgrades. When the
> first streams client using the new protocol joins the group, the group should
> be converted from a classic group to a streams group. Classic clients should
> receive REBALANCE\_IN\_PROGRESS errors triggering JoinGroup/SyncGroup calls
> while new protocol clients use StreamsGroupHeartbeat. The broker should
> internally translate between protocols during the migration period. The group
> should revert to classic when the last new\-protocol member leaves. This
> enables zero\-downtime migration between protocols.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
