[
https://issues.apache.org/jira/browse/KAFKA-20172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132847#comment-18132847
]
Ramu commented on KAFKA-20172:
------------------------------
## KAFKA-20172 — Implementation Update
**Attached patch:** Online migration between Classic and Streams rebalance
protocols
**Commit:**
[`947ad83e82b404aab5c3d7a21de2382ce162ac97`](https://github.com/ramu11/kafka/commit/947ad83e82b404aab5c3d7a21de2382ce162ac97)
### Implemented in this patch
- **Online migration:** Converts a classic Streams group to a Streams group
when a native Streams-protocol member joins.
- **Protocol bridging:** Translates classic JoinGroup, SyncGroup, and heartbeat
operations while classic and native Streams clients coexist.
- **Reverse migration:** Adds conversion back to a classic Streams group when
native members leave and migrated classic members remain.
- **Task reconciliation:** Preserves classic ownership metadata and translates
Streams assignments for classic clients, including representing warm-up tasks
as standbys.
- **Probing and offset metadata:** Extends subscription metadata to version 12
to carry the probing rebalance interval and known task changelog end-offset
sums. Unknown end offsets are not treated as caught up.
- **Regression coverage:** Adds coordinator, metadata, assignment, and
migration tests for the implemented paths.
### Remaining work
1. **P1 — Probing lifecycle:** Validate repeated probing cycles end to end,
including interval enforcement, warm-up completion, and cases where the
assignment epoch remains unchanged.
2. **P1 — Ownership transfer:** Add cross-member tests proving active tasks are
not transferred until the previous owner confirms revocation.
3. **P1 — Reverse-migration recovery:** Cover last-member departure edge cases,
fresh coordinator replay, append failures, pending revocations, and timer
recovery.
4. **P2/P3 — Compatibility and validation:** Expand static-membership, timeout,
LeaveGroup, mixed-version, and Kafka Streams system-test coverage. Verify that
end-offset telemetry produces the correct assignment decisions, not just that
the metadata is propagated.
### Scope and status
This is an **intermediate implementation checkpoint, not a claim of complete
KIP-1071 compliance**. Broader KIP work—including HA assignment and rack-aware
warm-up support, topology updates, and regex topic subscriptions—is tracked
separately under
[KAFKA-20116](https://issues.apache.org/jira/browse/KAFKA-20116),
[KAFKA-20170](https://issues.apache.org/jira/browse/KAFKA-20170), and
[KAFKA-20171](https://issues.apache.org/jira/browse/KAFKA-20171).
Focused Gradle tests passed at this checkpoint; comprehensive system-test and
recovery validation remains outstanding.
References:
[KIP-1071](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1071%3A+Streams+Rebalance+Protocol)
· [KAFKA-20172](https://issues.apache.org/jira/browse/KAFKA-20172)
> Support online migration between classic and streams rebalance protocols
> ------------------------------------------------------------------------
>
> Key: KAFKA-20172
> URL: https://issues.apache.org/jira/browse/KAFKA-20172
> Project: Kafka
> Issue Type: Task
> Components: group-coordinator, streams
> Reporter: Lucas Brutschy
> Priority: Major
> Labels: kip1071
> Attachments: KAFKA-20172-online-migration-947ad83-2026-10-27.zip
>
>
> The streams rebalance protocol \(KIP\-1071\) currently only supports offline
> migration between the classic and streams protocols. Users must shut down all
> application instances, wait for the session timeout to expire, update the
> configuration, and then restart. This requires a maintenance window and
> causes downtime for the streams application.
> As specified in KIP\-1071, online migration should allow converting classic
> consumer groups to streams groups during rolling client upgrades. When the
> first streams client using the new protocol joins the group, the group should
> be converted from a classic group to a streams group. Classic clients should
> receive REBALANCE\_IN\_PROGRESS errors triggering JoinGroup/SyncGroup calls
> while new protocol clients use StreamsGroupHeartbeat. The broker should
> internally translate between protocols during the migration period. The group
> should revert to classic when the last new\-protocol member leaves. This
> enables zero\-downtime migration between protocols.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
