[
https://issues.apache.org/jira/browse/KAFKA-20172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132566#comment-18132566
]
Ramu commented on KAFKA-20172:
------------------------------
Hi Matthias,
Thank you [~mjsax] for sharing your concerns. I understand that this is a
complex change spanning the classic and new rebalance protocols, the group
coordinator, and Kafka Streams, and I agree that the design and compatibility
details need to be handled carefully.
For context, I have been working at Red Hat since 2015, including on Red Hat
Streams for Apache Kafka. I believe I have the relevant engineering experience
and am prepared to invest the time needed to understand the existing
implementation, work through the edge cases, and take this work to completion.
I am following the agreed design in [KIP-1071: Streams Rebalance
Protocol](https://cwiki.apache.org/confluence/spaces/KAFKA/pages/311627992/KIP-1071+Streams+Rebalance+Protocol),
particularly its **Online Migration from a Classic Consumer Group to a Streams
Group** section, building on the group-coordinator model introduced by
[KIP-848](https://cwiki.apache.org/confluence/spaces/KAFKA/pages/217387038/KIP-848+The+Next+Generation+of+the+Consumer+Rebalance+Protocol).
So far, I have developed a work-in-progress implementation covering:
- Conversion of an existing classic Streams group when the first
Streams-protocol member joins, including translation of legacy member metadata
into the Streams group representation.
- A compatibility bridge for classic JoinGroup, SyncGroup, and Heartbeat
requests while classic and Streams-protocol clients coexist.
- Translation of broker-computed task assignments into the legacy consumer
assignment format, including active, standby, and warm-up task handling.
- Unit and integration tests covering metadata conversion, assignment
serialization, mixed-protocol operation, member validation, and migration
scenarios.
The work is available in my [current commit on
GitHub](https://github.com/ramu11/kafka/commit/bd58d3afaa83aa127b18ee3bac7eee66747c92dd).
The full group-coordinator test suite passes after the latest regression fix.
The migration integration tests also passed before that final fix, and I still
need to rerun them, along with the client-side assignment decoder test, against
the latest code.
I appreciate your concern about complex contributions being left unfinished.
That is not my intention. I would prefer to proceed incrementally, validate the
design with the community, and address feedback early rather than develop a
large change without sufficient review.
Could you please let me know whether you see any specific part of the current
approach that diverges from KIP-1071 or the intended group-coordinator design?
If there are architectural concerns or assumptions I should revisit before
continuing, I would genuinely appreciate your guidance. I am happy to adjust
the approach based on that feedback.
I respect your recommendation, but I would appreciate the opportunity to
continue working on this ticket and demonstrate that I can carry it through
responsibly.
Thank you for your time.
> Support online migration between classic and streams rebalance protocols
> ------------------------------------------------------------------------
>
> Key: KAFKA-20172
> URL: https://issues.apache.org/jira/browse/KAFKA-20172
> Project: Kafka
> Issue Type: Task
> Components: group-coordinator, streams
> Reporter: Lucas Brutschy
> Assignee: Ramu
> Priority: Major
> Labels: kip1071
>
> The streams rebalance protocol \(KIP\-1071\) currently only supports offline
> migration between the classic and streams protocols. Users must shut down all
> application instances, wait for the session timeout to expire, update the
> configuration, and then restart. This requires a maintenance window and
> causes downtime for the streams application.
> As specified in KIP\-1071, online migration should allow converting classic
> consumer groups to streams groups during rolling client upgrades. When the
> first streams client using the new protocol joins the group, the group should
> be converted from a classic group to a streams group. Classic clients should
> receive REBALANCE\_IN\_PROGRESS errors triggering JoinGroup/SyncGroup calls
> while new protocol clients use StreamsGroupHeartbeat. The broker should
> internally translate between protocols during the migration period. The group
> should revert to classic when the last new\-protocol member leaves. This
> enables zero\-downtime migration between protocols.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
