Hi all, As you may know, we've been working hard to graduate KIP-848 from preview in Apache Kafka 3.8 to production-ready in Apache Kafka 4.0 — exciting news!
KIP-848, the next-generation rebalance protocol, encompasses several significant updates in the consumer and group coordinator areas, including: - A completely new rebalance protocol for the consumer. - A redesigned group coordinator. - A deep refactoring of the consumer to introduce a background thread for handling all I/O operations, instead of performing them on the application thread. Note that this refactor is only applied when the new rebalance protocol is enabled — an important detail. Starting with Apache Kafka 4.0: - The new group coordinator will be the only option. - The new rebalance protocol will be enabled by default on the server side. - The consumer will continue to use the old rebalance protocol by default for backward compatibility with brokers older than 4.0. Our initial plan was to make the new rebalance protocol the default in the Consumer in Apache Kafka 4.0. However, after much discussion, we decided not to proceed with this approach — despite the feature being ready. The key concern is that the consumer shipped with Apache Kafka 4.0 would not work by default with brokers older than 4.0, which we believe would result in a poor user experience. Realistically, most deployments won’t upgrade to 4.0 immediately, and we anticipate a gradual adoption curve. Moreover, we feel that introducing such a change without prior notice (e.g. a deprecation warning) in earlier releases isn't ideal. We also considered implementing an automatic fallback mechanism in the consumer. However, the significant differences between the two protocols made this unfeasible. For example: - Static membership does not function identically in both protocols. - Consumer-triggered rebalances are not supported in the new protocol. - Default assignors are different. - Metrics and logs are different. - Some configs are no longer supported. - There are small behavioral differences due to the new internal architecture of the Consumer. Given these differences, we believe users must explicitly choose when to migrate from the old protocol to the new protocol as already planned in the KIP. To aid this process, we are providing a clear online upgrade path. Instead, our revised plan is as follows: - We will officially deprecate the old rebalance protocol in a 4.x release (likely 4.1) and introduce the appropriate deprecation message. - We plan to make the new rebalance protocol the default in Apache Kafka 5.0, giving the community sufficient time to adopt 4.x brokers before the switch. - We will drop the possibility of using the old rebalance protocol from the Consumer in Apache Kafka 6.0. I will formalize this revised plan in a KIP to discuss it further with the community. Regards, David, on behalf of the KIP-848 team P.S.: I started this new thread to ensure this update isn’t unnoticed in the long KIP-848 discussion thread.
