SRIKANTH PRATHIPATI created CAMEL-22811:
-------------------------------------------
Summary: camel-kafka: Support Kafka 4.0 KIP-848 consumer rebalance
protocol (group.protocol=consumer)
Key: CAMEL-22811
URL: https://issues.apache.org/jira/browse/CAMEL-22811
Project: Camel
Issue Type: New Feature
Components: camel-kafka
Reporter: SRIKANTH PRATHIPATI
h2. Background
Apache Kafka 4.0 introduces the “Next Generation of the Consumer Rebalance
Protocol” (KIP-848), which is GA and designed to be fully incremental (no
global sync barrier), improving rebalance times and scalability.
With the new protocol, some behavior and configuration moves to the broker:
heartbeat interval, session timeout, and the assignment strategy are controlled
server-side for {{Consumer}} groups.
h2. Problem statement (Camel impact)
KIP-848 expects the broker to manage {{{}partitionAssignor{}}},
{{{}heartbeatIntervalMs{}}}, and {{{}sessionTimeoutMs{}}}, while the current
Camel Kafka component “defaulted these if these are not supplied.”
Kafka’s own documentation for the new protocol states that
{{{}heartbeat.interval.ms{}}}, {{{}session.timeout.ms{}}}, and
{{partition.assignment.strategy}} are “no longer usable” when
{{group.protocol=consumer}} is enabled.
Please refer to the related thread - [#camel > Kafka | Next Generation of the
Consumer Rebalance
Protocol|https://camel.zulipchat.com/#narrow/channel/257298-camel/topic/Kafka.20.7C.20Next.20Generation.20of.20the.20Consumer.20Rebalance.20Protocol/with/541949206]
h2. Requested change (what to implement)
* Add/confirm first-class support for enabling the new protocol via consumer
property {{group.protocol=consumer}} (pass-through supported), and ensure Camel
does not inject defaults that conflict with the new protocol.
* When {{{}group.protocol=consumer{}}}, avoid setting (or warn/error if user
sets) classic-only properties such as {{{}heartbeat.interval.ms{}}},
{{{}session.timeout.ms{}}}, and {{partition.assignment.strategy}} (and
Camel-level equivalents like {{{}heartbeatIntervalMs{}}},
{{{}sessionTimeoutMs{}}}, {{partitionAssignor}} defaults).
* Optionally expose/document {{group.remote.assignor}} for selecting a
server-side assignor (override broker default) when using the consumer protocol.
h2. Acceptance criteria (suggested)
* With Kafka client 4.0+ and {{{}group.protocol=consumer{}}}, Camel consumer
starts successfully without requiring {{session.timeout.ms}} /
{{heartbeat.interval.ms}} / {{partition.assignment.strategy}} and without Camel
adding defaults for them.
* Add an integration test matrix (at least one) that runs Camel Kafka consumer
with {{group.protocol=consumer}} against a Kafka 4.0+ broker.
* Documentation update: clearly describe “Classic vs Consumer” protocol
selection, and call out unsupported configs under the consumer protocol.
[kip-848|[https://www.confluent.io/blog/kip-848-consumer-rebalance-protocol/]]
[Kafka4.0-DocLink|https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
