[
https://issues.apache.org/jira/browse/CAMEL-22811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen updated CAMEL-22811:
--------------------------------
Fix Version/s: 4.x
> camel-kafka: Support Kafka 4.0 KIP-848 consumer rebalance protocol
> (group.protocol=consumer)
> --------------------------------------------------------------------------------------------
>
> Key: CAMEL-22811
> URL: https://issues.apache.org/jira/browse/CAMEL-22811
> Project: Camel
> Issue Type: New Feature
> Components: camel-kafka
> Reporter: SRIKANTH PRATHIPATI
> Priority: Minor
> Fix For: 4.x
>
>
> h2. Background
> Apache Kafka 4.0 introduces the “Next Generation of the Consumer Rebalance
> Protocol” (KIP-848), which is GA and designed to be fully incremental (no
> global sync barrier), improving rebalance times and scalability.
> With the new protocol, some behavior and configuration moves to the broker:
> heartbeat interval, session timeout, and the assignment strategy are
> controlled server-side for {{Consumer}} groups.
> h2. Problem statement (Camel impact)
> KIP-848 expects the broker to manage {{{}partitionAssignor{}}},
> {{{}heartbeatIntervalMs{}}}, and {{{}sessionTimeoutMs{}}}, while the current
> Camel Kafka component “defaulted these if these are not supplied.”
> Kafka’s own documentation for the new protocol states that
> {{{}heartbeat.interval.ms{}}}, {{{}session.timeout.ms{}}}, and
> {{partition.assignment.strategy}} are “no longer usable” when
> {{group.protocol=consumer}} is enabled.
> Please refer to the related thread - [#camel > Kafka | Next Generation of the
> Consumer Rebalance
> Protocol|https://camel.zulipchat.com/#narrow/channel/257298-camel/topic/Kafka.20.7C.20Next.20Generation.20of.20the.20Consumer.20Rebalance.20Protocol/with/541949206]
> h2. Requested change (what to implement)
> * Add/confirm first-class support for enabling the new protocol via consumer
> property {{group.protocol=consumer}} (pass-through supported), and ensure
> Camel does not inject defaults that conflict with the new protocol.
> * When {{{}group.protocol=consumer{}}}, avoid setting (or warn/error if user
> sets) classic-only properties such as {{{}heartbeat.interval.ms{}}},
> {{{}session.timeout.ms{}}}, and {{partition.assignment.strategy}} (and
> Camel-level equivalents like {{{}heartbeatIntervalMs{}}},
> {{{}sessionTimeoutMs{}}}, {{partitionAssignor}} defaults).
> * Optionally expose/document {{group.remote.assignor}} for selecting a
> server-side assignor (override broker default) when using the consumer
> protocol.
> h2. Acceptance criteria (suggested)
> * With Kafka client 4.0+ and {{{}group.protocol=consumer{}}}, Camel consumer
> starts successfully without requiring {{session.timeout.ms}} /
> {{heartbeat.interval.ms}} / {{partition.assignment.strategy}} and without
> Camel adding defaults for them.
> * Add an integration test matrix (at least one) that runs Camel Kafka
> consumer with {{group.protocol=consumer}} against a Kafka 4.0+ broker.
> * Documentation update: clearly describe “Classic vs Consumer” protocol
> selection, and call out unsupported configs under the consumer protocol.
> [https://www.confluent.io/blog/kip-848-consumer-rebalance-protocol/]
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
