[
https://issues.apache.org/jira/browse/CAMEL-22811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
SRIKANTH PRATHIPATI updated CAMEL-22811:
----------------------------------------
Description:
h2. Background
Apache Kafka 4.0 introduces the “Next Generation of the Consumer Rebalance
Protocol” (KIP-848), which is GA and designed to be fully incremental (no
global sync barrier), improving rebalance times and scalability.
With the new protocol, some behavior and configuration moves to the broker:
heartbeat interval, session timeout, and the assignment strategy are controlled
server-side for {{Consumer}} groups.
h2. Problem statement (Camel impact)
KIP-848 expects the broker to manage {{{}partitionAssignor{}}},
{{{}heartbeatIntervalMs{}}}, and {{{}sessionTimeoutMs{}}}, while the current
Camel Kafka component “defaulted these if these are not supplied.”
Kafka’s own documentation for the new protocol states that
{{{}heartbeat.interval.ms{}}}, {{{}session.timeout.ms{}}}, and
{{partition.assignment.strategy}} are “no longer usable” when
{{group.protocol=consumer}} is enabled.
Please refer to the related thread - [#camel > Kafka | Next Generation of the
Consumer Rebalance
Protocol|https://camel.zulipchat.com/#narrow/channel/257298-camel/topic/Kafka.20.7C.20Next.20Generation.20of.20the.20Consumer.20Rebalance.20Protocol/with/541949206]
h2. Requested change (what to implement)
* Add/confirm first-class support for enabling the new protocol via consumer
property {{group.protocol=consumer}} (pass-through supported), and ensure Camel
does not inject defaults that conflict with the new protocol.
* When {{{}group.protocol=consumer{}}}, avoid setting (or warn/error if user
sets) classic-only properties such as {{{}heartbeat.interval.ms{}}},
{{{}session.timeout.ms{}}}, and {{partition.assignment.strategy}} (and
Camel-level equivalents like {{{}heartbeatIntervalMs{}}},
{{{}sessionTimeoutMs{}}}, {{partitionAssignor}} defaults).
* Optionally expose/document {{group.remote.assignor}} for selecting a
server-side assignor (override broker default) when using the consumer protocol.
h2. Acceptance criteria (suggested)
* With Kafka client 4.0+ and {{{}group.protocol=consumer{}}}, Camel consumer
starts successfully without requiring {{session.timeout.ms}} /
{{heartbeat.interval.ms}} / {{partition.assignment.strategy}} and without Camel
adding defaults for them.
* Add an integration test matrix (at least one) that runs Camel Kafka consumer
with {{group.protocol=consumer}} against a Kafka 4.0+ broker.
* Documentation update: clearly describe “Classic vs Consumer” protocol
selection, and call out unsupported configs under the consumer protocol.
[kip-848|[https://www.confluent.io/blog/kip-848-consumer-rebalance-protocol/]]
was:
h2. Background
Apache Kafka 4.0 introduces the “Next Generation of the Consumer Rebalance
Protocol” (KIP-848), which is GA and designed to be fully incremental (no
global sync barrier), improving rebalance times and scalability.
With the new protocol, some behavior and configuration moves to the broker:
heartbeat interval, session timeout, and the assignment strategy are controlled
server-side for {{Consumer}} groups.
h2. Problem statement (Camel impact)
KIP-848 expects the broker to manage {{{}partitionAssignor{}}},
{{{}heartbeatIntervalMs{}}}, and {{{}sessionTimeoutMs{}}}, while the current
Camel Kafka component “defaulted these if these are not supplied.”
Kafka’s own documentation for the new protocol states that
{{{}heartbeat.interval.ms{}}}, {{{}session.timeout.ms{}}}, and
{{partition.assignment.strategy}} are “no longer usable” when
{{group.protocol=consumer}} is enabled.
Please refer to the related thread - [#camel > Kafka | Next Generation of the
Consumer Rebalance
Protocol|https://camel.zulipchat.com/#narrow/channel/257298-camel/topic/Kafka.20.7C.20Next.20Generation.20of.20the.20Consumer.20Rebalance.20Protocol/with/541949206]
h2. Requested change (what to implement)
* Add/confirm first-class support for enabling the new protocol via consumer
property {{group.protocol=consumer}} (pass-through supported), and ensure Camel
does not inject defaults that conflict with the new protocol.
* When {{{}group.protocol=consumer{}}}, avoid setting (or warn/error if user
sets) classic-only properties such as {{{}heartbeat.interval.ms{}}},
{{{}session.timeout.ms{}}}, and {{partition.assignment.strategy}} (and
Camel-level equivalents like {{{}heartbeatIntervalMs{}}},
{{{}sessionTimeoutMs{}}}, {{partitionAssignor}} defaults).
* Optionally expose/document {{group.remote.assignor}} for selecting a
server-side assignor (override broker default) when using the consumer protocol.
h2. Acceptance criteria (suggested)
* With Kafka client 4.0+ and {{{}group.protocol=consumer{}}}, Camel consumer
starts successfully without requiring {{session.timeout.ms}} /
{{heartbeat.interval.ms}} / {{partition.assignment.strategy}} and without Camel
adding defaults for them.
* Add an integration test matrix (at least one) that runs Camel Kafka consumer
with {{group.protocol=consumer}} against a Kafka 4.0+ broker.
* Documentation update: clearly describe “Classic vs Consumer” protocol
selection, and call out unsupported configs under the consumer protocol.
[kip-848|[https://www.confluent.io/blog/kip-848-consumer-rebalance-protocol/]]
[Kafka4.0-DocLink|https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol]
> camel-kafka: Support Kafka 4.0 KIP-848 consumer rebalance protocol
> (group.protocol=consumer)
> --------------------------------------------------------------------------------------------
>
> Key: CAMEL-22811
> URL: https://issues.apache.org/jira/browse/CAMEL-22811
> Project: Camel
> Issue Type: New Feature
> Components: camel-kafka
> Reporter: SRIKANTH PRATHIPATI
> Priority: Minor
>
> h2. Background
> Apache Kafka 4.0 introduces the “Next Generation of the Consumer Rebalance
> Protocol” (KIP-848), which is GA and designed to be fully incremental (no
> global sync barrier), improving rebalance times and scalability.
> With the new protocol, some behavior and configuration moves to the broker:
> heartbeat interval, session timeout, and the assignment strategy are
> controlled server-side for {{Consumer}} groups.
> h2. Problem statement (Camel impact)
> KIP-848 expects the broker to manage {{{}partitionAssignor{}}},
> {{{}heartbeatIntervalMs{}}}, and {{{}sessionTimeoutMs{}}}, while the current
> Camel Kafka component “defaulted these if these are not supplied.”
> Kafka’s own documentation for the new protocol states that
> {{{}heartbeat.interval.ms{}}}, {{{}session.timeout.ms{}}}, and
> {{partition.assignment.strategy}} are “no longer usable” when
> {{group.protocol=consumer}} is enabled.
> Please refer to the related thread - [#camel > Kafka | Next Generation of the
> Consumer Rebalance
> Protocol|https://camel.zulipchat.com/#narrow/channel/257298-camel/topic/Kafka.20.7C.20Next.20Generation.20of.20the.20Consumer.20Rebalance.20Protocol/with/541949206]
> h2. Requested change (what to implement)
> * Add/confirm first-class support for enabling the new protocol via consumer
> property {{group.protocol=consumer}} (pass-through supported), and ensure
> Camel does not inject defaults that conflict with the new protocol.
> * When {{{}group.protocol=consumer{}}}, avoid setting (or warn/error if user
> sets) classic-only properties such as {{{}heartbeat.interval.ms{}}},
> {{{}session.timeout.ms{}}}, and {{partition.assignment.strategy}} (and
> Camel-level equivalents like {{{}heartbeatIntervalMs{}}},
> {{{}sessionTimeoutMs{}}}, {{partitionAssignor}} defaults).
> * Optionally expose/document {{group.remote.assignor}} for selecting a
> server-side assignor (override broker default) when using the consumer
> protocol.
> h2. Acceptance criteria (suggested)
> * With Kafka client 4.0+ and {{{}group.protocol=consumer{}}}, Camel consumer
> starts successfully without requiring {{session.timeout.ms}} /
> {{heartbeat.interval.ms}} / {{partition.assignment.strategy}} and without
> Camel adding defaults for them.
> * Add an integration test matrix (at least one) that runs Camel Kafka
> consumer with {{group.protocol=consumer}} against a Kafka 4.0+ broker.
> * Documentation update: clearly describe “Classic vs Consumer” protocol
> selection, and call out unsupported configs under the consumer protocol.
> [kip-848|[https://www.confluent.io/blog/kip-848-consumer-rebalance-protocol/]]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
