mjsax commented on code in PR #23486: URL: https://github.com/apache/kafka/pull/23486#discussion_r4032683177
########## docs/getting-started/upgrade.md: ########## @@ -53,10 +53,11 @@ type: docs * Share groups now support dead-letter queue functionality as outlined in [KIP-1191](https://cwiki.apache.org/confluence/x/fApJFg). Any records which are released (beyond max delivery count) or rejected by the share consumer become eligible for DLQ. Share group DLQ gets enabled when the Kafka feature `share.version` is upgraded to 2. The user can configure a DLQ topic on a share group by setting the dynamic config `errors.deadletterqueue.topic.name` (default `""`) to the name of the DLQ topic. The cluster can be configured to auto create the DLQ topics by setting the dynamic cluster config `errors.deadletterqueue.auto.create.topics.enable` to `true` (default `false`). If auto create is not enabled, the user must create the DLQ topic like a standard Kafka topic and set the dynamic config `errors.deadletterqueue.group.enable` to `true` on the DLQ topic. The DLQ topic name must be prefixed by the value set in the dynamic cluster config `errors.deadletterqueue.topic.name.prefix` (defa ult `dlq.`). The records sent to the DLQ topic by default only contain source record metadata like group, topic name, partition id, offset and delivery count. If original record data is also required, the user must set the dynamic config `errors.deadletterqueue.copy.record.enable` to `true` on the share group. * Kafka Connect distributed workers now support the `internal.topics.automatic.creation.enable` configuration (default: `true`). When set to `false`, Connect will not automatically create internal topics (offset, config, status, and connector-specific offset topics) and will instead fail at startup if any of these topics are missing. A new `connect-internal-topics.sh` tool is also available for manually creating these topics. For further details, please refer to [KIP-1209](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209:+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect). * Streams groups now support broker-side custom task assignors, registered via the new broker configuration `group.streams.assignors` and selected per group with the new group configuration `streams.assignor.name`. For further details, please refer to [KIP-1357](https://cwiki.apache.org/confluence/x/NoSnGQ). + * Kafka Streams now supports static membership (`group.instance.id`) with the Streams Rebalance Protocol (`group.protocol=streams`). This requires brokers running 4.4 or newer: brokers older than 4.4 reject `group.instance.id` on the streams group heartbeat with `INVALID_REQUEST`, which is a fatal error that shuts down the Kafka Streams client. Do not downgrade brokers below 4.4 while static members are running. For further details, please refer to the [Kafka Streams upgrade guide](/{version}/streams/upgrade-guide/). Review Comment: ```suggestion * Kafka Streams now supports static membership (`group.instance.id`) with the Streams Rebalance Protocol (`group.protocol=streams`). This requires brokers and clients running 4.4 or newer: brokers older than 4.4 reject `group.instance.id` on the streams group heartbeat with `INVALID_REQUEST`, which is a fatal error that shuts down the Kafka Streams client. Do not downgrade brokers below 4.4 while static members are running. For further details, please refer to the [Kafka Streams upgrade guide](/{version}/streams/upgrade-guide/). ``` ########## docs/streams/developer-guide/config-streams.md: ########## @@ -1809,7 +1809,7 @@ If you call `streamsBuilder.build()` without passing the `Properties` object, op #### group.instance.id {#group-instance-id} - `group.instance.id` is a consumer configuration that enables static membership. For Kafka Streams, it is configured at the Kafka Streams client level, and the configured value must be unique to each `KafkaStreams` instance. Internally, Kafka Streams appends the 1-based stream thread index to the configured `group.instance.id` to ensure that each stream thread's main consumer uses a unique value. For example, a configured value of `ks-client-A` results in `ks-client-A-1`, `ks-client-A-2`, and so on. Static membership is supported with both `group.protocol=classic` and `group.protocol=streams`. + `group.instance.id` is a consumer configuration that enables static membership. For Kafka Streams, it is configured at the Kafka Streams client level, and the configured value must be unique to each `KafkaStreams` instance. Internally, Kafka Streams appends the 1-based stream thread index to the configured `group.instance.id` to ensure that each stream thread's main consumer uses a unique value. For example, a configured value of `ks-client-A` results in `ks-client-A-1`, `ks-client-A-2`, and so on. Static membership is supported with both `group.protocol=classic` and `group.protocol=streams`. With `group.protocol=streams`, static membership requires brokers running Kafka 4.4 or newer; older brokers reject `group.instance.id` on the streams group heartbeat and the Kafka Streams client shuts down with a fatal error. Review Comment: ```suggestion `group.instance.id` is a consumer configuration that enables static membership. For Kafka Streams, it is configured at the Kafka Streams client level, and the configured value must be unique to each `KafkaStreams` instance. Internally, Kafka Streams appends the 1-based stream thread index to the configured `group.instance.id` to ensure that each stream thread's main consumer uses a unique value. For example, a configured value of `ks-client-A` results in `ks-client-A-1`, `ks-client-A-2`, and so on. Static membership is supported with both `group.protocol=classic` and `group.protocol=streams`. With `group.protocol=streams`, static membership requires clients and brokers running Kafka 4.4 or newer; older brokers reject `group.instance.id` on the streams group heartbeat and the Kafka Streams client shuts down with a fatal error. ``` ########## docs/streams/developer-guide/streams-rebalance-protocol.md: ########## @@ -53,7 +53,7 @@ The following features are available in the current release: * **Offline Migration**: After shutting down all members and waiting for their `session.timeout.ms` to expire (or forcing an explicit group leave), a classic group can be converted to a streams group and a streams group can be converted to a classic group. The only broker-side group data that will be preserved are the committed offsets. Internal topics (changelog and repartition topics) will continue to exist as regular Kafka topics. -* **Static Membership**: Streams applications can configure [`group.instance.id`](../config-streams#group-instance-id) when using `group.protocol=streams`. However, for topologies without persistent state stores, Kafka Streams generates a new process ID on each restart, causing the broker to recompute the group assignment and effectively negating the benefits of static membership across restarts. +* **Static Membership**: Streams applications can configure [`group.instance.id`](../config-streams#group-instance-id) when using `group.protocol=streams`. This requires brokers running Kafka 4.4 or newer: brokers on older versions reject `group.instance.id` on the streams group heartbeat with `INVALID_REQUEST`, which is a fatal error that shuts down the Kafka Streams client. Do not downgrade brokers below 4.4 while static members are running; if brokers have already been downgraded, remove `group.instance.id` from the application configuration and restart the application. Also note that for topologies without persistent state stores, Kafka Streams generates a new process ID on each restart, causing the broker to recompute the group assignment and effectively negating the benefits of static membership across restarts. Review Comment: ```suggestion * **Static Membership**: Streams applications can configure [`group.instance.id`](../config-streams#group-instance-id) when using `group.protocol=streams`. This requires clients and brokers running Kafka 4.4 or newer: brokers on older versions reject `group.instance.id` on the streams group heartbeat with `INVALID_REQUEST`, which is a fatal error that shuts down the Kafka Streams client. Do not downgrade brokers below 4.4 while static members are running; if brokers have already been downgraded, remove `group.instance.id` from the application configuration and restart the application. Also note that for topologies without persistent state stores, Kafka Streams generates a new process ID on each restart, causing the broker to recompute the group assignment and effectively negating the benefits of static membership across restarts. ``` ########## docs/streams/upgrade-guide.md: ########## @@ -89,7 +89,7 @@ For applications using the Streams Rebalance Protocol (`group.protocol=streams`) The buffer used by `suppress()` is now headers-aware, closing one of the gaps listed under the [KIP-1285 current limitations](#current-limitations) in 4.3.0. With [`dsl.store.format=HEADERS`](/{version}/streams/developer-guide/config-streams.html#dsl-store-format), each buffered value is stored together with the headers of the record it came from, so record headers are preserved across the suppression boundary: the record emitted when a buffered row is evicted carries the headers of the value being emitted. With the default `dsl.store.format=DEFAULT`, behavior is unchanged — the buffer stores plain values plus a single record context per buffered row, so a row that is updated by a later record before it is evicted preserves only the headers of that later record. The buffer's changelog records keep the existing V3 value format in both cases; under `HEADERS` the headers of the buffered old and prior values are carried in additional Kafka record headers (`vh.old`, `vh.prior`), which older versions ignore and restore the values without, so the changelog stays readable across a downgrade. -Kafka Streams now supports static membership with the Streams Rebalance Protocol. Applications using `group.protocol=streams` may configure `group.instance.id`. However, for topologies without persistent state stores, Kafka Streams generates a new process ID on each restart, causing the broker to recompute the group assignment and effectively negating the benefits of static membership across restarts. +Kafka Streams now supports static membership with the Streams Rebalance Protocol. Applications using `group.protocol=streams` may configure `group.instance.id`. However, for topologies without persistent state stores, Kafka Streams generates a new process ID on each restart, causing the broker to recompute the group assignment and effectively negating the benefits of static membership across restarts. Static membership with the Streams Rebalance Protocol requires brokers running Kafka 4.4 or newer: brokers on older versions reject `group.instance.id` on the streams group heartbeat with `INVALID_REQUEST`, which is a fatal error that shuts down the Kafka Streams client. Do not downgrade brokers below 4.4 while static members are running. If brokers have already been downgraded, remove `group.instance.id` from the application configuration and restart the application. Review Comment: ```suggestion Kafka Streams now supports static membership with the Streams Rebalance Protocol. Applications using `group.protocol=streams` may configure `group.instance.id`. However, for topologies without persistent state stores, Kafka Streams generates a new process ID on each restart, causing the broker to recompute the group assignment and effectively negating the benefits of static membership across restarts. Static membership with the Streams Rebalance Protocol requires clients and brokers running Kafka 4.4 or newer: brokers on older versions reject `group.instance.id` on the streams group heartbeat with `INVALID_REQUEST`, which is a fatal error that shuts down the Kafka Streams client. Do not downgrade brokers below 4.4 while static members are running. If brokers have already been downgraded, remove `group.instance.id` from the application configuration and restart the application. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
