Re: [PR] KAFKA-19977: Documentation section on the streams rebalance protocol [kafka]
lucasbru commented on PR #21170: URL: https://github.com/apache/kafka/pull/21170#issuecomment-3767276071 cherry-picked to 4.2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19977: Documentation section on the streams rebalance protocol [kafka]
lucasbru merged PR #21170: URL: https://github.com/apache/kafka/pull/21170 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19977: Documentation section on the streams rebalance protocol [kafka]
lucasbru commented on code in PR #21170: URL: https://github.com/apache/kafka/pull/21170#discussion_r2693844920 ## docs/streams/developer-guide/kafka-streams-group-sh.md: ## @@ -2,7 +2,7 @@ title: Kafka Streams Groups Tool type: docs description: -weight: 14 +weight: 15 Review Comment: It's used to order the pages in TOCs and sidebars. I wanted the streams group tool appear below the streams rebalance protocol page. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19977: Documentation section on the streams rebalance protocol [kafka]
mjsax commented on code in PR #21170:
URL: https://github.com/apache/kafka/pull/21170#discussion_r2691594080
##
docs/streams/developer-guide/kafka-streams-group-sh.md:
##
@@ -2,7 +2,7 @@
title: Kafka Streams Groups Tool
type: docs
description:
-weight: 14
+weight: 15
Review Comment:
For my own education: what is this and why does it get changed?
##
docs/streams/developer-guide/streams-rebalance-protocol.md:
##
@@ -0,0 +1,253 @@
+---
+title: Streams Rebalance Protocol
+description:
+weight: 14
+tags: ['kafka', 'docs']
+aliases:
+keywords:
+type: docs
+---
+
+
+
+The Streams Rebalance Protocol is a broker-driven rebalancing system designed
specifically for Kafka Streams applications. Following the pattern of KIP-848,
which moved rebalance coordination of plain consumers from clients to brokers,
KIP-1071 extends this model to Kafka Streams workloads.
+
+# Overview
+
+Instead of clients computing new assignments on the client during rebalance
events involving all members of the group, assignments are computed
continuously on the broker. Instead of using a consumer group, the streams
application registers as a **streams group** with the broker, which manages and
exposes all metadata required for coordination of the streams application
instances.
+
+This approach brings Kafka Streams coordination in line with the modern
broker-driven rebalance model introduced for consumers in KIP-848, providing a
dedicated group type with streams-specific semantics and metadata management.
+
+# What's Supported in This Version
+
+The following features are available in the current release:
+
+* **Core Streams Group Rebalance Protocol**: The `group.protocol=streams`
configuration enables the dedicated streams rebalance protocol. This separates
streams groups from consumer groups and provides a streams-specific group
membership lifecycle and metadata management on the broker.
+
+* **Sticky Task Assignor**: A basic task assignment strategy that minimizes
task movement during rebalances is included.
+
+* **Interactive Query Support**: IQ operations are compatible with the new
streams protocol.
+
+* **New Admin RPC**: The StreamsGroupDescribe RPC provides streams-specific
metadata separate from consumer group information, with corresponding access
via the [`Admin`](/{version}/javadoc/org/apache/kafka/clients/admin/Admin.html).
+
+* **CLI Integration**: You can list, describe, and delete streams groups via
the
[bin/kafka-streams-groups.sh](/{version}/streams/developer-guide/kafka-streams-group-sh/)
script.
+
+* **Offline Migration**: After shutting down all members and waiting for their
`session.timeout.ms` to expire (or forcing an explicit group leave), a classic
group can be converted to a streams group and a streams group can be converted
to a classic group. The only broker-side group data that will be preserved are
the committed offsets. Internal topics (changelog and repartition topics) will
continue to exist as regular Kafka topics.
+
+# What's Not Supported in This Version
+
+The following features are not yet available and should be avoided when using
the new protocol:
+
+* **Static Membership**: Setting a client `instance.id` will be rejected.
+
+* **Topology Updates**: If a topology is changed significantly (e.g., by
adding new source topics or changing the number of subtopologies), a new
streams group must be created.
+
+* **High Availability Assignor**: Only the sticky assignor is supported.
+
+* **Regular Expressions**: Pattern-based topic subscription is not supported.
+
+* **Online Migration**: Group migration while the application is running is
not available between the classic and new streams protocol.
+
+# Why Use the Streams Rebalance Protocol?
+
+The Streams Rebalance Protocol offers several key advantages over the classic
client-driven protocol:
+
+* **Broker-Driven Coordination**: Centralizes task assignment logic on brokers
instead of the client. This provides consistent, authoritative task assignment
decisions from a single coordination point and reduces the potential for
split-brain scenarios.
Review Comment:
```suggestion
* **Broker-Driven Coordination**: Centralizes task assignment logic on
brokers instead of the client. This provides consistent, authoritative task
assignment decisions from a single coordination point, and reduces the
potential for split-brain scenarios.
```
##
docs/streams/developer-guide/streams-rebalance-protocol.md:
##
@@ -0,0 +1,253 @@
+---
+title: Streams Rebalance Protocol
+description:
+weight: 14
+tags: ['kafka', 'docs']
+aliases:
+keywords:
+type: docs
+---
+
+
+
+The Streams Rebalance Protocol is a broker-driven rebalancing system designed
specifically for Kafka Streams applications. Following the pattern of KIP-848,
which moved rebalance coordination of plain consumers from clients to brokers,
KIP-1071 extends this model to Kafka Streams workloads.
+
+# Overview
+
+
Re: [PR] KAFKA-19977: Documentation section on the streams rebalance protocol [kafka]
lucasbru commented on PR #21170: URL: https://github.com/apache/kafka/pull/21170#issuecomment-3744211302 https://github.com/user-attachments/assets/a3e9e9a8-7dc9-4839-8d0c-e986a0065453"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19977: Documentation section on the streams rebalance protocol [kafka]
lucasbru commented on PR #21170: URL: https://github.com/apache/kafka/pull/21170#issuecomment-3744206852 @mjsax all addressed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19977: Documentation section on the streams rebalance protocol [kafka]
Copilot commented on code in PR #21170: URL: https://github.com/apache/kafka/pull/21170#discussion_r2661813617 ## docs/streams/developer-guide/streams-rebalance-protocol.md: ## @@ -0,0 +1,253 @@ +--- +title: Streams Rebalance Protocol +description: +weight: 14 +tags: ['kafka', 'docs'] +aliases: +keywords: +type: docs +--- + + + +The Streams Rebalance Protocol is a broker-driven rebalancing system designed specifically for Kafka Streams applications. Following the pattern of KIP-848, which moved rebalance coordination of plain consumers from clients to brokers, KIP-1071 extends this model to Kafka Streams workloads. + +## Overview + +Instead of clients computing new assignments on the client during rebalance events involving all members of the group, assignments are computed continuously on the broker. Instead of using a consumer group, the streams application registers as a **streams group** with the broker, which manages and exposes all metadata required for coordination of the streams application instances. + +This approach brings Kafka Streams coordination in line with the modern broker-driven rebalance model introduced for consumers in KIP-848, providing a dedicated group type with streams-specific semantics and metadata management. + +## What's Supported in This Version + +The following features are available in the current release: + +* **Core Streams Group Rebalance Protocol**: The `group.protocol=streams` configuration enables the dedicated streams rebalance protocol. This separates streams groups from consumer groups and provides a streams-specific group membership lifecycle and metadata management on the broker. + +* **Sticky Task Assignor**: A basic task assignment strategy that minimizes task movement during rebalances is included. + +* **Interactive Query Support**: IQ operations are compatible with the new streams protocol. + +* **New Admin RPC**: The StreamsGroupDescribe RPC provides streams-specific metadata separate from consumer group information, with corresponding access via the [`Admin`](/43/javadoc/org/apache/kafka/clients/admin/Admin.html). + +* **CLI Integration**: You can list, describe, and delete streams groups via the [kafka-streams-groups.sh](kafka-streams-group-sh.md) script. Review Comment: The link references a file 'kafka-streams-group-sh.md' which doesn't exist in the repository. This will result in broken links in the documentation. The documentation should either include the referenced file or use a different link target. ```suggestion * **CLI Integration**: You can list, describe, and delete streams groups via the `kafka-streams-groups.sh` script. ``` ## docs/streams/developer-guide/streams-rebalance-protocol.md: ## @@ -0,0 +1,253 @@ +--- +title: Streams Rebalance Protocol +description: +weight: 14 +tags: ['kafka', 'docs'] +aliases: +keywords: +type: docs +--- + + + +The Streams Rebalance Protocol is a broker-driven rebalancing system designed specifically for Kafka Streams applications. Following the pattern of KIP-848, which moved rebalance coordination of plain consumers from clients to brokers, KIP-1071 extends this model to Kafka Streams workloads. + +## Overview + +Instead of clients computing new assignments on the client during rebalance events involving all members of the group, assignments are computed continuously on the broker. Instead of using a consumer group, the streams application registers as a **streams group** with the broker, which manages and exposes all metadata required for coordination of the streams application instances. + +This approach brings Kafka Streams coordination in line with the modern broker-driven rebalance model introduced for consumers in KIP-848, providing a dedicated group type with streams-specific semantics and metadata management. + +## What's Supported in This Version + +The following features are available in the current release: + +* **Core Streams Group Rebalance Protocol**: The `group.protocol=streams` configuration enables the dedicated streams rebalance protocol. This separates streams groups from consumer groups and provides a streams-specific group membership lifecycle and metadata management on the broker. + +* **Sticky Task Assignor**: A basic task assignment strategy that minimizes task movement during rebalances is included. + +* **Interactive Query Support**: IQ operations are compatible with the new streams protocol. + +* **New Admin RPC**: The StreamsGroupDescribe RPC provides streams-specific metadata separate from consumer group information, with corresponding access via the [`Admin`](/43/javadoc/org/apache/kafka/clients/admin/Admin.html). + +* **CLI Integration**: You can list, describe, and delete streams groups via the [kafka-streams-groups.sh](kafka-streams-group-sh.md) script. + +* **Offline Migration**: After shutting down all members and waiting for their `session.timeout.ms` to expire, a classic group can be
Re: [PR] KAFKA-19977: Documentation section on the streams rebalance protocol [kafka]
lucasbru commented on PR #21170: URL: https://github.com/apache/kafka/pull/21170#issuecomment-3710582479 Updated to markdown, @mjsax . https://github.com/user-attachments/assets/7f7d9379-8cbb-4170-8389-998c2456cdea"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19977: Documentation section on the streams rebalance protocol [kafka]
mumrah commented on PR #21170: URL: https://github.com/apache/kafka/pull/21170#issuecomment-3679899501 Hey @lucasbru, we just merged the HTML to Markdown changes. You'll need to re-do these changes with the new system. Sorry for the inconvenience! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19977: Documentation section on the streams rebalance protocol [kafka]
lucasbru commented on PR #21170: URL: https://github.com/apache/kafka/pull/21170#issuecomment-3670063137 https://github.com/user-attachments/assets/8fa7a94f-0a83-466a-a2ec-bc4d5e3d9772"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19977: Documentation section on the streams rebalance protocol [kafka]
Copilot commented on code in PR #21170: URL: https://github.com/apache/kafka/pull/21170#discussion_r2630351868 ## docs/streams/developer-guide/streams-rebalance-protocol.html: ## @@ -0,0 +1,351 @@ + + + + + + +++ ++ +++Streams Rebalance Protocol
+The Streams Rebalance Protocol is a broker-driven rebalancing system designed specifically for Kafka Streams applications. Following the pattern of KIP-848, which moved rebalance coordination of plain consumers from clients to brokers, KIP-1071 extends this model to Kafka Streams workloads.
+ +++ +Table of Contents
+ +++ +Overview
+Instead of clients computing new assignments on the client during rebalance events involving all members of the group, assignments are computed continuously on the broker. Instead of using a consumer group, the streams application registers as a streams group with the broker, which manages and exposes all metadata required for coordination of the streams application instances.
+ +This approach brings Kafka Streams coordination in line with the modern broker-driven rebalance model introduced for consumers in KIP-848, providing a dedicated group type with streams-specific semantics and metadata management.
++What's Supported in This Version
+The following features are available in the current release:
++
- Core Streams Group Rebalance Protocol: The
+ +group.protocol=streamsconfiguration enables the dedicated streams rebalance protocol. This separates streams groups from consumer groups and provides a streams-specific group membership lifecycle and metadata management on the broker.- Stick
