AndrewJSchofield commented on code in PR #22950: URL: https://github.com/apache/kafka/pull/22950#discussion_r3655608075
########## docs/getting-started/upgrade.md: ########## @@ -48,6 +48,7 @@ type: docs * The `kafka-cluster.sh` tool now provides an `api-versions` command to display the API versions supported by the brokers or controllers, and it accepts both `--bootstrap-server` and `--bootstrap-controller`. As a result, `kafka-broker-api-versions.sh` is deprecated and will be removed in the next major release; use `kafka-cluster.sh api-versions` instead. For further details, please refer to [KIP-1220](https://cwiki.apache.org/confluence/x/-QkbFw). * Brokers can now record a human-readable description of each streams group's processing topology via a pluggable backend, retrievable through `Admin#describeStreamsGroups` and `kafka-streams-groups.sh --describe --topology`. The feature is disabled unless the new broker configuration `group.streams.topology.description.plugin.class` is set to a `StreamsGroupTopologyDescriptionPlugin` implementation; on the client side, the new Kafka Streams configuration `topology.description.push.enabled` (default `true`) controls whether the client pushes topology descriptions when requested. This adds a new RPC, `StreamsGroupTopologyDescriptionUpdate`, bumps `StreamsGroupDescribe` and `StreamsGroupHeartbeat` to version 1, and introduces the error codes `GROUP_DELETION_FAILED` (134) and `STREAMS_TOPOLOGY_DESCRIPTION_UPDATE_FAILED` (135). `DeleteGroups` is bumped to version 3, adding a per-group `ErrorMessage` field so brokers can report why a group deletion failed (for example, when the plugin fails to delete its stored topology description, the group is not deleted and `GROUP_DELETION_FAILED` is returned; retrying the deletion is safe). For further details, please refer to [KIP-1331](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1331%3A+Streams+Group+Topology+Description+Plugin) and the [Topology Description Plugin](/{version}/streams/developer-guide/topology-description-plugin/) documentation. * The `kafka-producer-perf-test.sh` tool now supports `--record-key-range`, `--key-distribution`, and `--random-seed` options to control the distribution of record keys. Use `--key-distribution range` for sequential key assignment (round-robin over the key range) or `--key-distribution random` for random key selection. The `--random-seed` option allows reproducible benchmark runs when using random key distribution. For further details, please refer to [KIP-1299](https://cwiki.apache.org/confluence/x/XpQ8G). + * Share groups now support dead letter queue functionality as outlined in [KIP-1191](https://cwiki.apache.org/confluence/spaces/KAFKA/pages/373885564/KIP-1191+Dead-letter+queues+for+share+groups). Any records which are released (beyond max delivery count) or rejected by the share consumer become eligible for DLQ. Share group DLQ gets enabled when the Kafka feature `share.version` is upgraded to 2. The user can configure a DLQ topic on a share group by setting the dynamic config `errors.deadletterqueue.topic.name` (default `""`) to the name of the DLQ topic. The cluster can be configured to auto create the DLQ topics by setting the dynamic cluster config `errors.deadletterqueue.auto.create.topics.enable` to `true` (default `false`). If auto create is not enabled, the user must create the DLQ topic like a standard Kafka topic and set the dynamic config `errors.deadletterqueue.group.enable` to `true` on the DLQ topic. The DLQ topic name must be prefixed by the value set in the dynami c cluster config `errors.deadletterqueue.topic.name.prefix` (default `dlq.`). The records sent to the DLQ topic by default only contain source record metadata like group, topic name, partition id, offset and delivery count. If original record data is also required, the user must set the dynamic config `errors.deadletterqueue.copy.record.enable` to `true` on the share group. Review Comment: nit: Let's use the short URL which is `https://cwiki.apache.org/confluence/x/fApJFg`. ########## docs/operations/basic-kafka-operations.md: ########## @@ -350,6 +350,50 @@ $ bin/kafka-share-groups.sh --bootstrap-server localhost:9092 --delete --group m Deletion of requested share groups ('my-share-group') was successful. ``` +## Configuring dead letter queues (DLQ) on share groups ([KIP-1191](https://cwiki.apache.org/confluence/spaces/KAFKA/pages/373885564/KIP-1191+Dead-letter+queues+for+share+groups)) + +Enable share group dead letter queues (DLQ) on the cluster: + +```bash +$ bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --feature share.version=2 +share.version was upgraded to 2. +``` + +Set DLQ topic on share group: + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config "errors.deadletterqueue.topic.name=dlq.gs1dlqtopic" --entity-type groups --entity-name my-share-group +Completed updating config for group my-share-group. +``` + +To enable share group DLQ topic auto creation (default disabled): + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config "errors.deadletterqueue.auto.create.topics.enable=true" --entity-type brokers --entity-default +Completed updating default config for brokers in the cluster. +``` + +To set your own Kafka topic as share group DLQ topic, you must set certain dynamic configs on the DLQ topic post creation. If cluster dynamic config to auto create share group DLQ topic is enabled (`errors.deadletterqueue.auto.create.topics.enable=true`), the configs are attached automatically to the auto created topics. + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config "errors.deadletterqueue.group.enable=true" --entity-type topics --entity-name dlq.gs1dlqtopic +Completed updating config for topic dlq.gs1dlqtopic. +``` + +To change the default `dlq.` share group DLQ topic prefix, set the dynamic cluster config `errors.deadletterqueue.topic.name.prefix` to desired value. The default value is `dlq.`. + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config "errors.deadletterqueue.topic.name.prefix=com.mycompany.dlq." --entity-type brokers --entity-default +Completed updating default config for brokers in the cluster. +``` + +To enable copying for source record key and value into the DLQ records (default disabled): Review Comment: Maybe `By default, the records written to the DLQ topic just include metadata about the source records such as the topic, partition and offset. You can set the errors.deadletterqueue.copy.read.enable configuration for the share group so the source record key and value are copied to the DLQ topic.` or similar. ########## docs/operations/basic-kafka-operations.md: ########## @@ -350,6 +350,50 @@ $ bin/kafka-share-groups.sh --bootstrap-server localhost:9092 --delete --group m Deletion of requested share groups ('my-share-group') was successful. ``` +## Configuring dead letter queues (DLQ) on share groups ([KIP-1191](https://cwiki.apache.org/confluence/spaces/KAFKA/pages/373885564/KIP-1191+Dead-letter+queues+for+share+groups)) Review Comment: I would not include the KIP in this heading. Including it in the upgrade nodes is fine, but really it's just part of AK at this point and the docs don't need to refer to the KIP. ########## docs/operations/basic-kafka-operations.md: ########## @@ -350,6 +350,50 @@ $ bin/kafka-share-groups.sh --bootstrap-server localhost:9092 --delete --group m Deletion of requested share groups ('my-share-group') was successful. ``` +## Configuring dead letter queues (DLQ) on share groups ([KIP-1191](https://cwiki.apache.org/confluence/spaces/KAFKA/pages/373885564/KIP-1191+Dead-letter+queues+for+share+groups)) + +Enable share group dead letter queues (DLQ) on the cluster: Review Comment: This is really ensuring the `share.version=2` on the cluster. A new cluster will already have it. I would just say something like `Share group dead-letter queues are enabled if the share.version feature at least 2.`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
