zJiaJun commented on code in PR #224: URL: https://github.com/apache/skywalking-go/pull/224#discussion_r2094760392
########## docs/en/advanced-features/kafka-reporter.md: ########## @@ -0,0 +1,147 @@ +# Kafka Reporter + +This document describes how to configure and use the Kafka reporter in the Apache SkyWalking Go agent. The Kafka reporter provides an alternative to the default gRPC reporter, allowing you to send trace, metrics, and log data to Apache Kafka. + +## Overview + +The SkyWalking Go agent can be configured to report collected telemetry data (traces, metrics, logs) to a Kafka cluster. This is useful for scenarios where Kafka is already part of your infrastructure or when you prefer Kafka's buffering and scalability features for handling observability data. + +## Enabling Kafka Reporter + +You can enable the Kafka reporter either through environment variables or by configuring the `agent.default.yaml` file. + +**Using Environment Variables:** + +Set the `SW_AGENT_REPORTER_TYPE` environment variable to `kafka`: +```bash +export SW_AGENT_REPORTER_TYPE=kafka +``` + +**Using `agent.default.yaml`:** + +Modify the `reporter.type` setting in your `agent.default.yaml` configuration file: +```yaml +reporter: + type: kafka # or grpc + # ... other global reporter settings +``` + +## Configuration + +The Kafka reporter requires specific configurations for connecting to your Kafka cluster and specifying topics for different data types. These can be set via environment variables or in the `agent.default.yaml` file. Environment variable names generally follow the pattern `SW_AGENT_REPORTER_KAFKA_OPTION_NAME_IN_UPPERCASE`. + +### Core Kafka Configuration + +These settings are typically found under the `reporter.kafka` section in `agent.default.yaml` or can be set using the corresponding environment variables. + +* **Broker Addresses:** + A comma-separated list of Kafka broker addresses. + * Environment Variable: `SW_AGENT_REPORTER_KAFKA_BROKERS` + * YAML: `reporter.kafka.brokers` + * Example: `kafka1:9092,kafka2:9092` + + +* **Topic for Segments:** + The Kafka topic where trace segments will be sent. + * Environment Variable: `SW_AGENT_REPORTER_KAFKA_TOPIC_SEGMENT` + * YAML: `reporter.kafka.topic_segment` + * Example: `skywalking-segments` + + +* **Topic for Metrics:** + The Kafka topic where metrics data will be sent. + * Environment Variable: `SW_AGENT_REPORTER_KAFKA_TOPIC_METER` + * YAML: `reporter.kafka.topic_meter` + * Example: `skywalking-meters` + + +* **Topic for Logs:** + The Kafka topic where log data will be sent. + * Environment Variable: `SW_AGENT_REPORTER_KAFKA_TOPIC_LOGGING` + * YAML: `reporter.kafka.topic_logging` + * Example: `skywalking-logs` + + +* **Topic for Management:** (Optional) + The Kafka topic for management-related messages (e.g., potentially for configurations or commands in future use). + * Environment Variable: `SW_AGENT_REPORTER_KAFKA_TOPIC_MANAGEMENT` + * YAML: `reporter.kafka.topic_management` + * Example: `skywalking-management` + +**Important Note:** Even when the primary data reporting is set to Kafka (`reporter.type: kafka`), the CDS functionality itself still relies on gRPC communication with the SkyWalking OAP (Observability Analysis Platform). Therefore, you **must** also configure the relevant gRPC settings under `reporter.grpc` (or their corresponding environment variables like `SW_AGENT_REPORTER_GRPC_BACKEND_SERVICE`) for CDS to work correctly. Review Comment: here -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@skywalking.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
