vernedeng commented on code in PR #1049: URL: https://github.com/apache/inlong-website/pull/1049#discussion_r1798752597
########## i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/sort/log_report.md: ########## @@ -0,0 +1,227 @@ +--- +title: OpenTelemetry 日志上报 +sidebar_position: 6 +--- + + + +## 概览 + +由于`InLong Sort`会运行在`Apache Flink`的不同`Task Manager`节点上，每个节点独立存储产生的日志，我们需要到每个节点上查看日志，维护效率低下。为此`InLong Sort`提供了基于OpenTelemetry的日志集中管理方案，用户可以高效地集中处理`Flink`日志。 + +`InLong Sort`可以将日志上报功能集成到各个`Connector`中，其日志处理流程如下图所示。日志通过[OpenTelemetry](https://opentelemetry.io/)进行上报，经由[OpenTelemetry Collector](https://opentelemetry.io/docs/collector/)收集处理后发往[Grafana Loki](https://grafana.com/oss/loki/)进行集中管理。 + + + +## Connector集成日志上报功能 + +`InLong Sort` 封装了[OpenTelemetryLogger](https://github.com/apache/inlong/blob/6e78dd2de8e917b9fc17a18d5e990b43089bb804/inlong-sort/sort-flink/base/src/main/java/org/apache/inlong/sort/base/util/OpenTelemetryLogger.java)类，其提供了一个` Builder`来帮助用户快速配置一个`OpenTelemetryLogger`，并可以通过调用`OpenTelemetryLogger`的`install`和`uninstall`方法来开启和关闭日志上报功能。借助`OpenTelemetryLogger`我们可以很便捷地为`Connector`赋予日志上报功能，以下介绍如何借助`OpenTelemetryLogger`类为符合[FLIP-27](https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface#FLIP27:RefactorSourceInterface-Motivation)标准的`Connector`集成日志上报功能： + +1. 在connector `SourceReader`类构造方法中使用`OpenTelemetryLogger.Builder()`构造一个`openTelemetryLogger`对象 +2. 在`SourceReader`的`Start`接口中调用`openTelemetryLogger`对象的`install()`方法 +3. 在`SourceReader`的`close`接口中调用`openTelemetryLogger`对象的`uninstall()`方法 + +使用示例如下： + +```java +import org.apache.inlong.sort.base.util.OpenTelemetryLogger; + +public class XXXSourceReader<T> +{ + + private static final Logger LOG = LoggerFactory.getLogger(XXXSourceReader.class); + + private final OpenTelemetryLogger openTelemetryLogger; + + public XXXSourceReader() { + ... + // 初始化 OpenTelemetryLogger + this.openTelemetryLogger = new OpenTelemetryLogger.Builder() + .setServiceName(this.getClass().getSimpleName()) + .setLocalHostIp(this.context.getLocalHostName()).build(); + } + + @Override + public void start() { + openTelemetryLogger.install(); // 开启日志上报功能 + ... + } + + @Override + public void close() throws Exception { + openTelemetryLogger.uninstall(); // 关闭日志上报功能 + super.close(); + } + + ... +} +``` + +目前`OpenTelemetryLogger`提供如下配置项： + +| 配置项 | 说明 | 默认值 | +| ----------- | ---------------------------------- | -------- | +|`endpoint` | `OpenTelemetry Collector`地址，如未指定则尝试从`OTEL_EXPORTER_ENDPOINT`环境变量获取；如环境变量也未配置则采用默认值 | `localhost:4317` | +| `serviceName` | `OpenTelemetry `服务名，可以用来区分不同的`connector` |`unnamed_service `| +| `layout` | `Log4j2` 的日志格式，应设置为一个`PatternLayout`对象 |`%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n`| +| `logLevel` | 上报的日志级别 |`Level.INFO`| +| `localHostIp` | 所在`Flink`节点IP，可在`SourceReader`中通过`this.context.getLocalHostName()`获取 |`null`| +## 使用说明 + +除了要为`Connector`集成日志上报功能外，还需要增加`opentelemetry-collector`、`grafana loki`、`grafana`三个docker容器，并为`Flink`容器配置`OTEL_EXPORTER_ENDPOINT`环境变量，`docker-compose.yml`文件参考如下： + +```yml +# flink jobmanager +jobmanager: + image: apache/flink:1.15-scala_2.12 + container_name: jobmanager + environment: + - | + FLINK_PROPERTIES= + jobmanager.rpc.address: jobmanager + - OTEL_EXPORTER_ENDPOINT=logcollector:4317 + ports: + - "8081:8081" + command: jobmanager + +# flink taskmanager +taskmanager: + image: apache/flink:1.15-scala_2.12 + container_name: taskmanager + environment: + - | + FLINK_PROPERTIES= + jobmanager.rpc.address: jobmanager + taskmanager.numberOfTaskSlots: 2 + - OTEL_EXPORTER_ENDPOINT=logcollector:4317 + command: taskmanager + +# opentelemetry collector +logcollector: + image: otel/opentelemetry-collector-contrib:0.110.0 + container_name: logcollector + volumes: + - ./log-system/otel-config.yaml:/otel-config.yaml + command: [ "--config=/otel-config.yaml"] + ports: + - "4317:4317" + +# grafana loki +loki: + image: grafana/loki:3.0.0 + ports: + - "3100:3100" + volumes: + - ./log-system/loki.yaml:/etc/loki/local-config.yaml + command: -config.file=/etc/loki/local-config.yaml + +# grafana +grafana: + environment: + - GF_PATHS_PROVISIONING=/etc/grafana/provisioning + - GF_AUTH_ANONYMOUS_ENABLED=true + - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin + entrypoint: + - sh + - -euc + - | + mkdir -p /etc/grafana/provisioning/datasources + cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml Review Comment: cat /etc/grafana/provisioning/datasources/ds.yaml -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
