This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong-website.git
The following commit(s) were added to refs/heads/master by this push:
new 61d139e45 [INLONG-439][Doc] Update the plugin and modules documents
(#440)
61d139e45 is described below
commit 61d139e45df018fa272eb03c1f93827bf07a25f5
Author: Charles Zhang <[email protected]>
AuthorDate: Wed Jun 22 10:06:04 2022 +0800
[INLONG-439][Doc] Update the plugin and modules documents (#440)
---
.../how_to_extend_data_node_for_manager.md | 16 +-
.../how_to_extend_data_node_for_sort.md | 32 +-
.../how_to_write_plugin_dashboard.md | 11 +-
docs/modules/dataproxy/overview.md | 435 ++++++++++-----------
docs/modules/sort/example.md | 4 +-
.../how_to_extend_data_node_for_manager.md | 10 +-
.../how_to_extend_data_node_for_sort.md | 30 +-
.../how_to_write_plugin_dashboard.md | 11 +-
.../current/modules/dataproxy/overview.md | 430 ++++++++++----------
.../current/modules/sort/example.md | 5 +-
10 files changed, 498 insertions(+), 486 deletions(-)
diff --git a/docs/design_and_concept/how_to_extend_data_node_for_manager.md
b/docs/design_and_concept/how_to_extend_data_node_for_manager.md
index a06cabbbe..13b67123c 100644
--- a/docs/design_and_concept/how_to_extend_data_node_for_manager.md
+++ b/docs/design_and_concept/how_to_extend_data_node_for_manager.md
@@ -1,18 +1,16 @@
---
-title: Manager Plugin for Data Node
+title: Manager Plugin
sidebar_position: 5
---
## Overview
Inlong is aimed at create dataflow between different data sources, now Inlong
has support several universal data sources such as **MySQL**, **Apache Kafka**,
**ClickHouse** on Input/Output respectively,
-You can refer to [Data Node](data_node/extract_node/overview.md) for specific
information.
-We Plan to support more data sources in the future, and this article is a
development manual to extend data nodes.
+You can refer to [Data Node](data_node/extract_node/overview.md) for specific
information. Each Data Node of InLong supports unified management through
Manager to simplify the use of users.
+This article describes how to extend a new data node through the Manager to
provide services.
## Extend Extract Node
-In order to extend an input data sources , also refered to **extract node** in
Inlong. We take **MySQL_BINLOG** for example.
-
- Develop extract node plugin in sort, refer to [Sort
Plugin](design_and_concept/how_to_extend_data_node_for_sort.md)
- Add **TaskType** in `org.apache.inlong.common.enums.TaskTypeEnum`
- Add **SourceType** in `org.apache.inlong.manager.common.enums.SourceType`
@@ -23,14 +21,8 @@ In order to extend an input data sources , also refered to
**extract node** in I
## Extend Load Node
-The way to extend data sink is nearly the same with data source
-
- Develop load node plugin in sort, refer to [Sort
Plugin](design_and_concept/how_to_extend_data_node_for_sort.md)
- Add **SinkType** in `org.apache.inlong.manager.common.enums.SinkType`
- Create new package under package path:
`org.apache.inlong.manager.common.pojo.sink`, develop every entity class needed.
- Create Operation class for new data source under package path:
`org.apache.inlong.manager.service.sink`.
-- Transfer data sink to **LoadNode** supported in **Sort**, refer to
`org.apache.inlong.manager.service.sort.util.LoadNodeUtils`
-
-## Last but not Least
-
-Any Optimization Suggestions are welcome through **email** or **pr**.
+- Transfer data sink to **LoadNode** supported in **Sort**, refer to
`org.apache.inlong.manager.service.sort.util.LoadNodeUtils`
\ No newline at end of file
diff --git a/docs/design_and_concept/how_to_extend_data_node_for_sort.md
b/docs/design_and_concept/how_to_extend_data_node_for_sort.md
index af6e6d944..0fdd3f47d 100644
--- a/docs/design_and_concept/how_to_extend_data_node_for_sort.md
+++ b/docs/design_and_concept/how_to_extend_data_node_for_sort.md
@@ -5,9 +5,12 @@ sidebar_position: 3
## Overview
-InLong Sort is known as a real-time ETL system. Currently, supported extract
or load includes FileSystemExtractNode, KafkaExtractNode, MongoExtractNode,
MySqlExtractNode, OracleExtractNode , PostgresExtractNode , PulsarExtractNode,
SqlServerExtractNode, ClickHouseLoadNode, ElasticsearchLoadNode,
FileSystemLoadNode, GreenplumLoadNode, HbaseLoadNode, HiveLoadNode,
IcebergLoadNode, KafkaLoadNode, MySqlLoadNode, OracleLoadNode,
PostgresLoadNode, SqlServerLoadNode, TDSQLPostgresLoadNode, etc. [...]
+InLong Sort is an ETL service based on Apache Flink SQL, the powerful
expressive power of Flink SQL brings high scalability and flexibility.
+Basically, the semantics supported by Flink SQL are supported by InLong Sort.
In some scenarios, when the built-in functions of Flink SQL do not meet the
requirements,
+they can also be extended through various UDFs in InLong Sort. At the same
time, it will be easier for those who have used SQL, especially Flink SQL, to
get started.
-This article describes how to extend a new source (abstracted as extract node
in inlong) or a new sink (abstracted as load node in inlong) in InLong Sort.
After understanding the InLong Sort architecture, you can understand how the
source corresponds to the extract node, and how the sink corresponds to the
load node. The architecture of inlong sort can be represented by UML object
relation diagram as:
+This article describes how to extend a new source (abstracted as extract node
in inlong) or a new sink (abstracted as load node in inlong) in InLong Sort.
+The architecture of inlong sort can be represented by UML object relation
diagram as:
@@ -43,7 +46,12 @@ In the second step, you can use the existing flick connector
or extend it yourse
There are three steps to extend an ExtractNode:
-**Step 1**：Inherit the ExtractNode class,the location of the class is:
`inlong/inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/ExtractNode.java`.
Specify the connector in the implemented ExtractNode.
+**Step 1**：Inherit the ExtractNode class,the location of the class is:
+```bash
+inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/ExtractNode.java
+```
+
+Specify the connector in the implemented ExtractNode.
```Java
// Inherit ExtractNode class and implement specific classes, such as
MongoExtractNode
@@ -88,13 +96,19 @@ public abstract class ExtractNode implements Node{...}
public interface Node {...}
```
-**Step 3**：Expand the Sort connector and check whether the corresponding
connector already exists in the
(`/inlong/inlong-sort/sort-connectors/mongodb-cdc`) directory. If you haven't
already, you need to refer to the official flink documentation [DataStream
Connectors](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/datastream/overview/#datastream-connectors)
to extend, directly call the existing flink-connector (such
as`inlong/inlong-sort/sort-connectors/mongo [...]
+**Step 3**：Expand the Sort connector and check whether the corresponding
connector already exists in the (`InLong
Agentinlong-sort/sort-connectors/mongodb-cdc`) directory. If you haven't
already,
+you need to refer to the official flink documentation [DataStream
Connectors](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/datastream/overview/#datastream-connectors)
to extend,
+directly call the existing flink-connector (such
as`inlong-sort/sort-connectors/mongodb-cdc`) or implement the related connector
by yourself.
## Extend a new load node
There are three steps to extend an LoadNode:
-**Step 1**：Inherit the LoadNode class, the location of the class is
`inlong/inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/LoadNode.java`,
specify the connector in the implemented LoadNode.
+**Step 1**：Inherit the LoadNode class, the location of the class is:
+```bash
+inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/LoadNode.java
+```
+specify the connector in the implemented LoadNode.
```java
// Inherit LoadNode class and implement specific classes, such as KafkaLoadNode
@@ -154,11 +168,15 @@ public abstract class LoadNode implements Node{...}
public interface Node {...}
```
-**Step 3**：Extend the Sort connector, Kafka's sort connector is in
inlong/inlong-sort/sort-connectors/kafka.
+**Step 3**：Extend the Sort connector, Kafka's sort connector is in
`inlong-sort/sort-connectors/kafka`.
## Bundle extract node and load node into InLong Sort
-To integrate extract and load into the InLong Sort mainstream, you need to
implement the semantics mentioned in the overview section: group, stream, node,
etc. The entry class of InLong Sort is in
`inlong-sort/sort-core/src/main/java/org/apache/inlong/sort/Entrance.java`. How
to integrate extract and load into InLong Sort can refer to the following ut.
First, build the corresponding extractnode and loadnode, then build
noderelation, streaminfo and groupinfo, and finally use FlinkSqlParse [...]
+To integrate extract and load into the InLong Sort mainstream, you need to
implement the semantics mentioned in the overview section: group, stream, node,
etc. The entry class of InLong Sort is in :
+```bash
+inlong-sort/sort-core/src/main/java/org/apache/inlong/sort/Entrance.java
+```
+How to integrate extract and load into InLong Sort can refer to the following
ut. First, build the corresponding extractnode and loadnode, then build
noderelation, streaminfo and groupinfo, and finally use FlinkSqlParser to
execute.
```java
public class MongoExtractToKafkaLoad extends AbstractTestBase {
diff --git a/docs/design_and_concept/how_to_write_plugin_dashboard.md
b/docs/design_and_concept/how_to_write_plugin_dashboard.md
index fe7ab6b3f..e7722c92e 100644
--- a/docs/design_and_concept/how_to_write_plugin_dashboard.md
+++ b/docs/design_and_concept/how_to_write_plugin_dashboard.md
@@ -3,17 +3,14 @@ title: Dashboard Plugin
sidebar_position: 4
---
-# Overview
-
-This article is aimed at InLong-Dashboard plug-in developers, trying to
describe the process of developing a Dashboard plug-in as comprehensively as
possible, helping developers quickly add a data storage LoadNode, and making
plug-in development easier.
-
-## Before
+## Overview
+This article is aimed at InLong Dashboard plug-in developers, trying to
describe the process of developing a Dashboard plug-in as comprehensively as
possible, helping developers quickly add a data storage LoadNode, and making
plug-in development easier.
The InLong Dashboard itself acts as a front-end console, built with the React
framework.
-## Extend a new sink
+## Extend a new Load Node
-In the `/inlong-dashboard/src/components/MetaData` directory, create a new
`StorageExample.tsx` file, and at the same time export the file in the
`index.ts` file in the current directory (refer to the existing LoadNode
writing method ), which completes a new sink named `Example`.
+In the `inlong-dashboard/src/components/MetaData` directory, create a new
`StorageExample.tsx` file, and at the same time export the file in the
`index.ts` file in the current directory (refer to the existing LoadNode
writing method ), which completes a new sink named `Example`.
````js
// export in index
diff --git a/docs/modules/dataproxy/overview.md
b/docs/modules/dataproxy/overview.md
index 4dce7fb79..c03d9cab3 100644
--- a/docs/modules/dataproxy/overview.md
+++ b/docs/modules/dataproxy/overview.md
@@ -1,218 +1,217 @@
----
-title: Overview
----
-
-Inlong-dataProxy belongs to the inlong proxy layer and is used for data
collection, reception and forwarding. Through format conversion, the data is
converted into TDMsg1 format that can be cached and processed by the cache layer
-InLong-dataProxy acts as a bridge from the InLong collection end to the InLong
buffer end. Dataproxy pulls the relationship between the business group id and
the corresponding topic name from the manager module, and internally manages
the producers of multiple topics
-The overall architecture of inlong-dataproxy is based on Apache Flume. On the
basis of this project, inlong-bus expands the source layer and sink layer, and
optimizes disaster tolerance forwarding, which improves the stability of the
system.
-
-## Architecture
-
-
-
-- The source layer opens port monitoring, which is realized through netty
server. The decoded data is sent to the channel layer
-- The channel layer has a selector, which is used to choose which type of
channel to go. If the memory is eventually full, the data will be processed.
-- The data of the channel layer will be forwarded through the sink layer. The
main purpose here is to convert the data to the TDMsg1 format and push it to
the cache layer (tube is more commonly used here)
-
-## DataProxy Configuration
-
-DataProxy supports configurable source-channel-sink, which is similar to
flume's configuration file structure. The configuration file name is such as
dataproxy-*.conf. Currently, dataproxy-pulsar.conf and dataproxy-tube.conf are
supported to distinguish different message middleware types. The specific type
can be specified when startup. The default (when not specified) ) using
dataproxy-pulsar.conf as configuration file.
-
-- Source configuration example:
-```shell
-agent1.sources.tcp-source.channels = ch-msg1 ch-msg2 ch-msg3 ch-more1 ch-more2
ch-more3 ch-msg5 ch-msg6 ch-msg7 ch-msg8 ch-msg9 ch-msg10 ch-transfer ch -Back
-Define the channel used in the source. Note that if the configuration below
this source uses the channel, it needs to be annotated here
-
-agent1.sources.tcp-source.type = org.apache.flume.source.SimpleTcpSource
-tcp resolution type definition, here provide the class name for instantiation,
SimpleTcpSource is mainly to initialize the configuration and start port
monitoring
-
-agent1.sources.tcp-source.msg-factory-name =
org.apache.flume.source.ServerMessageFactory
-Handler used for message structure analysis, and set read stream handler and
write stream handler
-
-agent1.sources.tcp-source.host = 0.0.0.0
-tcp ip binding monitoring, binding all network cards by default
-
-agent1.sources.tcp-source.port = 46801
-tcp port binding, port 46801 is bound by default
-
-agent1.sources.tcp-source.highWaterMark=2621440
-The concept of netty, set the netty high water level value
-
-agent1.sources.tcp-source.enableExceptionReturn=true
-The new function of v1.7 version, optional, the default is false, used to open
the exception channel, when an exception occurs, the data is written to the
exception channel to prevent other normal data transmission (the open source
version does not add this function), Details | Increase the local disk of
abnormal data landing
-
-agent1.sources.tcp-source.max-msg-length = 524288
-Limit the size of a single package, here if the compressed package is
transmitted, it is the compressed package size, the limit is 512KB
-
-agent1.sources.tcp-source.topic = test_token
-The default topic value, if the mapping relationship between groupId and topic
cannot be found, it will be sent to this topic
-
-agent1.sources.tcp-source.attr = m=9
-The default value of m is set, where the value of m is the version of inlong's
internal TdMsg protocol
-
-agent1.sources.tcp-source.connections = 5000
-Concurrent connections go online, new connections will be broken when the
upper limit is exceeded
-
-agent1.sources.tcp-source.max-threads = 64
-Netty thread pool work thread upper limit, generally recommended to choose
twice the cpu
-
-agent1.sources.tcp-source.receiveBufferSize = 524288
-Netty server tcp tuning parameters
-
-agent1.sources.tcp-source.sendBufferSize = 524288
-Netty server tcp tuning parameters
-
-agent1.sources.tcp-source.custom-cp = true
-Whether to use the self-developed channel process, the self-developed channel
process can select the alternate channel to send when the main channel is
blocked
-
-agent1.sources.tcp-source.selector.type =
org.apache.flume.channel.FailoverChannelSelector
-This channel selector is a self-developed channel selector, which is not much
different from the official website, mainly because of the channel master-slave
selection logic
-
-agent1.sources.tcp-source.selector.master = ch-msg5 ch-msg6 ch-msg7 ch-msg8
ch-msg9
-Specify the master channel, these channels will be preferentially selected for
data push. Those channels that are not in the master, transfer, fileMetric, and
slaMetric configuration items, but are in
-There are defined channels in channels, which are all classified as slave
channels. When the master channel is full, the slave channel will be selected.
Generally, the file channel type is recommended for the slave channel.
-
-agent1.sources.tcp-source.selector.transfer = ch-msg5 ch-msg6 ch-msg7 ch-msg8
ch-msg9
-Specify the transfer channel to accept the transfer type data. The transfer
here generally refers to the data pushed to the non-tube cluster, which is only
for forwarding, and it is reserved for subsequent functions.
-
-agent1.sources.tcp-source.selector.fileMetric = ch-back
-Specify the fileMetric channel to receive the metric data reported by the agent
-```
-
-- Channel configuration examples, memory channel:
-
-```shell
-agent1.channels.ch-more1.type = memory
-memory channel type
-
-agent1.channels.ch-more1.capacity = 10000000
-Memory channel queue size, the maximum number of messages that can be cached
-
-agent1.channels.ch-more1.keep-alive = 0
-
-agent1.channels.ch-more1.transactionCapacity = 20
-The maximum number of batches are processed in atomic operations, and the
memory channel needs to be locked when used, so there will be a batch process
to increase efficiency
-```
-
-- Channel configuration examples, file channel:
-
-```shell
-agent1.channels.ch-msg5.type = file
-file channel type
-
-agent1.channels.ch-msg5.capacity = 100000000
-The maximum number of messages that can be cached in a file channel
-
-agent1.channels.ch-msg5.maxFileSize = 1073741824
-file channel file maximum limit, the number of bytes
-
-agent1.channels.ch-msg5.minimumRequiredSpace = 1073741824
-The minimum free space of the disk where the file channel is located. Setting
this value can prevent the disk from being full
-
-agent1.channels.ch-msg5.checkpointDir = /data/work/file/ch-msg5/check
-file channel checkpoint path
-
-agent1.channels.ch-msg5.dataDirs = /data/work/file/ch-msg5/data
-file channel data path
-
-agent1.channels.ch-msg5.fsyncPerTransaction = false
-Whether to synchronize the disk for each atomic operation, it is recommended
to change it to false, otherwise it will affect the performance
-
-agent1.channels.ch-msg5.fsyncInterval = 5
-The time interval between data flush from memory to disk, in seconds
-```
-
-- Sink configuration example:
-
-```shell
-agent1.sinks.meta-sink-more1.channel = ch-msg1
-The upstream channel name of the sink
-
-agent1.sinks.meta-sink-more1.type = org.apache.flume.sink.MetaSink
-The sink class is implemented, where the message is implemented to push data
to the tube cluster
-
-agent1.sinks.meta-sink-more1.master-host-port-list =
-Tube cluster master node list
-
-agent1.sinks.meta-sink-more1.send_timeout = 30000
-Timeout limit when sending to tube
-
-agent1.sinks.meta-sink-more1.stat-interval-sec = 60
-Sink indicator statistics interval time, in seconds
-
-agent1.sinks.meta-sink-more1.thread-num = 8
-Sink class sends messages to the worker thread, 8 means to start 8 concurrent
threads
-
-agent1.sinks.meta-sink-more1.client-id-cache = true
-agent id cache, used to check the data reported by the agent to remove
duplicates
-
-agent1.sinks.meta-sink-more1.max-survived-time = 300000
-Maximum cache time
-
-agent1.sinks.meta-sink-more1.max-survived-size = 3000000
-Maximum number of caches
-```
-
-## Monitor Metrics configuration
-
- DataProxy provide monitor indicator based on JMX, user can implement the
code that read the metrics and report to user-defined monitor system.
-Source-module and Sink-module can add monitor metric class that is the
subclass of org.apache.inlong.commons.config.metrics.MetricItemSet, and
register it to MBeanServer. User-defined plugin can get module metric with JMX,
and report metric data to different monitor system.
-
- User can describe the configuration in the file "common.properties ". For
example:
-
-```shell
-metricDomains=DataProxy
-metricDomains.DataProxy.domainListeners=org.apache.inlong.dataproxy.metrics.prometheus.PrometheusMetricListener
-metricDomains.DataProxy.snapshotInterval=60000
-```
-
-- The JMX domain name of DataProxy is "DataProxy".
-- It is defined by the parameter "metricDomains".
-- The listeners of JMX domain is defined by the parameter
"metricDomains.$domainName.domainListeners".
-- The class names of the listeners is separated by the space char.
-- The listener class need to implement the interface
"org.apache.inlong.dataproxy.metrics.MetricListener".
-- The snapshot interval of the listeners is defined by the parameter
"metricDomains.$domainName.snapshotInterval", the parameter unit is
"millisecond".
-
-The method proto of org.apache.inlong.dataproxy.metrics.MetricListener is:
-```java
-public void snapshot(String domain, List itemValues);
-```
-
-The field of MetricItemValue.dimensions has these dimensions(The fields of
DataProxyMetricItem defined by the Annotation "@Dimension"):
-
-| property | description |
-| ---- | ---- |
-| clusterId | DataProxy cluster ID. |
-| sourceId | DataProxy source component name. |
-| sourceDataId | DataProxy source component data id, when source is a TCP
source, it will be port number. |
-| inlongGroupId | Inlong data group ID. |
-| inlongStreamId | Inlong data stream ID. |
-| sinkId | DataProxy sink component name. |
-| sinkDataId | DataProxy sink component data id, when sink is a pulsar
sink, it will be topic name. |
-
-The field of MetricItemValue.metrics has these metrics(The fields of
DataProxyMetricItem defined by the Annotation "@CountMetric"):
-
-| property | description |
-| ---- | ---- |
-| readSuccessCount | Successful event count reading from source component. |
-| readSuccessSize | Successful event body size reading from source
component. |
-| readFailCount | Failure event count reading from source component. |
-| readFailSize | Failure event body size reading from source component. |
-| sendCount | Event count sending to sink destination. |
-| sendSize | Event body size sending to sink destination. |
-| sendSuccessCount | Successful event count sending to sink destination. |
-| sendSuccessSize | Successful event body size sending to sink destination.
|
-| sendFailCount | Failure event count sending to sink destination. |
-| sendFailSize | Failure event body size sending to sink destination. |
-| sinkDuration | The unit is millisecond, the duration is between current
timepoint and the timepoint in sending to sink destination. |
-| nodeDuration | The unit is millisecond, the duration is between current
timepoint and the timepoint in getting event from source. |
-| wholeDuration | The unit is millisecond, the duration is between current
timepoint and the timepoint in generating event. |
-
-Monitor indicators have registered to MBeanServer, user can append JMX
parameters when running DataProxy, remote server can get monitor metrics with
RMI.
-```shell
--Dcom.sun.management.jmxremote
--Djava.rmi.server.hostname=127.0.0.1
--Dcom.sun.management.jmxremote.port=9999
--Dcom.sun.management.jmxremote.authenticate=false
--Dcom.sun.management.jmxremote.ssl=false
-```
+---
+title: Overview
+---
+
+InLong DataProxy mainly consists of connection convergence, routing, data
compression, and protocol conversion. DataProxy acts as a bridge from the
InLong Agent to the message queue, When the DataProxy pulls the metadata of the
data streams from the Manager module, the corresponding relationship between
the data streams and the topic name of the message queue is determined. When
DataProxy receives a message, it will first be sent to Memory Channel for
compression.
+And use the local Producer to send data to the back-end Cache layer (ie
message queue). When the message queue fails to send abnormally, DataProxy will
cache the message to the Disk Channel, the local disk.
+The overall architecture of InLong DataProxy is based on Apache Flume, which
extends the source layer and sinks layer. It optimizes disaster recovery
forwarding to improve the stability of the system.
+## Architecture
+
+
+
+- The source layer opens port monitoring, which is realized through netty
server. The decoded data is sent to the channel layer
+- The channel layer has a selector, which is used to choose which type of
channel to go. If the memory is eventually full, the data will be processed.
+- The data of the channel layer will be forwarded through the sink layer. The
main purpose here is to convert the data to the TDMsg1 format and push it to
the cache layer (tube is more commonly used here)
+
+## DataProxy Configuration
+
+DataProxy supports configurable source-channel-sink, which is similar to
flume's configuration file structure. The configuration file name is such as
dataproxy-*.conf. Currently, dataproxy-pulsar.conf and dataproxy-tube.conf are
supported to distinguish different message middleware types. The specific type
can be specified when startup. The default (when not specified) ) using
dataproxy-pulsar.conf as configuration file.
+
+- Source configuration example:
+```shell
+agent1.sources.tcp-source.channels = ch-msg1 ch-msg2 ch-msg3 ch-more1 ch-more2
ch-more3 ch-msg5 ch-msg6 ch-msg7 ch-msg8 ch-msg9 ch-msg10 ch-transfer ch -Back
+Define the channel used in the source. Note that if the configuration below
this source uses the channel, it needs to be annotated here
+
+agent1.sources.tcp-source.type = org.apache.flume.source.SimpleTcpSource
+tcp resolution type definition, here provide the class name for instantiation,
SimpleTcpSource is mainly to initialize the configuration and start port
monitoring
+
+agent1.sources.tcp-source.msg-factory-name =
org.apache.flume.source.ServerMessageFactory
+Handler used for message structure analysis, and set read stream handler and
write stream handler
+
+agent1.sources.tcp-source.host = 0.0.0.0
+tcp ip binding monitoring, binding all network cards by default
+
+agent1.sources.tcp-source.port = 46801
+tcp port binding, port 46801 is bound by default
+
+agent1.sources.tcp-source.highWaterMark=2621440
+The concept of netty, set the netty high water level value
+
+agent1.sources.tcp-source.enableExceptionReturn=true
+The new function of v1.7 version, optional, the default is false, used to open
the exception channel, when an exception occurs, the data is written to the
exception channel to prevent other normal data transmission (the open source
version does not add this function), Details | Increase the local disk of
abnormal data landing
+
+agent1.sources.tcp-source.max-msg-length = 524288
+Limit the size of a single package, here if the compressed package is
transmitted, it is the compressed package size, the limit is 512KB
+
+agent1.sources.tcp-source.topic = test_token
+The default topic value, if the mapping relationship between groupId and topic
cannot be found, it will be sent to this topic
+
+agent1.sources.tcp-source.attr = m=9
+The default value of m is set, where the value of m is the version of inlong's
internal TdMsg protocol
+
+agent1.sources.tcp-source.connections = 5000
+Concurrent connections go online, new connections will be broken when the
upper limit is exceeded
+
+agent1.sources.tcp-source.max-threads = 64
+Netty thread pool work thread upper limit, generally recommended to choose
twice the cpu
+
+agent1.sources.tcp-source.receiveBufferSize = 524288
+Netty server tcp tuning parameters
+
+agent1.sources.tcp-source.sendBufferSize = 524288
+Netty server tcp tuning parameters
+
+agent1.sources.tcp-source.custom-cp = true
+Whether to use the self-developed channel process, the self-developed channel
process can select the alternate channel to send when the main channel is
blocked
+
+agent1.sources.tcp-source.selector.type =
org.apache.flume.channel.FailoverChannelSelector
+This channel selector is a self-developed channel selector, which is not much
different from the official website, mainly because of the channel master-slave
selection logic
+
+agent1.sources.tcp-source.selector.master = ch-msg5 ch-msg6 ch-msg7 ch-msg8
ch-msg9
+Specify the master channel, these channels will be preferentially selected for
data push. Those channels that are not in the master, transfer, fileMetric, and
slaMetric configuration items, but are in
+There are defined channels in channels, which are all classified as slave
channels. When the master channel is full, the slave channel will be selected.
Generally, the file channel type is recommended for the slave channel.
+
+agent1.sources.tcp-source.selector.transfer = ch-msg5 ch-msg6 ch-msg7 ch-msg8
ch-msg9
+Specify the transfer channel to accept the transfer type data. The transfer
here generally refers to the data pushed to the non-tube cluster, which is only
for forwarding, and it is reserved for subsequent functions.
+
+agent1.sources.tcp-source.selector.fileMetric = ch-back
+Specify the fileMetric channel to receive the metric data reported by the agent
+```
+
+- Channel configuration examples, memory channel:
+
+```shell
+agent1.channels.ch-more1.type = memory
+memory channel type
+
+agent1.channels.ch-more1.capacity = 10000000
+Memory channel queue size, the maximum number of messages that can be cached
+
+agent1.channels.ch-more1.keep-alive = 0
+
+agent1.channels.ch-more1.transactionCapacity = 20
+The maximum number of batches are processed in atomic operations, and the
memory channel needs to be locked when used, so there will be a batch process
to increase efficiency
+```
+
+- Channel configuration examples, file channel:
+
+```shell
+agent1.channels.ch-msg5.type = file
+file channel type
+
+agent1.channels.ch-msg5.capacity = 100000000
+The maximum number of messages that can be cached in a file channel
+
+agent1.channels.ch-msg5.maxFileSize = 1073741824
+file channel file maximum limit, the number of bytes
+
+agent1.channels.ch-msg5.minimumRequiredSpace = 1073741824
+The minimum free space of the disk where the file channel is located. Setting
this value can prevent the disk from being full
+
+agent1.channels.ch-msg5.checkpointDir = /data/work/file/ch-msg5/check
+file channel checkpoint path
+
+agent1.channels.ch-msg5.dataDirs = /data/work/file/ch-msg5/data
+file channel data path
+
+agent1.channels.ch-msg5.fsyncPerTransaction = false
+Whether to synchronize the disk for each atomic operation, it is recommended
to change it to false, otherwise it will affect the performance
+
+agent1.channels.ch-msg5.fsyncInterval = 5
+The time interval between data flush from memory to disk, in seconds
+```
+
+- Sink configuration example:
+
+```shell
+agent1.sinks.meta-sink-more1.channel = ch-msg1
+The upstream channel name of the sink
+
+agent1.sinks.meta-sink-more1.type = org.apache.flume.sink.MetaSink
+The sink class is implemented, where the message is implemented to push data
to the tube cluster
+
+agent1.sinks.meta-sink-more1.master-host-port-list =
+Tube cluster master node list
+
+agent1.sinks.meta-sink-more1.send_timeout = 30000
+Timeout limit when sending to tube
+
+agent1.sinks.meta-sink-more1.stat-interval-sec = 60
+Sink indicator statistics interval time, in seconds
+
+agent1.sinks.meta-sink-more1.thread-num = 8
+Sink class sends messages to the worker thread, 8 means to start 8 concurrent
threads
+
+agent1.sinks.meta-sink-more1.client-id-cache = true
+agent id cache, used to check the data reported by the agent to remove
duplicates
+
+agent1.sinks.meta-sink-more1.max-survived-time = 300000
+Maximum cache time
+
+agent1.sinks.meta-sink-more1.max-survived-size = 3000000
+Maximum number of caches
+```
+
+## Monitor Metrics configuration
+
+ DataProxy provide monitor indicator based on JMX, user can implement the
code that read the metrics and report to user-defined monitor system.
+Source-module and Sink-module can add monitor metric class that is the
subclass of org.apache.inlong.commons.config.metrics.MetricItemSet, and
register it to MBeanServer. User-defined plugin can get module metric with JMX,
and report metric data to different monitor system.
+
+ User can describe the configuration in the file "common.properties ". For
example:
+
+```shell
+metricDomains=DataProxy
+metricDomains.DataProxy.domainListeners=org.apache.inlong.dataproxy.metrics.prometheus.PrometheusMetricListener
+metricDomains.DataProxy.snapshotInterval=60000
+```
+
+- The JMX domain name of DataProxy is "DataProxy".
+- It is defined by the parameter "metricDomains".
+- The listeners of JMX domain is defined by the parameter
"metricDomains.$domainName.domainListeners".
+- The class names of the listeners is separated by the space char.
+- The listener class need to implement the interface
"org.apache.inlong.dataproxy.metrics.MetricListener".
+- The snapshot interval of the listeners is defined by the parameter
"metricDomains.$domainName.snapshotInterval", the parameter unit is
"millisecond".
+
+The method proto of org.apache.inlong.dataproxy.metrics.MetricListener is:
+```java
+public void snapshot(String domain, List itemValues);
+```
+
+The field of MetricItemValue.dimensions has these dimensions(The fields of
DataProxyMetricItem defined by the Annotation "@Dimension"):
+
+| property | description |
+| ---- | ---- |
+| clusterId | DataProxy cluster ID. |
+| sourceId | DataProxy source component name. |
+| sourceDataId | DataProxy source component data id, when source is a TCP
source, it will be port number. |
+| inlongGroupId | Inlong data group ID. |
+| inlongStreamId | Inlong data stream ID. |
+| sinkId | DataProxy sink component name. |
+| sinkDataId | DataProxy sink component data id, when sink is a pulsar
sink, it will be topic name. |
+
+The field of MetricItemValue.metrics has these metrics(The fields of
DataProxyMetricItem defined by the Annotation "@CountMetric"):
+
+| property | description |
+| ---- | ---- |
+| readSuccessCount | Successful event count reading from source component. |
+| readSuccessSize | Successful event body size reading from source
component. |
+| readFailCount | Failure event count reading from source component. |
+| readFailSize | Failure event body size reading from source component. |
+| sendCount | Event count sending to sink destination. |
+| sendSize | Event body size sending to sink destination. |
+| sendSuccessCount | Successful event count sending to sink destination. |
+| sendSuccessSize | Successful event body size sending to sink destination.
|
+| sendFailCount | Failure event count sending to sink destination. |
+| sendFailSize | Failure event body size sending to sink destination. |
+| sinkDuration | The unit is millisecond, the duration is between current
timepoint and the timepoint in sending to sink destination. |
+| nodeDuration | The unit is millisecond, the duration is between current
timepoint and the timepoint in getting event from source. |
+| wholeDuration | The unit is millisecond, the duration is between current
timepoint and the timepoint in generating event. |
+
+Monitor indicators have registered to MBeanServer, user can append JMX
parameters when running DataProxy, remote server can get monitor metrics with
RMI.
+```shell
+-Dcom.sun.management.jmxremote
+-Djava.rmi.server.hostname=127.0.0.1
+-Dcom.sun.management.jmxremote.port=9999
+-Dcom.sun.management.jmxremote.authenticate=false
+-Dcom.sun.management.jmxremote.ssl=false
+```
diff --git a/docs/modules/sort/example.md b/docs/modules/sort/example.md
index 8b8fa75ce..9a3317c65 100644
--- a/docs/modules/sort/example.md
+++ b/docs/modules/sort/example.md
@@ -1,11 +1,11 @@
---
-title: DataStream Example
+title: Example
sidebar_position: 3
---
## Overview
-To make it easier for you to create InLong-Sort jobs, here we list some data
stream configuration examples.
+To make it easier for you to create InLong Sort jobs, here we list some data
stream configuration examples.
The following will introduce SQL, Dashboard, Manager Client Tools methods to
use Inlong Sort.
## Environment Requirements
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_extend_data_node_for_manager.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_extend_data_node_for_manager.md
index f06171a7e..64bc2e0eb 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_extend_data_node_for_manager.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_extend_data_node_for_manager.md
@@ -1,18 +1,16 @@
---
-title: Manager 数据节点插件
+title: Manager 插件
sidebar_position: 5
---
## 总览
Inlong 设计初衷即是为了在不同数据源之间创建数据流，截止目前，Inlong 已经支持多种常用数据源的读取和写入，如 **MySQL**,
**Apache Kafka**, **ClickHouse** 等，
-详细内容可参考 [数据节点](data_node/extract_node/overview.md)。
-我们会在未来支持更多的常用数据源，故本文会简短介绍如何在现有框架下扩展数据源。
+详细内容可参考 [数据节点](data_node/extract_node/overview.md)。 InLong 的每种数据节点，支持通过
Manager 提供统一的管理，以简化用户的使用。
+本文介绍通过 Manager 如何扩展一个新的数据节点，实现提供服务。
## 扩展读取节点
-以 **MySQL_BINLOG** 为例，介绍如何在 Inlong 框架下扩展读取节点。
-
- 首先需要在 Sort 组件内支持该数据源，详情参考 [Sort
插件](design_and_concept/how_to_extend_data_node_for_sort.md)
- 在枚举类`org.apache.inlong.common.enums.TaskTypeEnum`中增加对应的枚举
- 在枚举类`org.apache.inlong.manager.common.enums.SourceType`中同样增加对应枚举
@@ -22,8 +20,6 @@ Inlong 设计初衷即是为了在不同数据源之间创建数据流，截止
## 扩展写入节点
-扩展写入节点的方式与读取节点类似
-
- 首先需要在 Sort 组件内支持该数据源，详情参考 [Sort
插件](design_and_concept/how_to_extend_data_node_for_sort.md)
- 在枚举类`org.apache.inlong.manager.common.enums.SinkType`中增加对应枚举
- 在`org.apache.inlong.manager.common.pojo.sink`路径下创建文件夹, 创建对应实体类
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_extend_data_node_for_sort.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_extend_data_node_for_sort.md
index b26fe5179..3263a4ca6 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_extend_data_node_for_sort.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_extend_data_node_for_sort.md
@@ -5,9 +5,11 @@ sidebar_position: 3
## 总览
-InLong Sort 是一个 ETL 系统，当前支持的 extract 或 load 包括
FileSystemExtractNode，KafkaExtractNode，MongoExtractNode，MySqlExtractNode，OracleExtractNode，PostgresExtractNode，PulsarExtractNode，SqlServerExtractNode，ClickHouseLoadNode
，ElasticsearchLoadNode，FileSystemLoadNode，GreenplumLoadNode，HbaseLoadNode，HiveLoadNode，IcebergLoadNode，KafkaLoadNode，MySqlLoadNode，OracleLoadNode，PostgresLoadNode，SqlServerLoadNode，TDSQLPostgresLoadNode
等。InLong Sort是基于Flink SQL的ETL方案，Flink SQL强大的表达能力带来的高可扩展性、灵活性，基本上 Flink SQ [...]
+InLong Sort 是一个基于 Apache Flink SQL 的 ETL 服务。Flink SQL 强大的表达能力带来的高可扩展性、灵活性，基本上
Flink SQL 支持的语意，InLong Sort 都支持。
+当 Flink SQL 内置的函数不满足需求时，还可通过 UDF 来扩展。这对于曾经使用过 SQL 尤其是 Flink SQL 的开发者非常友好。
-本文介绍如何在 InLong Sort 中扩展一个新的 source（在 InLong 中抽象为 Extract Node）或一个新的
sink（在InLong中抽象为 Load Node ）。在弄清楚 InLong 的架构之后，就可以明白 Source 与 Extract Node
如何对应，Sink 与 Load Node 如何对应。InLong Sort 架构的 UML 对象关系图如下：
+本文介绍如何在 InLong Sort 中扩展一个新的 source（在 InLong 中抽象为 Extract Node）或一个新的
sink（在InLong中抽象为 Load Node ）。
+InLong Sort 架构的 UML 对象关系图如下：
@@ -43,7 +45,11 @@ InLong Sort 是一个 ETL 系统，当前支持的 extract 或 load 包括 FileS
扩展一个 ExtractNode 分为三个步骤：
-**第一步**：继承 ExtractNode 类，类的位置在
`inlong/inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/ExtractNode.java`；在实现的
ExtractNode 中指定 connecter；
+**第一步**：继承 ExtractNode 类，类的位置在:
+```bash
+inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/ExtractNode.java
+```
+在实现的 ExtractNode 中指定 connector；
```Java
// 继承 ExtractNode 类，实现具体的类，例如 MongoExtractNode
@@ -88,13 +94,18 @@ public abstract class ExtractNode implements Node{...}
public interface Node {...}
```
-**第三步**：扩展 Sort
Connector，查看此（`/inlong/inlong-sort/sort-connectors/mongodb-cdc`）目录下是否已经存在对应的
connector。如果没有，则需要参考 Flink 官方文档 [DataStream Connectors
](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/datastream/overview/#datastream-connectors)
来扩展，调用已有的
Flink-connector（例如`inlong/inlong-sort/sort-connectors/mongodb-cdc`）或自行实现相关的
connecter 均可。
+**第三步**：扩展 Sort
Connector，查看此（`inlong-sort/sort-connectors/mongodb-cdc`）目录下是否已经存在对应的
connector。如果没有，则需要参考 Flink 官方文档 [DataStream Connectors
](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/datastream/overview/#datastream-connectors)
来扩展，
+调用已有的 Flink-connector（例如`inlong-sort/sort-connectors/mongodb-cdc`）或自行实现相关的
connector 均可。
## 扩展 Load Node
扩展一个 LoadNode 分为三个步骤：
-**第一步**：继承 LoadNode 类，类的位置在
`inlong/inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/LoadNode.java`；在实现的LoadNode
中指定 connecter；
+**第一步**：继承 LoadNode 类，类的位置在:
+```bash
+inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/LoadNode.java
+```
+在实现的LoadNode 中指定 connector；
```java
// 继承 LoadNode 类，实现具体的类，例如 KafkaLoadNode
@@ -155,12 +166,15 @@ public abstract class LoadNode implements Node{...}
public interface Node {...}
```
-**第三步**：扩展 Sort Connector，Kafka 的 sort connector 在
`inlong/inlong-sort/sort-connectors/kafka` 目录下。
+**第三步**：扩展 Sort Connector，Kafka 的 sort connector 在
`inlong-sort/sort-connectors/kafka` 目录下。
-## 集成 Extract 和 Load 到 InLong Sort 主流程
+## 集成 Extract 和 Load Node 到 InLong Sort 主流程
将 Extract 和 Load 集成到 InLong Sort 主流程中，需要构建总览小节中提到的语意：Group、Stream、Node 等。
-InLong Sort
的入口类在`inlong-sort/sort-core/src/main/java/org/apache/inlong/sort/Entrance.java`。
+InLong Sort 的入口类在:
+```bash
+inlong-sort/sort-core/src/main/java/org/apache/inlong/sort/Entrance.java
+```
Extract 和 Load 如何集成至 InLong Sort，可参考下面的 UT，首先构建对应的 ExtractNode、LoadNode，再构建
NodeRelation、StreamInfo、GroupInfo，最后通过 FlinkSqlParser 执行。
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_dashboard.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_dashboard.md
index 34e023ca8..59d1cfc6b 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_dashboard.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/how_to_write_plugin_dashboard.md
@@ -3,17 +3,14 @@ title: Dashboard 插件
sidebar_position: 4
---
-# 总览
-
-本文面向 InLong-Dashboard 插件开发人员，尝试尽可能全面地阐述开发一个 Dashboard
插件所经过的历程，帮助开发者快速新增一个数据存储流向，让插件开发变得简单。
-
-## 开发之前
+## 总览
+本文面向 InLong Dashboard 插件开发人员，尝试尽可能全面地阐述开发一个 Dashboard 插件所经过的历程，帮助开发者快速新增一个
Load Node，让插件开发变得简单。
InLong Dashboard 本身作为前端控制台，采用 React 框架构建。
-## 集成新的 LoadNode 到 InLong-Dashboard 的主流程
+## 集成新的 Load Node 到 InLong Dashboard 的主流程
-在 `/inlong-dashboard/src/components/MetaData` 目录下，新建一个
`StorageExampleNode.tsx` 文件，同时讲该文件在当前目录的 `index.ts` 文件内部进行导出（可参考已有 LoadNode
的写法），这样便完成了新增一种名为 `ExampleNode` 的 LoadNode，接下来，我们将介绍怎么定义该 LoadNode 的内部结构。
+在 `inlong-dashboard/src/components/MetaData` 目录下，新建一个 `StorageExampleNode.tsx`
文件，同时讲该文件在当前目录的 `index.ts` 文件内部进行导出（可参考已有 LoadNode 的写法），这样便完成了新增一种名为
`ExampleNode` 的 LoadNode，接下来，我们将介绍怎么定义该 LoadNode 的内部结构。
在 LoadNode 的定义中，可通过 `import type { GetStorageFormFieldsType,
GetStorageColumnsType } from '@/utils/metaData';`
文件中的类型声明查看我们约定的统一规范，这里我们展示了一个最简单的 LoadNode 定义（重点在于`--关注点--`标签内部）:
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/dataproxy/overview.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/dataproxy/overview.md
index d590c3b1f..1917508d9 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/dataproxy/overview.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/dataproxy/overview.md
@@ -1,215 +1,215 @@
----
-title: 总览
----
-
-InLong-dataProxy属于inlong proxy层，用于数据的汇集接收以及转发。通过格式转换，将数据转为cache层可以缓存处理的TDMsg1格式
-InLong-dataProxy充当了InLong采集端到InLong缓冲端的桥梁，dataproxy从manager模块拉取业务id与对应topic名称的关系，内部管理多个topic的生产者
-当dataproxy收到消息时，会首先缓存到本地的Channel中，并使用本地的producer往后端即cache层发送数据
-InLong-dataProxy整体架构基于Apache
Flume。inlong-dataproxy在该项目的基础上，扩展了source层和sink层，并对容灾转发做了优化处理，提升了系统的稳定性。
-
-## 架构
-
-
-
-- Source层开启端口监听，通过netty server实现。解码之后的数据发到channel层
-- channel层有一个selector，用于选择走哪种类型的channel，如果memory最终满了，会对数据做落地处理
-- channel层的数据会通过sink层做转发，这里主要是将数据转为TDMsg1的格式，并推送到cache层（这里用的比较多的是tube）
-
-
-## DataProxy功能配置说明
-
-DataProxy 支持配置化的 source-channel-sink，配置方式与 flume 的配置文件结构类似。配置文件放在
dataproxy-*.conf 文件中，目前支持 dataproxy-pulsar.conf 和 dataproxy-tube.conf
两种，用于区分不同的中间件类型，具体类型可以在启动时指定，默认（未指定时）使用 dataproxy-pulsar.conf 作为配置文件
-
-- Source配置示例：
-
-```shell
-agent1.sources.tcp-source.channels = ch-msg1 ch-msg2 ch-msg3 ch-more1 ch-more2
ch-more3 ch-msg5 ch-msg6 ch-msg7 ch-msg8 ch-msg9 ch-msg10 ch-transfer ch-back
-定义source中使用到的channel，注意此source下面的配置如果有使用到channel，均需要在此注释
-
-agent1.sources.tcp-source.type = org.apache.flume.source.SimpleTcpSource
-tcp解析类型定义，这里提供类名用于实例化，SimpleTcpSource主要是初始化配置并启动端口监听
-
-agent1.sources.tcp-source.msg-factory-name =
org.apache.flume.source.ServerMessageFactory
-用于构造消息解析的handler，并设置read stream handler和write stream handler
-
-agent1.sources.tcp-source.host = 0.0.0.0
-tcp ip绑定监听，默认绑定所有网卡
-
-agent1.sources.tcp-source.port = 46801
-tcp 端口绑定，默认绑定46801端口
-
-agent1.sources.tcp-source.highWaterMark=2621440
-netty概念，设置netty高水位值
-
-agent1.sources.tcp-source.max-msg-length = 524288
-限制单个包大小，这里如果传输的是压缩包，则是压缩包大小，限制512KB
-
-agent1.sources.tcp-source.topic = test_token
-默认topic值，如果groupId和topic的映射关系找不到，则发送到此topic中
-
-agent1.sources.tcp-source.attr = m=9
-默认m值设置，这里的m值是inlong内部TdMsg协议的版本
-
-agent1.sources.tcp-source.connections = 5000
-并发连接上线，超过上限值时会对新连接做断链处理
-
-agent1.sources.tcp-source.max-threads = 64
-netty线程池工作线程上限，一般推荐选择cpu的两倍
-
-agent1.sources.tcp-source.receiveBufferSize = 524288
-netty server tcp调优参数
-
-agent1.sources.tcp-source.sendBufferSize = 524288
-netty server tcp调优参数
-
-agent1.sources.tcp-source.custom-cp = true
-是否使用自研的channel process，自研channel process可在主channel阻塞时，选择备用channel发送
-
-agent1.sources.tcp-source.selector.type =
org.apache.flume.channel.FailoverChannelSelector
-这个channel selector就是自研的channel selector，和官网的差别不大，主要是有channel主从选择逻辑
-
-agent1.sources.tcp-source.selector.master = ch-msg5 ch-msg6 ch-msg7 ch-msg8
ch-msg9
-指定master
channel，这些channel会被优先选择用于数据推送。那些不在master、transfer、fileMetric、slaMetric配置项里的channel，但在
-channels里面有定义的channel，统归为slave channel，当master channel都被占满时，就会选择使用slave
channel，slave channel一般建议使用file channel类型
-
-agent1.sources.tcp-source.selector.transfer = ch-msg5 ch-msg6 ch-msg7 ch-msg8
ch-msg9
-指定transfer
channel，承接transfer类型的数据，这里的transfer一般是指推送到非tube集群的数据，仅做转发，这里预留出来供后续功能使用
-
-agent1.sources.tcp-source.selector.fileMetric = ch-back
-指定fileMetric channel，用于接收agent上报的指标数据
-```
-
-- Channel配置示例，memory channel：
-
-```shell
-agent1.channels.ch-more1.type = memory
-memory channel类型
-
-agent1.channels.ch-more1.capacity = 10000000
-memory channel 队列大小，可缓存最大消息条数
-
-agent1.channels.ch-more1.keep-alive = 0
-
-agent1.channels.ch-more1.transactionCapacity = 20
-原子操作时批量处理最大条数，memory channel使用时需要用到加锁，因此会有批处理流程增加效率
-```
-
-- Channel配置示例，file channel：
-
-```shell
-agent1.channels.ch-msg5.type = file
-file channel类型
-
-agent1.channels.ch-msg5.capacity = 100000000
-file channel最大可缓存消息条数
-
-agent1.channels.ch-msg5.maxFileSize = 1073741824
-file channel文件最大上限，字节数
-
-agent1.channels.ch-msg5.minimumRequiredSpace = 1073741824
-file channel所在磁盘最小可用空间，设置此值可以防止磁盘写满
-
-agent1.channels.ch-msg5.checkpointDir = /data/work/file/ch-msg5/check
-file channel checkpoint路径
-
-agent1.channels.ch-msg5.dataDirs = /data/work/file/ch-msg5/data
-file channel数据路径
-
-agent1.channels.ch-msg5.fsyncPerTransaction = false
-是否对每个原子操作做同步磁盘，建议改false，否则会对性能有影响
-
-agent1.channels.ch-msg5.fsyncInterval = 5
-数据从内存flush到磁盘的时间间隔，单位秒
-```
-
-- Sink配置示例：
-
-```shell
-agent1.sinks.meta-sink-more1.channel = ch-msg1
-sink的上游channel名称
-
-agent1.sinks.meta-sink-more1.type = org.apache.flume.sink.MetaSink
-sink类实现，此处实现消息向tube集群推送数据
-
-agent1.sinks.meta-sink-more1.master-host-port-list =
-tube集群master节点列表
-
-agent1.sinks.meta-sink-more1.send_timeout = 30000
-发送到tube时超时时间限制
-
-agent1.sinks.meta-sink-more1.stat-interval-sec = 60
-sink指标统计间隔时间，单位秒
-
-agent1.sinks.meta-sink-more1.thread-num = 8
-Sink类发送消息的工作线程，8表示启动8个并发线程
-
-agent1.sinks.meta-sink-more1.client-id-cache = true
-agent id缓存，用于检查agent上报数据去重
-
-agent1.sinks.meta-sink-more1.max-survived-time = 300000
-缓存最大时间
-
-agent1.sinks.meta-sink-more1.max-survived-size = 3000000
-缓存最大个数
-```
-
-## 监控指标配置
-
-
DataProxy提供了JMX方式的监控指标Listener能力，用户可以实现MetricListener接口，注册后可以定期接收监控指标，用户选择将指标上报自定义的监控系统。Source和Sink模块可以通过将指标数据统计到org.apache.inlong.commons.config.metrics.MetricItemSet的子类中，并注册到MBeanServer。用户自定义的MetricListener通过JMX方式收集指标数据并上报到外部监控系统
-
- 用户能在配置文件common.propetiese增加如下配置，例如：
-
-```shell
-metricDomains=DataProxy
-metricDomains.DataProxy.domainListeners=org.apache.inlong.dataproxy.metrics.prometheus.PrometheusMetricListener
-metricDomains.DataProxy.snapshotInterval=60000
-```
-
-- 统一的JMX域名：DataProxy，并定义在参数metricDomains下；自定义的Source、Sink等组件也可以上报到不同的JMX域名。
--
对一个JMX域名的监控指标MetricListener可以配置在metricDomains.$domainName.domainListeners参数里，可以配置多个，用空格分隔类名。
--
这些监控指标MetricListener需要实现接口：org.apache.inlong.dataproxy.metrics.MetricListener。
-- 快照参数：metricDomains.$domainName.snapshotInterval，定义拉取一次监控指标数据的间隔时间，参数单位是毫秒。
-
-org.apache.inlong.dataproxy.metrics.MetricListener接口的方法原型：
-```java
-public void snapshot(String domain, List<MetricItemValue> itemValues);
-```
-
-监控指标项的MetricItemValue.dimensions有这些维度(DataProxyMetricItem的这些字段通过注解Annotation
"@Dimension"定义):
-
-| property | description |
-| ---- | ---- |
-| clusterId | DataProxy集群ID |
-| sourceId| DataProxy的Source组件名 |
-| sourceDataId| DataProxy的Source组件数据流ID，如果Source是一个TCPSource，那么这个ID会是一个端口号 |
-| inlongGroupId| Inlong数据ID |
-| inlongStreamId| Inlong数据流ID |
-| sinkId| DataProxy的Sink组件名 |
-| sinkDataId| DataProxy的Sink组件数据流ID，如果Sink是一个Pulsar发送组件，这个ID会是一个Topic名。 |
-
-监控指标项的MetricItemValue.metrics有这些指标(DataProxyMetricItem的这些字段通过注解Annotation
"@CountMetric"定义):
-
-| property | description |
-| ---- | ---- |
-| readSuccessCount | 接收成功条数 |
-| readSuccessSize | 接收成功大小，单位：byte |
-| readFailCount | 接收失败条数 |
-| readFailSize | 接收失败大小，单位：byte |
-| sendCount | 发送条数 |
-| sendSize | 发送大小，单位：byte |
-| sendSuccessCount | 发送成功条数 |
-| sendSuccessSize | 发送成功大小，单位：byte |
-| sendFailCount | 发送失败条数 |
-| sendFailSize | 发送失败大小，单位：byte |
-| sinkDuration | 发送成功回调时间和发送开始时间的时间差，用于评估目标集群的处理时延和健康状况，单位：毫秒 |
-| nodeDuration | 发送成功回调时间和接收成功时间的时间差，用于评估DataProxy内部处理耗时和健康状况，单位：毫秒 |
-| wholeDuration | 发送成功回调时间和事件生成时间的时间差，单位：毫秒 |
-
-监控指标已经注册到MBeanServer，用户可以在DataProxy的启动参数中增加如下类似JMX定义（端口和鉴权根据情况进行调整），实现监控指标从远端采集。
-```shell
- -Dcom.sun.management.jmxremote
- -Djava.rmi.server.hostname=127.0.0.1
- -Dcom.sun.management.jmxremote.port=9999
- -Dcom.sun.management.jmxremote.authenticate=false
- -Dcom.sun.management.jmxremote.ssl=false
-```
+---
+title: 总览
+---
+
+InLong DataProxy 主要有连接收敛、路由、数据压缩和协议转换等作用。DataProxy 充当了 InLong 采集端到消息队列的桥梁，
+当 DataProxy 从 Manager 模块拉取数据流元数据后，数据流和消息队列 Topic 名称对应关系也就确定了。当 DataProxy
收到消息时，会首先发送到 Memory Channel 中进行压缩，
+并使用本地的 Producer 往后端 Cache 层（即消息队列）发送数据。当消息队列异常出现发送失败时，DataProxy 会将消息缓存到 Disk
Channel，也就是本地磁盘中。
+InLong DataProxy 整体架构基于 Apache Flume，扩展了 Source 层和 Sink
层，并对容灾转发做了优化处理，提升了系统的稳定性。
+
+## 架构
+
+
+
+- Source层开启端口监听，通过netty server实现。解码之后的数据发到channel层
+- channel层有一个selector，用于选择走哪种类型的channel，如果memory最终满了，会对数据做落地处理
+- channel层的数据会通过sink层做转发，这里主要是将数据转为TDMsg1的格式，并推送到cache层（这里用的比较多的是tube）
+
+
+## DataProxy功能配置说明
+
+DataProxy 支持配置化的 source-channel-sink，配置方式与 flume 的配置文件结构类似。配置文件放在
dataproxy-*.conf 文件中，目前支持 dataproxy-pulsar.conf 和 dataproxy-tube.conf
两种，用于区分不同的中间件类型，具体类型可以在启动时指定，默认（未指定时）使用 dataproxy-pulsar.conf 作为配置文件
+
+- Source配置示例：
+
+```shell
+agent1.sources.tcp-source.channels = ch-msg1 ch-msg2 ch-msg3 ch-more1 ch-more2
ch-more3 ch-msg5 ch-msg6 ch-msg7 ch-msg8 ch-msg9 ch-msg10 ch-transfer ch-back
+定义source中使用到的channel，注意此source下面的配置如果有使用到channel，均需要在此注释
+
+agent1.sources.tcp-source.type = org.apache.flume.source.SimpleTcpSource
+tcp解析类型定义，这里提供类名用于实例化，SimpleTcpSource主要是初始化配置并启动端口监听
+
+agent1.sources.tcp-source.msg-factory-name =
org.apache.flume.source.ServerMessageFactory
+用于构造消息解析的handler，并设置read stream handler和write stream handler
+
+agent1.sources.tcp-source.host = 0.0.0.0
+tcp ip绑定监听，默认绑定所有网卡
+
+agent1.sources.tcp-source.port = 46801
+tcp 端口绑定，默认绑定46801端口
+
+agent1.sources.tcp-source.highWaterMark=2621440
+netty概念，设置netty高水位值
+
+agent1.sources.tcp-source.max-msg-length = 524288
+限制单个包大小，这里如果传输的是压缩包，则是压缩包大小，限制512KB
+
+agent1.sources.tcp-source.topic = test_token
+默认topic值，如果groupId和topic的映射关系找不到，则发送到此topic中
+
+agent1.sources.tcp-source.attr = m=9
+默认m值设置，这里的m值是inlong内部TdMsg协议的版本
+
+agent1.sources.tcp-source.connections = 5000
+并发连接上线，超过上限值时会对新连接做断链处理
+
+agent1.sources.tcp-source.max-threads = 64
+netty线程池工作线程上限，一般推荐选择cpu的两倍
+
+agent1.sources.tcp-source.receiveBufferSize = 524288
+netty server tcp调优参数
+
+agent1.sources.tcp-source.sendBufferSize = 524288
+netty server tcp调优参数
+
+agent1.sources.tcp-source.custom-cp = true
+是否使用自研的channel process，自研channel process可在主channel阻塞时，选择备用channel发送
+
+agent1.sources.tcp-source.selector.type =
org.apache.flume.channel.FailoverChannelSelector
+这个channel selector就是自研的channel selector，和官网的差别不大，主要是有channel主从选择逻辑
+
+agent1.sources.tcp-source.selector.master = ch-msg5 ch-msg6 ch-msg7 ch-msg8
ch-msg9
+指定master
channel，这些channel会被优先选择用于数据推送。那些不在master、transfer、fileMetric、slaMetric配置项里的channel，但在
+channels里面有定义的channel，统归为slave channel，当master channel都被占满时，就会选择使用slave
channel，slave channel一般建议使用file channel类型
+
+agent1.sources.tcp-source.selector.transfer = ch-msg5 ch-msg6 ch-msg7 ch-msg8
ch-msg9
+指定transfer
channel，承接transfer类型的数据，这里的transfer一般是指推送到非tube集群的数据，仅做转发，这里预留出来供后续功能使用
+
+agent1.sources.tcp-source.selector.fileMetric = ch-back
+指定fileMetric channel，用于接收agent上报的指标数据
+```
+
+- Channel配置示例，memory channel：
+
+```shell
+agent1.channels.ch-more1.type = memory
+memory channel类型
+
+agent1.channels.ch-more1.capacity = 10000000
+memory channel 队列大小，可缓存最大消息条数
+
+agent1.channels.ch-more1.keep-alive = 0
+
+agent1.channels.ch-more1.transactionCapacity = 20
+原子操作时批量处理最大条数，memory channel使用时需要用到加锁，因此会有批处理流程增加效率
+```
+
+- Channel配置示例，file channel：
+
+```shell
+agent1.channels.ch-msg5.type = file
+file channel类型
+
+agent1.channels.ch-msg5.capacity = 100000000
+file channel最大可缓存消息条数
+
+agent1.channels.ch-msg5.maxFileSize = 1073741824
+file channel文件最大上限，字节数
+
+agent1.channels.ch-msg5.minimumRequiredSpace = 1073741824
+file channel所在磁盘最小可用空间，设置此值可以防止磁盘写满
+
+agent1.channels.ch-msg5.checkpointDir = /data/work/file/ch-msg5/check
+file channel checkpoint路径
+
+agent1.channels.ch-msg5.dataDirs = /data/work/file/ch-msg5/data
+file channel数据路径
+
+agent1.channels.ch-msg5.fsyncPerTransaction = false
+是否对每个原子操作做同步磁盘，建议改false，否则会对性能有影响
+
+agent1.channels.ch-msg5.fsyncInterval = 5
+数据从内存flush到磁盘的时间间隔，单位秒
+```
+
+- Sink配置示例：
+
+```shell
+agent1.sinks.meta-sink-more1.channel = ch-msg1
+sink的上游channel名称
+
+agent1.sinks.meta-sink-more1.type = org.apache.flume.sink.MetaSink
+sink类实现，此处实现消息向tube集群推送数据
+
+agent1.sinks.meta-sink-more1.master-host-port-list =
+tube集群master节点列表
+
+agent1.sinks.meta-sink-more1.send_timeout = 30000
+发送到tube时超时时间限制
+
+agent1.sinks.meta-sink-more1.stat-interval-sec = 60
+sink指标统计间隔时间，单位秒
+
+agent1.sinks.meta-sink-more1.thread-num = 8
+Sink类发送消息的工作线程，8表示启动8个并发线程
+
+agent1.sinks.meta-sink-more1.client-id-cache = true
+agent id缓存，用于检查agent上报数据去重
+
+agent1.sinks.meta-sink-more1.max-survived-time = 300000
+缓存最大时间
+
+agent1.sinks.meta-sink-more1.max-survived-size = 3000000
+缓存最大个数
+```
+
+## 监控指标配置
+
+
DataProxy提供了JMX方式的监控指标Listener能力，用户可以实现MetricListener接口，注册后可以定期接收监控指标，用户选择将指标上报自定义的监控系统。Source和Sink模块可以通过将指标数据统计到org.apache.inlong.commons.config.metrics.MetricItemSet的子类中，并注册到MBeanServer。用户自定义的MetricListener通过JMX方式收集指标数据并上报到外部监控系统
+
+ 用户能在配置文件common.propetiese增加如下配置，例如：
+
+```shell
+metricDomains=DataProxy
+metricDomains.DataProxy.domainListeners=org.apache.inlong.dataproxy.metrics.prometheus.PrometheusMetricListener
+metricDomains.DataProxy.snapshotInterval=60000
+```
+
+- 统一的JMX域名：DataProxy，并定义在参数metricDomains下；自定义的Source、Sink等组件也可以上报到不同的JMX域名。
+-
对一个JMX域名的监控指标MetricListener可以配置在metricDomains.$domainName.domainListeners参数里，可以配置多个，用空格分隔类名。
+-
这些监控指标MetricListener需要实现接口：org.apache.inlong.dataproxy.metrics.MetricListener。
+- 快照参数：metricDomains.$domainName.snapshotInterval，定义拉取一次监控指标数据的间隔时间，参数单位是毫秒。
+
+org.apache.inlong.dataproxy.metrics.MetricListener接口的方法原型：
+```java
+public void snapshot(String domain, List<MetricItemValue> itemValues);
+```
+
+监控指标项的MetricItemValue.dimensions有这些维度(DataProxyMetricItem的这些字段通过注解Annotation
"@Dimension"定义):
+
+| property | description |
+| ---- | ---- |
+| clusterId | DataProxy集群ID |
+| sourceId| DataProxy的Source组件名 |
+| sourceDataId| DataProxy的Source组件数据流ID，如果Source是一个TCPSource，那么这个ID会是一个端口号 |
+| inlongGroupId| Inlong数据ID |
+| inlongStreamId| Inlong数据流ID |
+| sinkId| DataProxy的Sink组件名 |
+| sinkDataId| DataProxy的Sink组件数据流ID，如果Sink是一个Pulsar发送组件，这个ID会是一个Topic名。 |
+
+监控指标项的MetricItemValue.metrics有这些指标(DataProxyMetricItem的这些字段通过注解Annotation
"@CountMetric"定义):
+
+| property | description |
+| ---- | ---- |
+| readSuccessCount | 接收成功条数 |
+| readSuccessSize | 接收成功大小，单位：byte |
+| readFailCount | 接收失败条数 |
+| readFailSize | 接收失败大小，单位：byte |
+| sendCount | 发送条数 |
+| sendSize | 发送大小，单位：byte |
+| sendSuccessCount | 发送成功条数 |
+| sendSuccessSize | 发送成功大小，单位：byte |
+| sendFailCount | 发送失败条数 |
+| sendFailSize | 发送失败大小，单位：byte |
+| sinkDuration | 发送成功回调时间和发送开始时间的时间差，用于评估目标集群的处理时延和健康状况，单位：毫秒 |
+| nodeDuration | 发送成功回调时间和接收成功时间的时间差，用于评估DataProxy内部处理耗时和健康状况，单位：毫秒 |
+| wholeDuration | 发送成功回调时间和事件生成时间的时间差，单位：毫秒 |
+
+监控指标已经注册到MBeanServer，用户可以在DataProxy的启动参数中增加如下类似JMX定义（端口和鉴权根据情况进行调整），实现监控指标从远端采集。
+```shell
+ -Dcom.sun.management.jmxremote
+ -Djava.rmi.server.hostname=127.0.0.1
+ -Dcom.sun.management.jmxremote.port=9999
+ -Dcom.sun.management.jmxremote.authenticate=false
+ -Dcom.sun.management.jmxremote.ssl=false
+```
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/sort/example.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/sort/example.md
index d3958f7cb..5800e9835 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/sort/example.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/sort/example.md
@@ -1,12 +1,11 @@
---
-title: 数据流示例
+title: Example
sidebar_position: 3
---
## 示例
-为了更容易创建InLong-Sort作业，这里我们列出了一些数据流配置示例。
-下面将介绍InLong-Sort的SQL、Dashboard、Manager客户端工具的使用。
+为了更容易创建 InLong Sort 作业，这里我们列出了一些数据流配置示例。下面将介绍 InLong Sort 的
SQL、Dashboard、Manager 客户端工具的使用。
## 环境要求
- JDK 1.8.x
