This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-inlong-website.git
The following commit(s) were added to refs/heads/master by this push:
new 34369f0 [INLONG-2271] move TDMsg to InLongMsg (#263)
34369f0 is described below
commit 34369f0e91cfb49d23b109c4b6d777679a20c60c
Author: dockerzhang <[email protected]>
AuthorDate: Sun Jan 23 11:37:37 2022 +0800
[INLONG-2271] move TDMsg to InLongMsg (#263)
Co-authored-by: dockerzhang(张超) <[email protected]>
---
docs/administration/_category_.json | 2 +-
docs/design_and_concept/basic_concept.md | 21 +++++-----
docs/development/_category_.json | 4 ++
docs/development/inlong_msg.md | 47 ++++++++++++++++++++++
.../current/design_and_concept/basic_concept.md | 17 ++++----
.../current/development/inlong_msg.md | 47 ++++++++++++++++++++++
6 files changed, 119 insertions(+), 19 deletions(-)
diff --git a/docs/administration/_category_.json
b/docs/administration/_category_.json
index 2aaed72..cc9b71e 100644
--- a/docs/administration/_category_.json
+++ b/docs/administration/_category_.json
@@ -1,4 +1,4 @@
{
"label": "Administration",
- "position": 8
+ "position": 9
}
\ No newline at end of file
diff --git a/docs/design_and_concept/basic_concept.md
b/docs/design_and_concept/basic_concept.md
index 888ea0e..d8b33a2 100644
--- a/docs/design_and_concept/basic_concept.md
+++ b/docs/design_and_concept/basic_concept.md
@@ -3,14 +3,15 @@ title: Basic Concept
sidebar_position: 1
---
-| Name | Description | Other |
-| ---- | ---- | ---- |
-| Group | Data Streams Group, it contains multiple data streams, and one Group
represents one data ingestion. | Group has attributes such as ID and Name. |
-| Stream | Data Stream, a stream has a specific flow direction. | Stream has
attributes such as ID, Name, and data fields. |
-| Agent | Represents various collection capabilities. | It contains File
Agent, SQL Agent, Binlog Agent, etc. |
-| DataProxy | Forward received data to different message queues. | Supports
data transmission blocking, placing retransmission. |
-| Sort | Data stream sorting | Sort-flink based on Flink, sort-standalone for
local sorting. |
-| TubeMQ | InLong's self-developed message queuing service | It can also be
called Tube, with low-cost, high-performance features. |
-| Pulsar | [Apache Pulsar](https://pulsar.apache.org/), a high-performance,
high-consistency message queue service |
-| Hive | [Apache Hive](https://hive.apache.org/), a data warehouse built on
the Hadoop architecture |
+| Name | Description | Other |
+|------------| ---- | ---- |
+| Group | Data Streams Group, it contains multiple data streams, and one
Group represents one data ingestion. | Group has attributes such as ID and
Name. |
+| Stream | Data Stream, a stream has a specific flow direction. | Stream
has attributes such as ID, Name, and data fields. |
+| InLongMsg | InLong data format, if you consume message directly from the
message queue, you need to perform `InLongMsg` parsing first. |
+| Agent | Represents various collection capabilities. | It contains File
Agent, SQL Agent, Binlog Agent, etc. |
+| DataProxy | Forward received data to different message queues. | Supports
data transmission blocking, placing retransmission. |
+| Sort | Data stream sorting | Sort-flink based on Flink,
sort-standalone for local sorting. |
+| TubeMQ | InLong's self-developed message queuing service | It can also
be called Tube, with low-cost, high-performance features. |
+| Pulsar | [Apache Pulsar](https://pulsar.apache.org/), a
high-performance, high-consistency message queue service |
+| Hive | [Apache Hive](https://hive.apache.org/), a data warehouse built
on the Hadoop architecture |
| ClickHouse | [ClickHouse](https://clickhouse.com/), a high performance
columnar OLAP database | |
\ No newline at end of file
diff --git a/docs/development/_category_.json b/docs/development/_category_.json
new file mode 100644
index 0000000..6244591
--- /dev/null
+++ b/docs/development/_category_.json
@@ -0,0 +1,4 @@
+{
+ "label": "Development",
+ "position": 8
+}
\ No newline at end of file
diff --git a/docs/development/inlong_msg.md b/docs/development/inlong_msg.md
new file mode 100644
index 0000000..14abfd8
--- /dev/null
+++ b/docs/development/inlong_msg.md
@@ -0,0 +1,47 @@
+---
+title: Parse InLongMsg
+sidebar_position: 1
+---
+
+## Overview
+If you consume data directly from a message queue (InLong TubeMQ or Pulsar),
you need to parse `InLongMsg` first. Origin data can be parsed in the following
ways.
+
+## Dependency
+- Add Maven Dependency
+```java
+<dependency>
+ <groupId>org.apache.inlong</groupId>
+ <artifactId>inlong-common</artifactId>
+ <version>inlong_version</version>
+</dependency>
+```
+
+- Add Parse Method
+```java
+public static List<byte[]> parserInLongMsg(byte[] bytes) {
+ List<byte[]> originalContentByteList = new ArrayList<>();
+ InLongMsg inLongMsg = InLongMsg.parseFrom(bytes);
+ Set<String> attrs = inLongMsg.getAttrs();
+ if (CollectionUtils.isEmpty(attrs)) {
+ return originalContentByteList;
+ }
+ for (String attr : attrs) {
+ if (attr == null) {
+ continue;
+ }
+ Iterator<byte[]> iterator = inLongMsg.getIterator(attr);
+ if (iterator == null) {
+ continue;
+ }
+ while (iterator.hasNext()) {
+ byte[] bodyBytes = iterator.next();
+ if (bodyBytes == null || bodyBytes.length == 0) {
+ continue;
+ }
+ // Origin data sended by InLong agent
+ originalContentByteList.add(bodyBytes);
+ }
+ }
+ return originalContentByteList;
+}
+```
\ No newline at end of file
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/basic_concept.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/basic_concept.md
index f63b643..e54e435 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/basic_concept.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/design_and_concept/basic_concept.md
@@ -3,14 +3,15 @@ title: 基本概念
sidebar_position: 1
---
-| Name | Description | Other |
-| ---- | ---- | ---- |
-| Group | 数据流组，包含多个数据流，一个Group 代表一个数据接入 | Group 有ID、Name 等属性 |
-| Stream | 数据流，一个数据流有具体的流向 | Stream 有ID、Name、数据字段等属性 |
-| Agent | 代表各种采集能力 | 包含文件Agent、SQL Agent、Binlog Agent 等 |
-| DataProxy | 将接收到的数据转发到不同的消息队列 | 支持数据发送阻塞和落盘重发 |
-| Sort | 数据流分拣 | 主要有基于Flink的sort-flink，sort-standalone 本地分拣 |
-| TubeMQ | InLong自带的消息队列服务 | 也可以叫Tube，拥有低成本、高性能特性 |
+| Name | Description | Other |
+| ---- |----------------------------------------------------------| ---- |
+| Group | 数据流组，包含多个数据流，一个Group 代表一个数据接入 | Group
有ID、Name 等属性 |
+| Stream | 数据流，一个数据流有具体的流向 | Stream
有ID、Name、数据字段等属性 |
+| InLongMsg | InLong 数据格式，如果从消息队列中直接消费，需要先进行`InLongMsg` 解析 |
+| Agent | 代表各种采集能力 |
包含文件Agent、SQL Agent、Binlog Agent 等 |
+| DataProxy | 将接收到的数据转发到不同的消息队列 |
支持数据发送阻塞和落盘重发 |
+| Sort | 数据流分拣 |
主要有基于Flink的sort-flink，sort-standalone 本地分拣 |
+| TubeMQ | InLong自带的消息队列服务 |
也可以叫Tube，拥有低成本、高性能特性 |
| Pulsar | 即[Apache Pulsar](https://pulsar.apache.org/), 高性能、高一致性消息队列服务 |
| Hive | 即[Apache Hive](https://hive.apache.org/)，一个建立在Hadoop架构之上的数据仓库 |
| ClickHouse | [ClickHouse](https://clickhouse.com/)，高性能列式OLAP 数据库 | |
\ No newline at end of file
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/development/inlong_msg.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/development/inlong_msg.md
new file mode 100644
index 0000000..c4996b6
--- /dev/null
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/development/inlong_msg.md
@@ -0,0 +1,47 @@
+---
+title: 解析 InLongMsg
+sidebar_position: 1
+---
+
+## 总览
+如果直接从消息队列（InLong TubeMQ 或Pulsar)消费数据，需要先对`InLongMsg` 进行解析。可通过以下方式可以解析出源数据。
+
+## 解析
+- 增加maven 依赖
+```java
+<dependency>
+ <groupId>org.apache.inlong</groupId>
+ <artifactId>inlong-common</artifactId>
+ <version>inlong_version</version>
+</dependency>
+```
+
+- 增加解析逻辑
+```java
+public static List<byte[]> parserInLongMsg(byte[] bytes) {
+ List<byte[]> originalContentByteList = new ArrayList<>();
+ InLongMsg inLongMsg = InLongMsg.parseFrom(bytes);
+ Set<String> attrs = inLongMsg.getAttrs();
+ if (CollectionUtils.isEmpty(attrs)) {
+ return originalContentByteList;
+ }
+ for (String attr : attrs) {
+ if (attr == null) {
+ continue;
+ }
+ Iterator<byte[]> iterator = inLongMsg.getIterator(attr);
+ if (iterator == null) {
+ continue;
+ }
+ while (iterator.hasNext()) {
+ byte[] bodyBytes = iterator.next();
+ if (bodyBytes == null || bodyBytes.length == 0) {
+ continue;
+ }
+ // agent 发送的原始用户数据
+ originalContentByteList.add(bodyBytes);
+ }
+ }
+ return originalContentByteList;
+}
+```
\ No newline at end of file
