This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong-website.git
The following commit(s) were added to refs/heads/master by this push:
new 4e60d61f30 [INLONG-822][Doc] Add the quick start guide for collecting
file and writing to ClickHouse (#838)
4e60d61f30 is described below
commit 4e60d61f30d4502559c555ce89616f22c28a1636
Author: Amelie <[email protected]>
AuthorDate: Thu Aug 24 17:21:23 2023 +0800
[INLONG-822][Doc] Add the quick start guide for collecting file and writing
to ClickHouse (#838)
Co-authored-by: joytzli <[email protected]>
Co-authored-by: Charles Zhang <[email protected]>
---
docs/quick_start/file_pulsar_clickhouse_example.md | 101 +++++++++++++++++++++
.../img/file_clickhouse/create_cluster_label.png | Bin 0 -> 22127 bytes
.../img/file_clickhouse/create_pulsar_cluster.png | Bin 0 -> 36568 bytes
docs/quick_start/img/file_clickhouse/datanode.png | Bin 0 -> 101757 bytes
.../img/file_clickhouse/pulsar_approval.png | Bin 0 -> 109310 bytes
.../img/file_clickhouse/pulsar_approve_result.png | Bin 0 -> 163376 bytes
.../img/file_clickhouse/pulsar_audit.png | Bin 0 -> 161957 bytes
.../img/file_clickhouse/pulsar_file_source.png | Bin 0 -> 88091 bytes
.../img/file_clickhouse/pulsar_group.png | Bin 0 -> 355321 bytes
.../img/file_clickhouse/pulsar_sink.png | Bin 0 -> 252822 bytes
.../img/file_clickhouse/pulsar_stream.png | Bin 0 -> 192767 bytes
.../img/file_clickhouse/pulsar_table.png | Bin 0 -> 47031 bytes
.../quick_start/file_pulsar_clickhouse_example.md | 98 ++++++++++++++++++++
.../img/file_clickhouse/create_cluster_label.png | Bin 0 -> 5211 bytes
.../img/file_clickhouse/create_pulsar_cluster.png | Bin 0 -> 30222 bytes
.../quick_start/img/file_clickhouse/datanode.png | Bin 0 -> 27843 bytes
.../img/file_clickhouse/pulsar_approval.png | Bin 0 -> 36519 bytes
.../img/file_clickhouse/pulsar_approval_result.png | Bin 0 -> 53846 bytes
.../img/file_clickhouse/pulsar_audit.png | Bin 0 -> 57854 bytes
.../img/file_clickhouse/pulsar_ingestion.png | Bin 0 -> 91016 bytes
.../img/file_clickhouse/pulsar_sink.png | Bin 0 -> 241208 bytes
.../img/file_clickhouse/pulsar_source.png | Bin 0 -> 93424 bytes
.../img/file_clickhouse/pulsar_stream.png | Bin 0 -> 79289 bytes
.../img/file_clickhouse/pulsar_table.png | Bin 0 -> 47031 bytes
24 files changed, 199 insertions(+)
diff --git a/docs/quick_start/file_pulsar_clickhouse_example.md
b/docs/quick_start/file_pulsar_clickhouse_example.md
new file mode 100644
index 0000000000..f2a5c22022
--- /dev/null
+++ b/docs/quick_start/file_pulsar_clickhouse_example.md
@@ -0,0 +1,101 @@
+---
+title: File -> Pulsar -> ClickHouse Example
+sidebar_position: 4
+---
+
+Here we use an example to introduce how to create File -> Pulsar -> ClickHouse
data ingestion.
+
+## Deployment
+### Install InLong
+
+Before we begin, we need to install InLong. Here we provide two ways:
+1. Install InLong with Docker by according to the [instructions
here](deployment/docker.md).(Recommanded)
+2. Install InLong binary according to the [instructions
here](deployment/bare_metal.md).
+
+### Install ClickHouse
+```shell
+docker run -d --rm --net=host --name clickhouse -e CLICKHOUSE_USER=admin -e
CLICKHOUSE_PASSWORD=inlong -e CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1
clickhouse/clickhouse-server:22.8
+```
+
+## Cluster Initialize
+When all containers are successfully started, you can access the Inlong
dashboard address http://localhost, and use the following default account to
log in.
+```
+User: admin
+Password: inlong
+```
+
+### Create Cluster Tag
+Click [Clusters] -> [ClusterTags] -> [Create] on the page to specify the
cluster label name and person in charge.
+
+
+:::caution
+Since each component reports the ClusterTags as `default_cluster` by default,
do not use other names.
+:::
+
+### Register Pulsar Cluster
+Click [Clusters] -> [ClusterTags] -> [Create] on the page to register Pulsar
Cluster.
+
+
+:::note
+The ClusterTags selects the newly created `default_cluster`, the Pulsar
cluster deployed by docker:
+
+Service URL is `pulsar://pulsar:6650`, Admin URL is `http://pulsar:8080`.
+:::
+
+### Register Clickhouse DataNodes
+Click [DataNodes] -> [Create] on the page to register ClickHouse DataNodes.
+
+
+## Create Task
+### Create Data Streams Group
+Click [Ingestion] -> [Create], inpit ID and select Pulsar Type.
+
+
+:::note
+If you want to use Kafka, you need to modify the MQ_TYPE in docker-compose.yml
+:::
+
+### Create Data Stream
+Click [Next] -> [Create], config data stream.
+
+
+### Create Data Source
+Click [Sources create] -> [File], config data source informations.
+
+
+### Create Data Sink
+Click [Sinks create] -> [ClickHouse], input Name, DbName, TableName and select
created ck DataNode and so on, then click [save].
+
+
+### Approve Data Stream
+Click [Approval] -> [MyApproval] -> [Detail] -> [Select Cluster tag] -> [Ok].
+
+
+Back to [Ingestion] page, wait for [configuration success].
+
+
+## Test Data
+### Enter Agent Container
+```
+docker exec -it agent /bin/bash
+```
+
+### Send Data
+```
+# collect file /data/collect-data/test.log data source configuration path and
total of 10001 data packets are sent.
+for i in {0..10000};do echo "name_$i | $i" >> /data/collect-data/test.log ;done
+```
+
+### Verify Data
+Check data in [Audit] page.
+
+
+Then enter ClickHouse container, check data in table.
+
+
+## Questions
+### "specify container image platform" when start Docker Compose
+The API is incompatible, you can delete platform: "linux/x86_64" in
docker-compose.
+
+### Task Configuration Failed
+Generally, the MQ or Flink group configuration is incorrect. You can view the
error information on the page, or enter the Manager container to view detailed
logs.
\ No newline at end of file
diff --git a/docs/quick_start/img/file_clickhouse/create_cluster_label.png
b/docs/quick_start/img/file_clickhouse/create_cluster_label.png
new file mode 100644
index 0000000000..8fb4d7e16f
Binary files /dev/null and
b/docs/quick_start/img/file_clickhouse/create_cluster_label.png differ
diff --git a/docs/quick_start/img/file_clickhouse/create_pulsar_cluster.png
b/docs/quick_start/img/file_clickhouse/create_pulsar_cluster.png
new file mode 100644
index 0000000000..1cc548f34a
Binary files /dev/null and
b/docs/quick_start/img/file_clickhouse/create_pulsar_cluster.png differ
diff --git a/docs/quick_start/img/file_clickhouse/datanode.png
b/docs/quick_start/img/file_clickhouse/datanode.png
new file mode 100644
index 0000000000..47df5ca8ce
Binary files /dev/null and b/docs/quick_start/img/file_clickhouse/datanode.png
differ
diff --git a/docs/quick_start/img/file_clickhouse/pulsar_approval.png
b/docs/quick_start/img/file_clickhouse/pulsar_approval.png
new file mode 100644
index 0000000000..ae31072d95
Binary files /dev/null and
b/docs/quick_start/img/file_clickhouse/pulsar_approval.png differ
diff --git a/docs/quick_start/img/file_clickhouse/pulsar_approve_result.png
b/docs/quick_start/img/file_clickhouse/pulsar_approve_result.png
new file mode 100644
index 0000000000..fac86c1957
Binary files /dev/null and
b/docs/quick_start/img/file_clickhouse/pulsar_approve_result.png differ
diff --git a/docs/quick_start/img/file_clickhouse/pulsar_audit.png
b/docs/quick_start/img/file_clickhouse/pulsar_audit.png
new file mode 100644
index 0000000000..83c359d40f
Binary files /dev/null and
b/docs/quick_start/img/file_clickhouse/pulsar_audit.png differ
diff --git a/docs/quick_start/img/file_clickhouse/pulsar_file_source.png
b/docs/quick_start/img/file_clickhouse/pulsar_file_source.png
new file mode 100644
index 0000000000..0043e7651e
Binary files /dev/null and
b/docs/quick_start/img/file_clickhouse/pulsar_file_source.png differ
diff --git a/docs/quick_start/img/file_clickhouse/pulsar_group.png
b/docs/quick_start/img/file_clickhouse/pulsar_group.png
new file mode 100644
index 0000000000..f49cee868d
Binary files /dev/null and
b/docs/quick_start/img/file_clickhouse/pulsar_group.png differ
diff --git a/docs/quick_start/img/file_clickhouse/pulsar_sink.png
b/docs/quick_start/img/file_clickhouse/pulsar_sink.png
new file mode 100644
index 0000000000..4ab433a250
Binary files /dev/null and
b/docs/quick_start/img/file_clickhouse/pulsar_sink.png differ
diff --git a/docs/quick_start/img/file_clickhouse/pulsar_stream.png
b/docs/quick_start/img/file_clickhouse/pulsar_stream.png
new file mode 100644
index 0000000000..c59a22c371
Binary files /dev/null and
b/docs/quick_start/img/file_clickhouse/pulsar_stream.png differ
diff --git a/docs/quick_start/img/file_clickhouse/pulsar_table.png
b/docs/quick_start/img/file_clickhouse/pulsar_table.png
new file mode 100644
index 0000000000..7df435d327
Binary files /dev/null and
b/docs/quick_start/img/file_clickhouse/pulsar_table.png differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/file_pulsar_clickhouse_example.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/file_pulsar_clickhouse_example.md
new file mode 100644
index 0000000000..9efddab797
--- /dev/null
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/file_pulsar_clickhouse_example.md
@@ -0,0 +1,98 @@
+---
+title: File -> Pulsar -> ClickHouse 示例
+sidebar_position: 4
+---
+
+在下面的内容中，我们将通过一个完整的示例介绍如何使用 Apache InLong 创建 File -> Pulsar -> ClickHouse 数据流。
+
+## 环境部署
+### 安装 InLong
+在开始之前，我们需要安装 InLong 的全部组件，这里提供两种方式：
+1. 按照 [这里的说明](deployment/docker.md)，使用 Docker 进行快速部署。（推荐）
+2. 按照 [这里的说明](deployment/bare_metal.md)，使用二进制包依次安装各组件。
+
+### 安装 ClickHouse
+```shell
+docker run -d --rm --net=host --name clickhouse -e CLICKHOUSE_USER=admin -e
CLICKHOUSE_PASSWORD=inlong -e CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1
clickhouse/clickhouse-server:22.8
+```
+
+## 集群初始化
+容器启动成功后，访问 InLong Dashboard 地址 http://localhost，并使用以下默认账号登录：
+```
+User: admin
+Password: inlong
+```
+
+### 创建集群标签
+页面点击 【集群管理】->【标签管理】->【新建】，指定集群标签名称和负责人：
+
+
+**注：default_cluster 是各个组件默认上报集群标签，请勿使用其它名称。**
+
+### 注册 Pulsar 集群
+页面点击 [集群管理] -> [集群管理] -> [新建集群]，注册 Pulsar 集群：
+
+
+:::note
+集群标签选择刚创建的 `default_cluster`，配置 Docker 部署的 Pulsar 集群：
+
+Service URL 为 `pulsar://pulsar:6650`，Admin URL 为 `http://pulsar:8080`。
+:::
+
+### 注册 Clickhouse 数据节点
+页面点击【数据节点】->【新建】，新增 ClickHouse 数据节点：
+
+
+## 任务创建
+### 新建数据流组
+页面点击【数据接入】->【新建接入】，只用填写 ID 接选择 Pulsar 即可：
+
+
+:::note
+注：如果需要使用 Kafka，需要修改 docker-compose.yml 中的 MQ_TYPE。
+:::
+
+### 新建数据流
+点击【下一步】->【新建数据流】，配置 ID 及测试源字段信息：
+
+
+### 为数据流添加文件数据源
+点击【新建数据源】，选择 File 类型，配置数据源信息，包括名称、选择采集 Agent 集群及 IP 、采集文件路径等：
+
+
+### 新建数据目标
+点击【新建数据目标】，选择 ClickHouse 类型，配置包括名称、库表名称、选择已创建的节点及 Schema 映射信息等，然后点击【提交审批】：
+
+
+### 审批数据流
+页面点击【审批管理】->【我的审批】->【详情】->【选择集群标签】->【通过】：
+
+
+返回【数据集成】，等待任务配置成功：
+
+
+## 测试数据
+### 进入 Agent 容器
+```
+docker exec -it agent /bin/bash
+```
+
+### 发送数据
+```
+# 采集文件 /data/collect-data/test.log 数据源配置路径，总共发送 10001 条
+for i in {0..10000};do echo "name_$i | $i" >> /data/collect-data/test.log ;done
+```
+
+### 数据验证
+查看审计页面发送数据：
+
+
+然后进入 ClickHouse 容器，查看库表数据：
+
+
+## 常见问题
+### Docker Compose 启动出现“specify container image platform”
+API 不兼容导致，可以删除 docker-compose.yml 中的 platform: "linux/x86_64"，执行。
+
+### 数据流配置错误
+一般是 MQ 或者 Flink 集群配置错误导致，可以在页面查看错误信息，或者进入 Manager 容器查看详细日志。
\ No newline at end of file
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/create_cluster_label.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/create_cluster_label.png
new file mode 100644
index 0000000000..34abe0b592
Binary files /dev/null and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/create_cluster_label.png
differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/create_pulsar_cluster.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/create_pulsar_cluster.png
new file mode 100644
index 0000000000..881162522b
Binary files /dev/null and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/create_pulsar_cluster.png
differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/datanode.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/datanode.png
new file mode 100644
index 0000000000..1b7e77b239
Binary files /dev/null and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/datanode.png
differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_approval.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_approval.png
new file mode 100644
index 0000000000..c195bde008
Binary files /dev/null and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_approval.png
differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_approval_result.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_approval_result.png
new file mode 100644
index 0000000000..9ae0e91b4b
Binary files /dev/null and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_approval_result.png
differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_audit.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_audit.png
new file mode 100644
index 0000000000..4fa3dc7562
Binary files /dev/null and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_audit.png
differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_ingestion.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_ingestion.png
new file mode 100644
index 0000000000..9c6e864336
Binary files /dev/null and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_ingestion.png
differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_sink.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_sink.png
new file mode 100644
index 0000000000..81c342b102
Binary files /dev/null and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_sink.png
differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_source.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_source.png
new file mode 100644
index 0000000000..f6606b6bc2
Binary files /dev/null and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_source.png
differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_stream.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_stream.png
new file mode 100644
index 0000000000..69f6774c40
Binary files /dev/null and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_stream.png
differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_table.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_table.png
new file mode 100644
index 0000000000..7df435d327
Binary files /dev/null and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/img/file_clickhouse/pulsar_table.png
differ
