This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong-website.git
The following commit(s) were added to refs/heads/master by this push:
new 6571b24174 [INLONG-904][DOC] Add MySQL to Iceberg example document for
data sync (#905)
6571b24174 is described below
commit 6571b24174cf5a63062ebde528c13c6bcf9c00dc
Author: haifxu <[email protected]>
AuthorDate: Wed Dec 20 09:50:40 2023 +0800
[INLONG-904][DOC] Add MySQL to Iceberg example document for data sync (#905)
---
.../data_sync/img/mysql_iceberg/approval.png | Bin 0 -> 28940 bytes
.../data_sync/img/mysql_iceberg/audit_iceberg.png | Bin 0 -> 49261 bytes
.../img/mysql_iceberg/create_cluster_tag.png | Bin 0 -> 28397 bytes
.../img/mysql_iceberg/create_group_stream.png | Bin 0 -> 23186 bytes
.../img/mysql_iceberg/create_iceberg_datanode.png | Bin 0 -> 26816 bytes
.../img/mysql_iceberg/create_pulsar_cluster.png | Bin 0 -> 26969 bytes
.../img/mysql_iceberg/create_stream_sink.png | Bin 0 -> 27503 bytes
.../img/mysql_iceberg/create_stream_source.png | Bin 0 -> 29689 bytes
.../data_sync/img/mysql_iceberg/result_iceberg.png | Bin 0 -> 9991 bytes
.../data_sync/img/mysql_iceberg/result_mysql.png | Bin 0 -> 12227 bytes
.../data_sync/img/mysql_iceberg/success.png | Bin 0 -> 6067 bytes
.../quick_start/data_sync/mysql_iceberg_example.md | 126 +++++++++++++++++++++
.../data_sync/img/mysql_iceberg/approval.png | Bin 0 -> 26805 bytes
.../data_sync/img/mysql_iceberg/audit_iceberg.png | Bin 0 -> 52910 bytes
.../img/mysql_iceberg/create_cluster_tag.png | Bin 0 -> 14019 bytes
.../img/mysql_iceberg/create_group_stream.png | Bin 0 -> 23186 bytes
.../img/mysql_iceberg/create_iceberg_datanode.png | Bin 0 -> 25181 bytes
.../img/mysql_iceberg/create_pulsar_cluster.png | Bin 0 -> 28881 bytes
.../img/mysql_iceberg/create_stream_sink.png | Bin 0 -> 26234 bytes
.../img/mysql_iceberg/create_stream_source.png | Bin 0 -> 30799 bytes
.../data_sync/img/mysql_iceberg/result_iceberg.png | Bin 0 -> 9991 bytes
.../data_sync/img/mysql_iceberg/result_mysql.png | Bin 0 -> 12227 bytes
.../data_sync/img/mysql_iceberg/success.png | Bin 0 -> 6300 bytes
.../quick_start/data_sync/mysql_iceberg_example.md | 125 ++++++++++++++++++++
24 files changed, 251 insertions(+)
diff --git a/docs/quick_start/data_sync/img/mysql_iceberg/approval.png
b/docs/quick_start/data_sync/img/mysql_iceberg/approval.png
new file mode 100644
index 0000000000..b7186460af
Binary files /dev/null and
b/docs/quick_start/data_sync/img/mysql_iceberg/approval.png differ
diff --git a/docs/quick_start/data_sync/img/mysql_iceberg/audit_iceberg.png
b/docs/quick_start/data_sync/img/mysql_iceberg/audit_iceberg.png
new file mode 100644
index 0000000000..964fcfc71d
Binary files /dev/null and
b/docs/quick_start/data_sync/img/mysql_iceberg/audit_iceberg.png differ
diff --git
a/docs/quick_start/data_sync/img/mysql_iceberg/create_cluster_tag.png
b/docs/quick_start/data_sync/img/mysql_iceberg/create_cluster_tag.png
new file mode 100644
index 0000000000..bc1fc8a16d
Binary files /dev/null and
b/docs/quick_start/data_sync/img/mysql_iceberg/create_cluster_tag.png differ
diff --git
a/docs/quick_start/data_sync/img/mysql_iceberg/create_group_stream.png
b/docs/quick_start/data_sync/img/mysql_iceberg/create_group_stream.png
new file mode 100644
index 0000000000..68c85968d7
Binary files /dev/null and
b/docs/quick_start/data_sync/img/mysql_iceberg/create_group_stream.png differ
diff --git
a/docs/quick_start/data_sync/img/mysql_iceberg/create_iceberg_datanode.png
b/docs/quick_start/data_sync/img/mysql_iceberg/create_iceberg_datanode.png
new file mode 100644
index 0000000000..7d3220972f
Binary files /dev/null and
b/docs/quick_start/data_sync/img/mysql_iceberg/create_iceberg_datanode.png
differ
diff --git
a/docs/quick_start/data_sync/img/mysql_iceberg/create_pulsar_cluster.png
b/docs/quick_start/data_sync/img/mysql_iceberg/create_pulsar_cluster.png
new file mode 100644
index 0000000000..eb725498e7
Binary files /dev/null and
b/docs/quick_start/data_sync/img/mysql_iceberg/create_pulsar_cluster.png differ
diff --git
a/docs/quick_start/data_sync/img/mysql_iceberg/create_stream_sink.png
b/docs/quick_start/data_sync/img/mysql_iceberg/create_stream_sink.png
new file mode 100644
index 0000000000..5312a9f9df
Binary files /dev/null and
b/docs/quick_start/data_sync/img/mysql_iceberg/create_stream_sink.png differ
diff --git
a/docs/quick_start/data_sync/img/mysql_iceberg/create_stream_source.png
b/docs/quick_start/data_sync/img/mysql_iceberg/create_stream_source.png
new file mode 100644
index 0000000000..ce9ca70254
Binary files /dev/null and
b/docs/quick_start/data_sync/img/mysql_iceberg/create_stream_source.png differ
diff --git a/docs/quick_start/data_sync/img/mysql_iceberg/result_iceberg.png
b/docs/quick_start/data_sync/img/mysql_iceberg/result_iceberg.png
new file mode 100644
index 0000000000..411617808c
Binary files /dev/null and
b/docs/quick_start/data_sync/img/mysql_iceberg/result_iceberg.png differ
diff --git a/docs/quick_start/data_sync/img/mysql_iceberg/result_mysql.png
b/docs/quick_start/data_sync/img/mysql_iceberg/result_mysql.png
new file mode 100644
index 0000000000..0757cf5104
Binary files /dev/null and
b/docs/quick_start/data_sync/img/mysql_iceberg/result_mysql.png differ
diff --git a/docs/quick_start/data_sync/img/mysql_iceberg/success.png
b/docs/quick_start/data_sync/img/mysql_iceberg/success.png
new file mode 100644
index 0000000000..8887cf35de
Binary files /dev/null and
b/docs/quick_start/data_sync/img/mysql_iceberg/success.png differ
diff --git a/docs/quick_start/data_sync/mysql_iceberg_example.md
b/docs/quick_start/data_sync/mysql_iceberg_example.md
new file mode 100644
index 0000000000..5e1b19e3b7
--- /dev/null
+++ b/docs/quick_start/data_sync/mysql_iceberg_example.md
@@ -0,0 +1,126 @@
+---
+title: MySQL to Iceberg Example
+sidebar_position: 3
+---
+
+Here we use an example to introduce how to use Apache InLong creating `MySQL
-> Iceberg` full database migration.
+
+## Deployment
+### Install InLong
+
+Before we begin, we need to install InLong. Here we provide two ways:
+- [Docker Deployment](deployment/docker.md) (Recommended)
+- [Bare Metal Deployment](deployment/bare_metal.md)
+
+### Add Connectors
+Download the [connectors](https://inlong.apache.org/downloads/) corresponding
to Flink 1.13, and after decompression, place
`sort-connector-iceberg-[version]-SNAPSHOT.jar` in `/inlong-sort/connectors/`
directory.
+
+### Install Iceberg
+Please refer to the [Installation
Tutorial](https://iceberg.apache.org/hive-quickstart) on the Apache Iceberg
official website.
+
+## Cluster Initialize
+When all containers are successfully started, you can access the InLong
dashboard address http://localhost, and use the following default account to
log in.
+```
+User: admin
+Password: inlong
+```
+
+### Create Cluster Tag
+Click [Clusters] -> [ClusterTags] -> [Create] on the page to specify the
cluster label name and responsible person.
+
+
+:::caution
+`default_cluster` is the default ClusterTags reported by each component. If
you decide to use a different name, make sure to update the corresponding tag
configuration accordingly.
+:::
+
+### Register Pulsar Cluster
+Click [Clusters] -> [Cluster] -> [Create] on the page to register Pulsar
Cluster.
+
+
+:::note
+The ClusterTags selects the newly created `default_cluster`, the Pulsar
cluster deployed by docker:
+
+Service URL is `pulsar://pulsar:6650`, Admin URL is `http://pulsar:8080`.
+:::
+
+### Register Iceberg DataNodes
+Click [DataNodes] -> [Create] on the page to register Iceberg DataNodes.
+
+
+## Create Task
+### Create Data Streams Group
+Click [Synchronization] → [Create] on the page and input the Group ID, Stream
ID and Full database migration:
+
+
+### Create Data Source
+In the data source, click [New] → [MySQL] to configure the source name,
address, databases and tables information.
+
+
+:::note
+- If the read mode is `Full amount + Incremental`, the existing data in the
source table will also be collected, but the `Incremental` mode will not.
+- The table white list format is `<dbName>.<tableName>` and supports regular
expressions.
+:::
+
+### Create Data Sink
+In the data sink, click [New] → [Iceberg] to configure the sink name and
created Iceberg data node.
+We can choose the data sink to have the same database table name as the data
source, or customize it.
+
+
+:::note
+When customizing the names of database tables, you can use built-in parameters
and string combinations to generate the target table names.
+
+Built-in parameters include:
+- Source database name: ${database}
+- Source table name: ${table}
+
+For example, if the source table name is `table1` and the mapping rule is
`${table}_inlong`, the data from `table1` will be ultimately mapped and written
into `table1_inlong`.
+:::
+
+### Approve Data Stream
+Click [Approval] -> [MyApproval] -> [Approval] -> [Ok].
+
+
+Back to [Synchronization] page, wait for [success].
+
+
+## Test Data
+### Send Data
+```
+#!/bin/bash
+
+# MySQL info
+DB_HOST="mysql"
+DB_USER="root"
+DB_PASS="inlong"
+DB_NAME="test"
+DB_TABLE1="source_table"
+DB_TABLE2="source_table2"
+
+# Insert data in a loop
+for ((i=1; i<=500; i++))
+do
+ # Generate data
+ id=$i
+ name="name_$i"
+
+ # Build an insert SQL
+ query1="INSERT INTO $DB_TABLE1 (id, name) VALUES ($id, '$name');"
+ query2="INSERT INTO $DB_TABLE2 (id, name) VALUES ($id, '$name');"
+
+ # Execute insert SQL
+ mysql -h $DB_HOST -u $DB_USER -p$DB_PASS $DB_NAME -e "$query1"
+ mysql -h $DB_HOST -u $DB_USER -p$DB_PASS $DB_NAME -e "$query2"
+done
+```
+
+Modify the variables in the script according to the actual environment, and
insert 500 pieces of data into each table(There is a piece of existing data in
`source_table`):
+
+
+### Verify Data
+Enter Iceberg, check data in table.
+
+
+
+You can also view audit data on the page:
+
+
\ No newline at end of file
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/approval.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/approval.png
new file mode 100644
index 0000000000..ccc7967503
Binary files /dev/null and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/approval.png
differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/audit_iceberg.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/audit_iceberg.png
new file mode 100644
index 0000000000..de102bcf94
Binary files /dev/null and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/audit_iceberg.png
differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/create_cluster_tag.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/create_cluster_tag.png
new file mode 100644
index 0000000000..aa4b45fdc8
Binary files /dev/null and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/create_cluster_tag.png
differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/create_group_stream.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/create_group_stream.png
new file mode 100644
index 0000000000..68c85968d7
Binary files /dev/null and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/create_group_stream.png
differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/create_iceberg_datanode.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/create_iceberg_datanode.png
new file mode 100644
index 0000000000..10a3c6f385
Binary files /dev/null and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/create_iceberg_datanode.png
differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/create_pulsar_cluster.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/create_pulsar_cluster.png
new file mode 100644
index 0000000000..fb75c178aa
Binary files /dev/null and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/create_pulsar_cluster.png
differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/create_stream_sink.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/create_stream_sink.png
new file mode 100644
index 0000000000..91d5ab4f38
Binary files /dev/null and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/create_stream_sink.png
differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/create_stream_source.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/create_stream_source.png
new file mode 100644
index 0000000000..3fe5b80687
Binary files /dev/null and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/create_stream_source.png
differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/result_iceberg.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/result_iceberg.png
new file mode 100644
index 0000000000..411617808c
Binary files /dev/null and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/result_iceberg.png
differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/result_mysql.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/result_mysql.png
new file mode 100644
index 0000000000..0757cf5104
Binary files /dev/null and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/result_mysql.png
differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/success.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/success.png
new file mode 100644
index 0000000000..6ef56342fe
Binary files /dev/null and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/img/mysql_iceberg/success.png
differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/mysql_iceberg_example.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/mysql_iceberg_example.md
new file mode 100644
index 0000000000..7196d65967
--- /dev/null
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/data_sync/mysql_iceberg_example.md
@@ -0,0 +1,125 @@
+---
+title: MySQL 到 Iceberg 示例
+sidebar_position: 3
+---
+
+在下面的内容中，我们将通过一个完整的示例介绍如何使用 Apache InLong 创建 MySQL -> Iceberg 整库数据同步。
+
+## 环境部署
+### 安装 InLong
+
+在开始之前，我们需要安装 InLong 的全部组件，这里提供两种方式：
+- [Docker 部署](deployment/docker.md)（推荐）
+- [Bare Metal 部署](deployment/bare_metal.md)
+
+### 添加 Connectors
+下载 Flink 1.13 对应版本的
[connectors](https://inlong.apache.org/zh-CN/downloads)，解压后将
`sort-connector-iceberg-[version]-SNAPSHOT.jar` 放在 `/inlong-sort/connectors/`
目录下。
+
+### 安装 Iceberg
+请参考 Apache Iceberg 官网的[安装教程](https://iceberg.apache.org/hive-quickstart)
+
+## 集群初始化
+容器启动成功后，访问 InLong Dashboard 地址 http://localhost，并使用以下默认账号登录：
+```
+User: admin
+Password: inlong
+```
+
+### 创建集群标签
+页面点击 【集群管理】->【标签管理】->【新建】，指定集群标签名称和负责人：
+
+
+**注：default_cluster 是各个组件默认上报集群标签，如果使用其它名称，确认对应标签配置已修改。**
+
+### 注册 Pulsar 集群
+页面点击 [集群管理] -> [集群管理] -> [新建集群]，注册 Pulsar 集群：
+
+
+:::note
+集群标签选择刚创建的 `default_cluster`，配置 Docker 部署的 Pulsar 集群：
+
+Service URL 为 `pulsar://pulsar:6650`, Admin URL 为 `http://pulsar:8080`.
+:::
+
+### 注册 Iceberg 数据节点
+页面点击 [数据节点] -> [创建] ，新增 Iceberg 数据节点.
+
+
+## 任务创建
+### 新建数据流组
+页面点击[数据同步] → [创建]，输入 Group ID、Steam ID 和 是否整库迁移：
+
+
+### 创建数据源
+数据源中点击 [新建] → [MySQL] 配置数据源名称、地址、库表信息等。
+
+
+:::note
+- 这里读取模式选择了 `全量+增量`，表中的存量数据也会被采集，`仅增量` 模式则不会。
+- 表名白名单格式为 `<dbName>.<tableName>`，支持正则表达。
+:::
+
+### 创建数据目标
+数据目标中点击 [新建] → [Iceberg]，设置数据目标名称并选择创建好的 Iceberg 数据节点,
+库表名称我们可以选择与数据源一致，或者自定义。
+
+
+:::note
+自定义库表名称时，支持使用内置参数和字符串组合生成目标库表名称。
+
+内置参数包括：
+- 来源库名：${database}
+- 来源表名：${table}
+
+如：来源表名称为 `table1`，映射规则为 `${table}_inlong`，则 `table1` 的数据将被最终映射写入至
`table1_inlong` 中。
+:::
+
+### 审批数据流
+点击 [审批管理] -> [我的审批] -> [审批] -> [Ok].
+
+
+返回【数据集成】，等待任务配置成功：
+
+
+## 测试数据
+### 发送数据
+```
+#!/bin/bash
+
+# MySQL info
+DB_HOST="mysql"
+DB_USER="root"
+DB_PASS="inlong"
+DB_NAME="test"
+DB_TABLE1="source_table"
+DB_TABLE2="source_table2"
+
+# Insert data in a loop
+for ((i=1; i<=500; i++))
+do
+ # Generate data
+ id=$i
+ name="name_$i"
+
+ # Build an insert SQL
+ query1="INSERT INTO $DB_TABLE1 (id, name) VALUES ($id, '$name');"
+ query2="INSERT INTO $DB_TABLE2 (id, name) VALUES ($id, '$name');"
+
+ # Execute insert SQL
+ mysql -h $DB_HOST -u $DB_USER -p$DB_PASS $DB_NAME -e "$query1"
+ mysql -h $DB_HOST -u $DB_USER -p$DB_PASS $DB_NAME -e "$query2"
+done
+```
+
+根据实际环境修改脚本中的变量，向每个表中插入 500 条数据(这里 `source_table` 中有一条存量数据):
+
+
+
+### 验证数据
+进入 Iceberg，查看表数据
+
+
+
+也可以在页面查看审计数据:
+
+
\ No newline at end of file
