This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong-website.git
The following commit(s) were added to refs/heads/master by this push:
new 8867dd5b32 [INLONG-849][Doc] Add Iceberg extract node documentation
(#850)
8867dd5b32 is described below
commit 8867dd5b329d57f3e2b1dc3237e49a1e66799295
Author: haifxu <[email protected]>
AuthorDate: Fri Sep 15 14:55:01 2023 +0800
[INLONG-849][Doc] Add Iceberg extract node documentation (#850)
---
docs/data_node/extract_node/iceberg.md | 155 +++++++++++++++++++++
docs/data_node/extract_node/img/iceberg-source.png | Bin 0 -> 21082 bytes
.../current/data_node/extract_node/iceberg.md | 152 ++++++++++++++++++++
.../data_node/extract_node/img/iceberg-source.png | Bin 0 -> 22936 bytes
4 files changed, 307 insertions(+)
diff --git a/docs/data_node/extract_node/iceberg.md
b/docs/data_node/extract_node/iceberg.md
new file mode 100644
index 0000000000..7840136b23
--- /dev/null
+++ b/docs/data_node/extract_node/iceberg.md
@@ -0,0 +1,155 @@
+---
+title: Iceberg
+sidebar_position: 14
+---
+
+import {siteVariables} from '../../version';
+
+## Overview
+
+[Apache Iceberg](https://iceberg.apache.org/) is a high-performance format for
huge analytic tables.
+
+## Version
+
+| Extract Node | Version
|
+|-------------------------|--------------------------------------------------------------|
+| [Iceberg](./iceberg.md) | [Iceberg](https://iceberg.apache.org/): 0.12.x,
0.13.x <br/> |
+
+## Dependencies
+
+<pre><code parentName="pre">
+{`<dependency>
+ <groupId>org.apache.inlong</groupId>
+ <artifactId>sort-connector-iceberg</artifactId>
+ <version>${siteVariables.inLongVersion}</version>
+</dependency>
+`}
+</code></pre>
+
+## Usage
+
+Before creating the Iceberg task, we need a Flink environment integrated with
Hadoop.
+
+- Download [`Apache Hadoop`](https://hadoop.apache.org/releases.html).
+- Modify `jobmanager.sh` and `taskmanager.sh` and add `Hadoop` environment
variables.
+For commands, please refer to [Apache
Flink](https://github.com/apache/flink/tree/master/flink-dist/src/main/flink-bin/bin).
+
+```shell
+export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`
+```
+
+- Modify `docker-compose.yml` in the `docker/docker-compose` and mount
`Hadoop` and `Flink startup commands` into the container:
+
+```shell
+ jobmanager:
+ image: apache/flink:1.13-scala_2.11
+ container_name: jobmanager
+ user: root
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ volumes:
+ # Mount Hadoop
+ - HADOOP_HOME:HADOOP_HOME
+ # Mount the modified jobmanager.sh which adds the HADOOP_HOME env
correctly
+ - /jobmanager.sh:/opt/flink/bin/jobmanager.sh
+ ports:
+ - "8081:8081"
+ command: jobmanager
+
+ taskmanager:
+ image: apache/flink:1.13-scala_2.11
+ container_name: taskmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ taskmanager.numberOfTaskSlots: 2
+ volumes:
+ # Mount Hadoop
+ - HADOOP_HOME:HADOOP_HOME
+ # Mount the modified taskmanager.sh which adds the HADOOP_HOME env
correctly
+ - /taskmanager.sh:/opt/flink/bin/taskmanager.sh
+ command: taskmanager
+```
+
+### Flink SQL API
+
+Before using `Flink sql client`, `sql-client.sh` also needs to add Hadoop
environment variables and mounted to the container.
+For commands, please refer to [Apache
Flink](https://github.com/apache/flink/blob/master/flink-table/flink-sql-client/bin/sql-client.sh).
+
+```shell
+export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`
+```
+
+使用 `Flink sql cli`：
+
+```
+CREATE TABLE `iceberg_table_source`(
+ PRIMARY KEY (`_id`) NOT ENFORCED,
+ `_id` STRING,
+ `id` INT,
+ `name` STRING,
+ `age` INT)
+ WITH (
+ 'connector' = 'iceberg-inlong',
+ 'catalog-database' = 'DATABASES',
+ 'catalog-table' = 'TABLE',
+ 'catalog-type' = 'HIVE',
+ 'catalog-name' = 'HIVE',
+ 'streaming' = 'true',
+ 'uri' = 'thrift://127.0.0.1:9083'
+);
+```
+
+### Dashboard
+
+Source → Create → Iceberg
+
+
+
+### Manager Client
+
+TODO
+
+## Options
+
+| Options | Required | Type | Description
|
+|----------------------|----------|--------|---------------------------------------------------------------------------------------------------------------------------------------------------|
+| connector | required | String | Specify what connector to use,
here should be 'iceberg-inlong'
|
+| catalog-database | required | String | Database name managed in the
Iceberg directory
|
+| catalog-table | required | String | Table name managed in Iceberg
catalogs and databases
|
+| catalog-type | required | String | `hive` or `hadoop` for built-in
directories
|
+| catalog-name | required | String | directory name
|
+| uri | required | String | The thrift URI of Hive metastore,
such as: `thrift://127.0.0.1:9083`
|
+| warehouse | optional | String | For a Hive directory, the Hive
repository location. For the hadoop directory, it is the HDFS directory that
stores metadata files and data files. |
+| inlong.metric.labels | optional | String | In long metric label, the format
of value is groupId=xxgroup&streamId=xxstream&nodeId=xxnode
|
+
+## Data Type Mapping
+
+| Flink SQL Type | Iceberg Type |
+|----------------|--------------|
+| CHAR | STRING |
+| VARCHAR | STRING |
+| STRING | STRING |
+| BOOLEAN | BOOLEAN |
+| BINARY | FIXED(L) |
+| VARBINARY | BINARY |
+| DECIMAL | DECIMAL(P,S) |
+| TINYINT | INT |
+| SMALLINT | INT |
+| INTEGER | INT |
+| BIGINT | LONG |
+| FLOAT | FLOAT |
+| DOUBLE | DOUBLE |
+| DATE | DATE |
+| TIME | TIME |
+| TIMESTAMP | TIMESTAMP |
+| TIMESTAMP_LTZ | TIMESTAMPTZ |
+| INTERVAL | - |
+| ARRAY | LIST |
+| MULTISET | MAP |
+| MAP | MAP |
+| ROW | STRUCT |
+| RAW | - |
\ No newline at end of file
diff --git a/docs/data_node/extract_node/img/iceberg-source.png
b/docs/data_node/extract_node/img/iceberg-source.png
new file mode 100644
index 0000000000..a1de4ffe3a
Binary files /dev/null and b/docs/data_node/extract_node/img/iceberg-source.png
differ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/extract_node/iceberg.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/extract_node/iceberg.md
new file mode 100644
index 0000000000..4b9eb6e3cd
--- /dev/null
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/extract_node/iceberg.md
@@ -0,0 +1,152 @@
+---
+title: Iceberg
+sidebar_position: 14
+---
+
+import {siteVariables} from '../../version';
+
+## 概览
+
+[Apache Iceberg](https://iceberg.apache.org/) 是一种用于大型分析表的高性能格式。
+
+## 版本
+
+| 提取节点 | 版本
|
+|-------------------------|------------------------------------------------------|
+| [Iceberg](./iceberg.md) |
[Iceberg](https://iceberg.apache.org/)：0.12.x，0.13.x |
+
+## 依赖项
+
+<pre><code parentName="pre">
+{`<dependency>
+ <groupId>org.apache.inlong</groupId>
+ <artifactId>sort-connector-iceberg</artifactId>
+ <version>${siteVariables.inLongVersion}</version>
+</dependency>
+`}
+</code></pre>
+
+## 配置 Iceberg 数据抽取节点
+
+- 下载 [`Apache Hadoop`](https://hadoop.apache.org/releases.html)
+- 修改 `jobmanager.sh` 和 `taskmanager.sh`，加入 `Hadoop` 环境变量。启动命令可以参考 [Apache
Flink](https://github.com/apache/flink/tree/master/flink-dist/src/main/flink-bin/bin)
+
+```shell
+export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`
+```
+
+- 修改 `docker/docker-compose` 目录下的 `docker-compose.yml`，将 `Hadoop` 和 `Flink
启动命令` 挂载至容器中:
+
+```shell
+ jobmanager:
+ image: apache/flink:1.13-scala_2.11
+ container_name: jobmanager
+ user: root
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ volumes:
+ # Mount Hadoop
+ - HADOOP_HOME:HADOOP_HOME
+ # Mount the modified jobmanager.sh which adds the HADOOP_HOME env
correctly
+ - /jobmanager.sh:/opt/flink/bin/jobmanager.sh
+ ports:
+ - "8081:8081"
+ command: jobmanager
+
+ taskmanager:
+ image: apache/flink:1.13-scala_2.11
+ container_name: taskmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ taskmanager.numberOfTaskSlots: 2
+ volumes:
+ # Mount Hadoop
+ - HADOOP_HOME:HADOOP_HOME
+ # Mount the modified taskmanager.sh which adds the HADOOP_HOME env
correctly
+ - /taskmanager.sh:/opt/flink/bin/taskmanager.sh
+ command: taskmanager
+```
+
+### Flink SQL API
+
+使用 Flink sql client 之前，`sql-client.sh` 启动命令也需要添加 Hadoop 环境变量，并挂载至容器。
+启动命令可以参考 [Apache
Flink](https://github.com/apache/flink/blob/master/flink-table/flink-sql-client/bin/sql-client.sh)
+
+```shell
+export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`
+```
+
+使用 `Flink sql cli`：
+
+```
+CREATE TABLE `iceberg_table_source`(
+ PRIMARY KEY (`_id`) NOT ENFORCED,
+ `_id` STRING,
+ `id` INT,
+ `name` STRING,
+ `age` INT)
+ WITH (
+ 'connector' = 'iceberg-inlong',
+ 'catalog-database' = 'DATABASES',
+ 'catalog-table' = 'TABLE',
+ 'catalog-type' = 'HIVE',
+ 'catalog-name' = 'HIVE',
+ 'streaming' = 'true',
+ 'uri' = 'thrift://127.0.0.1:9083'
+);
+```
+
+### Dashboard
+
+页面点击 数据源 → 新建 → Iceberg
+
+
+
+### Manager Client
+
+TODO
+
+## 参数信息
+
+| 选项 | 必填 | 类型 | 描述
|
+|----------------------|-----|--------|----------------------------------------------------------------------------------|
+| connector | 必填 | String | 指定要使用的 Connector，这里应该是
'iceberg-inlong' |
+| catalog-database | 必填 | String | 在 Iceberg 目录中管理的数据库名称
|
+| catalog-table | 必填 | String | 在 Iceberg 目录和数据库中管理的表名
|
+| catalog-type | 必填 | String | `hive` 或 `hadoop` 用于内置目录
|
+| catalog-name | 必填 | String | 目录名称
|
+| uri | 必填 | String | Hive 元存储的 thrift
URI，如：`thrift://127.0.0.1:9083` |
+| warehouse | 可选 | String | 对于 Hive 目录，是 Hive 仓库位置。对于 hadoop 目录，是
HDFS 目录存放元数据文件和数据文件 |
+| inlong.metric.labels | 可选 | String | 在 long metric label 中，value 的格式为
groupId=xxgroup&streamId=xxstream&nodeId=xxnode |
+
+## 数据类型映射
+
+| Flink SQL Type | Iceberg Type |
+|----------------|--------------|
+| CHAR | STRING |
+| VARCHAR | STRING |
+| STRING | STRING |
+| BOOLEAN | BOOLEAN |
+| BINARY | FIXED(L) |
+| VARBINARY | BINARY |
+| DECIMAL | DECIMAL(P,S) |
+| TINYINT | INT |
+| SMALLINT | INT |
+| INTEGER | INT |
+| BIGINT | LONG |
+| FLOAT | FLOAT |
+| DOUBLE | DOUBLE |
+| DATE | DATE |
+| TIME | TIME |
+| TIMESTAMP | TIMESTAMP |
+| TIMESTAMP_LTZ | TIMESTAMPTZ |
+| INTERVAL | - |
+| ARRAY | LIST |
+| MULTISET | MAP |
+| MAP | MAP |
+| ROW | STRUCT |
+| RAW | - |
\ No newline at end of file
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/extract_node/img/iceberg-source.png
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/extract_node/img/iceberg-source.png
new file mode 100644
index 0000000000..2444882e80
Binary files /dev/null and
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/extract_node/img/iceberg-source.png
differ
