This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 7cc0e1b27cc [docs] add new data types in the Flink connector. (#599)
7cc0e1b27cc is described below
commit 7cc0e1b27ccbbd9464b66e502af937a53f53a507
Author: Petrichor <[email protected]>
AuthorDate: Mon Apr 29 15:51:47 2024 +0800
[docs] add new data types in the Flink connector. (#599)
---
docs/ecosystem/flink-doris-connector.md | 18 +++++++++++++-----
.../current/ecosystem/flink-doris-connector.md | 14 ++++++++++----
.../version-2.1/ecosystem/flink-doris-connector.md | 20 +++++++++++++-------
.../version-2.1/ecosystem/flink-doris-connector.md | 17 +++++++++++------
4 files changed, 47 insertions(+), 22 deletions(-)
diff --git a/docs/ecosystem/flink-doris-connector.md
b/docs/ecosystem/flink-doris-connector.md
index 52738e1238c..b9666394a3f 100644
--- a/docs/ecosystem/flink-doris-connector.md
+++ b/docs/ecosystem/flink-doris-connector.md
@@ -57,7 +57,7 @@ Add flink-doris-connector
<dependency>
<groupId>org.apache.doris</groupId>
<artifactId>flink-doris-connector-1.16</artifactId>
- <version>1.5.2</version>
+ <version>1.6.0</version>
</dependency>
```
@@ -106,7 +106,7 @@ DorisOptions.Builder builder = DorisOptions.builder()
.setUsername("root")
.setPassword("password");
-DorisSource<List<?>> dorisSource = DorisSourceBuilder.<List<?>>builder()
+DorisSource<List<?>> dorisSource = DorisSource.<List<?>>builder()
.setDorisOptions(builder.build())
.setDorisReadOptions(DorisReadOptions.builder().build())
.setDeserializer(new SimpleListDeserializationSchema())
@@ -387,8 +387,15 @@ ON a.city = c.city
| VARCHAR | STRING |
| STRING | STRING |
| DECIMALV2 | DECIMAL |
-| TIME | DOUBLE |
-| HLL | Unsupported datatype |
+| ARRAY | ARRAY |
+| MAP | MAP |
+| JSON | STRING |
+| VARIANT | STRING |
+| IPV4 | STRING |
+| IPV6 | STRING |
+
+> Starting from version connector-1.6.1, support is added for reading three
data types: Variant, IPV6, and IPV4. Reading IPV6 and Variant requires Doris
version 2.1.1 or higher.
+
## Flink write Metrics
Where the metrics value of type Counter is the cumulative value of the
imported task from the beginning to the current time, you can observe each
metric in each table in the Flink Webui metrics.
@@ -533,10 +540,11 @@ insert into doris_sink select id,name,bank,age from
cdc_mysql_source;
| --sink-conf | All configurations of Doris Sink can be found
[here](https://doris.apache.org/zh-CN/docs/dev/ecosystem/flink-doris-connector/#%E9%80%9A%E7%94%A8%E9%85%8D%E7%BD%AE%E9%A1%B9)
View the complete configuration items. |
| --table-conf | The configuration items of the Doris table(The
exception is table-buckets, non-properties attributes), that is, the content
contained in properties. For example `--table-conf replication_num=1`, and the
`--table-conf table-buckets="tbl1:10,tbl2:20,a.*:30,b.*:40,.*:50"` option
specifies the number of buckets for different tables based on the order of
regular expressions. If there is no match, the table is created with the
default setting of BUCKETS AUTO. |
| --ignore-default-value | Turn off the default value of synchronizing mysql
table structure. It is suitable for synchronizing mysql data to doris when the
field has a default value but the actual inserted data is null. Reference
[here](https://github.com/apache/doris-flink-connector/pull/152) |
-| --use-new-schema-change | Whether to use the new schema change to support
synchronization of MySQL multi-column changes and default values. Reference
[here](https://github.com/apache/doris-flink-connector/pull/167) |
+| --use-new-schema-change | Whether to use the new schema change to support
synchronization of MySQL multi-column changes and default values. since version
1.6.0, the default value has been set to true. Reference
[here](https://github.com/apache/doris-flink-connector/pull/167) |
| --single-sink | Whether to use a single Sink to synchronize all
tables. When turned on, newly created tables in the upstream can also be
automatically recognized and tables automatically created. |
| --multi-to-one-origin | When writing multiple upstream tables into the
same table, the configuration of the source table, for example:
--multi-to-one-origin="a\_.\*｜b_.\*"， Reference
[here](https://github.com/apache/doris-flink-connector/pull/208) |
| --multi-to-one-target | Used with multi-to-one-origin, the configuration
of the target table, such as: --multi-to-one-target="a\|b" |
+| --create-table-only | Whether only the table schema should be
synchronized |
>Note: When synchronizing, you need to add the corresponding Flink CDC
>dependencies in the $FLINK_HOME/lib directory, such as
>flink-sql-connector-mysql-cdc-${version}.jar,
>flink-sql-connector-oracle-cdc-${version}.jar
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/ecosystem/flink-doris-connector.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/ecosystem/flink-doris-connector.md
index feef8590138..2c48e3c2fda 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/ecosystem/flink-doris-connector.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/ecosystem/flink-doris-connector.md
@@ -60,7 +60,7 @@ under the License.
<dependency>
<groupId>org.apache.doris</groupId>
<artifactId>flink-doris-connector-1.16</artifactId>
- <version>1.5.2</version>
+ <version>1.6.0</version>
</dependency>
```
@@ -109,7 +109,7 @@ DorisOptions.Builder builder = DorisOptions.builder()
.setUsername("root")
.setPassword("password");
-DorisSource<List<?>> dorisSource = DorisSourceBuilder.<List<?>>builder()
+DorisSource<List<?>> dorisSource = DorisSource.<List<?>>builder()
.setDorisOptions(builder.build())
.setDorisReadOptions(DorisReadOptions.builder().build())
.setDeserializer(new SimpleListDeserializationSchema())
@@ -399,6 +399,11 @@ ON a.city = c.city
| ARRAY | ARRAY |
| MAP | MAP |
| JSON | STRING |
+| VARIANT | STRING |
+| IPV4 | STRING |
+| IPV6 | STRING |
+
+> 自connector-1.6.1开始支持读取Variant,IPV6,IPV4三种数据类型，其中读取IPV6，Variant
需Doris版本2.1.1及以上。
## Flink 写入指标
@@ -542,10 +547,11 @@ insert into doris_sink select id,name,bank,age from
cdc_mysql_source;
| --sink-conf | Doris Sink
的所有配置，可以在[这里](https://doris.apache.org/zh-CN/docs/dev/ecosystem/flink-doris-connector/#%E9%80%9A%E7%94%A8%E9%85%8D%E7%BD%AE%E9%A1%B9)查看完整的配置项。
|
| --table-conf | Doris 表的配置项，即 properties 中包含的内容（其中 table-buckets
例外，非 properties 属性）。例如 `--table-conf replication_num=1`，而 `--table-conf
table-buckets="tbl1:10,tbl2:20,a.*:30,b.*:40,.*:50"`表示按照正则表达式顺序指定不同表的 buckets
数量，如果没有匹配到则采用 BUCKETS AUTO 建表。 |
| --ignore-default-value | 关闭同步 mysql 表结构的默认值。适用于同步 mysql 数据到 doris
时，字段有默认值，但实际插入数据为 null
情况。参考[#152](https://github.com/apache/doris-flink-connector/pull/152) |
-| --use-new-schema-change | 是否使用新的 schema change，支持同步 mysql
多列变更、默认值。参考[#167](https://github.com/apache/doris-flink-connector/pull/167) |
+| --use-new-schema-change | 是否使用新的 schema change，支持同步 mysql 多列变更、默认值，1.6.0
开始该参数默认为true。参考[#167](https://github.com/apache/doris-flink-connector/pull/167)
|
| --single-sink | 是否使用单个 Sink 同步所有表，开启后也可自动识别上游新创建的表，自动创建表。 |
-| --multi-to-one-origin |
将上游多张表写入同一张表时，源表的配置，比如：--multi-to-one-origin="a\_.\*\|b_.\*"，具体参考[这里](https://github.com/apache/doris-flink-connector/pull/208)
|
+| --multi-to-one-origin |
将上游多张表写入同一张表时，源表的配置，比如：--multi-to-one-origin="a\_.\*\|b_.\*"，具体参考[#208](https://github.com/apache/doris-flink-connector/pull/208)
|
| --multi-to-one-target | 与 multi-to-one-origin
搭配使用，目标表的配置，比如：--multi-to-one-target="a\|b" |
+| --create-table-only | 是否只仅仅同步表结构
>注：同步时需要在$FLINK_HOME/lib 目录下添加对应的 Flink CDC 依赖，比如
>flink-sql-connector-mysql-cdc-${version}.jar，flink-sql-connector-oracle-cdc-${version}.jar
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/ecosystem/flink-doris-connector.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/ecosystem/flink-doris-connector.md
index feef8590138..facea6b19c6 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/ecosystem/flink-doris-connector.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/ecosystem/flink-doris-connector.md
@@ -47,7 +47,7 @@ under the License.
| 1.3.0 | 1.16 | 1.0+ | 8 | - |
| 1.4.0 | 1.15,1.16,1.17 | 1.0+ | 8 |- |
| 1.5.2 | 1.15,1.16,1.17,1.18 | 1.0+ | 8 |- |
-| 1.6.0 | 1.15,1.16,1.17,1.18,1.19 | 1.0+ | 8 |- |
+| 1.6.0 | 1.15,1.16,1.17,1.18,1.19 | 1.0+ | 8 | - |
## 使用
@@ -60,7 +60,7 @@ under the License.
<dependency>
<groupId>org.apache.doris</groupId>
<artifactId>flink-doris-connector-1.16</artifactId>
- <version>1.5.2</version>
+ <version>1.6.0</version>
</dependency>
```
@@ -109,7 +109,7 @@ DorisOptions.Builder builder = DorisOptions.builder()
.setUsername("root")
.setPassword("password");
-DorisSource<List<?>> dorisSource = DorisSourceBuilder.<List<?>>builder()
+DorisSource<List<?>> dorisSource = DorisSource.<List<?>>builder()
.setDorisOptions(builder.build())
.setDorisReadOptions(DorisReadOptions.builder().build())
.setDeserializer(new SimpleListDeserializationSchema())
@@ -397,8 +397,13 @@ ON a.city = c.city
| STRING | STRING |
| DECIMALV2 | DECIMAL |
| ARRAY | ARRAY |
-| MAP | MAP |
-| JSON | STRING |
+| MAP | MAP |
+| JSON | STRING |
+| VARIANT | STRING |
+| IPV4 | STRING |
+| IPV6 | STRING |
+
+> 自connector-1.6.1开始支持读取Variant,IPV6,IPV4三种数据类型，其中读取IPV6，Variant
需Doris版本2.1.1及以上。
## Flink 写入指标
@@ -542,10 +547,11 @@ insert into doris_sink select id,name,bank,age from
cdc_mysql_source;
| --sink-conf | Doris Sink
的所有配置，可以在[这里](https://doris.apache.org/zh-CN/docs/dev/ecosystem/flink-doris-connector/#%E9%80%9A%E7%94%A8%E9%85%8D%E7%BD%AE%E9%A1%B9)查看完整的配置项。
|
| --table-conf | Doris 表的配置项，即 properties 中包含的内容（其中 table-buckets
例外，非 properties 属性）。例如 `--table-conf replication_num=1`，而 `--table-conf
table-buckets="tbl1:10,tbl2:20,a.*:30,b.*:40,.*:50"`表示按照正则表达式顺序指定不同表的 buckets
数量，如果没有匹配到则采用 BUCKETS AUTO 建表。 |
| --ignore-default-value | 关闭同步 mysql 表结构的默认值。适用于同步 mysql 数据到 doris
时，字段有默认值，但实际插入数据为 null
情况。参考[#152](https://github.com/apache/doris-flink-connector/pull/152) |
-| --use-new-schema-change | 是否使用新的 schema change，支持同步 mysql
多列变更、默认值。参考[#167](https://github.com/apache/doris-flink-connector/pull/167) |
+| --use-new-schema-change | 是否使用新的 schema change，支持同步 mysql 多列变更、默认值。1.6.0
默认为true。参考[#167](https://github.com/apache/doris-flink-connector/pull/167) |
| --single-sink | 是否使用单个 Sink 同步所有表，开启后也可自动识别上游新创建的表，自动创建表。 |
-| --multi-to-one-origin |
将上游多张表写入同一张表时，源表的配置，比如：--multi-to-one-origin="a\_.\*\|b_.\*"，具体参考[这里](https://github.com/apache/doris-flink-connector/pull/208)
|
+| --multi-to-one-origin |
将上游多张表写入同一张表时，源表的配置，比如：--multi-to-one-origin="a\_.\*\|b_.\*"，具体参考[#208](https://github.com/apache/doris-flink-connector/pull/208)
|
| --multi-to-one-target | 与 multi-to-one-origin
搭配使用，目标表的配置，比如：--multi-to-one-target="a\|b" |
+| --create-table-only | 是否只仅仅创建表的结构
>注：同步时需要在$FLINK_HOME/lib 目录下添加对应的 Flink CDC 依赖，比如
>flink-sql-connector-mysql-cdc-${version}.jar，flink-sql-connector-oracle-cdc-${version}.jar
diff --git a/versioned_docs/version-2.1/ecosystem/flink-doris-connector.md
b/versioned_docs/version-2.1/ecosystem/flink-doris-connector.md
index 52738e1238c..baa33ca8c7f 100644
--- a/versioned_docs/version-2.1/ecosystem/flink-doris-connector.md
+++ b/versioned_docs/version-2.1/ecosystem/flink-doris-connector.md
@@ -45,6 +45,7 @@ under the License.
| 1.3.0 | 1.16 | 1.0+ | 8 | - |
| 1.4.0 | 1.15,1.16,1.17 | 1.0+ | 8 |- |
| 1.5.2 | 1.15,1.16,1.17,1.18 | 1.0+ | 8 |- |
+| 1.6.0 | 1.15,1.16,1.17,1.18,1.19| 1.0+ | 8 | - |
## USE
@@ -57,7 +58,7 @@ Add flink-doris-connector
<dependency>
<groupId>org.apache.doris</groupId>
<artifactId>flink-doris-connector-1.16</artifactId>
- <version>1.5.2</version>
+ <version>1.6.0</version>
</dependency>
```
@@ -106,7 +107,7 @@ DorisOptions.Builder builder = DorisOptions.builder()
.setUsername("root")
.setPassword("password");
-DorisSource<List<?>> dorisSource = DorisSourceBuilder.<List<?>>builder()
+DorisSource<List<?>> dorisSource = DorisSource.<List<?>>builder()
.setDorisOptions(builder.build())
.setDorisReadOptions(DorisReadOptions.builder().build())
.setDeserializer(new SimpleListDeserializationSchema())
@@ -387,9 +388,12 @@ ON a.city = c.city
| VARCHAR | STRING |
| STRING | STRING |
| DECIMALV2 | DECIMAL |
-| TIME | DOUBLE |
-| HLL | Unsupported datatype |
+| JSON | STRING |
+| VARIANT | STRING |
+| IPV4 | STRING |
+| IPV6 | STRING |
+> Starting from version connector-1.6.1, support is added for reading three
data types: Variant, IPV6, and IPV4. Reading IPV6 and Variant requires Doris
version 2.1.1 or higher.
## Flink write Metrics
Where the metrics value of type Counter is the cumulative value of the
imported task from the beginning to the current time, you can observe each
metric in each table in the Flink Webui metrics.
@@ -533,10 +537,11 @@ insert into doris_sink select id,name,bank,age from
cdc_mysql_source;
| --sink-conf | All configurations of Doris Sink can be found
[here](https://doris.apache.org/zh-CN/docs/dev/ecosystem/flink-doris-connector/#%E9%80%9A%E7%94%A8%E9%85%8D%E7%BD%AE%E9%A1%B9)
View the complete configuration items. |
| --table-conf | The configuration items of the Doris table(The
exception is table-buckets, non-properties attributes), that is, the content
contained in properties. For example `--table-conf replication_num=1`, and the
`--table-conf table-buckets="tbl1:10,tbl2:20,a.*:30,b.*:40,.*:50"` option
specifies the number of buckets for different tables based on the order of
regular expressions. If there is no match, the table is created with the
default setting of BUCKETS AUTO. |
| --ignore-default-value | Turn off the default value of synchronizing mysql
table structure. It is suitable for synchronizing mysql data to doris when the
field has a default value but the actual inserted data is null. Reference
[here](https://github.com/apache/doris-flink-connector/pull/152) |
-| --use-new-schema-change | Whether to use the new schema change to support
synchronization of MySQL multi-column changes and default values. Reference
[here](https://github.com/apache/doris-flink-connector/pull/167) |
+| --use-new-schema-change | Whether to use the new schema change to support
synchronization of MySQL multi-column changes and default values. since version
1.6.0, the default value has been set to true. Reference
[here](https://github.com/apache/doris-flink-connector/pull/167) |
| --single-sink | Whether to use a single Sink to synchronize all
tables. When turned on, newly created tables in the upstream can also be
automatically recognized and tables automatically created. |
| --multi-to-one-origin | When writing multiple upstream tables into the
same table, the configuration of the source table, for example:
--multi-to-one-origin="a\_.\*｜b_.\*"， Reference
[here](https://github.com/apache/doris-flink-connector/pull/208) |
-| --multi-to-one-target | Used with multi-to-one-origin, the configuration
of the target table, such as: --multi-to-one-target="a\|b" |
+| --multi-to-one-target | Used with multi-to-one-origin, the configuration
of the target table, such as: --multi-to-one-target="a\|b" |
+| --create-table-only | Whether only the table schema should be
synchronized |
>Note: When synchronizing, you need to add the corresponding Flink CDC
>dependencies in the $FLINK_HOME/lib directory, such as
>flink-sql-connector-mysql-cdc-${version}.jar,
>flink-sql-connector-oracle-cdc-${version}.jar
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
