This is an automated email from the ASF dual-hosted git repository.
davidzollo pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new cf2a3e1c8f [Docs][Connector-V2] Improve Assert ClickHouse and Doris
docs (#11299)
cf2a3e1c8f is described below
commit cf2a3e1c8f787fded994ca65a037f7136acdf475
Author: Daniel Carter <[email protected]>
AuthorDate: Fri Jul 3 22:59:25 2026 +0800
[Docs][Connector-V2] Improve Assert ClickHouse and Doris docs (#11299)
Co-authored-by: DanielCarter-stack
<[email protected]>
---
docs/en/connectors/sink/Assert.md | 10 ++++++++--
docs/en/connectors/sink/Clickhouse.md | 4 ++--
docs/en/connectors/sink/Doris.md | 2 +-
docs/en/connectors/source/Clickhouse.md | 3 +++
docs/en/connectors/source/Doris.md | 11 +++++++----
docs/zh/connectors/sink/Assert.md | 12 +++++++++---
docs/zh/connectors/sink/Clickhouse.md | 6 +++---
docs/zh/connectors/sink/Doris.md | 5 +++--
docs/zh/connectors/source/Clickhouse.md | 11 +++++++----
docs/zh/connectors/source/Doris.md | 11 +++++++----
10 files changed, 50 insertions(+), 25 deletions(-)
diff --git a/docs/en/connectors/sink/Assert.md
b/docs/en/connectors/sink/Assert.md
index 2f7e5ddfce..c69f2139ec 100644
--- a/docs/en/connectors/sink/Assert.md
+++ b/docs/en/connectors/sink/Assert.md
@@ -12,20 +12,21 @@ A sink plugin which can assert illegal data by user defined
rules
- [ ] [exactly-once](../../introduction/concepts/connector-v2-features.md)
- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md)
+- [x] [support multiple table
write](../../introduction/concepts/connector-v2-features.md)
## Options
| Name
| Type | Required
| Default |
|------------------------------------------------------------------------------------------------|-------------------------------------------------|----------|---------|
| rules
| ConfigMap | yes
| - |
-| rules.field_rules
| string | yes
| - |
+| rules.field_rules
| ConfigList | no
| - |
| rules.field_rules.field_name
| string\|ConfigMap | yes
| - |
| rules.field_rules.field_type
| string | no
| - |
| rules.field_rules.field_value
| ConfigList | no
| - |
| rules.field_rules.field_value.rule_type
| string | no
| - |
| rules.field_rules.field_value.rule_value
| numeric | no
| - |
| rules.field_rules.field_value.equals_to
| boolean\|numeric\|string\|ConfigList\|ConfigMap | no
| - |
-| rules.row_rules
| string | yes
| - |
+| rules.row_rules
| ConfigList | no
| - |
| rules.row_rules.rule_type
| string | no
| - |
| rules.row_rules.rule_value
| string | no
| - |
| rules.catalog_table_rule
| ConfigMap | no
| - |
@@ -48,6 +49,7 @@ A sink plugin which can assert illegal data by user defined
rules
| rules.table-names
| ConfigList | no
| - |
| rules.tables_configs
| ConfigList | no
| - |
| rules.tables_configs.table_path
| String | no
| - |
+| multi_table_sink_replica
| int | no
| - |
| common-options
| | no
| - |
### rules [ConfigMap]
@@ -110,6 +112,10 @@ Used to assert the multiple tables should be in the data.
The path of the table.
+### multi_table_sink_replica [int]
+
+The replica number used by the multi-table sink common option. Configure it
only when the job needs more than one sink replica per table.
+
### common options
Sink plugin common parameters, please refer to [Sink Common
Options](../common-options/sink-common-options.md) for details
diff --git a/docs/en/connectors/sink/Clickhouse.md
b/docs/en/connectors/sink/Clickhouse.md
index cef70da88f..615d252013 100644
--- a/docs/en/connectors/sink/Clickhouse.md
+++ b/docs/en/connectors/sink/Clickhouse.md
@@ -58,8 +58,8 @@ They can be downloaded via install-plugin.sh or from the
Maven central repositor
| username | String | Yes | - |
`ClickHouse` user username.
|
| password | String | Yes | - |
`ClickHouse` user password.
|
| clickhouse.config | Map | No | | In
addition to the above mandatory parameters that must be specified by
`clickhouse-jdbc` , users can also specify multiple optional parameters, which
cover all the
[parameters](https://github.com/ClickHouse/clickhouse-jdbc/tree/master/clickhouse-client#configuration)
provided by `clickhouse-jdbc`. |
-| bulk_size | String | No | 20000 | The
number of rows written through
[Clickhouse-jdbc](https://github.com/ClickHouse/clickhouse-jdbc) each time, the
`default is 20000`.
|
-| split_mode | String | No | false | This
mode only support clickhouse table which engine is 'Distributed'.And
`internal_replication` option-should be `true`.They will split distributed
table data in seatunnel and perform write directly on each shard. The shard
weight define is clickhouse will counted. |
+| bulk_size | int | No | 20000 | The
number of rows written through
[Clickhouse-jdbc](https://github.com/ClickHouse/clickhouse-jdbc) each time, the
`default is 20000`.
|
+| split_mode | Boolean | No | false | This
mode only support clickhouse table which engine is 'Distributed'.And
`internal_replication` option-should be `true`.They will split distributed
table data in seatunnel and perform write directly on each shard. The shard
weight define is clickhouse will counted. |
| sharding_key | String | No | - | When
use split_mode, which node to send data to is a problem, the default is random
selection, but the 'sharding_key' parameter can be used to specify the field
for the sharding algorithm. This option only worked when 'split_mode' is true.
|
| primary_key | String | No | - | Mark
the primary key column from clickhouse table, and based on primary key execute
INSERT/UPDATE/DELETE to clickhouse table.
|
| support_upsert | Boolean | No | false |
Support upsert row by query primary key.
|
diff --git a/docs/en/connectors/sink/Doris.md b/docs/en/connectors/sink/Doris.md
index ab622f0657..f2b06f5c0f 100644
--- a/docs/en/connectors/sink/Doris.md
+++ b/docs/en/connectors/sink/Doris.md
@@ -50,7 +50,7 @@ The internal implementation of Doris sink connector is cached
and imported by st
| password | String | Yes | -
| `Doris` user password
|
| database | String | Yes | -
| The database name of `Doris` table, use `${database_name}` to represent
the upstream table name
|
| table | String | Yes | -
| The table name of `Doris` table, use `${table_name}` to represent the
upstream table name
|
-| table.identifier | String | Yes | -
| The name of `Doris` table, it will deprecate after version 2.3.5,
please use `database` and `table` instead.
|
+| table.identifier | String | No | -
| Deprecated table identifier. Please use `database` and `table` instead.
|
| sink.label-prefix | String | Yes | -
| The label prefix used by stream load imports. In the 2pc scenario,
global uniqueness is required to ensure the EOS semantics of SeaTunnel.
|
| sink.enable-2pc | bool | No | false
| Whether to enable two-phase commit (2pc), the default is false. For
two-phase commit, please refer to
[here](https://doris.apache.org/docs/data-operate/transaction?_highlight=two&_highlight=phase#stream-load-2pc).
|
| sink.enable-delete | bool | No | -
| Whether to enable deletion. This option requires Doris table to enable
batch delete function (0.15+ version is enabled by default), and only supports
Unique model. you can get more detail at this
[link](https://doris.apache.org/docs/dev/data-operate/delete/batch-delete-manual/)
|
diff --git a/docs/en/connectors/source/Clickhouse.md
b/docs/en/connectors/source/Clickhouse.md
index 3ab0eaace4..9964bed115 100644
--- a/docs/en/connectors/source/Clickhouse.md
+++ b/docs/en/connectors/source/Clickhouse.md
@@ -70,6 +70,7 @@ Table list configuration:
| sql | String | NO | - | The query
sql used to search data though Clickhouse server.
|
| filter_query | String | NO | - | Data
filtering in Clickhouse. the format is "field = value", example : filter_query
= "id > 2 and type = 1"
|
| partition_list | Array | NO | - | Table
partition list to filter the specified partition. If it is a partitioned table,
this field can be configured to filter the data of the specified partition.
example: partition_list = ["20250615", "20250616"]
|
+| split.size | int | NO | Integer.MAX_VALUE | The number
of ClickHouse parts grouped into each SeaTunnel split. The minimum value is
`1`. Smaller values create more splits for higher parallelism.
|
| batch_size | int | NO | 1024 | The maximum
rows of data that can be obtained by reading from Clickhouse once.
|
Note: When this configuration corresponds to a single table, you can flatten
the configuration items in table_list to the outer layer.
@@ -88,6 +89,7 @@ If both the `table_path` and `sql` parameters are set, it
will be executed in sq
In query table mode, if you don't want to read the entire table, you can
specify the `partition_list` or `filter_query` parameter.
* `partition_list`: filter the data of the specified partition
* `filter_query`: filter the data based on the specified conditions
+* `split.size`: control how many ClickHouse parts are grouped into one
SeaTunnel split when reading by `table_path`
The `batch_size` parameter can be used to control the amount of data read each
time to avoid OOM exception when reading a large amount of data. Appropriately
increasing this value will help to improve the performance of the reading
process.
@@ -114,6 +116,7 @@ source {
server_time_zone = "UTC"
partition_list = ["20250615", "20250616"]
filter_query = "id > 2 and type = 1"
+ split.size = 1
batch_size = 1024
clickhouse.config = {
"socket_timeout": "300000"
diff --git a/docs/en/connectors/source/Doris.md
b/docs/en/connectors/source/Doris.md
index af1843bada..9f2bafb340 100644
--- a/docs/en/connectors/source/Doris.md
+++ b/docs/en/connectors/source/Doris.md
@@ -68,11 +68,14 @@ Base configuration:
| username | string | yes | - | User
username
|
| password | string | yes | - | User
password
|
| doris.request.retries | int | no | 3 | Number
of retries to send requests to Doris FE.
|
-| doris.request.read.timeout.ms | int | no | 30000 |
|
-| doris.request.connect.timeout.ms | int | no | 30000 |
|
+| doris.request.read.timeout.ms | int | no | 30000 | Socket
read timeout for requests sent to Doris BE.
|
+| doris.request.connect.timeout.ms | int | no | 30000 |
Connection timeout for requests sent to Doris FE or BE.
|
| query-port | string | no | 9030 | Doris
QueryPort
|
| doris.request.query.timeout.s | int | no | 3600 | Timeout
period of Doris scan data, expressed in seconds.
|
-| table_list | string | 否 | - | table
list
|
+| doris.request.tablet.size | int | no | Integer.MAX_VALUE |
The number of Doris tablets grouped into each SeaTunnel split. The minimum
value is `1`. |
+| doris.deserialize.arrow.async | boolean | no | false | Whether
to deserialize Arrow data asynchronously.
|
+| doris.request.retriesdoris.deserialize.queue.size | int | no | 64 | Queue
size used by asynchronous Arrow deserialization.
|
+| table_list | Array | no | - | List of
Doris tables to read.
|
Table list configuration:
@@ -225,4 +228,4 @@ sink{
## Changelog
-<ChangeLog />
\ No newline at end of file
+<ChangeLog />
diff --git a/docs/zh/connectors/sink/Assert.md
b/docs/zh/connectors/sink/Assert.md
index fd2ce77ba3..7bfc7c2e14 100644
--- a/docs/zh/connectors/sink/Assert.md
+++ b/docs/zh/connectors/sink/Assert.md
@@ -11,21 +11,22 @@ Assert 数据接收器是一个用于断言数据是否符合用户定义规则
## 核心特性
- [ ] [精准一次](../../introduction/concepts/connector-v2-features.md)
-- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md)
+- [ ] [定时刷新](../../introduction/concepts/connector-v2-features.md)
+- [x] [支持多表写入](../../introduction/concepts/connector-v2-features.md)
## 配置
| Name
| Type | Required
| Default |
|------------------------------------------------------------------------------------------------|-------------------------------------------------|----------|---------|
| rules
| ConfigMap | yes
| - |
-| rules.field_rules
| string | yes
| - |
+| rules.field_rules
| ConfigList | no
| - |
| rules.field_rules.field_name
| string\|ConfigMap | yes
| - |
| rules.field_rules.field_type
| string | no
| - |
| rules.field_rules.field_value
| ConfigList | no
| - |
| rules.field_rules.field_value.rule_type
| string | no
| - |
| rules.field_rules.field_value.rule_value
| numeric | no
| - |
| rules.field_rules.field_value.equals_to
| boolean\|numeric\|string\|ConfigList\|ConfigMap | no
| - |
-| rules.row_rules
| string | yes
| - |
+| rules.row_rules
| ConfigList | no
| - |
| rules.row_rules.rule_type
| string | no
| - |
| rules.row_rules.rule_value
| string | no
| - |
| rules.catalog_table_rule
| ConfigMap | no
| - |
@@ -48,6 +49,7 @@ Assert 数据接收器是一个用于断言数据是否符合用户定义规则
| rules.table-names
| ConfigList | no
| - |
| rules.tables_configs
| ConfigList | no
| - |
| rules.tables_configs.table_path
| String | no
| - |
+| multi_table_sink_replica
| int | no
| - |
| common-options
| | no
| - |
### rules [ConfigMap]
@@ -111,6 +113,10 @@ catalog_table_rule用于断言Catalog表是否与用户定义的表相同。
表的路径。
+### multi_table_sink_replica [int]
+
+多表写入通用参数中的副本数。只有在每张表需要多个 sink 副本时才需要配置。
+
### common options
Sink 插件的通用参数，请参考 [Sink Common
Options](../common-options/sink-common-options.md) 了解详情
diff --git a/docs/zh/connectors/sink/Clickhouse.md
b/docs/zh/connectors/sink/Clickhouse.md
index c4ff751f65..f016248c33 100644
--- a/docs/zh/connectors/sink/Clickhouse.md
+++ b/docs/zh/connectors/sink/Clickhouse.md
@@ -17,7 +17,7 @@ import ChangeLog from '../changelog/connector-clickhouse.md';
> Clickhouse sink 插件通过实现幂等写入可以达到精准一次，需要配合 aggregating merge tree 支持重复数据删除的引擎。
- [x] [支持多表写入](../../introduction/concepts/connector-v2-features.md)
-- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md)
+- [ ] [定时刷新](../../introduction/concepts/connector-v2-features.md)
## 描述
@@ -57,8 +57,8 @@ import ChangeLog from '../changelog/connector-clickhouse.md';
| username | String | Yes | - |
`ClickHouse` 用户账号.
|
| password | String | Yes | - |
`ClickHouse` 用户密码.
|
| clickhouse.config | Map | No | | 除了上述必须由
`clickhouse-jdbc` 指定的必填参数外，用户还可以指定多个可选参数，这些参数涵盖了 `clickhouse-jdbc`
提供的所有[参数](https://github.com/ClickHouse/clickhouse-jdbc/tree/master/clickhouse-client#configuration).
|
-| bulk_size | String | No | 20000 |
每次通过[Clickhouse-jdbc](https://github.com/ClickHouse/clickhouse-jdbc)
写入的行数，即默认是20000.
|
-| split_mode | String | No | false |
此模式仅支持引擎为`Distributed`的 `clickhouse` 表。选项 `internal_replication` 应该是 `true`
。他们将在 seatunnel 中拆分分布式表数据，并直接对每个分片进行写入。分片权重定义为 `clickhouse` 将计算在内。
|
+| bulk_size | int | No | 20000 |
每次通过[Clickhouse-jdbc](https://github.com/ClickHouse/clickhouse-jdbc)
写入的行数，即默认是20000.
|
+| split_mode | Boolean | No | false |
此模式仅支持引擎为`Distributed`的 `clickhouse` 表。选项 `internal_replication` 应该是 `true`
。他们将在 seatunnel 中拆分分布式表数据，并直接对每个分片进行写入。分片权重定义为 `clickhouse` 将计算在内。
|
| sharding_key | String | No | - | 使用
`split_mode`
时，将数据发送到哪个节点是个问题，默认为随机选择，但可以使用`sharding_key`参数来指定分片算法的字段。此选项仅在`split_mode`为
`true` 时有效.
|
| primary_key | String | No | - |
标记`clickhouse`表中的主键列，并根据主键执行INSERT/UPDATE/DELETE到`clickhouse`表.
|
| support_upsert | Boolean | No | false |
支持按查询主键更新插入行.
|
diff --git a/docs/zh/connectors/sink/Doris.md b/docs/zh/connectors/sink/Doris.md
index 5891115521..e9cbd32144 100644
--- a/docs/zh/connectors/sink/Doris.md
+++ b/docs/zh/connectors/sink/Doris.md
@@ -20,7 +20,8 @@ import ChangeLog from '../changelog/connector-doris.md';
- [x] [精确一次](../../introduction/concepts/connector-v2-features.md)
- [x] [cdc](../../introduction/concepts/connector-v2-features.md)
-- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md)
+- [x] [支持多表写入](../../introduction/concepts/connector-v2-features.md)
+- [ ] [定时刷新](../../introduction/concepts/connector-v2-features.md)
## 描述
@@ -49,7 +50,7 @@ Doris Sink连接器的内部实现是通过stream load批量缓存和导入的
| password | String | Yes | -
| `Doris` 密码
|
| database | String | Yes | -
| `Doris`数据库名称 , 使用 `${database_name}` 表示上游数据库名称。
|
| table | String | Yes | -
| `Doris` 表名, 使用 `${table_name}` 表示上游表名。
|
-| table.identifier | String | Yes | -
| `Doris` 表的名称，2.3.5 版本后将弃用，请使用 `database` 和 `table` 代替。
|
+| table.identifier | String | No | -
| 已弃用的表标识，建议改用 `database` 和 `table`。
|
| sink.label-prefix | String | Yes | -
| stream load导入使用的标签前缀。 在2pc场景下，需要全局唯一性来保证SeaTunnel的EOS语义。
|
| sink.enable-2pc | bool | No | false
| 是否启用两阶段提交（2pc），默认为 false。
对于两阶段提交，请参考[此处](https://doris.apache.org/docs/data-operate/transaction?_highlight=two&_highlight=phase#stream-load-2pc)。
|
| sink.enable-delete | bool | No | -
| 是否启用删除。 该选项需要Doris表开启批量删除功能（0.15+版本默认开启），且仅支持Unique模型。
您可以在此[link](https://doris.apache.org/docs/dev/data-operate/delete/batch-delete-manual/)获得更多详细信息
|
diff --git a/docs/zh/connectors/source/Clickhouse.md
b/docs/zh/connectors/source/Clickhouse.md
index 91a00b1376..9d52a4ddce 100644
--- a/docs/zh/connectors/source/Clickhouse.md
+++ b/docs/zh/connectors/source/Clickhouse.md
@@ -16,8 +16,8 @@ import ChangeLog from '../changelog/connector-clickhouse.md';
- [ ] [流处理](../../introduction/concepts/connector-v2-features.md)
- [ ] [精确一次](../../introduction/concepts/connector-v2-features.md)
- [x] [列映射](../../introduction/concepts/connector-v2-features.md)
-- [ ] [并行度](../../introduction/concepts/connector-v2-features.md)
-- [ ] [支持用户自定义拆分](../../introduction/concepts/connector-v2-features.md)
+- [x] [并行度](../../introduction/concepts/connector-v2-features.md)
+- [x] [支持用户自定义拆分](../../introduction/concepts/connector-v2-features.md)
- [x] [支持多表读](../../introduction/concepts/connector-v2-features.md)
> 支持查询SQL，可以实现投影效果。
@@ -56,7 +56,7 @@ import ChangeLog from '../changelog/connector-clickhouse.md';
| host | String | 是 | - | `ClickHouse`
集群地址, 格式是`host:port` , 允许多个`hosts`配置. 例如 `"host1:8123,host2:8123"` . |
| username | String | 是 | - | `ClickHouse`
user 用户账号. |
| password | String | 是 | - | `ClickHouse`
user 用户密码. |
-| table_list | Array | NO | - |
要读取的数据表列表，支持配置多表.
|
+| table_list | Array | 否 | - |
要读取的数据表列表，支持配置多表.
|
| clickhouse.config | Map | 否 | - | 除了上述必须由
`clickhouse-jdbc` 指定的必填参数外，用户还可以指定多个可选参数，这些参数涵盖了 `clickhouse-jdbc`
提供的所有[参数](https://github.com/ClickHouse/clickhouse-jdbc/tree/master/clickhouse-client#configuration).
|
| server_time_zone | String | 否 | ZoneId.systemDefault() |
数据库服务中的会话时区。如果未设置，则使用ZoneId.systemDefault（）设置服务时区.
|
| common-options | | 否 | - | 源插件常用参数，详见
[源通用选项](../common-options/source-common-options.md).
|
@@ -69,6 +69,7 @@ import ChangeLog from '../changelog/connector-clickhouse.md';
| sql | String | 否 | - | 用于通过Clickhouse服务搜索数据的查询sql.
|
| filter_query | String | 否 | - | 数据过滤条件. 格式为: "field = value", 例如 :
filter_query = "id > 2 and type = 1" |
| partition_list | Array | 否 | - | 指定分区列表过滤数据.
如果是分区表，该字段可以配置为过滤指定分区的数据。. 例如: partition_list = ["20250615", "20250616"] |
+| split.size | int | 否 | Integer.MAX_VALUE | 每个 SeaTunnel split 包含的
ClickHouse part 数量，最小值为 `1`。值越小，拆分越多，并行读取粒度越细。 |
| batch_size | int | 否 | 1024 | 从Clickhouse读取一次可以获得的最大数据行数。
|
注意: 当此配置对应于单个表时，您可以将table_list中的配置项展平到外层。
@@ -87,6 +88,7 @@ Clickhouse源连接器支持并行读取数据。
当指定`table_path`参数时，如果不想读取整个表，可以指定`partition_list`或`filter_query`参数过滤指定条件或分区的数据。
* `partition_list`: 过滤指定分区的数据
* `filter_query`: 根据指定条件对数据进行过滤
+* `split.size`: 使用 `table_path` 读取时，控制每个 SeaTunnel split 包含多少个 ClickHouse part
`batch_size`参数可用于控制每次查询读取的数据量，以避免在读取大量数据时出现OOM异常。适当增加这个值将有助于提高读取过程的性能。
@@ -113,6 +115,7 @@ source {
server_time_zone = "UTC"
partition_list = ["20250615", "20250616"]
filter_query = "id > 2 and type = 1"
+ split.size = 1
batch_size = 1024
clickhouse.config = {
"socket_timeout": "300000"
@@ -230,4 +233,4 @@ sink {
## 变更日志
-<ChangeLog />
\ No newline at end of file
+<ChangeLog />
diff --git a/docs/zh/connectors/source/Doris.md
b/docs/zh/connectors/source/Doris.md
index f15bdf584a..d507988285 100644
--- a/docs/zh/connectors/source/Doris.md
+++ b/docs/zh/connectors/source/Doris.md
@@ -68,11 +68,14 @@ import ChangeLog from '../changelog/connector-doris.md';
| username | string | yes | - | 用户名
|
| password | string | yes | - | 密码
|
| doris.request.retries | int | no | 3 | 请求Doris
FE的重试次数
|
-| doris.request.read.timeout.ms | int | no | 30000 |
|
-| doris.request.connect.timeout.ms | int | no | 30000 |
|
+| doris.request.read.timeout.ms | int | no | 30000 | 请求 Doris
BE 的 socket 读取超时时间。
|
+| doris.request.connect.timeout.ms | int | no | 30000 | 请求 Doris
FE 或 BE 的连接超时时间。
|
| query-port | string | no | 9030 |
Doris查询端口
|
| doris.request.query.timeout.s | int | no | 3600 |
Doris扫描数据的超时时间，单位秒
|
-| table_list | string | 否 | - | 表清单
|
+| doris.request.tablet.size | int | no | Integer.MAX_VALUE |
每个 SeaTunnel split 包含的 Doris tablet 数量，最小值为 `1`。
|
+| doris.deserialize.arrow.async | boolean | no | false | 是否异步反序列化
Arrow 数据。
|
+| doris.request.retriesdoris.deserialize.queue.size | int | no | 64 | 异步反序列化
Arrow 数据时使用的队列大小。
|
+| table_list | Array | no | - | 要读取的
Doris 表清单。
|
表清单配置:
@@ -225,4 +228,4 @@ sink{
## 变更日志
-<ChangeLog />
\ No newline at end of file
+<ChangeLog />
