This is an automated email from the ASF dual-hosted git repository.
zhangshenghang pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new beb56d412f [Docs][Connector-V2] Improve ClickHouse Databend and Doris
docs (#11318)
beb56d412f is described below
commit beb56d412fff225335dcfe843d2a29b977a2876a
Author: Daniel Carter <[email protected]>
AuthorDate: Sat Jul 4 09:51:46 2026 +0800
[Docs][Connector-V2] Improve ClickHouse Databend and Doris docs (#11318)
Co-authored-by: DanielCarter-stack
<[email protected]>
---
docs/en/connectors/sink/Clickhouse.md | 12 +++++++-----
docs/en/connectors/sink/Databend.md | 3 ++-
docs/en/connectors/source/Clickhouse.md | 7 ++++---
docs/en/connectors/source/Databend.md | 5 ++---
docs/en/connectors/source/Doris.md | 7 +++++--
docs/zh/connectors/sink/Clickhouse.md | 12 +++++++-----
docs/zh/connectors/sink/Databend.md | 3 ++-
docs/zh/connectors/source/Clickhouse.md | 7 ++++---
docs/zh/connectors/source/Databend.md | 5 ++---
docs/zh/connectors/source/Doris.md | 7 +++++--
10 files changed, 40 insertions(+), 28 deletions(-)
diff --git a/docs/en/connectors/sink/Clickhouse.md
b/docs/en/connectors/sink/Clickhouse.md
index 615d252013..ad9e9d4860 100644
--- a/docs/en/connectors/sink/Clickhouse.md
+++ b/docs/en/connectors/sink/Clickhouse.md
@@ -59,11 +59,11 @@ They can be downloaded via install-plugin.sh or from the
Maven central repositor
| password | String | Yes | - |
`ClickHouse` user password.
|
| clickhouse.config | Map | No | | In
addition to the above mandatory parameters that must be specified by
`clickhouse-jdbc` , users can also specify multiple optional parameters, which
cover all the
[parameters](https://github.com/ClickHouse/clickhouse-jdbc/tree/master/clickhouse-client#configuration)
provided by `clickhouse-jdbc`. |
| bulk_size | int | No | 20000 | The
number of rows written through
[Clickhouse-jdbc](https://github.com/ClickHouse/clickhouse-jdbc) each time, the
`default is 20000`.
|
-| split_mode | Boolean | No | false | This
mode only support clickhouse table which engine is 'Distributed'.And
`internal_replication` option-should be `true`.They will split distributed
table data in seatunnel and perform write directly on each shard. The shard
weight define is clickhouse will counted. |
-| sharding_key | String | No | - | When
use split_mode, which node to send data to is a problem, the default is random
selection, but the 'sharding_key' parameter can be used to specify the field
for the sharding algorithm. This option only worked when 'split_mode' is true.
|
-| primary_key | String | No | - | Mark
the primary key column from clickhouse table, and based on primary key execute
INSERT/UPDATE/DELETE to clickhouse table.
|
-| support_upsert | Boolean | No | false |
Support upsert row by query primary key.
|
-| allow_experimental_lightweight_delete | Boolean | No | false | Allow
experimental lightweight delete based on `*MergeTree` table engine.
|
+| split_mode | Boolean | No | false | Only
applies when the target ClickHouse table uses the `Distributed` engine and
`internal_replication` is `true`. SeaTunnel splits distributed-table writes and
writes directly to each shard.
|
+| sharding_key | String | No | - | Field
used by the sharding algorithm when `split_mode=true`. If it is not configured,
the target shard is selected randomly.
|
+| primary_key | String | No | - |
Primary key column list used to handle INSERT/UPDATE/DELETE changelog rows. Use
commas for multiple columns, for example `id,name`.
|
+| support_upsert | Boolean | No | false |
Whether to query by `primary_key` before writing and perform upsert-style
writes.
|
+| allow_experimental_lightweight_delete | Boolean | No | false |
Allows DELETE changelog rows to use ClickHouse lightweight delete on
`*MergeTree` table engines.
|
| schema_save_mode | Enum | no |
CREATE_SCHEMA_WHEN_NOT_EXIST | Schema save mode. Please refer to the
`schema_save_mode` section below.
|
| data_save_mode | Enum | no | APPEND_DATA
| Data save mode. Please refer to the `data_save_mode` section below.
|
| custom_sql | String | no | -
| When data_save_mode selects CUSTOM_PROCESSING, you should fill in the
CUSTOM_SQL parameter. This parameter usually fills in a SQL that can be
executed. SQL will be executed before synchronization tasks. |
@@ -233,6 +233,8 @@ sink {
### CDC(Change data capture) for *MergeTree engine
+For CDC-style update/delete rows, configure `primary_key`. Set
`allow_experimental_lightweight_delete=true` only when the target `*MergeTree`
table can use ClickHouse lightweight delete.
+
```hocon
sink {
Clickhouse {
diff --git a/docs/en/connectors/sink/Databend.md
b/docs/en/connectors/sink/Databend.md
index 1d43a39b4e..4f96e5e83c 100644
--- a/docs/en/connectors/sink/Databend.md
+++ b/docs/en/connectors/sink/Databend.md
@@ -37,7 +37,7 @@ The Databend sink internally implements bulk data import
through stage attachmen
| Name | Type | Required | Default Value | Description
|
|---------------------|------|----------|---------------|---------------------------------------------|
-| url | String | Yes | - | Databend JDBC connection URL
|
+| url | String | Yes | - | Databend JDBC connection URL. It
must start with `jdbc:databend://` |
| username | String | Yes | - | Databend database username
|
| password | String | Yes | - | Databend database password
|
| database | String | No | - | Databend database name, defaults to
the database name specified in the connection URL |
@@ -159,6 +159,7 @@ sink {
Set `conflict_key` to the primary-key column used to merge update/delete
events. Set
`enable_delete = true` only when DELETE events should remove rows from
Databend.
+If `conflict_key` is not configured, the sink writes normal insert-style
batches.
```hocon
sink {
diff --git a/docs/en/connectors/source/Clickhouse.md
b/docs/en/connectors/source/Clickhouse.md
index 9964bed115..f15ce69bca 100644
--- a/docs/en/connectors/source/Clickhouse.md
+++ b/docs/en/connectors/source/Clickhouse.md
@@ -70,10 +70,10 @@ Table list configuration:
| sql | String | NO | - | The query
sql used to search data though Clickhouse server.
|
| filter_query | String | NO | - | Data
filtering in Clickhouse. the format is "field = value", example : filter_query
= "id > 2 and type = 1"
|
| partition_list | Array | NO | - | Table
partition list to filter the specified partition. If it is a partitioned table,
this field can be configured to filter the data of the specified partition.
example: partition_list = ["20250615", "20250616"]
|
-| split.size | int | NO | Integer.MAX_VALUE | The number
of ClickHouse parts grouped into each SeaTunnel split. The minimum value is
`1`. Smaller values create more splits for higher parallelism.
|
+| split_size | int | NO | Integer.MAX_VALUE | The number
of ClickHouse parts grouped into each SeaTunnel split when configured inside
`table_list`. Use `split.size` only when this option is flattened to the outer
source level. The minimum value is `1`. Smaller values create more splits for
higher parallelism. |
| batch_size | int | NO | 1024 | The maximum
rows of data that can be obtained by reading from Clickhouse once.
|
-Note: When this configuration corresponds to a single table, you can flatten
the configuration items in table_list to the outer layer.
+Note: When this configuration corresponds to a single table, you can flatten
the configuration items in table_list to the outer layer. In that flattened
form, configure `split.size` instead of `split_size`.
## Parallel Reader
The Clickhouse source connector supports parallel reading of data.
@@ -88,7 +88,7 @@ If both the `table_path` and `sql` parameters are set, it
will be executed in sq
## Tips
In query table mode, if you don't want to read the entire table, you can
specify the `partition_list` or `filter_query` parameter.
* `partition_list`: filter the data of the specified partition
-* `filter_query`: filter the data based on the specified conditions
+* `filter_query`: filter the data based on the specified conditions. It can
also be used together with `sql`; SeaTunnel applies it as an additional
ClickHouse-side filter.
* `split.size`: control how many ClickHouse parts are grouped into one
SeaTunnel split when reading by `table_path`
The `batch_size` parameter can be used to control the amount of data read each
time to avoid OOM exception when reading a large amount of data. Appropriately
increasing this value will help to improve the performance of the reading
process.
@@ -216,6 +216,7 @@ source {
{
table_path = "default.table2"
sql = "select * from default.table2 where age > 18"
+ split_size = 1
}
]
server_time_zone = "UTC"
diff --git a/docs/en/connectors/source/Databend.md
b/docs/en/connectors/source/Databend.md
index e2df8c82a4..0a23d701e5 100644
--- a/docs/en/connectors/source/Databend.md
+++ b/docs/en/connectors/source/Databend.md
@@ -65,7 +65,7 @@ Basic Configuration:
| Name | Type | Required | Default Value | Description |
|------|------|----------|---------------|-------------|
-| url | String | Yes | - | Databend JDBC connection URL |
+| url | String | Yes | - | Databend JDBC connection URL. It must start with
`jdbc:databend://` |
| username | String | Yes | - | Databend database username |
| password | String | Yes | - | Databend database password |
| database | String | No | - | Databend database name, defaults to the
database name specified in the connection URL |
@@ -76,8 +76,7 @@ Basic Configuration:
| ssl | Boolean | No | false | Whether to use SSL for the Databend connection |
| jdbc_config | Map | No | - | Additional JDBC connection configuration, such
as load balancing strategies |
-You must configure either `sql`, `query`, or both `database` and `table`. The
connector does not
-currently support `table_list`, so configure one Databend source block for
each table.
+You must configure either `sql`, `query`, or both `database` and `table`. When
more than one of them is configured, the read SQL is chosen in this order:
`sql`, then `query`, then `SELECT * FROM database.table`. The connector does
not currently support `table_list`, so configure one Databend source block for
each table.
## Task Examples
diff --git a/docs/en/connectors/source/Doris.md
b/docs/en/connectors/source/Doris.md
index 9f2bafb340..58d04434b8 100644
--- a/docs/en/connectors/source/Doris.md
+++ b/docs/en/connectors/source/Doris.md
@@ -70,7 +70,7 @@ Base configuration:
| doris.request.retries | int | no | 3 | Number
of retries to send requests to Doris FE.
|
| doris.request.read.timeout.ms | int | no | 30000 | Socket
read timeout for requests sent to Doris BE.
|
| doris.request.connect.timeout.ms | int | no | 30000 |
Connection timeout for requests sent to Doris FE or BE.
|
-| query-port | string | no | 9030 | Doris
QueryPort
|
+| query-port | int | no | 9030 | Doris
query port.
|
| doris.request.query.timeout.s | int | no | 3600 | Timeout
period of Doris scan data, expressed in seconds.
|
| doris.request.tablet.size | int | no | Integer.MAX_VALUE |
The number of Doris tablets grouped into each SeaTunnel split. The minimum
value is `1`. |
| doris.deserialize.arrow.async | boolean | no | false | Whether
to deserialize Arrow data asynchronously.
|
@@ -85,10 +85,11 @@ Table list configuration:
| table | string | yes | - | The name
of Doris table
|
| doris.read.field | string | no | - | Use the
'doris.read.field' parameter to select the doris table columns to read
|
| doris.filter.query | string | no | - | Data
filtering in doris. the format is "field = value",example : doris.filter.query
= "F_ID > 2" |
+| doris.request.tablet.size | int | no | Integer.MAX_VALUE |
The number of Doris tablets grouped into each SeaTunnel split for this table.
The minimum value is `1`. |
| doris.batch.size | int | no | 1024 | The
maximum value that can be obtained by reading Doris BE once.
|
| doris.exec.mem.limit | long | no | 2147483648 | Maximum
memory that can be used by a single be scan request. The default memory is 2G
(2147483648). |
-Note: When this configuration corresponds to a single table, you can flatten
the configuration items in table_list to the outer layer.
+Note: When this configuration corresponds to a single table, you can flatten
the configuration items in table_list to the outer layer. If `table_list` is
not configured, `database` and `table` must be configured at the outer source
level.
### Tips
@@ -197,6 +198,8 @@ source{
table = "doris_table_0"
doris.read.field = "F_ID,F_INT,F_BIGINT,F_TINYINT"
doris.filter.query = "F_ID >= 50"
+ doris.request.tablet.size = 1
+ doris.exec.mem.limit = 2147483648
},
{
database = "st_source_1"
diff --git a/docs/zh/connectors/sink/Clickhouse.md
b/docs/zh/connectors/sink/Clickhouse.md
index f016248c33..d5e6cb2f0f 100644
--- a/docs/zh/connectors/sink/Clickhouse.md
+++ b/docs/zh/connectors/sink/Clickhouse.md
@@ -58,11 +58,11 @@ import ChangeLog from
'../changelog/connector-clickhouse.md';
| password | String | Yes | - |
`ClickHouse` 用户密码.
|
| clickhouse.config | Map | No | | 除了上述必须由
`clickhouse-jdbc` 指定的必填参数外，用户还可以指定多个可选参数，这些参数涵盖了 `clickhouse-jdbc`
提供的所有[参数](https://github.com/ClickHouse/clickhouse-jdbc/tree/master/clickhouse-client#configuration).
|
| bulk_size | int | No | 20000 |
每次通过[Clickhouse-jdbc](https://github.com/ClickHouse/clickhouse-jdbc)
写入的行数，即默认是20000.
|
-| split_mode | Boolean | No | false |
此模式仅支持引擎为`Distributed`的 `clickhouse` 表。选项 `internal_replication` 应该是 `true`
。他们将在 seatunnel 中拆分分布式表数据，并直接对每个分片进行写入。分片权重定义为 `clickhouse` 将计算在内。
|
-| sharding_key | String | No | - | 使用
`split_mode`
时，将数据发送到哪个节点是个问题，默认为随机选择，但可以使用`sharding_key`参数来指定分片算法的字段。此选项仅在`split_mode`为
`true` 时有效.
|
-| primary_key | String | No | - |
标记`clickhouse`表中的主键列，并根据主键执行INSERT/UPDATE/DELETE到`clickhouse`表.
|
-| support_upsert | Boolean | No | false |
支持按查询主键更新插入行.
|
-| allow_experimental_lightweight_delete | Boolean | No | false |
允许基于`MergeTree`表引擎实验性轻量级删除.
|
+| split_mode | Boolean | No | false | 仅当目标
ClickHouse 表使用 `Distributed` 引擎且 `internal_replication` 为 `true` 时生效。SeaTunnel
会拆分分布式表写入，并直接写入各个分片。 |
+| sharding_key | String | No | - |
`split_mode=true` 时用于分片算法的字段。不配置时会随机选择目标分片。 |
+| primary_key | String | No | - | 用于处理
INSERT/UPDATE/DELETE 变更数据的主键列。多列用英文逗号分隔，例如 `id,name`。 |
+| support_upsert | Boolean | No | false | 是否按
`primary_key` 查询后再写入，从而实现类似 upsert 的写入效果。 |
+| allow_experimental_lightweight_delete | Boolean | No | false | 允许 DELETE
变更数据在 `*MergeTree` 表引擎上使用 ClickHouse lightweight delete。 |
| schema_save_mode | Enum | no |
CREATE_SCHEMA_WHEN_NOT_EXIST | schema保存模式，请参考下面的`schema_save_mode`
|
| data_save_mode | Enum | no | APPEND_DATA
| 数据保存模式，请参考下面的`data_save_mode`。
|
| custom_sql | String | no | -
|
当data_save_mode设置为CUSTOM_PROCESSING时，必须同时设置CUSTOM_SQL参数。CUSTOM_SQL的值为可执行的SQL语句，在同步任务开启前SQL将会被执行
|
@@ -234,6 +234,8 @@ sink {
### CDC(Change data capture) for *MergeTree engine
+处理 CDC 更新/删除数据时，需要配置 `primary_key`。只有目标 `*MergeTree` 表可以使用 ClickHouse
lightweight delete 时，才设置 `allow_experimental_lightweight_delete=true`。
+
```hocon
sink {
Clickhouse {
diff --git a/docs/zh/connectors/sink/Databend.md
b/docs/zh/connectors/sink/Databend.md
index 259af9e62c..92c2f46478 100644
--- a/docs/zh/connectors/sink/Databend.md
+++ b/docs/zh/connectors/sink/Databend.md
@@ -37,7 +37,7 @@ Databend sink 内部通过 stage attachment 实现数据的批量导入。
| 名称 | 类型 | 是否必须 | 默认值 | 描述 |
|---------------------|------|----------|--------|------------------------------------|
-| url | String | 是 | - | Databend JDBC 连接 URL |
+| url | String | 是 | - | Databend JDBC 连接 URL，必须以
`jdbc:databend://` 开头 |
| username | String | 是 | - | Databend 数据库用户名 |
| password | String | 是 | - | Databend 数据库密码 |
| database | String | 否 | - | Databend 数据库名称，默认使用连接 URL 中指定的数据库名 |
@@ -159,6 +159,7 @@ sink {
`conflict_key` 用来指定合并更新/删除事件的主键列。只有当 DELETE 事件需要删除
Databend 中的数据时，才需要设置 `enable_delete = true`。
+如果不配置 `conflict_key`，sink 会按普通批量插入方式写入。
```hocon
sink {
diff --git a/docs/zh/connectors/source/Clickhouse.md
b/docs/zh/connectors/source/Clickhouse.md
index 9d52a4ddce..2625a97ed2 100644
--- a/docs/zh/connectors/source/Clickhouse.md
+++ b/docs/zh/connectors/source/Clickhouse.md
@@ -69,10 +69,10 @@ import ChangeLog from
'../changelog/connector-clickhouse.md';
| sql | String | 否 | - | 用于通过Clickhouse服务搜索数据的查询sql.
|
| filter_query | String | 否 | - | 数据过滤条件. 格式为: "field = value", 例如 :
filter_query = "id > 2 and type = 1" |
| partition_list | Array | 否 | - | 指定分区列表过滤数据.
如果是分区表，该字段可以配置为过滤指定分区的数据。. 例如: partition_list = ["20250615", "20250616"] |
-| split.size | int | 否 | Integer.MAX_VALUE | 每个 SeaTunnel split 包含的
ClickHouse part 数量，最小值为 `1`。值越小，拆分越多，并行读取粒度越细。 |
+| split_size | int | 否 | Integer.MAX_VALUE | 在 `table_list` 内配置时，每个
SeaTunnel split 包含的 ClickHouse part 数量。只有把配置展开到 source 外层时才使用 `split.size`。最小值为
`1`，值越小，拆分越多，并行读取粒度越细。 |
| batch_size | int | 否 | 1024 | 从Clickhouse读取一次可以获得的最大数据行数。
|
-注意: 当此配置对应于单个表时，您可以将table_list中的配置项展平到外层。
+注意: 当此配置对应于单个表时，您可以将table_list中的配置项展平到外层。展开到外层时，分片大小参数使用 `split.size`，而不是
`split_size`。
## 并行读取
@@ -87,7 +87,7 @@ Clickhouse源连接器支持并行读取数据。
## Tips
当指定`table_path`参数时，如果不想读取整个表，可以指定`partition_list`或`filter_query`参数过滤指定条件或分区的数据。
* `partition_list`: 过滤指定分区的数据
-* `filter_query`: 根据指定条件对数据进行过滤
+* `filter_query`: 根据指定条件对数据进行过滤。它也可以和 `sql` 一起使用，SeaTunnel 会把它作为额外的 ClickHouse
侧过滤条件。
* `split.size`: 使用 `table_path` 读取时，控制每个 SeaTunnel split 包含多少个 ClickHouse part
`batch_size`参数可用于控制每次查询读取的数据量，以避免在读取大量数据时出现OOM异常。适当增加这个值将有助于提高读取过程的性能。
@@ -214,6 +214,7 @@ source {
{
table_path = "default.table2"
sql = "select * from default.table2 where age > 18"
+ split_size = 1
}
]
server_time_zone = "UTC"
diff --git a/docs/zh/connectors/source/Databend.md
b/docs/zh/connectors/source/Databend.md
index 2b3c3b25a4..6a6bf584d8 100644
--- a/docs/zh/connectors/source/Databend.md
+++ b/docs/zh/connectors/source/Databend.md
@@ -64,7 +64,7 @@ import ChangeLog from '../changelog/connector-databend.md';
| 名称 | 类型 | 是否必须 | 默认值 | 描述 |
|------|------|----------|--------|------|
-| url | String | 是 | - | Databend JDBC 连接 URL |
+| url | String | 是 | - | Databend JDBC 连接 URL，必须以 `jdbc:databend://` 开头 |
| username | String | 是 | - | Databend 数据库用户名 |
| password | String | 是 | - | Databend 数据库密码 |
| database | String | 否 | - | Databend 数据库名称，默认使用连接 URL 中指定的数据库名 |
@@ -75,8 +75,7 @@ import ChangeLog from '../changelog/connector-databend.md';
| ssl | Boolean | 否 | false | 是否使用 SSL 连接 Databend |
| jdbc_config | Map | 否 | - | 额外的 JDBC 连接配置，如加载均衡策略等 |
-必须配置 `sql`、`query`、或同时配置 `database` 和 `table`。当前连接器不支持
-`table_list`，如果要读多张表，请为每张表分别配置一个 Databend source。
+必须配置 `sql`、`query`、或同时配置 `database` 和 `table`。如果同时配置了多个读取入口，实际读取 SQL
的优先级是：`sql`、`query`、最后是 `SELECT * FROM database.table`。当前连接器不支持
`table_list`，如果要读多张表，请为每张表分别配置一个 Databend source。
## 任务示例
diff --git a/docs/zh/connectors/source/Doris.md
b/docs/zh/connectors/source/Doris.md
index d507988285..02f9218672 100644
--- a/docs/zh/connectors/source/Doris.md
+++ b/docs/zh/connectors/source/Doris.md
@@ -70,7 +70,7 @@ import ChangeLog from '../changelog/connector-doris.md';
| doris.request.retries | int | no | 3 | 请求Doris
FE的重试次数
|
| doris.request.read.timeout.ms | int | no | 30000 | 请求 Doris
BE 的 socket 读取超时时间。
|
| doris.request.connect.timeout.ms | int | no | 30000 | 请求 Doris
FE 或 BE 的连接超时时间。
|
-| query-port | string | no | 9030 |
Doris查询端口
|
+| query-port | int | no | 9030 | Doris
查询端口。
|
| doris.request.query.timeout.s | int | no | 3600 |
Doris扫描数据的超时时间，单位秒
|
| doris.request.tablet.size | int | no | Integer.MAX_VALUE |
每个 SeaTunnel split 包含的 Doris tablet 数量，最小值为 `1`。
|
| doris.deserialize.arrow.async | boolean | no | false | 是否异步反序列化
Arrow 数据。
|
@@ -85,10 +85,11 @@ import ChangeLog from '../changelog/connector-doris.md';
| table | string | yes | - | 表名
|
| doris.read.field | string | no | - |
选择要读取的Doris表字段
|
| doris.filter.query | string | no | - | 数据过滤.
格式："字段 = 值", 例如：doris.filter.query = "F_ID > 2"
|
+| doris.request.tablet.size | int | no | Integer.MAX_VALUE |
当前表每个 SeaTunnel split 包含的 Doris tablet 数量，最小值为 `1`。
|
| doris.batch.size | int | no | 1024 |
每次能够从BE中读取到的最大行数
|
| doris.exec.mem.limit | long | no | 2147483648 |
单个be扫描请求可以使用的最大内存。默认内存为2G（2147483648）
|
-注意: 当此配置对应于单个表时，您可以将table_list中的配置项展平到外层。
+注意: 当此配置对应于单个表时，您可以将table_list中的配置项展平到外层。如果不配置 `table_list`，必须在 source 外层配置
`database` 和 `table`。
### 提示
@@ -197,6 +198,8 @@ source{
table = "doris_table_0"
doris.read.field = "F_ID,F_INT,F_BIGINT,F_TINYINT"
doris.filter.query = "F_ID >= 50"
+ doris.request.tablet.size = 1
+ doris.exec.mem.limit = 2147483648
},
{
database = "st_source_1"
