This is an automated email from the ASF dual-hosted git repository.
zhangshenghang pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 5e5e1715f3 [Docs][Connector-V2] Improve Paimon MQTT and Redis docs
(#11310)
5e5e1715f3 is described below
commit 5e5e1715f304fdaac47a5ed36ed3634e4c5961c2
Author: Daniel Carter <[email protected]>
AuthorDate: Sat Jul 4 09:54:40 2026 +0800
[Docs][Connector-V2] Improve Paimon MQTT and Redis docs (#11310)
Co-authored-by: DanielCarter-stack
<[email protected]>
---
docs/en/connectors/sink/Mqtt.md | 8 +++++---
docs/en/connectors/sink/Paimon.md | 8 +++++---
docs/en/connectors/sink/Redis.md | 5 +++++
docs/en/connectors/source/Paimon.md | 6 +++---
docs/en/connectors/source/Redis.md | 4 ++--
docs/zh/connectors/sink/Mqtt.md | 5 +++--
docs/zh/connectors/sink/Paimon.md | 12 +++++++-----
docs/zh/connectors/sink/Redis.md | 5 +++++
docs/zh/connectors/source/Paimon.md | 8 ++++----
docs/zh/connectors/source/Redis.md | 4 ++--
10 files changed, 41 insertions(+), 24 deletions(-)
diff --git a/docs/en/connectors/sink/Mqtt.md b/docs/en/connectors/sink/Mqtt.md
index c75f728254..e00b0e31ce 100644
--- a/docs/en/connectors/sink/Mqtt.md
+++ b/docs/en/connectors/sink/Mqtt.md
@@ -13,12 +13,14 @@ This connector is suitable for publishing SeaTunnel
pipeline data to IoT endpoin
## Key features
- [ ] [exactly-once](../../introduction/concepts/connector-v2-features.md)
+- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md)
**Delivery Semantics Notice**:
This connector provides **at-most-once** delivery when QoS=0, and
**best-effort at-least-once** when QoS=1.
Due to `clean_session=true` (the default, required for stateless operation),
unacknowledged messages may be lost during
-client disconnections. For stronger guarantees, consider setting
`clean_session=false` (with proper clientId management)
-or enabling source replay capabilities in SeaTunnel.
+client disconnections. Setting `clean_session=false` lets the broker keep
session state while the writer is running, but
+the sink currently generates a unique client id for each writer and does not
expose a `client_id` option. For recovery
+from job restarts, rely on upstream replay and MQTT QoS rather than a stable
sink client id.
## Supported Engines
@@ -102,7 +104,7 @@ The MQTT connection establishment timeout in seconds.
Whether to use a clean MQTT session. Default is `true`.
- `true` — Broker discards any previous session state. Suitable for stateless
operation (recommended for most use cases).
-- `false` — Broker retains session state (subscriptions, unacknowledged QoS 1
messages). Enables stronger at-least-once guarantees but may cause broker-side
state accumulation. Requires stable, unique `clientId` per writer.
+- `false` — Broker retains session state for the generated writer client id
while the writer is running. This can help with transient disconnects, but it
may cause broker-side state accumulation and does not provide a stable client
id across job restarts.
### common options
diff --git a/docs/en/connectors/sink/Paimon.md
b/docs/en/connectors/sink/Paimon.md
index 7aec327e60..7147f3660c 100644
--- a/docs/en/connectors/sink/Paimon.md
+++ b/docs/en/connectors/sink/Paimon.md
@@ -56,7 +56,9 @@ libfb303-xxx.jar
## Key features
- [x] [exactly-once](../../introduction/concepts/connector-v2-features.md)
+- [x] [cdc](../../introduction/concepts/connector-v2-features.md)
- [x] [support multiple table
write](../../introduction/concepts/connector-v2-features.md)
+- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md)
## Options
@@ -70,7 +72,7 @@ libfb303-xxx.jar
| table | String | Yes | -
| The table you want to access
|
| user | String | No | -
| Paimon user to access table
|
| password | String | No | -
| Paimon user password to access table
|
-| hdfs_site_path | String | No | -
| The path of hdfs-site.xml
|
+| hdfs_site_path | String | No | -
| Deprecated. The path of hdfs-site.xml. Prefer `paimon.hadoop.conf` or
`paimon.hadoop.conf-path` for new jobs
|
| schema_save_mode | Enum | No |
CREATE_SCHEMA_WHEN_NOT_EXIST | The schema save mode
|
| data_save_mode | Enum | No | APPEND_DATA
| The data save mode
|
| paimon.table.primary-keys | String | No | -
| Default comma-separated list of columns (primary key) that identify a row
in tables.(Notice: The partition field needs to be included in the primary key
fields) |
@@ -78,8 +80,8 @@ libfb303-xxx.jar
| paimon.table.write-props | Map | No | -
| Properties passed through to paimon table initialization,
[reference](https://paimon.apache.org/docs/master/maintenance/configurations/#coreoptions).
|
| paimon.hadoop.conf | Map | No | -
| Properties in hadoop conf
|
| paimon.hadoop.conf-path | String | No | -
| The specified loading path for the 'core-site.xml', 'hdfs-site.xml',
'hive-site.xml' files
|
-| paimon.table.non-primary-key | Boolean | false | -
| Switch to create `table with PK` or `table without PK`. true : `table
without PK`, false : `table with PK`
|
-| branch | String | No | main
| The branch name of Paimon table to write data to. For non-main branches,
the main table and target branch must already exist, and
`schema_save_mode=RECREATE_SCHEMA` or `data_save_mode=DROP_DATA` is not
supported. |
+| paimon.table.non-primary-key | Boolean | No | false
| Switch to create `table with PK` or `table without PK`. true : `table
without PK`, false : `table with PK`
|
+| branch | String | No | -
| The branch name of Paimon table to write data to. If omitted, data is
written to the main branch. For non-main branches, the main table and target
branch must already exist, and `schema_save_mode=RECREATE_SCHEMA` or
`data_save_mode=DROP_DATA` is not supported. |
## Checkpoint in batch mode
diff --git a/docs/en/connectors/sink/Redis.md b/docs/en/connectors/sink/Redis.md
index 16d610631f..bb693d3163 100644
--- a/docs/en/connectors/sink/Redis.md
+++ b/docs/en/connectors/sink/Redis.md
@@ -40,6 +40,7 @@ the connector can build the Redis key from one or more
upstream fields, for exam
| value_field | string | No | - |
Upstream field used as the Redis value for `KEY`/`STRING`, `LIST`, `SET`, and
`ZSET`. |
| hash_key_field | string | No | - |
Upstream field used as the Redis hash field when `data_type = HASH`. |
| hash_value_field | string | No | - |
Upstream field used as the Redis hash value when `data_type = HASH`. |
+| multi_table_sink_replica | int | No | 1 |
Writer replica count for multi-table writes. |
| common-options | config | No | - | Sink
plugin common parameters. See [Sink Common
Options](../common-options/sink-common-options.md). |
## Write Rules
@@ -73,6 +74,10 @@ For `HASH`, `hash_key_field` chooses the Redis hash field.
If `hash_value_field`
written as the Redis hash value. If `hash_value_field` is not configured, the
connector serializes the whole upstream
row as the hash value.
+### multi_table_sink_replica
+
+Replica count for multi-table sink writers. It applies when upstream rows
carry table identifiers and the job writes multiple Redis tables in one
pipeline.
+
## Examples
### Write Rows To A Redis List
diff --git a/docs/en/connectors/source/Paimon.md
b/docs/en/connectors/source/Paimon.md
index 4a43e027ee..195165741f 100644
--- a/docs/en/connectors/source/Paimon.md
+++ b/docs/en/connectors/source/Paimon.md
@@ -84,11 +84,11 @@ The table you want to access
### table_list [array]
-The list of tables to be read, you can use this configuration instead of
`table`
+The list of tables to be read. Use this configuration instead of `table` when
one source needs to read multiple Paimon tables. Each item must contain
`table`, and can contain its own `query`.
### hdfs_site_path [string]
-The file path of `hdfs-site.xml`
+The file path of `hdfs-site.xml`. This option is deprecated; prefer
`paimon.hadoop.conf` or `paimon.hadoop.conf-path` for new jobs.
### query [string]
@@ -126,7 +126,7 @@ Properties in hadoop conf
The specified loading path for the 'core-site.xml', 'hdfs-site.xml',
'hive-site.xml' files
## Filesystems
-The Paimon connector supports writing data to multiple file systems.
Currently, the supported file systems are hdfs and s3.
+The Paimon connector supports reading data from multiple file systems.
Currently, the supported file systems are hdfs and s3.
If you use the s3 filesystem. You can configure the
`fs.s3a.access-key`、`fs.s3a.secret-key`、`fs.s3a.endpoint`、`fs.s3a.path.style.access`、`fs.s3a.aws.credentials.provider`
properties in the `paimon.hadoop.conf` option.
Besides, the warehouse should start with `s3a://`.
diff --git a/docs/en/connectors/source/Redis.md
b/docs/en/connectors/source/Redis.md
index 5277fb7a61..e7e0d665eb 100644
--- a/docs/en/connectors/source/Redis.md
+++ b/docs/en/connectors/source/Redis.md
@@ -49,7 +49,7 @@ When using `tables_configs` to read multiple key patterns,
each table configurat
| single_field_name | string | no | - | Field name for
single-value types |
| field_delimiter | string | no | ',' | Delimiter for text
format |
-**Note:** When this configuration corresponds to a single table, you can
flatten the configuration items in tables_configs to the outer layer (backward
compatible).
+**Note:** When this configuration corresponds to a single table, configure
these table-level options at the outer layer instead of using `tables_configs`.
For example, put `keys`, `data_type`, `format`, and `schema` directly under
`Redis`.
**Important:** In multi-table mode, the above table-level parameters should be
configured within each item of `tables_configs`.
@@ -125,7 +125,7 @@ schema {
each kv that in hash key it will be treated as a row and send it to upstream.
-**Tips: connector will use the first field information of schema config as the
field name of each k that in each kv**
+**Tips: connector will use the first field in the schema config as the field
name for each key in each hash kv pair.**
### keys [string]
diff --git a/docs/zh/connectors/sink/Mqtt.md b/docs/zh/connectors/sink/Mqtt.md
index 22b2f6c4de..1de1cadc4c 100644
--- a/docs/zh/connectors/sink/Mqtt.md
+++ b/docs/zh/connectors/sink/Mqtt.md
@@ -13,12 +13,13 @@ import ChangeLog from '../changelog/connector-mqtt.md';
## 主要特性
- [ ] [精确一次](../../introduction/concepts/connector-v2-features.md)
+- [ ] [定时刷新](../../introduction/concepts/connector-v2-features.md)
:::caution 投递语义
当 `qos = 0` 时，该连接器提供最多一次投递；当 `qos = 1` 时，提供尽力而为的至少一次投递。
-默认 `clean_session = true`，连接器按无状态方式运行。客户端断开连接时，未确认的消息可能丢失。如果需要更强的投递保证，可以设置
`clean_session = false`，并确保每个 writer 使用稳定且唯一的 client id；也可以配合 SeaTunnel
的上游重放能力使用。
+默认 `clean_session = true`，连接器按无状态方式运行。客户端断开连接时，未确认的消息可能丢失。设置 `clean_session =
false` 后，broker 可以在 writer 运行期间保留会话状态；但当前 Sink 会为每个 writer 自动生成唯一 client
id，尚不提供 `client_id` 配置。作业重启后的恢复主要依赖上游重放能力和 MQTT QoS，而不是稳定的 Sink client id。
:::
@@ -104,7 +105,7 @@ MQTT broker 认证密码。匿名访问时可以不配置。
是否使用 clean MQTT session。默认值为 `true`。
- `true`：broker 会丢弃之前的会话状态，适合大多数无状态写入场景。
-- `false`：broker 可以保留会话状态和未确认的 QoS 1 消息，可以增强至少一次语义，但可能造成 broker
端状态堆积。使用时需要确保每个 writer 的 client id 稳定且唯一。
+- `false`：broker 可以在 writer 运行期间保留自动生成的 client id 对应的会话状态。它有助于处理短暂断连，但可能造成
broker 端状态堆积，也不提供跨作业重启的稳定 client id。
### common options
diff --git a/docs/zh/connectors/sink/Paimon.md
b/docs/zh/connectors/sink/Paimon.md
index 491c61d80d..a46228cb28 100644
--- a/docs/zh/connectors/sink/Paimon.md
+++ b/docs/zh/connectors/sink/Paimon.md
@@ -55,8 +55,10 @@ libfb303-xxx.jar
## 主要特性
-- [x] [exactly-once](../../introduction/concepts/connector-v2-features.md)
-- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md)
+- [x] [精确一次](../../introduction/concepts/connector-v2-features.md)
+- [x] [变更数据捕获](../../introduction/concepts/connector-v2-features.md)
+- [x] [支持多表写入](../../introduction/concepts/connector-v2-features.md)
+- [ ] [定时刷新](../../introduction/concepts/connector-v2-features.md)
## 连接器选项
@@ -70,7 +72,7 @@ libfb303-xxx.jar
| table | 字符串 | 是 | - |
表名
|
| user | 字符串 | 否 | - |
paimon开启权限后，用户名
|
| password | 字符串 | 否 | - |
paimon开启权限后，用户名对应密码
|
-| hdfs_site_path | 字符串 | 否 | - |
hdfs-site.xml文件路径
|
+| hdfs_site_path | 字符串 | 否 | - |
已废弃。hdfs-site.xml 文件路径。新作业建议使用 `paimon.hadoop.conf` 或 `paimon.hadoop.conf-path`
|
| schema_save_mode | 枚举 | 否 | CREATE_SCHEMA_WHEN_NOT_EXIST |
Schema保存模式
|
| data_save_mode | 枚举 | 否 | APPEND_DATA |
数据保存模式
|
| paimon.table.primary-keys | 字符串 | 否 | - |
主键字段列表，联合主键使用逗号分隔(注意：分区字段需要包含在主键字段中)
|
@@ -78,8 +80,8 @@ libfb303-xxx.jar
| paimon.table.write-props | Map | 否 | - |
Paimon表初始化指定的属性,
[参考](https://paimon.apache.org/docs/master/maintenance/configurations/#coreoptions)
|
| paimon.hadoop.conf | Map | 否 | - |
Hadoop配置文件属性信息
|
| paimon.hadoop.conf-path | 字符串 | 否 | - |
Hadoop配置文件目录，用于加载'core-site.xml', 'hdfs-site.xml', 'hive-site.xml'文件配置
|
-| paimon.table.non-primary-key | Boolean | false | -
| 控制创建主键表或者非主键表. 当为true时,创建非主键表, 为false时,创建主键表
|
-| branch | 字符串 | 否 | main |
要写入数据的Paimon表分支名称。非 main 分支要求 main 表和目标分支已存在，且不支持
`schema_save_mode=RECREATE_SCHEMA` 或 `data_save_mode=DROP_DATA`。 |
+| paimon.table.non-primary-key | Boolean | 否 | false
| 控制创建主键表或者非主键表. 当为true时,创建非主键表, 为false时,创建主键表
|
+| branch | 字符串 | 否 | - |
要写入数据的 Paimon 表分支名称。不配置时写入 main 分支。非 main 分支要求 main 表和目标分支已存在，且不支持
`schema_save_mode=RECREATE_SCHEMA` 或 `data_save_mode=DROP_DATA`。 |
## 批模式下的checkpoint
diff --git a/docs/zh/connectors/sink/Redis.md b/docs/zh/connectors/sink/Redis.md
index 28edf2571c..53e7a3df5f 100644
--- a/docs/zh/connectors/sink/Redis.md
+++ b/docs/zh/connectors/sink/Redis.md
@@ -40,6 +40,7 @@ Redis 接收器连接器可以在批处理或流处理作业中把上游数据
| value_field | string | 否 | - | 写入
`KEY`/`STRING`、`LIST`、`SET`、`ZSET` 时，作为 Redis value 的上游字段。 |
| hash_key_field | string | 否 | - |
`data_type = HASH` 时，作为 Redis hash field 的上游字段。 |
| hash_value_field | string | 否 | - |
`data_type = HASH` 时，作为 Redis hash value 的上游字段。 |
+| multi_table_sink_replica | int | 否 | 1 |
多表写入时的写入器副本数。 |
| common-options | config | 否 | - |
接收器插件通用参数，详情请参考[接收器通用选项](../common-options/sink-common-options.md)。 |
## 写入规则
@@ -71,6 +72,10 @@ Redis 接收器连接器可以在批处理或流处理作业中把上游数据
写入 `HASH` 时，`hash_key_field` 用来指定 Redis hash field。如果配置了
`hash_value_field`，则把这个字段的值
作为 Redis hash value；如果不配置，连接器会把整行数据序列化后作为 hash value。
+### multi_table_sink_replica
+
+多表写入时的写入器副本数。当上游数据带有表标识，并且一个作业需要写入多张 Redis 表时使用。
+
## 示例
### 写入 Redis List
diff --git a/docs/zh/connectors/source/Paimon.md
b/docs/zh/connectors/source/Paimon.md
index 1f92139e8f..5c0b6fc854 100644
--- a/docs/zh/connectors/source/Paimon.md
+++ b/docs/zh/connectors/source/Paimon.md
@@ -84,11 +84,11 @@ Paimon 的 catalog uri，仅当 catalog_type 为 hive 时需要
### table_list [array]
-`Paimon` 表名列表，当需要同时读取多表时使用此配置代替 table
+`Paimon` 表名列表。当一个 Source 需要读取多张 Paimon 表时，使用此配置代替 `table`。每个配置项必须包含
`table`，也可以配置该表自己的 `query`。
### hdfs_site_path [string]
-`hdfs-site.xml` 文件地址
+`hdfs-site.xml` 文件地址。该选项已废弃，新作业建议使用 `paimon.hadoop.conf` 或
`paimon.hadoop.conf-path`。
### query [string]
@@ -130,9 +130,9 @@ hadoop conf 属性
指定 'core-site.xml', 'hdfs-site.xml', 'hive-site.xml' 文件加载路径。
-## Filesystems
+## 文件系统
-Paimon 连接器支持向多个文件系统写入数据。目前，支持的文件系统有 `hdfs` 和 `s3`。
+Paimon 连接器支持从多个文件系统读取数据。目前，支持的文件系统有 `hdfs` 和 `s3`。
如果使用 `s3` 文件系统，可以在 `paimon.hadoop.conf`
中配置`fs.s3a.access-key`、`fs.s3a.secret-key`、`fs.s3a.endpoint`、`fs.s3a.path.style.access`、`fs.s3a.aws.credentials.provider`
属性，数仓地址应该以 `s3a://` 开头。
## 示例
diff --git a/docs/zh/connectors/source/Redis.md
b/docs/zh/connectors/source/Redis.md
index 39416dac71..847e8ff4d9 100644
--- a/docs/zh/connectors/source/Redis.md
+++ b/docs/zh/connectors/source/Redis.md
@@ -49,7 +49,7 @@ import ChangeLog from '../changelog/connector-redis.md';
| single_field_name | string | 否 | - | 单值类型的字段名称
|
| field_delimiter | string | 否 | ',' | 文本格式的分隔符
|
-**注意：** 当配置对应单个表时，可以将 tables_configs 中的配置项平铺到外层（向后兼容）。
+**注意：** 当只读取单表时，请把这些表级参数直接配置在 `Redis` 外层，例如
`keys`、`data_type`、`format`、`schema`，不需要使用 `tables_configs`。
**重要提示：** 在多表模式下，上述表级参数需要配置在 `tables_configs` 的每个表项中。
@@ -125,7 +125,7 @@ schema {
hash key 中的每个 kv 将会被视为一行并被发送给上游。
-**提示：连接器将使用 scheme config 的第一个字段信息作为每个 kv 中每个 k 的字段名称**
+**提示：连接器会使用 schema 配置中的第一个字段作为每个 hash kv 中 key 的字段名。**
### keys [string]
