This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-12941-9c50d224e5948fb2291b69b2ba468fc8069e0e2b in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit 80f4ec15dc96d32040c032833db75e21249c00ba Author: Jast <[email protected]> AuthorDate: Mon Oct 26 14:19:00 2026 +0000 [Docs] Fix connector, transform and common-option doc inaccuracies verified against source code (#12941) Co-authored-by: zhangshenghang <[email protected]> Co-authored-by: jast <[email protected]> --- .../en/connectors/common-options/sink-common-options.md | 17 +++++++++++------ docs/en/connectors/common-options/sink-write-modes.md | 4 +++- .../connectors/common-options/source-common-options.md | 9 +++++++-- docs/en/connectors/sink/Kudu.md | 4 +++- docs/en/connectors/sink/Prometheus.md | 2 +- docs/en/connectors/source/Kudu.md | 4 +++- docs/en/connectors/source/Pulsar.md | 4 ++-- docs/en/connectors/source/Splunk.md | 2 +- .../zh/connectors/common-options/sink-common-options.md | 17 +++++++++++------ docs/zh/connectors/common-options/sink-write-modes.md | 4 +++- .../connectors/common-options/source-common-options.md | 9 +++++++-- docs/zh/connectors/sink/Fluss.md | 7 ++++--- docs/zh/connectors/sink/Kudu.md | 4 +++- docs/zh/connectors/sink/Prometheus.md | 2 +- docs/zh/connectors/source/Kudu.md | 4 +++- docs/zh/connectors/source/Pulsar.md | 4 ++-- docs/zh/connectors/source/Splunk.md | 2 +- 17 files changed, 66 insertions(+), 33 deletions(-) diff --git a/docs/en/connectors/common-options/sink-common-options.md b/docs/en/connectors/common-options/sink-common-options.md index d86b8cb742..1c9110ad5d 100644 --- a/docs/en/connectors/common-options/sink-common-options.md +++ b/docs/en/connectors/common-options/sink-common-options.md @@ -17,14 +17,14 @@ The old configuration name `source_table_name` is deprecated, please migrate to | Name | Type | Required | Default | Description | |---------------|--------|----------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| plugin_input | String | No | - | When `plugin_input` is not specified, the current plug-in processes the data set `dataset` output by the previous plugin in the configuration file <br/> When `plugin_input` is specified, the current plug-in is processing the data set corresponding to this parameter. | +| plugin_input | list | No | - | When `plugin_input` is not specified, the current plug-in processes the data set `dataset` output by the previous plugin in the configuration file <br/> When `plugin_input` is specified, the current plug-in processes the data set corresponding to this parameter. The value can be a single upstream dataset name or an array of names; with several names the Zeta engine unions those upstream data sets into this sink. | | metadata_datasource_id | String | No | - | The data source ID for retrieving connection configuration from Metadata Center. When specified, the connector will fetch connection details (e.g., URL, username, password) from the external metadata service instead of using direct configuration. See [Metadata SPI](../../introduction/concepts/metadata-spi.md) for more information. | -### plugin_input [string] +### plugin_input [list] When `plugin_input` is not specified, the current plug-in processes the data set `dataset` output by the previous plugin in the configuration file. -When `plugin_input` is specified, the current plug-in is processing the data set corresponding to this parameter. +When `plugin_input` is specified, the current plug-in processes the data set corresponding to this parameter. The value can be a single upstream dataset name (a string) or an array of names. ### metadata_datasource_id [string] @@ -32,7 +32,7 @@ The data source ID for retrieving connection configuration from Metadata Center. # Important note -When the job configuration `plugin_input` you must set the `plugin_output` parameter +A sink only consumes data sets, so it has no `plugin_output` option. The `plugin_input` value must match the `plugin_output` declared by an upstream source or transform. `plugin_input` is only mandatory when the job has more than one source, transform or sink (see the note at the end of this page). ## Task Example @@ -42,10 +42,15 @@ When the job configuration `plugin_input` you must set the `plugin_output` param ```bash source { - FakeSourceStream { + FakeSource { parallelism = 2 plugin_output = "fake" - field_name = "name,age" + schema = { + fields { + name = "string" + age = "int" + } + } } } diff --git a/docs/en/connectors/common-options/sink-write-modes.md b/docs/en/connectors/common-options/sink-write-modes.md index 162b74692e..18be1888d0 100644 --- a/docs/en/connectors/common-options/sink-write-modes.md +++ b/docs/en/connectors/common-options/sink-write-modes.md @@ -99,9 +99,11 @@ Current connector support differs by file connector: | SmbFile | Yes | Handles existing SMB directories and files. | | S3File | Yes | Handles existing S3 paths and objects through the file sink save mode flow. | | OssFile | Yes | Handles existing OSS paths and objects through the file sink save mode flow. | +| GcsFile | Yes | Handles existing GCS paths and objects through the file sink save mode flow. | | ObsFile | No | The current sink option rule does not expose `schema_save_mode` or `data_save_mode`. | | CosFile | No | The current sink option rule does not expose `schema_save_mode` or `data_save_mode`. | | BosFile | No | The current sink option rule does not expose `schema_save_mode` or `data_save_mode`. | +| OssJindoFile | No | The current sink option rule does not expose `schema_save_mode` or `data_save_mode`. | If a file connector page does not list `schema_save_mode` or `data_save_mode`, do not assume the option is accepted by that connector. @@ -176,7 +178,7 @@ Check whether the sink uses `query`. JDBC custom query mode does not apply save ### A file sink rejects `data_save_mode` -Check the specific connector option table. `S3File`, `OssFile`, `HdfsFile`, `FtpFile`, `SftpFile`, and `LocalFile` expose file save mode options. `ObsFile`, `CosFile`, and `BosFile` currently do not. +Check the specific connector option table. `LocalFile`, `HdfsFile`, `FtpFile`, `SftpFile`, `SmbFile`, `S3File`, `OssFile`, and `GcsFile` expose file save mode options. `ObsFile`, `CosFile`, `BosFile`, and `OssJindoFile` currently do not. ### I only want to create the target table diff --git a/docs/en/connectors/common-options/source-common-options.md b/docs/en/connectors/common-options/source-common-options.md index 0b7e605e0c..c0fe537bee 100644 --- a/docs/en/connectors/common-options/source-common-options.md +++ b/docs/en/connectors/common-options/source-common-options.md @@ -20,7 +20,7 @@ The old configuration name `result_table_name` is deprecated, please migrate to # Important note -When the job configuration `plugin_output` you must set the `plugin_input` parameter +`plugin_output` registers the data set produced by this source as a temporary table that downstream transforms or sinks can reference through `plugin_input`. A source itself never takes `plugin_input`. If the job has only one source, at most one transform and one sink, both options are optional; if any of source, transform or sink appears more than once, every plugin must declare `plugin_output`/`plugin_input` so the engine can build the DAG. ## Task Example @@ -30,8 +30,13 @@ When the job configuration `plugin_output` you must set the `plugin_input` param ```bash source { - FakeSourceStream { + FakeSource { plugin_output = "fake_table" + schema = { + fields { + id = int + } + } } } ``` diff --git a/docs/en/connectors/sink/Kudu.md b/docs/en/connectors/sink/Kudu.md index 8be27b8ad7..6b44a1fc06 100644 --- a/docs/en/connectors/sink/Kudu.md +++ b/docs/en/connectors/sink/Kudu.md @@ -26,7 +26,9 @@ import ChangeLog from '../changelog/connector-kudu.md'; | SeaTunnel Data Type | Kudu Data Type | |---------------------|--------------------------| | BOOLEAN | BOOL | -| INT | INT8<br/>INT16<br/>INT32 | +| TINYINT | INT8 | +| SMALLINT | INT16 | +| INT | INT32 | | BIGINT | INT64 | | DECIMAL | DECIMAL | | FLOAT | FLOAT | diff --git a/docs/en/connectors/sink/Prometheus.md b/docs/en/connectors/sink/Prometheus.md index b0408fc741..69d066a1ec 100644 --- a/docs/en/connectors/sink/Prometheus.md +++ b/docs/en/connectors/sink/Prometheus.md @@ -50,7 +50,7 @@ downloaded from Maven Central. | key_value | String | Yes | - | Name of the upstream field that contains the Prometheus sample value. A `double` field is recommended. | | key_timestamp | String | No | - | Name of the upstream field that contains the Prometheus sample timestamp. If omitted, the sink uses the current system time. | | headers | Map | No | - | HTTP request headers. | -| retry | Int | No | 3 | Maximum retry attempts for a failed remote-write request. Retries transport `IOException`s and retryable HTTP statuses (`5xx` and `429`); other `4xx` responses fail fast. Set to `0` to disable retries. | +| retry | Int | No | 3 | Maximum retry attempts for a failed remote-write request. The sink applies an effective default of `3` when the option is omitted; set to `0` to disable retries. Retries transport `IOException`s and retryable HTTP statuses (`5xx` and `429`); other `4xx` responses fail fast. | | retry_backoff_multiplier_ms | Int | No | 100 | Retry backoff multiplier in milliseconds. | | retry_backoff_max_ms | Int | No | 10000 | Maximum retry backoff in milliseconds. | | batch_size | Int | No | 1024 | Positive number of rows buffered before writing to Prometheus. | diff --git a/docs/en/connectors/source/Kudu.md b/docs/en/connectors/source/Kudu.md index d7431883f0..5a2e8c81f5 100644 --- a/docs/en/connectors/source/Kudu.md +++ b/docs/en/connectors/source/Kudu.md @@ -33,7 +33,9 @@ The tested kudu version is 1.11.1. | kudu Data Type | SeaTunnel Data Type | |--------------------------|---------------------| | BOOL | BOOLEAN | -| INT8<br/>INT16<br/>INT32 | INT | +| INT8 | TINYINT | +| INT16 | SMALLINT | +| INT32 | INT | | INT64 | BIGINT | | DECIMAL | DECIMAL | | FLOAT | FLOAT | diff --git a/docs/en/connectors/source/Pulsar.md b/docs/en/connectors/source/Pulsar.md index a670d0fdd4..77810416f3 100644 --- a/docs/en/connectors/source/Pulsar.md +++ b/docs/en/connectors/source/Pulsar.md @@ -399,9 +399,9 @@ source { When subscribing to partitioned topics or topic patterns (`topic-pattern`), SeaTunnel automatically discovers partitions and dynamically assigns topic splits across reader subtasks. You can tune `topic-discovery.interval` to control how frequently new partitions are detected. -### Which cursor reset strategies and subscription modes are supported? +### Which cursor startup modes and reset strategies are supported? -The source supports `EARLIEST` and `LATEST` startup modes through `cursor.startup.mode`. Consumers join the subscription with configurable subscription types (e.g. `Failover` or `Exclusive`), and SeaTunnel checkpoints message IDs to ensure reliable playback on failure. +The source supports `EARLIEST`, `LATEST`, `SUBSCRIPTION` and `TIMESTAMP` startup modes through `cursor.startup.mode`. `TIMESTAMP` requires `cursor.startup.timestamp`, and `SUBSCRIPTION` requires `cursor.reset.mode` (`EARLIEST` or `LATEST`): if the subscription already exists the consumer resumes from its committed position, otherwise the subscription is created at the earliest or latest message according to `cursor.reset.mode`. The subscription type is not configurable; the connector onl [...] ### How do I configure authentication for secure Pulsar clusters? diff --git a/docs/en/connectors/source/Splunk.md b/docs/en/connectors/source/Splunk.md index 2f5d5a9bb6..9e763b9cf7 100644 --- a/docs/en/connectors/source/Splunk.md +++ b/docs/en/connectors/source/Splunk.md @@ -22,7 +22,7 @@ The `Http-Splunk` connector allows batch reading data from Splunk REST API endpo | method | String | No | `POST` | HTTP request method | | keep_params_as_form | Boolean | No | `true` | Keep parameters as form urlencoded | | max_response_size_bytes | Long | No | `20971520` | Maximum allowed HTTP response size in bytes before failing fast, to avoid unbounded memory use on large exports. | -| params | Map | Yes | - | Request parameters including `search` and `output_mode` | +| params | Map | No | - | Request parameters such as `search` and `output_mode`. The option is not validated as required, but a `search` parameter is needed for a meaningful export; `output_mode` defaults to `json` when not set. | ## Example Configuration diff --git a/docs/zh/connectors/common-options/sink-common-options.md b/docs/zh/connectors/common-options/sink-common-options.md index af97fd29b2..dfe6cd3ef0 100644 --- a/docs/zh/connectors/common-options/sink-common-options.md +++ b/docs/zh/connectors/common-options/sink-common-options.md @@ -17,27 +17,32 @@ sidebar_position: 4 | 名称 | 类型 | 是否需要 | 默认值 | |--------------|--------|------|-----| -| plugin_input | string | 否 | - | +| plugin_input | list | 否 | - | | metadata_datasource_id | string | 否 | - | ### metadata_datasource_id [string] 用于从元数据中心获取连接配置的数据源 ID。当指定此参数时，连接器将从外部元数据服务获取连接详细信息（如 URL、用户名、密码），而不是使用直接配置。详见 [元数据 SPI](../../introduction/concepts/metadata-spi.md)。 -### plugin_input [string] +### plugin_input [list] -当不指定 `plugin_input` 时，当前插件处理配置文件中上一个插件输出的数据集 `dataset` +当不指定 `plugin_input` 时，当前插件处理配置文件中上一个插件输出的数据集 `dataset`。 -当指定了 `plugin_input` 时，当前插件正在处理该参数对应的数据集 +当指定了 `plugin_input` 时，当前插件处理该参数对应的数据集。该值可以是单个上游数据集名称（字符串），也可以是名称数组；配置多个名称时，Zeta 引擎会将这些上游数据集合并后写入当前 sink。 ## Examples ```bash source { - FakeSourceStream { + FakeSource { parallelism = 2 plugin_output = "fake" - field_name = "name,age" + schema = { + fields { + name = "string" + age = "int" + } + } } } diff --git a/docs/zh/connectors/common-options/sink-write-modes.md b/docs/zh/connectors/common-options/sink-write-modes.md index de3f906094..5985bbbd5e 100644 --- a/docs/zh/connectors/common-options/sink-write-modes.md +++ b/docs/zh/connectors/common-options/sink-write-modes.md @@ -99,9 +99,11 @@ File Sink 写的是文件，因此不使用 `generate_sink_sql`、`query` 或数 | SmbFile | 是 | 处理 SMB 目录和文件。 | | S3File | 是 | 通过 File Sink save mode 流程处理 S3 路径和对象。 | | OssFile | 是 | 通过 File Sink save mode 流程处理 OSS 路径和对象。 | +| GcsFile | 是 | 通过 File Sink save mode 流程处理 GCS 路径和对象。 | | ObsFile | 否 | 当前 sink option rule 没有暴露 `schema_save_mode` 或 `data_save_mode`。 | | CosFile | 否 | 当前 sink option rule 没有暴露 `schema_save_mode` 或 `data_save_mode`。 | | BosFile | 否 | 当前 sink option rule 没有暴露 `schema_save_mode` 或 `data_save_mode`。 | +| OssJindoFile | 否 | 当前 sink option rule 没有暴露 `schema_save_mode` 或 `data_save_mode`。 | 如果某个文件 connector 页面没有列出 `schema_save_mode` 或 `data_save_mode`，不要默认认为该 connector 可以接收这些参数。 @@ -176,7 +178,7 @@ sink { ### File Sink 不接受 `data_save_mode` -检查具体 connector 参数表。`S3File`、`OssFile`、`HdfsFile`、`FtpFile`、`SftpFile`、`LocalFile` 暴露文件 save mode 参数；`ObsFile`、`CosFile` 和 `BosFile` 当前未暴露。 +检查具体 connector 参数表。`LocalFile`、`HdfsFile`、`FtpFile`、`SftpFile`、`SmbFile`、`S3File`、`OssFile`、`GcsFile` 暴露文件 save mode 参数；`ObsFile`、`CosFile`、`BosFile` 和 `OssJindoFile` 当前未暴露。 ### 我只想建表，不想抽取数据 diff --git a/docs/zh/connectors/common-options/source-common-options.md b/docs/zh/connectors/common-options/source-common-options.md index d61aa85351..911c0f396b 100644 --- a/docs/zh/connectors/common-options/source-common-options.md +++ b/docs/zh/connectors/common-options/source-common-options.md @@ -20,7 +20,7 @@ sidebar_position: 3 # 重要提示 -在作业配置中使用 `plugin_output` 时，必须设置 `plugin_input` 参数。 +`plugin_output` 会把当前 source 产生的数据集注册为临时表，供下游 transform 或 sink 通过 `plugin_input` 引用。source 自身不需要也不会使用 `plugin_input`。如果作业只有一个 source、最多一个 transform 和一个 sink，这两个参数都可以省略；如果 source、transform 或 sink 中任意一类的数量大于 1，则每个插件都必须声明 `plugin_output`/`plugin_input`，引擎才能正确构建 DAG。 ## 任务示例 @@ -30,8 +30,13 @@ sidebar_position: 3 ```bash source { - FakeSourceStream { + FakeSource { plugin_output = "fake_table" + schema = { + fields { + id = int + } + } } } ``` diff --git a/docs/zh/connectors/sink/Fluss.md b/docs/zh/connectors/sink/Fluss.md index 3ea5832c41..206c0e8cb1 100644 --- a/docs/zh/connectors/sink/Fluss.md +++ b/docs/zh/connectors/sink/Fluss.md @@ -244,8 +244,9 @@ sink { ### 跨库复制 CDC 变更 可以使用占位符把一个 Fluss 命名空间下的 CDC 变更复制到另一个命名空间。 -`schema_name` 占位符会解析为上游数据库名，`table_name` 解析为上游表名， -这样单个 Sink 配置就能扇出多张表。 +Fluss 只有 database + table 两级标识（没有 schema 层级），因此在 Fluss-to-Fluss +链路中上游的 `schemaName` 始终为 null。应使用 `${database_name}` 获取上游 Fluss +database、`${table_name}` 获取上游 Fluss 表，这样单个 Sink 配置就能扇出多张表。 ```hocon env { @@ -268,7 +269,7 @@ sink { Fluss { plugin_input = "fluss_cdc" bootstrap.servers = "fluss-coordinator:9123" - database = "sink_db_${schema_name}" + database = "sink_db_${database_name}" table = "sink_${table_name}" multi_table_sink_replica = 2 } diff --git a/docs/zh/connectors/sink/Kudu.md b/docs/zh/connectors/sink/Kudu.md index 3fb48cb8d7..2665a8bc24 100644 --- a/docs/zh/connectors/sink/Kudu.md +++ b/docs/zh/connectors/sink/Kudu.md @@ -26,7 +26,9 @@ import ChangeLog from '../changelog/connector-kudu.md'; | SeaTunnel 数据类型 | Kudu 数据类型 | |---------------------|--------------------------| | BOOLEAN | BOOL | -| INT | INT8<br/>INT16<br/>INT32 | +| TINYINT | INT8 | +| SMALLINT | INT16 | +| INT | INT32 | | BIGINT | INT64 | | DECIMAL | DECIMAL | | FLOAT | FLOAT | diff --git a/docs/zh/connectors/sink/Prometheus.md b/docs/zh/connectors/sink/Prometheus.md index 04d4125b34..b57be92e0d 100644 --- a/docs/zh/connectors/sink/Prometheus.md +++ b/docs/zh/connectors/sink/Prometheus.md @@ -47,7 +47,7 @@ Prometheus 数据接收器把上游数据写入 Prometheus remote write API。 | key_value | String | 是 | - | 上游数据中保存 Prometheus 指标值的字段名。推荐使用 `double` 类型字段。 | | key_timestamp | String | 否 | - | 上游数据中保存 Prometheus 指标时间戳的字段名。不配置时使用当前系统时间。 | | headers | Map | 否 | - | HTTP 请求头。 | -| retry | Int | 否 | 3 | remote-write 请求失败时的最大重试次数。会重试传输层 `IOException` 以及可重试的 HTTP 状态码（`5xx` 和 `429`）；其他 `4xx` 响应会快速失败。设为 `0` 可禁用重试。 | +| retry | Int | 否 | 3 | remote-write 请求失败时的最大重试次数。未配置该选项时，Sink 生效的默认值为 `3`；配置为 `0` 可禁用重试。会重试传输层 `IOException` 以及可重试的 HTTP 状态码（`5xx` 和 `429`）；其他 `4xx` 响应会快速失败。 | | retry_backoff_multiplier_ms | Int | 否 | 100 | 重试退避时间倍数，单位毫秒。 | | retry_backoff_max_ms | Int | 否 | 10000 | 最大重试退避时间，单位毫秒。 | | batch_size | Int | 否 | 1024 | 写入 Prometheus 前缓存的行数，必须大于 0。 | diff --git a/docs/zh/connectors/source/Kudu.md b/docs/zh/connectors/source/Kudu.md index 4c7fd23e2e..905e455c58 100644 --- a/docs/zh/connectors/source/Kudu.md +++ b/docs/zh/connectors/source/Kudu.md @@ -33,7 +33,9 @@ import ChangeLog from '../changelog/connector-kudu.md'; | Kudu 数据类型 | SeaTunnel 数据类型 | |-------------|------------------| | BOOL | BOOLEAN | -| INT8<br/>INT16<br/>INT32 | INT | +| INT8 | TINYINT | +| INT16 | SMALLINT | +| INT32 | INT | | INT64 | BIGINT | | DECIMAL | DECIMAL | | FLOAT | FLOAT | diff --git a/docs/zh/connectors/source/Pulsar.md b/docs/zh/connectors/source/Pulsar.md index be59880ab3..d909f205cd 100644 --- a/docs/zh/connectors/source/Pulsar.md +++ b/docs/zh/connectors/source/Pulsar.md @@ -392,9 +392,9 @@ source { 当订阅分区 Topic 或使用正则表达式匹配 Topic 时（`topic-pattern`），SeaTunnel 会自动感知分区信息并将分片均匀分配给并发任务。可通过 `topic-discovery.interval` 调整新分区检测的探测频率。 -### 支持哪些消费游标（Cursor）位点恢复与订阅模式？ +### 支持哪些消费游标（Cursor）启动模式与重置策略？ -通过 `cursor.startup.mode` 支持配置为 `EARLIEST`（从最早未消费位点）或 `LATEST`（从最新位点开始消费）。消费者以配置的订阅类型（如 `Failover` 或 `Exclusive`）注册，并在 Checkpoint 时同步保存消息 ID，确保故障时精准回溯。 +通过 `cursor.startup.mode` 支持配置为 `EARLIEST`（从最早位点）、`LATEST`（从最新位点）、`SUBSCRIPTION`（从订阅已提交的位点恢复）或 `TIMESTAMP`（从指定时间戳开始消费）。使用 `TIMESTAMP` 时必须配置 `cursor.startup.timestamp`；使用 `SUBSCRIPTION` 时必须配置 `cursor.reset.mode`（`EARLIEST` 或 `LATEST`）：如果订阅已存在，则从其已提交的位点继续消费；否则按 `cursor.reset.mode` 在最早或最新消息处创建订阅。订阅类型（Subscription Type）不可配置，连接器只设置订阅名称并使用 Pulsar 客户端默认值。SeaTunnel 会在 Checkpoint 时保存消息 ID，确保故障时精准回溯。 ### 如何在连接受安全防护的 Pulsar 集群时配置身份认证？ diff --git a/docs/zh/connectors/source/Splunk.md b/docs/zh/connectors/source/Splunk.md index 758fcdb947..fdc1f25674 100644 --- a/docs/zh/connectors/source/Splunk.md +++ b/docs/zh/connectors/source/Splunk.md @@ -22,7 +22,7 @@ import ChangeLog from '../changelog/connector-http-splunk.md'; | method | String | 否 | `POST` | HTTP 请求方法 | | keep_params_as_form | Boolean | 否 | `true` | 以表单编码（form urlencoded）形式传递参数 | | max_response_size_bytes | Long | 否 | `20971520` | 允许的最大 HTTP 响应大小（字节），超过后快速失败，以避免大数据量导出时内存无限增长。 | -| params | Map | 是 | - | 请求参数，包括 `search` 和 `output_mode` | +| params | Map | 否 | - | 请求参数，例如 `search` 和 `output_mode`。该选项不会被校验为必填，但要得到有意义的导出结果需要提供 `search` 参数；`output_mode` 未配置时默认为 `json`。 | ## 示例配置
