This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-13062-fcb404cd947da03a712c4ae9ee9068744bd946a5 in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit c70765c2301b367b72a1d9340cc8894b677723c8 Author: Jast <[email protected]> AuthorDate: Sun Nov 1 07:15:44 2026 +0000 [Docs] Fix connector and common-option doc inaccuracies verified against source code (#13062) Co-authored-by: zhangshenghang <[email protected]> Co-authored-by: jast <[email protected]> Co-authored-by: multica-agent <[email protected]> --- .../common-options/source-common-options.md | 1 + docs/en/connectors/sink/MongoDB.md | 20 ++++++++++++-------- docs/en/connectors/source/GoogleSheets.md | 4 ++-- docs/en/connectors/source/Hive.md | 2 +- docs/en/connectors/source/Jira.md | 2 +- docs/en/connectors/source/LocalFile.md | 4 +++- docs/en/connectors/source/OneSignal.md | 21 ++++++++------------- docs/en/connectors/source/Opengauss-CDC.md | 12 +++++++----- docs/en/connectors/source/SqlServer-CDC.md | 2 +- .../common-options/source-common-options.md | 1 + docs/zh/connectors/sink/LocalFile.md | 6 ++---- docs/zh/connectors/sink/MongoDB.md | 16 +++++++++------- docs/zh/connectors/sink/Paimon.md | 2 +- docs/zh/connectors/source/GoogleSheets.md | 4 ++-- docs/zh/connectors/source/Jira.md | 2 +- docs/zh/connectors/source/LocalFile.md | 4 +++- docs/zh/connectors/source/OneSignal.md | 19 ++++++------------- docs/zh/connectors/source/Opengauss-CDC.md | 12 +++++++----- docs/zh/connectors/source/Paimon.md | 4 +++- docs/zh/connectors/source/SqlServer-CDC.md | 2 +- 20 files changed, 72 insertions(+), 68 deletions(-) diff --git a/docs/en/connectors/common-options/source-common-options.md b/docs/en/connectors/common-options/source-common-options.md index c0fe537bee..f1d10d446c 100644 --- a/docs/en/connectors/common-options/source-common-options.md +++ b/docs/en/connectors/common-options/source-common-options.md @@ -17,6 +17,7 @@ The old configuration name `result_table_name` is deprecated, please migrate to | plugin_output | String | No | - | When `plugin_output` is not specified, the data processed by this plugin will not be registered as a data set `(dataStream/dataset)` that can be directly accessed by other plugins, or called a temporary table `(table)` <br/>When `plugin_output` is specified, the data processed by this plugin will be registered as a data set `(dataStream/dataset)` that can be directly accessed by other plugins, or called a temporary table `(table)` . The dat [...] | parallelism | Int | No | - | When `parallelism` is not specified, the `parallelism` in env is used by default. <br/>When parallelism is specified, it will override the parallelism in env. [...] | metadata_datasource_id | String | No | - | The data source ID for retrieving connection configuration from Metadata Center. When specified, the connector will fetch connection details (e.g., URL, username, password) from the external metadata service instead of using direct configuration. See [Metadata SPI](../../introduction/concepts/metadata-spi.md) for more information. [...] +| dag-parsing.mode | Enum | No | SINGLENESS | Whether to enable parsing support for multi-table source. `SINGLENESS` processes each table with a separate source and sink; `MULTIPLEXING` processes multiple tables (such as sharded tables) with one source and one sink. | # Important note diff --git a/docs/en/connectors/sink/MongoDB.md b/docs/en/connectors/sink/MongoDB.md index c3d39a2555..fe97c3a360 100644 --- a/docs/en/connectors/sink/MongoDB.md +++ b/docs/en/connectors/sink/MongoDB.md @@ -32,7 +32,9 @@ The connector supports two write semantics: - **Append writes** — every row produces a new document. Fast, but not idempotent across retries. - **Upsert writes** — when `upsert-enable = true` and `primary-key` is configured, the connector - uses the primary key as the MongoDB `_id` (or compound `_id`) and upserts. Combined with + matches existing documents on the configured primary key fields and upserts. On insert, MongoDB + auto-generates the document `_id` only when the primary key does not include `_id`; with + `primary-key = _id` the supplied `_id` value is kept as the document identity. Combined with checkpoint-based recovery this gives at-least-once with idempotent retries, which is the standard way to deliver exactly-once into MongoDB. @@ -84,15 +86,15 @@ The following table lists the field data type mapping from SeaTunnel data type t | FLOAT | Double | | DECIMAL | Decimal128 | | Date | Date | -| Timestamp | Timestamp[Date] | +| Timestamp | Date | | ROW | Object | | ARRAY | Array | **Tips** -> 1. When SeaTunnel writes `Date` and `Timestamp` types to MongoDB, both become MongoDB `Date` -> fields, but at different precisions: SeaTunnel `Date` is second precision; SeaTunnel -> `Timestamp` is millisecond precision. +> 1. When SeaTunnel writes `Date` and `Timestamp` types to MongoDB, both are written as MongoDB +> `Date` fields (`BsonDateTime`, millisecond precision). A SeaTunnel `Date` is stored as +> midnight of that day; a SeaTunnel `Timestamp` keeps its time-of-day. > 2. When using the `DECIMAL` type in SeaTunnel, the maximum range cannot > exceed 34 digits. Use > `decimal(34, 18)` to stay within the supported precision and scale. @@ -311,9 +313,11 @@ By specifying a clear primary key and using the upsert method, exactly-once writ achieved. If `primary-key` and `upsert-enable` are defined in the configuration, the MongoDB sink uses -upsert semantics instead of regular INSERT statements. The connector combines the primary keys -declared in `primary-key` as the MongoDB reserved primary key and writes via upsert mode to ensure -idempotent writes. In the event of a failure, SeaTunnel jobs recover from the last successful +upsert semantics instead of regular INSERT statements. The connector uses the primary keys +declared in `primary-key` as the upsert match filter and writes via upsert mode to ensure +idempotent writes; when no matching document exists, the inserted document gets an auto-generated +MongoDB `_id` unless the primary key itself includes `_id`, in which case the filter's `_id` value +is kept as the inserted document's identity. In the event of a failure, SeaTunnel jobs recover from the last successful checkpoint and reprocess, which may result in duplicate processing during recovery. It is highly recommended to use upsert mode because it avoids violating database primary key constraints and generating duplicate data if records need to be reprocessed. diff --git a/docs/en/connectors/source/GoogleSheets.md b/docs/en/connectors/source/GoogleSheets.md index b52ce2ab26..4cdf248c11 100644 --- a/docs/en/connectors/source/GoogleSheets.md +++ b/docs/en/connectors/source/GoogleSheets.md @@ -88,7 +88,7 @@ the available types. ```hocon source { GoogleSheets { - service_account_key = "seatunnel-test" + service_account_key = "<base64-encoded service account JSON>" sheet_id = "1VI0DvyZK-NIdssSdsDSsSSSC-_-rYMi7ppJiI_jhE" sheet_name = "sheets01" range = "A1:C3" @@ -115,7 +115,7 @@ env { source { GoogleSheets { - service_account_key = "seatunnel-test" + service_account_key = "<base64-encoded service account JSON>" sheet_id = "1VI0DvyZK-NIdssSdsDSsSSSC-_-rYMi7ppJiI_jhE" sheet_name = "sheets01" range = "A1:C100" diff --git a/docs/en/connectors/source/Hive.md b/docs/en/connectors/source/Hive.md index 9cd8fb2baa..db262f9388 100644 --- a/docs/en/connectors/source/Hive.md +++ b/docs/en/connectors/source/Hive.md @@ -12,7 +12,7 @@ import ChangeLog from '../changelog/connector-hive.md'; ## Description -Read data from Apache Hive tables. The connector talks to Hive Metastore for schema discovery and reads the underlying files from HDFS (or S3/OSS when configured). The supported table storage formats are text, parquet and ORC; the format is detected from the Hive table's storage format (`STORED AS`) and tables stored in any other format are rejected. Each table can be read as one batch split, with parallelism and snapshot/offset resume supported through the checkpoint mechanism. +Read data from Apache Hive tables. The connector talks to Hive Metastore for schema discovery and reads the underlying files from HDFS (or S3/OSS when configured). The supported table storage formats are text, parquet and ORC; the format is detected from the Hive table's storage format (`STORED AS`) and tables stored in any other format are rejected. The data files of each table are enumerated as one split per file, with parallelism and snapshot/offset resume supported through the checkp [...] :::tip diff --git a/docs/en/connectors/source/Jira.md b/docs/en/connectors/source/Jira.md index d495aa284b..8ed3f02921 100644 --- a/docs/en/connectors/source/Jira.md +++ b/docs/en/connectors/source/Jira.md @@ -31,7 +31,7 @@ The Jira source is a batch-only connector. It throws an error when the job runs | email | String | Yes | - | Jira account email used for Basic authentication. | | api_token | String | Yes | - | Jira API token used for Basic authentication. | | method | String | No | GET | HTTP method. Supported values are `GET` and `POST`. | -| headers | Map | No | - | Extra HTTP request headers. Do not put the Jira `Authorization` header here unless you intentionally want to override the generated header. | +| headers | Map | No | - | Extra HTTP request headers. Note: any `Authorization` entry configured here is overwritten by the Basic authentication header generated from `email` and `api_token`. | | params | Map | No | - | HTTP query parameters. | | body | String | No | - | HTTP request body, usually used with `POST`. | | format | String | No | TEXT | Response format. Use `json` when the response should be parsed by `schema`, `json_field`, or `content_field`. | diff --git a/docs/en/connectors/source/LocalFile.md b/docs/en/connectors/source/LocalFile.md index e98f7cc586..290c7d6822 100644 --- a/docs/en/connectors/source/LocalFile.md +++ b/docs/en/connectors/source/LocalFile.md @@ -58,7 +58,7 @@ For bounded Zeta jobs, restore resumes pending file splits after readers registe | name | type | required | default value | |----------------------------|---------|----------|--------------------------------------| | path | string | yes | - | -| file_format_type | string | yes | - | +| file_format_type | string | conditionally yes | - | | read_columns | list | no | - | | read_partitions | list | no | - | | delimiter/field_delimiter | string | no | \001 | @@ -122,6 +122,8 @@ File type, supported as the following file types: If you assign file type to `json`, you should also assign schema option to tell connector how to parse data to the row you want. +Required when reading a single table with `path`. When reading multiple tables with `tables_configs`, the top-level option is not needed because each table entry defines its own `file_format_type`. + For example: upstream data is the following: diff --git a/docs/en/connectors/source/OneSignal.md b/docs/en/connectors/source/OneSignal.md index c72e5ea591..1c63164a69 100644 --- a/docs/en/connectors/source/OneSignal.md +++ b/docs/en/connectors/source/OneSignal.md @@ -26,7 +26,7 @@ Use this connector to ingest OneSignal resources such as apps, players, segments | url | String | Yes | - | OneSignal REST API endpoint. Common endpoints include `https://onesignal.com/api/v1/apps` and `https://onesignal.com/api/v1/players`. | | password | String | Yes | - | OneSignal user auth key. The connector sends it as the HTTP `Authorization: Basic <password>` header. Create one at [OneSignal Accounts and Keys](https://documentation.onesignal.com/docs/accounts-and-keys#user-auth-key). | | method | String | No | get | HTTP request method. Supported values are `GET` and `POST`. | -| headers | Map | No | - | Extra HTTP headers. Do not put `Authorization` here unless you want to override the header generated from `password`. | +| headers | Map | No | - | Extra HTTP headers. Note: any `Authorization` entry configured here is overwritten by the `Authorization: Basic <password>` header generated from `password`. | | params | Map | No | - | HTTP query parameters, such as `limit`, `offset`, or other OneSignal API parameters. | | body | String | No | - | HTTP request body. Useful for endpoints that accept a JSON payload. | | format | String | No | text | Response format. Use `json` with `schema` to read OneSignal JSON as SeaTunnel rows with named fields. Use `text` to keep the raw response. | @@ -34,7 +34,7 @@ Use this connector to ingest OneSignal resources such as apps, players, segments | schema.fields | Config | No | - | Field names and SeaTunnel data types used to parse the JSON response. | | json_field | Config | No | - | Field-level JSONPath mapping. Use it with `schema` when each output field lives at a different JSON path. | | content_field | String | No | - | JSONPath expression that selects a JSON fragment before `schema` parses it. For example, use `$.players[*]` to flatten a list response. | -| pageing | Config | No | - | HTTP pagination settings inherited from the HTTP source connector. OneSignal paged endpoints use `page` / `per_page` parameters. | +| pageing | Config | No | - | HTTP pagination settings inherited from the HTTP source connector. Currently not effective for OneSignal: the reader is created without pagination support, so a job reads only a single response per request. | | poll_interval_millis | Int | No | - | Request interval in milliseconds for streaming jobs. In batch mode the connector reads once and finishes. | | retry | Int | No | - | Maximum retry count when an HTTP request fails with `IOException`. | | retry_backoff_multiplier_ms | Int | No | 100 | Retry backoff multiplier in milliseconds. | @@ -50,7 +50,7 @@ Use this connector to ingest OneSignal resources such as apps, players, segments - Set `format = "json"` and define `schema` when you want typed SeaTunnel rows. - Use `content_field` when OneSignal wraps records in a nested array such as `$.players[*]`. - Use `json_field` only when each output field needs its own JSONPath expression. -- OneSignal paged endpoints accept `page` and `per_page` query parameters; configure them through `params` and `pageing`. +- The OneSignal reader currently ignores `pageing`; each job reads a single response. Use `params` (for example `limit`) to control how many records that response returns. ## Task Examples @@ -90,9 +90,12 @@ sink { } ``` -### Read Players With Pagination +### Read Players -Use `params` together with `pageing` to walk through paged OneSignal endpoints: +Query the players endpoint with `params`. Note that the OneSignal reader currently does not +paginate: a `pageing` block is accepted but ignored, and the job reads only the single response +returned for the configured `params`. Use `limit` to control how many records that response +returns: ```hocon env { @@ -108,14 +111,6 @@ source { params = { app_id = "<your-app-id>" limit = "50" - offset = "0" - } - pageing = { - page_field = "offset" - start_page_number = 0 - page_step = 50 - total_page_size = 10 - use_placeholder_replacement = false } format = "json" content_field = "$.players[*]" diff --git a/docs/en/connectors/source/Opengauss-CDC.md b/docs/en/connectors/source/Opengauss-CDC.md index 83e2b4c8d8..c833a4912b 100644 --- a/docs/en/connectors/source/Opengauss-CDC.md +++ b/docs/en/connectors/source/Opengauss-CDC.md @@ -186,11 +186,13 @@ Opengauss CDC exposes metadata fields that can be used by the `Metadata` transfo | Field | Type | Description | |-------|------|-------------| -| database | STRING | Source database name. | -| table | STRING | Source table name. | -| rowKind | STRING | Change type, such as insert, update, or delete. | -| ts_ms | LONG | Source event timestamp in milliseconds. | -| delay | LONG | Delay between event time and processing time in milliseconds. | +| Database | STRING | Source database name. | +| Table | STRING | Source table name. | +| RowKind | STRING | Change type, such as insert, update, or delete. | +| EventTime | LONG | The time the connector fetched and processed the event, in milliseconds. | +| Delay | LONG | Delay between event time and processing time in milliseconds. | + +The metadata field keys are case-sensitive and must be written exactly as in the table above. Example: diff --git a/docs/en/connectors/source/SqlServer-CDC.md b/docs/en/connectors/source/SqlServer-CDC.md index e7f76b465a..d246d875a1 100644 --- a/docs/en/connectors/source/SqlServer-CDC.md +++ b/docs/en/connectors/source/SqlServer-CDC.md @@ -103,7 +103,7 @@ case-sensitive databases, make sure the configured identifier case matches the d | split.allow-sampling | Boolean | No | true | Whether to allow sampling-based sharding strategy. When set to false, the system will fall back to unevenly-sized chunk splitting (iterative query approach) regardless of the shard count. The default value is true. | | enable_concurrent_read | Boolean | No | true | Whether to enable concurrent read with split during the snapshot phase. When set to false, the source skips split analysis and reads the table as a single split, which is useful for tables without indexes. The default value is true. | | exactly_once | Boolean | No | false | Enable exactly once semantic. [...] -| debezium.* | config | No | - | Pass-through Debezium's properties to Debezium Embedded Engine which is used to capture data changes from SqlServer server.<br/>See more about<br/>the [Debezium's SqlServer Connector properties](https://github.com/debezium/debezium/blob/1.6/documentation/modules/ROOT/pages/connectors/sqlserver.adoc#connector-properties) [...] +| debezium.* | config | No | - | Pass-through Debezium's properties to Debezium Embedded Engine which is used to capture data changes from SqlServer server.<br/>See more about<br/>the [Debezium's SqlServer Connector properties](https://debezium.io/documentation/reference/1.9/connectors/sqlserver.html) [...] | format | Enum | No | DEFAULT | Optional output format for SqlServer CDC, valid enumerations are "DEFAULT"、"COMPATIBLE_DEBEZIUM_JSON". [...] | schema-changes.enabled | Boolean | No | false | Schema evolution is disabled by default. Now we only support `add column`、`drop column`、`rename column` and `modify column`. [...] | schema-changes.include | List | No | - | Only the listed schema change event types are sent downstream (when `schema-changes.enabled = true`). Empty means all are eligible. See [Schema change event filtering](#schema-change-event-filtering). [...] diff --git a/docs/zh/connectors/common-options/source-common-options.md b/docs/zh/connectors/common-options/source-common-options.md index 911c0f396b..243b43ace3 100644 --- a/docs/zh/connectors/common-options/source-common-options.md +++ b/docs/zh/connectors/common-options/source-common-options.md @@ -17,6 +17,7 @@ sidebar_position: 3 | plugin_output | String | 否 | - | 当未指定 `plugin_output` 时，此插件处理的数据将不会被注册为可由其他插件直接访问的数据集 `(dataStream/dataset)`，或称为临时表 `(table)`。<br/>当指定了 `plugin_output` 时，此插件处理的数据将被注册为可由其他插件直接访问的数据集 `(dataStream/dataset)`，或称为临时表 `(table)`。此处注册的数据集 `(dataStream/dataset)` 可通过指定 `plugin_input` 直接被其他插件访问。 | | parallelism | Int | 否 | - | 当未指定 `parallelism` 时，默认使用环境中的 `parallelism`。<br/>当指定了 `parallelism` 时，将覆盖环境中的 `parallelism` 设置。 | | metadata_datasource_id | String | 否 | - | 用于从元数据中心获取连接配置的数据源 ID。当指定此参数时，连接器将从外部元数据服务获取连接详细信息（如 URL、用户名、密码），而不是使用直接配置。详见 [元数据 SPI](../../introduction/concepts/metadata-spi.md)。 | +| dag-parsing.mode | Enum | 否 | SINGLENESS | 是否启用多表 source 的解析支持。`SINGLENESS` 表示每张表由独立的 source 和 sink 处理；`MULTIPLEXING` 表示多张表（例如分片表）由一个 source 和一个 sink 处理。 | # 重要提示 diff --git a/docs/zh/connectors/sink/LocalFile.md b/docs/zh/connectors/sink/LocalFile.md index 22dc30926e..b1f50400ce 100644 --- a/docs/zh/connectors/sink/LocalFile.md +++ b/docs/zh/connectors/sink/LocalFile.md @@ -60,7 +60,7 @@ import ChangeLog from '../changelog/connector-file-local.md'; | filename_time_format | string | 否 | "yyyy.MM.dd" | 仅在 custom_filename 为 true 时使用 | | file_format_type | string | 否 | "csv" | 文件格式类型 | | filename_extension | string | 否 | - | 使用自定义的文件扩展名覆盖默认的文件扩展名。 例如：`.xml`, `.json`, `dat`, `.customtype` | -| field_delimiter | string | 否 | '\001' for text and ',' for csv | 仅在 file_format_type 为 text 时使用 | +| field_delimiter | string | 否 | '\001' for text and ',' for csv | 仅在 file_format_type 为 text 和 csv 时使用 | | row_delimiter | string | 否 | "\n" | 仅在 file_format_type 为 `text`、`csv`、`json` 时使用 | | have_partition | boolean | 否 | false | 是否需要处理分区 | | partition_by | array | 否 | - | 仅在 have_partition 为 true 时使用 | @@ -216,9 +216,7 @@ Sink 插件的常见参数，请参阅 [Sink 常见选项](../common-options/sin - MINIMAL：仅为包含特殊字符（如字段分隔符、引号字符或行分隔符字符串中的任何字符）的字段加引号。 - NONE：从不为字段加引号。当数据中包含分隔符时，输出会在前面加上转义字符。如果未设置转义字符，则格式验证会抛出异常。 -### xml - -_root_tag [string] +### xml_root_tag [string] 指定 XML 文件中根元素的标签名。 diff --git a/docs/zh/connectors/sink/MongoDB.md b/docs/zh/connectors/sink/MongoDB.md index cbf4865630..cdd2c4fbca 100644 --- a/docs/zh/connectors/sink/MongoDB.md +++ b/docs/zh/connectors/sink/MongoDB.md @@ -31,8 +31,9 @@ MongoDB Sink 连接器将 SeaTunnel 行记录写入 MongoDB 集合。每行记 连接器支持两种写入语义： - **追加写入**：每行生成一条新文档。性能最好，但在失败重试场景下不幂等。 -- **Upsert 写入**：当 `upsert-enable = true` 且配置了 `primary-key` 时，连接器会把主键作为 - MongoDB 的 `_id`（或复合 `_id`）并以 upsert 方式写入。结合 checkpoint 恢复机制，可以实现 +- **Upsert 写入**：当 `upsert-enable = true` 且配置了 `primary-key` 时，连接器会把主键字段作为 + upsert 的匹配过滤条件并以 upsert 方式写入。插入新文档时，仅当主键不包含 `_id` 的情况下文档 + `_id` 才由 MongoDB 自动生成；若主键为 `_id`，则沿用过滤条件中传入的 `_id` 值。结合 checkpoint 恢复机制，可以实现 at-least-once + 幂等重试，这是把 exactly-once 落到 MongoDB 的标准做法。 缓存、重试以及可选的事务都可以通过下文的配置项进行调节。 @@ -79,14 +80,14 @@ MongoDB Sink 连接器将 SeaTunnel 行记录写入 MongoDB 集合。每行记 | FLOAT | Double | | DECIMAL | Decimal128 | | Date | Date | -| Timestamp | Timestamp / Date | +| Timestamp | Date | | ROW | Object | | ARRAY | Array | **提示** -> 1. 使用 SeaTunnel 将 `Date` 和 `Timestamp` 类型写入 MongoDB 时，结果都是 `Date` 类型，但 -> 精度不同：SeaTunnel 的 `Date` 为秒级精度，`Timestamp` 为毫秒级精度。 +> 1. 使用 SeaTunnel 将 `Date` 和 `Timestamp` 类型写入 MongoDB 时，都会写入为 MongoDB 的 `Date` +> 类型（BsonDateTime，毫秒精度）。SeaTunnel 的 `Date` 会存储为当天零点，`Timestamp` 则保留时分秒信息。 > 2. 在 SeaTunnel 中使用 `DECIMAL` 类型时，最大精度不能超过 34 位。建议使用 > `decimal(34, 18)` 以满足支持的精度与标度。 @@ -298,8 +299,9 @@ sink { 通过定义明确的主键并启用 `upsert` 模式，可以实现精准一次写入（exactly-once）语义。 当配置中定义了 `primary-key` 且启用了 `upsert-enable`，MongoDB Sink 将使用 Upsert 语义而非 -普通 INSERT 语句。SeaTunnel 会将定义的主键作为 MongoDB 的复合主键，在 Upsert 模式下写入， -以确保幂等性。 +普通 INSERT 语句。SeaTunnel 会将定义的主键作为 Upsert 的匹配过滤条件，在 Upsert 模式下写入 +以确保幂等性；插入新文档时，仅当主键不包含 `_id` 的情况下 `_id` 才由 MongoDB 自动生成 +（若主键为 `_id`，则新文档沿用该 `_id` 值）。 若作业在运行过程中失败，SeaTunnel 会从上一个成功的 checkpoint 恢复并重新处理数据，这可能 导致重复数据。强烈建议启用 Upsert 模式，以避免主键冲突或重复插入。 diff --git a/docs/zh/connectors/sink/Paimon.md b/docs/zh/connectors/sink/Paimon.md index cc817cf012..61e25aecd3 100644 --- a/docs/zh/connectors/sink/Paimon.md +++ b/docs/zh/connectors/sink/Paimon.md @@ -160,7 +160,7 @@ Cdc采集支持有限数量的模式更改。目前支持的模式更改包括 * 删除列。 -* 更改列。 +* 重命名列（可与上文“修改列类型”中列出的类型加宽、列位置、注释、可空性变更同时进行——当新旧列名不同时，变更处理器会在同一步骤中应用这些更新并重命名该列）。 ## 示例 diff --git a/docs/zh/connectors/source/GoogleSheets.md b/docs/zh/connectors/source/GoogleSheets.md index 694a4cb43b..4be8807fbe 100644 --- a/docs/zh/connectors/source/GoogleSheets.md +++ b/docs/zh/connectors/source/GoogleSheets.md @@ -82,7 +82,7 @@ Google Cloud 服务账号密钥 JSON 文件经过 Base64 编码后的内容。 ```hocon source { GoogleSheets { - service_account_key = "seatunnel-test" + service_account_key = "<base64-encoded service account JSON>" sheet_id = "1VI0DvyZK-NIdssSdsDSsSSSC-_-rYMi7ppJiI_jhE" sheet_name = "sheets01" range = "A1:C3" @@ -109,7 +109,7 @@ env { source { GoogleSheets { - service_account_key = "seatunnel-test" + service_account_key = "<base64-encoded service account JSON>" sheet_id = "1VI0DvyZK-NIdssSdsDSsSSSC-_-rYMi7ppJiI_jhE" sheet_name = "sheets01" range = "A1:C100" diff --git a/docs/zh/connectors/source/Jira.md b/docs/zh/connectors/source/Jira.md index 263223b6fc..25c9272271 100644 --- a/docs/zh/connectors/source/Jira.md +++ b/docs/zh/connectors/source/Jira.md @@ -31,7 +31,7 @@ Jira Source 只支持批处理。如果作业以流模式运行，连接器会 | email | String | 是 | - | 用于 Basic 认证的 Jira 账号邮箱。 | | api_token | String | 是 | - | 用于 Basic 认证的 Jira API Token。 | | method | String | 否 | GET | HTTP 请求方法，支持 `GET` 和 `POST`。 | -| headers | Map | 否 | - | 额外的 HTTP 请求头。除非需要覆盖自动生成的 Jira 认证头，否则不要在这里配置 `Authorization`。 | +| headers | Map | 否 | - | 额外的 HTTP 请求头。注意：此处配置的 `Authorization` 会被连接器根据 `email` 和 `api_token` 生成的 Basic 认证头覆盖。 | | params | Map | 否 | - | HTTP 查询参数。 | | body | String | 否 | - | HTTP 请求体，通常和 `POST` 一起使用。 | | format | String | 否 | TEXT | 返回内容格式。如果要用 `schema`、`json_field` 或 `content_field` 解析 JSON，请设置为 `json`。 | diff --git a/docs/zh/connectors/source/LocalFile.md b/docs/zh/connectors/source/LocalFile.md index c721289c8c..97c146bbe5 100644 --- a/docs/zh/connectors/source/LocalFile.md +++ b/docs/zh/connectors/source/LocalFile.md @@ -58,7 +58,7 @@ import ChangeLog from '../changelog/connector-file-local.md'; | 名称 | 类型 | 是否必须 | 默认值 | |----------------------------|---------|------|---------------------| | path | string | 是 | - | -| file_format_type | string | 是 | - | +| file_format_type | string | 视情况而定 | - | | read_columns | list | 否 | - | | read_partitions | list | 否 | - | | delimiter/field_delimiter | string | 否 | \001 | @@ -122,6 +122,8 @@ import ChangeLog from '../changelog/connector-file-local.md'; 如果您将文件类型指定为 `json`，您还应该指定 schema 选项来告诉连接器如何将数据解析为您想要的行。 +使用 `path` 读取单表时必填。使用 `tables_configs` 读取多表时，每个表条目都会定义自己的 `file_format_type`，无需在顶层配置该选项。 + 例如： 上游数据如下： diff --git a/docs/zh/connectors/source/OneSignal.md b/docs/zh/connectors/source/OneSignal.md index 217e736fd5..a56549dfe0 100644 --- a/docs/zh/connectors/source/OneSignal.md +++ b/docs/zh/connectors/source/OneSignal.md @@ -26,7 +26,7 @@ OneSignal 源连接器用于从 OneSignal 的 REST API 读取数据。它基于 | url | String | 是 | - | OneSignal REST API 地址，常见接口如 `https://onesignal.com/api/v1/apps`、`https://onesignal.com/api/v1/players`。 | | password | String | 是 | - | OneSignal 用户 Auth Key，连接器会将其作为 `Authorization: Basic <password>` 请求头发送。可在 [OneSignal 账号与密钥](https://documentation.onesignal.com/docs/accounts-and-keys#user-auth-key) 中创建。 | | method | String | 否 | get | HTTP 请求方法，支持 `GET` 和 `POST`。 | -| headers | Map | 否 | - | 额外的 HTTP 请求头。除非需要覆盖由 `password` 生成的头，否则不要在这里配置 `Authorization`。 | +| headers | Map | 否 | - | 额外的 HTTP 请求头。注意：此处配置的 `Authorization` 会被连接器根据 `password` 生成的 `Authorization: Basic <password>` 请求头覆盖。 | | params | Map | 否 | - | HTTP 查询参数，例如 `limit`、`offset` 等 OneSignal API 参数。 | | body | String | 否 | - | HTTP 请求体，对支持 JSON 负载的接口有用。 | | format | String | 否 | text | 响应格式，`json` 时需要配合 `schema`；`text` 时返回原始响应。 | @@ -34,7 +34,7 @@ OneSignal 源连接器用于从 OneSignal 的 REST API 读取数据。它基于 | schema.fields | Config | 否 | - | 字段名与 SeaTunnel 数据类型，用于解析 JSON 响应。 | | json_field | Config | 否 | - | 字段级 JSONPath 映射，与 `schema` 配合使用。 | | content_field | String | 否 | - | 在 `schema` 解析之前先通过 JSONPath 抽取一段 JSON，例如 `$.players[*]` 可展开列表响应。 | -| pageing | Config | 否 | - | HTTP 分页配置，继承自 HTTP 源连接器。OneSignal 分页接口通常使用 `page` / `per_page` 参数。 | +| pageing | Config | 否 | - | HTTP 分页配置，继承自 HTTP 源连接器。对 OneSignal 当前不生效：连接器创建的读取器不带分页能力，任务每次请求只读取单次响应。 | | poll_interval_millis | int | 否 | - | 流式任务下两次请求之间的间隔（毫秒）。批模式下连接器读取一次后即结束。 | | retry | int | 否 | - | HTTP 请求返回 `IOException` 时的最大重试次数。 | | retry_backoff_multiplier_ms | int | 否 | 100 | HTTP 请求失败时的重试退避倍数（毫秒）。 | @@ -50,7 +50,7 @@ OneSignal 源连接器用于从 OneSignal 的 REST API 读取数据。它基于 - 需要按字段读取时，把 `format` 设置为 `json` 并配置 `schema`。 - 当 OneSignal 把记录嵌套在数组中（例如 `players` 列表）时，使用 `content_field` 抽取数组元素。 - 只有当不同字段位于不同 JSON 路径时，才需要使用 `json_field`。 -- OneSignal 分页接口使用 `page` 和 `per_page` 查询参数，可以通过 `params` 与 `pageing` 配合来逐页读取。 +- OneSignal 读取器目前会忽略 `pageing`，每个任务只读取单次响应；可通过 `params`（例如 `limit`）控制该次响应返回的记录数。 ## 任务示例 @@ -90,9 +90,10 @@ sink { } ``` -### 读取 Players 列表（分页） +### 读取 Players 列表 -通过 `params` 配合 `pageing` 读取 OneSignal 分页接口： +通过 `params` 查询 Players 接口。注意：OneSignal 读取器目前不支持自动翻页，`pageing` 配置会被 +忽略，任务只读取按当前 `params` 发出的单次响应；可通过 `limit` 控制该次响应返回的记录数： ```hocon env { @@ -108,14 +109,6 @@ source { params = { app_id = "<your-app-id>" limit = "50" - offset = "0" - } - pageing = { - page_field = "offset" - start_page_number = 0 - page_step = 50 - total_page_size = 10 - use_placeholder_replacement = false } format = "json" content_field = "$.players[*]" diff --git a/docs/zh/connectors/source/Opengauss-CDC.md b/docs/zh/connectors/source/Opengauss-CDC.md index 768e03827b..287f310faa 100644 --- a/docs/zh/connectors/source/Opengauss-CDC.md +++ b/docs/zh/connectors/source/Opengauss-CDC.md @@ -183,11 +183,13 @@ Opengauss CDC 会提供以下元数据字段，可配合 `Metadata` 转换使用 | 字段 | 类型 | 说明 | |------|------|------| -| database | STRING | 源数据库名称。 | -| table | STRING | 源表名称。 | -| rowKind | STRING | 变更类型，例如 insert、update 或 delete。 | -| ts_ms | LONG | 源事件时间，单位为毫秒。 | -| delay | LONG | 事件时间和处理时间之间的延迟，单位为毫秒。 | +| Database | STRING | 源数据库名称。 | +| Table | STRING | 源表名称。 | +| RowKind | STRING | 变更类型，例如 insert、update 或 delete。 | +| EventTime | LONG | 连接器读取并处理该事件的时间，单位为毫秒。 | +| Delay | LONG | 事件时间和处理时间之间的延迟，单位为毫秒。 | + +元数据字段的键名区分大小写，必须按照上表原样书写。 示例： diff --git a/docs/zh/connectors/source/Paimon.md b/docs/zh/connectors/source/Paimon.md index 0c4bc651c6..a9e8add00b 100644 --- a/docs/zh/connectors/source/Paimon.md +++ b/docs/zh/connectors/source/Paimon.md @@ -90,7 +90,7 @@ Paimon 的 catalog URI。当 `catalog_type` 为 `hive` 时必须配置。 ### table_list [array] -`Paimon` 表名列表。当一个 Source 需要读取多张 Paimon 表时，使用此配置代替 `table`。`table` 和 `table_list` 必须二选一配置。每个配置项必须包含 `table`，也可以配置该表自己的 `query`。 +`Paimon` 表名列表。当一个 Source 需要读取多张 Paimon 表时，使用此配置代替 `table`。`table` 和 `table_list` 必须二选一配置。每个配置项必须包含 `database` 和 `table`，也可以配置该表自己的 `query`。 ### hdfs_site_path [string] @@ -164,10 +164,12 @@ source { database = "default" table_list = [ { + database = "default" table = "table1" query = "select * from table1 where id > 100" }, { + database = "default" table = "table2" query = "select * from table2 where id > 100" } diff --git a/docs/zh/connectors/source/SqlServer-CDC.md b/docs/zh/connectors/source/SqlServer-CDC.md index 9e4579286b..8b22afd602 100644 --- a/docs/zh/connectors/source/SqlServer-CDC.md +++ b/docs/zh/connectors/source/SqlServer-CDC.md @@ -97,7 +97,7 @@ Sql Server CDC 连接器允许从 SqlServer 数据库读取快照数据和增量 | split.allow-sampling | Boolean | 否 | true | 是否启用基于采样的分片策略。当设置为 false 时，无论预估分片数是否超过阈值，系统都将回退到非均匀分片方式（迭代查询方式）。默认值为 true。 | | enable_concurrent_read | Boolean | 否 | true | 是否在快照阶段启用基于分片的并发读取。当设置为 false 时，source 会跳过分片分析，并以单个 split 读取整张表，适合没有索引的表。默认值为 true。 | | exactly_once | Boolean | 否 | false | 启用精确一次语义。 | -| debezium.* | config | 否 | - | 将 Debezium 的属性传递给 Debezium Embedded Engine，用于捕获来自 SqlServer 服务器的数据变更。<br/>了解更多关于<br/>[Debezium 的 SqlServer 连接器属性](https://github.com/debezium/debezium/blob/1.6/documentation/modules/ROOT/pages/connectors/sqlserver.adoc#connector-properties) | +| debezium.* | config | 否 | - | 将 Debezium 的属性传递给 Debezium Embedded Engine，用于捕获来自 SqlServer 服务器的数据变更。<br/>了解更多关于<br/>[Debezium 的 SqlServer 连接器属性](https://debezium.io/documentation/reference/1.9/connectors/sqlserver.html) | | format | Enum | 否 | DEFAULT | SqlServer CDC 的可选输出格式，有效枚举为 "DEFAULT"、"COMPATIBLE_DEBEZIUM_JSON"。 | | schema-changes.enabled | Boolean | 否 | false | 模式演进默认是禁用的。当前我们只支持 `add column`、`drop column`、`rename column` 和 `modify column`。 | | schema-changes.include | List | 否 | - | 仅向下游发送列出的 schema change 事件类型（需 `schema-changes.enabled = true`）。为空表示全部允许。详见 [Schema change 事件过滤](#schema-change-事件过滤)。 |
