This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-12993-a48d4ddb67d886d1bbac79fba35673eeff822a9c in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit 2abfce0fc682f86785bfc49e8f43fbe5d219efb6 Author: Jast <[email protected]> AuthorDate: Tue Nov 17 15:49:54 2026 +0000 [Docs] Fix HTTP-based connector, CDC and sink doc inaccuracies verified against source code (#12993) Co-authored-by: zhangshenghang <[email protected]> Co-authored-by: jast <[email protected]> --- docs/en/connectors/sink/Cassandra.md | 15 ++++---- docs/en/connectors/sink/Couchbase.md | 2 +- docs/en/connectors/sink/Enterprise-WeChat.md | 37 ++++++++++---------- docs/en/connectors/sink/Feishu.md | 16 +++------ docs/en/connectors/sink/IoTDB.md | 4 +-- docs/en/connectors/sink/Lance.md | 2 +- docs/en/connectors/source/Github.md | 51 ++++++++++++---------------- docs/en/connectors/source/Gitlab.md | 50 ++++++++++++--------------- docs/en/connectors/source/MongoDB-CDC.md | 5 +++ docs/en/connectors/source/MySQL-CDC.md | 4 ++- docs/en/connectors/source/PostgreSQL-CDC.md | 5 +-- docs/en/connectors/source/Shopify.md | 2 +- docs/en/connectors/source/Zendesk.md | 9 ++--- docs/zh/connectors/sink/Cassandra.md | 12 +++---- docs/zh/connectors/sink/Couchbase.md | 2 +- docs/zh/connectors/sink/Enterprise-WeChat.md | 32 ++++++++--------- docs/zh/connectors/sink/Feishu.md | 16 +++------ docs/zh/connectors/sink/IoTDB.md | 4 +-- docs/zh/connectors/sink/Lance.md | 2 +- docs/zh/connectors/sink/Zendesk.md | 2 +- docs/zh/connectors/source/Github.md | 51 ++++++++++++---------------- docs/zh/connectors/source/Gitlab.md | 50 ++++++++++++--------------- docs/zh/connectors/source/MongoDB-CDC.md | 4 +++ docs/zh/connectors/source/MySQL-CDC.md | 4 ++- docs/zh/connectors/source/PostgreSQL-CDC.md | 5 +-- docs/zh/connectors/source/Shopify.md | 2 +- docs/zh/connectors/source/Zendesk.md | 7 ++-- 27 files changed, 183 insertions(+), 212 deletions(-) diff --git a/docs/en/connectors/sink/Cassandra.md b/docs/en/connectors/sink/Cassandra.md index a88449e3a7..8dd8165eb5 100644 --- a/docs/en/connectors/sink/Cassandra.md +++ b/docs/en/connectors/sink/Cassandra.md @@ -41,7 +41,7 @@ schema before starting the job. | username | String | No | - | Cassandra username. Configure it together with `password`. | | password | String | No | - | Cassandra password. Configure it together with `username`. | | datacenter | String | No | datacenter1 | Local datacenter name used by the Cassandra Java driver. | -| consistency_level | String | No | LOCAL_ONE | Write consistency level, such as `LOCAL_ONE`, `ONE`, `QUORUM`, or `LOCAL_QUORUM`. | +| consistency_level | String | No | LOCAL_ONE | Accepted for compatibility and validated, but currently NOT applied to write statements by this sink; writes use the driver default consistency level. | | fields | Array | No | - | Target columns to write. If not set, all target table columns are used. | | batch_size | int | No | 5000 | Maximum number of rows buffered before one flush. | | batch_type | String | No | UNLOGGED | Cassandra batch type. Supported driver values include `LOGGED`, `UNLOGGED`, and `COUNTER`. | @@ -77,7 +77,8 @@ The `Cassandra` datacenter, default is `datacenter1`. ### consistency_level [String] -The `Cassandra` write consistency level, default is `LOCAL_ONE`. +This option is parsed and validated (default `LOCAL_ONE`), but the current sink writer does not +apply it to write statements; writes use the driver default consistency level. ### fields [array] @@ -110,11 +111,10 @@ Sink plugin common parameters. For details, see [Sink Common Options](../common- - The target keyspace and table must already exist before the job starts. - `fields` is useful when the upstream row has extra columns. It is not a schema creation option. -- `async_write = true` improves throughput, while `batch_size` controls how many rows are grouped - before a flush. -- The sink uses the Cassandra Java driver. Authentication and consistency options match the - driver's terminology; configure `consistency_level` together with the cluster's replication - strategy when stronger guarantees are required. +- `async_write = true` (default) sends each row as an individual async statement, while `batch_size` controls how often the buffer is flushed. With `async_write = false`, rows are grouped into one `batch_type` batch of at most `batch_size` statements per flush. +- The sink uses the Cassandra Java driver. Authentication options match the driver's terminology. + Note that `consistency_level` is currently accepted but not applied to write statements; to + change the write consistency level, adjust the driver default in your cluster/client setup. - `batch_type = "UNLOGGED"` is the typical default. Use `LOGGED` for atomic batches when correctness outweighs throughput, or `COUNTER` for counter tables. @@ -200,7 +200,6 @@ sink { keyspace = "test" table = "orders" fields = ["id", "order_id", "customer", "amount", "is_deleted"] - consistency_level = "LOCAL_QUORUM" batch_size = 2000 batch_type = "UNLOGGED" async_write = true diff --git a/docs/en/connectors/sink/Couchbase.md b/docs/en/connectors/sink/Couchbase.md index 57767d6f7c..46e17e5d48 100644 --- a/docs/en/connectors/sink/Couchbase.md +++ b/docs/en/connectors/sink/Couchbase.md @@ -80,7 +80,7 @@ Couchbase stores JSON documents. The connector maps SeaTunnel types to JSON valu | collection | String | Yes | - | Target collection name. | | ready.timeout | Integer | No | `30` | Maximum seconds to wait for the target bucket to become ready during writer initialization. Must be greater than zero. | | primary-key | `List<String>` | No | - | Field names used to build the document key (length-prefixed encoding: `<len>:<value>` components separated by `#`). A random UUID is used when not set. | -| upsert-enable | Boolean | No | `false` | Enable upsert (insert-or-replace) mode. When `false`, duplicate keys will cause an error. | +| upsert-enable | Boolean | No | `false` | Enable upsert (insert-or-replace) mode. When `false`, duplicate keys will cause an error. Requires `primary-key` to be configured; the job fails at startup when `upsert-enable = true` without `primary-key`. | | buffer-flush.max-rows | Integer | No | `1000` | Maximum rows to buffer before a batch write is triggered. Use `-1` to disable. | | retry.max | Integer | No | `3` | Maximum retry attempts on transient write failure. | | retry.interval | Long | No | `1000` | Base milliseconds for linear retry delay. Attempt `n` waits `retry.interval × n` ms. | diff --git a/docs/en/connectors/sink/Enterprise-WeChat.md b/docs/en/connectors/sink/Enterprise-WeChat.md index 4d12717f5e..73d8e3e809 100644 --- a/docs/en/connectors/sink/Enterprise-WeChat.md +++ b/docs/en/connectors/sink/Enterprise-WeChat.md @@ -14,18 +14,18 @@ import ChangeLog from '../changelog/connector-http-wechat.md'; A sink plugin that sends SeaTunnel rows to an Enterprise WeChat robot webhook. The connector identifier in job configuration is `WeChat`. Each row is serialized into a plain-text message with -the fields rendered as `fieldName: fieldValue` lines before the request is sent to the webhook. +the fields rendered as `fieldName: fieldValue`, joined by the literal two-character sequence `\n` +(not real newlines) before the request is sent to the webhook. -The connector extends the [Http sink](Http.md) so it inherits the standard HTTP retry behaviour -(`retry`, `retry_backoff_multiplier_ms`, `retry_backoff_max_ms`) and the generic -`multi_table_sink_replica` option for fan-out writes. +The connector extends the [Http sink](Http.md). It also accepts the generic +`multi_table_sink_replica` option for fan-out writes. Note that `retry`, +`retry_backoff_multiplier_ms` and `retry_backoff_max_ms` are accepted by the option rule but are +not read by the current sink writer, so no retries are performed. -> For example, if the data from upstream is `{"alarmStatus": "firing", "alarmTime": "2022-08-03 01:38:49", "alarmContent": "The disk usage exceeds the threshold"}`, the output content to WeChat Robot is the following: +> For example, if the data from upstream is `{"alarmStatus": "firing", "alarmTime": "2022-08-03 01:38:49", "alarmContent": "The disk usage exceeds the threshold"}`, the text content sent to the WeChat robot is the following (the `\n` sequences below are the literal two characters backslash + `n`, not real newlines): > > ``` -> alarmStatus: firing -> alarmTime: 2022-08-03 01:38:49 -> alarmContent: The disk usage exceeds the threshold +> alarmStatus: firing\nalarmTime: 2022-08-03 01:38:49\nalarmContent: The disk usage exceeds the threshold > ``` ## Key features @@ -36,8 +36,8 @@ The connector extends the [Http sink](Http.md) so it inherits the standard HTTP ## Data Type Mapping The connector renders every upstream row as one plain-text message. Each field is converted to its -string representation (`String.valueOf(value)`) and emitted on its own line together with the -field name, so no per-type JSON structure exists on the wire. +string representation (`String.valueOf(value)`) and joined with its field name using the literal +sequence `\n`, so no per-type JSON structure exists on the wire. | SeaTunnel Data Type | Enterprise WeChat Message Field | |---------------------|---------------------------------| @@ -55,9 +55,9 @@ field name, so no per-type JSON structure exists on the wire. | url | String | Yes | - | Enterprise WeChat robot webhook URL, format `https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=XXXXXX`. | | mentioned_list | array | No | - | User IDs to mention in the group. Use `@all` to mention everyone. | | mentioned_mobile_list | array | No | - | Mobile phone numbers to mention in the group. Use `@all` to mention everyone. | -| retry | int | No | - | Maximum retry times when the HTTP request throws `IOException`. | -| retry_backoff_multiplier_ms | int | No | 100 | Retry backoff multiplier in milliseconds. | -| retry_backoff_max_ms | int | No | 10000 | Maximum retry backoff in milliseconds. | +| retry | int | No | - | Accepted by the option rule, but the current sink writer never reads this option, so no retries are performed. | +| retry_backoff_multiplier_ms | int | No | 100 | Accepted by the option rule, but not applied by the current sink writer. | +| retry_backoff_max_ms | int | No | 10000 | Accepted by the option rule, but not applied by the current sink writer. | | multi_table_sink_replica | int | No | 1 | Number of writer replicas used when writing multiple tables. | | common-options | | no | - | Sink plugin common parameters, please refer to [Sink Common Options](../common-options/sink-common-options.md) for details. | @@ -78,19 +78,16 @@ Mobile phone numbers to mention in the group. Use `@all` to mention everyone. ### retry [int] -Maximum retry times when the HTTP request throws `IOException`. There is no retry by default. The -retry loop uses `retry_backoff_multiplier_ms` and `retry_backoff_max_ms` to compute the wait -between attempts. +Accepted by the option rule, but the current sink writer never reads this option, so no retries are +performed regardless of the configured value. ### retry_backoff_multiplier_ms [int] -Base unit (in milliseconds) for the retry backoff. The wait between attempts grows across retries -up to `retry_backoff_max_ms`. The growth curve is not a fixed multiplier per attempt — see -`HttpClientProvider` (`connector-http-base`) for the exact Fibonacci-based strategy. Default is `100`. +Accepted by the option rule, but not applied by the current sink writer. ### retry_backoff_max_ms [int] -Maximum wait between retries, in milliseconds. Default is `10000`. +Accepted by the option rule, but not applied by the current sink writer. ### multi_table_sink_replica [int] diff --git a/docs/en/connectors/sink/Feishu.md b/docs/en/connectors/sink/Feishu.md index d5d420f7ba..22b54baebc 100644 --- a/docs/en/connectors/sink/Feishu.md +++ b/docs/en/connectors/sink/Feishu.md @@ -59,9 +59,9 @@ Feishu webhook URLs and custom authentication headers are sensitive. Do not prin | url | String | Yes | - | Feishu webhook URL. Must not be empty or whitespace-only. The current sink writer sends requests to this fixed URL and does not replace table-name placeholders. | | headers | Map | No | - | HTTP request headers. Use it when the webhook gateway requires extra headers. | | params | Map | No | - | Accepted by the option rule, but the current sink writer does not pass it to requests. Put non-sensitive query parameters directly in `url` when needed. | -| retry | Int | No | - | The maximum retry times when the HTTP request fails with an `IOException`. | -| retry_backoff_multiplier_ms | Int | No | 100 | Retry backoff multiplier in milliseconds. | -| retry_backoff_max_ms | Int | No | 10000 | Maximum retry backoff in milliseconds. | +| retry | Int | No | - | Accepted by the option rule, but the current sink writer never reads this option, so no retries are performed. | +| retry_backoff_multiplier_ms | Int | No | 100 | Accepted by the option rule, but not applied by the current sink writer. | +| retry_backoff_max_ms | Int | No | 10000 | Accepted by the option rule, but not applied by the current sink writer. | | array_mode | Boolean | No | false | Send rows as a JSON array when true, or as one JSON object per request when false. | | batch_size | Int | No | 1 | The maximum number of rows sent in one request. Only works when `array_mode` is true. | | request_interval_ms | Int | No | 0 | Interval in milliseconds between two HTTP requests, used to avoid sending requests too frequently. | @@ -74,7 +74,7 @@ Feishu webhook URLs and custom authentication headers are sensitive. Do not prin - If the webhook URL needs query parameters, include non-sensitive parameters directly in `url`. Prefer `headers` for authentication material when the gateway supports it, because the full URL, including its query string, may appear in logs or job metadata. - Multi-table jobs can use `multi_table_sink_replica`, but the Feishu sink sends every row to the configured fixed `url`. It does not replace `${database_name}`, `${schema_name}`, or `${table_name}` in the URL. - `array_mode` is useful when the receiver accepts a JSON array and you want fewer HTTP requests. -- Feishu webhook delivery is not exactly-once. If a retry happens after the remote service has already handled a request, the receiver may see duplicate messages. +- Feishu webhook delivery is not exactly-once. If a request is retried by an external gateway after the remote service has already handled it, the receiver may see duplicate messages. Note that the `retry` options of this sink are accepted but not applied by the current sink writer. ## Task Example @@ -111,7 +111,7 @@ sink { } ``` -### With headers and retry +### With custom headers ```hocon Feishu { @@ -119,9 +119,6 @@ Feishu { headers { Content-Type = "application/json" } - retry = 3 - retry_backoff_multiplier_ms = 200 - retry_backoff_max_ms = 5000 } ``` @@ -181,9 +178,6 @@ sink { array_mode = true batch_size = 20 request_interval_ms = 1000 - retry = 5 - retry_backoff_multiplier_ms = 200 - retry_backoff_max_ms = 10000 } } ``` diff --git a/docs/en/connectors/sink/IoTDB.md b/docs/en/connectors/sink/IoTDB.md index 8031581613..d675693910 100644 --- a/docs/en/connectors/sink/IoTDB.md +++ b/docs/en/connectors/sink/IoTDB.md @@ -53,7 +53,7 @@ Used to write data to IoTDB. | key_timestamp | String | No | processing time | Specify field-name of the IoTDB timestamp in SeaTunnelRow. If not specified, use processing-time as timestamp | | key_measurement_fields | Array | No | exclude device and timestamp fields | Specify field names of the IoTDB measurement list in SeaTunnelRow. If not specified, include all fields except `key_device` and `key_timestamp` fields. | | storage_group | String | No | - | Specify device storage group(path prefix) <br/> example: deviceId = \${storage_group} + "." + \${key_device} | -| batch_size | Integer | No | 1024 | For batch writing, data is flushed into IoTDB when the buffered row count reaches `batch_size`. | +| batch_size | Integer | No | 1024 (see note) | For batch writing, data is flushed into IoTDB when the buffered row count reaches `batch_size`. NOTE: the built-in default is currently NOT applied when the option is omitted — if unset, rows are only flushed on checkpoint/close. Set `batch_size` explicitly to enable size-based flushing. | | max_retries | Integer | No | - | The number of retries to flush failed | | retry_backoff_multiplier_ms | Integer | No | - | Using as a multiplier for generating the next delay for backoff | | max_retry_backoff_ms | Integer | No | - | The amount of time to wait before attempting to retry a request to `IoTDB` | @@ -207,7 +207,7 @@ IoTDB> SELECT * FROM root.test_group.* align by device; ### Case4: Streaming writes with explicit batch flush -For long-running streaming jobs, increase `batch_size` to reduce per-row RPC overhead. The connector flushes the buffered rows when either the buffer fills up to `batch_size` or the checkpoint completes. Set `max_retries` and `max_retry_backoff_ms` to keep the job resilient against transient RPC failures. +For long-running streaming jobs, increase `batch_size` to reduce per-row RPC overhead. When `batch_size` is set explicitly, the connector flushes the buffered rows when either the buffer fills up to `batch_size` or the checkpoint completes; if `batch_size` is omitted, rows are only flushed on checkpoint/close. Set `max_retries` and `max_retry_backoff_ms` to keep the job resilient against transient RPC failures. ```hocon env { diff --git a/docs/en/connectors/sink/Lance.md b/docs/en/connectors/sink/Lance.md index 681cc3322f..203d673b97 100644 --- a/docs/en/connectors/sink/Lance.md +++ b/docs/en/connectors/sink/Lance.md @@ -139,7 +139,7 @@ Lance uses the Apache Arrow type system. The sink creates the Lance schema from | SMALLINT | int32 | | INT | int32 | | BIGINT | int64 | -| FLOAT | float32 | +| FLOAT | float64 (values are widened to double) | | DOUBLE | float64 | | DECIMAL | decimal128 | | NULL | null | diff --git a/docs/en/connectors/source/Github.md b/docs/en/connectors/source/Github.md index 21510047c3..18e6afab55 100644 --- a/docs/en/connectors/source/Github.md +++ b/docs/en/connectors/source/Github.md @@ -33,9 +33,9 @@ The Github source connector reads data from the GitHub REST API. It is built on | json_field | Config | No | - | | content_field | String | No | - | | pageing | Config | No | - | -| page_type | String | No | PageNumber | -| cursor_field | String | No | - | -| cursor_response_field | String | No | - | +| pageing.page_type | String | No | PageNumber | +| pageing.cursor_field | String | No | - | +| pageing.cursor_response_field | String | No | - | | poll_interval_millis | int | No | - | | retry | int | No | - | | retry_backoff_multiplier_ms | int | No | 100 | @@ -43,10 +43,10 @@ The Github source connector reads data from the GitHub REST API. It is built on | enable_multi_lines | boolean | No | false | | keep_params_as_form | boolean | No | false | | keep_page_param_as_http_param | boolean | No | false | -| batch_size | int | No | 100 | -| start_page_number | long | No | 1 | -| total_page_size | long | No | 0 | -| use_placeholder_replacement | boolean | No | false | +| pageing.batch_size | int | No | 100 | +| pageing.start_page_number | long | No | 1 | +| pageing.total_page_size | long | No | 0 | +| pageing.use_placeholder_replacement | boolean | No | false | | connect_timeout_ms | int | No | 12000 | | socket_timeout_ms | int | No | 60000 | | json_filed_missed_return_null | boolean | No | false | @@ -94,19 +94,19 @@ JSONPath expression used to select a JSON fragment before schema parsing, for ex ### pageing [Config] -Pagination settings inherited from the HTTP connector. Keep the option name `pageing` in job configs. Configure `page_type = "Cursor"` for cursor-based GitHub pagination when needed. +Pagination settings inherited from the HTTP connector option rule. **These options are not applied by this connector**: `GithubSource` builds its reader without a pagination context, so the source always reads a single page regardless of `pageing` settings, and the job still succeeds. Do not rely on `pageing` here; passing more data requires connector support to be added. -### page_type [String] +### pageing.page_type [String] -Pagination type. Supported values are `PageNumber` (default) and `Cursor`. Use `Cursor` for endpoints that return a `next` cursor in the response. +Pagination type. Supported values are `PageNumber` (default) and `Cursor`. These options describe how pagination *would* be configured, but as noted above they are **not applied by this connector**, so the source reads a single page regardless of this setting. -### cursor_field [String] +### pageing.cursor_field [String] -The request parameter name that carries the cursor value. Used together with `page_type = "Cursor"`. +The request parameter name that carries the cursor value. Used together with `pageing.page_type = "Cursor"`. -### cursor_response_field [String] +### pageing.cursor_response_field [String] -The JSONPath of the cursor in the response body. Used together with `page_type = "Cursor"`. +The JSONPath of the cursor in the response body. Used together with `pageing.page_type = "Cursor"`. ### poll_interval_millis [int] @@ -136,19 +136,19 @@ When `true`, request parameters are sent as form-encoded body parameters instead When `true`, the page parameter remains in the request URL when paginating instead of being replaced inside the body. -### batch_size [int] +### pageing.batch_size [int] The number of records returned per page request when the total number of pages is unknown. -### start_page_number [long] +### pageing.start_page_number [long] Which page number to start synchronizing from. -### total_page_size [long] +### pageing.total_page_size [long] -Total page size to read. `0` means use `batch_size` until the API stops returning new pages. +Total page size to read. `0` means use `pageing.batch_size` until the API stops returning new pages. -### use_placeholder_replacement [boolean] +### pageing.use_placeholder_replacement [boolean] When `true`, use `${field}` placeholder replacement for headers, parameters and body values; otherwise use key-based replacement. @@ -174,8 +174,7 @@ Source plugin common parameters. See [Source Common Options](../common-options/s - The connector always adds an `Authorization: Bearer <access_token>` header from `access_token`. Put other custom headers in `headers`. - Set `format = "json"` and define `schema` when you want typed SeaTunnel rows. - Use `content_field` when the GitHub response wraps records in a nested array such as `$.items[*]`. -- For traditional page-number pagination, keep `page_type = "PageNumber"` and use `params` with `page` / `per_page`. -- For cursor-based endpoints (such as the GitHub Events API), set `page_type = "Cursor"` and configure `cursor_field` / `cursor_response_field`. +- This connector currently reads a single page only: the `pageing` options are accepted by the option rule but not applied by the reader (see the `pageing` section). Do not configure `params` with `page` / `per_page` expecting the connector to walk pages. ## Task Examples @@ -212,7 +211,7 @@ sink { } ``` -### Read Paged GitHub API Results +### Read GitHub API Results (Single Page) ```hocon env { @@ -227,13 +226,7 @@ source { method = "GET" params = { per_page = "100" - page = "${page}" - } - pageing = { - page_field = "page" - total_page_size = 5 - start_page_number = 1 - use_placeholder_replacement = true + page = "1" } format = "json" schema = { diff --git a/docs/en/connectors/source/Gitlab.md b/docs/en/connectors/source/Gitlab.md index 53dbcc9c37..0bdff99c0c 100644 --- a/docs/en/connectors/source/Gitlab.md +++ b/docs/en/connectors/source/Gitlab.md @@ -33,9 +33,9 @@ The Gitlab source connector reads data from the GitLab REST API. It is built on | json_field | Config | No | - | | content_field | String | No | - | | pageing | Config | No | - | -| page_type | String | No | PageNumber | -| cursor_field | String | No | - | -| cursor_response_field | String | No | - | +| pageing.page_type | String | No | PageNumber | +| pageing.cursor_field | String | No | - | +| pageing.cursor_response_field | String | No | - | | poll_interval_millis | int | No | - | | retry | int | No | - | | retry_backoff_multiplier_ms | int | No | 100 | @@ -43,10 +43,10 @@ The Gitlab source connector reads data from the GitLab REST API. It is built on | enable_multi_lines | boolean | No | false | | keep_params_as_form | boolean | No | false | | keep_page_param_as_http_param | boolean | No | false | -| batch_size | int | No | 100 | -| start_page_number | long | No | 1 | -| total_page_size | long | No | 0 | -| use_placeholder_replacement | boolean | No | false | +| pageing.batch_size | int | No | 100 | +| pageing.start_page_number | long | No | 1 | +| pageing.total_page_size | long | No | 0 | +| pageing.use_placeholder_replacement | boolean | No | false | | connect_timeout_ms | int | No | 12000 | | socket_timeout_ms | int | No | 60000 | | json_filed_missed_return_null | boolean | No | false | @@ -94,19 +94,19 @@ JSONPath expression used to select a JSON fragment before schema parsing, for ex ### pageing [Config] -Pagination settings inherited from the HTTP connector. Keep the option name `pageing` in job configs. +Pagination settings inherited from the HTTP connector option rule. **These options are not applied by this connector**: `GitlabSource` builds its reader without a pagination context, so the source always reads a single page regardless of `pageing` settings, and the job still succeeds. Do not rely on `pageing` here; passing more data requires connector support to be added. -### page_type [String] +### pageing.page_type [String] -Pagination type. Supported values are `PageNumber` (default) and `Cursor`. Use `Cursor` for GitLab endpoints that return a `X-Next-Page` cursor header. +Pagination type. Supported values are `PageNumber` (default) and `Cursor`. These options describe how pagination *would* be configured, but as noted above they are **not applied by this connector**, so the source reads a single page regardless of this setting. -### cursor_field [String] +### pageing.cursor_field [String] -The request parameter name that carries the cursor value. Used together with `page_type = "Cursor"`. +The request parameter name that carries the cursor value. Used together with `pageing.page_type = "Cursor"`. -### cursor_response_field [String] +### pageing.cursor_response_field [String] -The JSONPath of the cursor in the response body. Used together with `page_type = "Cursor"`. +The JSONPath of the cursor in the response body. Used together with `pageing.page_type = "Cursor"`. ### poll_interval_millis [int] @@ -136,19 +136,19 @@ When `true`, request parameters are sent as form-encoded body parameters instead When `true`, the page parameter remains in the request URL when paginating instead of being replaced inside the body. -### batch_size [int] +### pageing.batch_size [int] The number of records returned per page request when the total number of pages is unknown. -### start_page_number [long] +### pageing.start_page_number [long] Which page number to start synchronizing from. -### total_page_size [long] +### pageing.total_page_size [long] -Total page size to read. `0` means use `batch_size` until the API stops returning new pages. +Total page size to read. `0` means use `pageing.batch_size` until the API stops returning new pages. -### use_placeholder_replacement [boolean] +### pageing.use_placeholder_replacement [boolean] When `true`, use `${field}` placeholder replacement for headers, parameters and body values; otherwise use key-based replacement. @@ -174,7 +174,7 @@ Source plugin common parameters. See [Source Common Options](../common-options/s - The connector always adds a `PRIVATE-TOKEN` header from `access_token`. Put other custom headers in `headers`. - Set `format = "json"` and define `schema` when you want typed SeaTunnel rows. - Use `content_field` when the GitLab response wraps records in a nested array such as `$.items[*]`. -- For traditional page-number pagination, keep `page_type = "PageNumber"` and use `params` with `page` / `per_page`. +- This connector currently reads a single page only: the `pageing` options are accepted by the option rule but not applied by the reader (see the `pageing` section). Do not configure `params` with `page` / `per_page` expecting the connector to walk pages. - The GitLab source supports batch mode only; `poll_interval_millis` does not enable streaming behavior. ## Task Examples @@ -212,7 +212,7 @@ sink { } ``` -### Read Paged GitLab API Results +### Read GitLab API Results (Single Page) ```hocon env { @@ -227,13 +227,7 @@ source { method = "GET" params = { per_page = "100" - page = "${page}" - } - pageing = { - page_field = "page" - total_page_size = 5 - start_page_number = 1 - use_placeholder_replacement = true + page = "1" } format = "json" schema = { diff --git a/docs/en/connectors/source/MongoDB-CDC.md b/docs/en/connectors/source/MongoDB-CDC.md index a891e47049..f0efa8e3d4 100644 --- a/docs/en/connectors/source/MongoDB-CDC.md +++ b/docs/en/connectors/source/MongoDB-CDC.md @@ -112,6 +112,11 @@ For specific types in MongoDB, we use Extended JSON format to map them to Seatun ## Source Options +> **Note:** MongoDB-CDC shares the CDC base option rule, so a few JDBC-oriented snapshot options +> (`snapshot.split.size`, `snapshot.fetch.size`, `enable_concurrent_read`) are accepted by +> validation but are not used by MongoDB-CDC. Snapshot chunking is controlled by +> `incremental.snapshot.chunk.size.mb` instead. + | Name | Type | Required | Default | Description | |------------------------------------|---------|----------|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | hosts | String | Yes | - | The comma-separated list of hostname and port pairs of the MongoDB servers, or a standard MongoDB connection URI using `mongodb://` or `mongodb+srv://`. For example: `localhost:27017,localhost:27018` or `mongodb+srv://cluster.example.net`. | diff --git a/docs/en/connectors/source/MySQL-CDC.md b/docs/en/connectors/source/MySQL-CDC.md index b7307cc616..388f2ccf84 100644 --- a/docs/en/connectors/source/MySQL-CDC.md +++ b/docs/en/connectors/source/MySQL-CDC.md @@ -208,7 +208,7 @@ When an initial consistent snapshot is made for large databases, your establishe | snapshot.fetch.size | Integer | No | 1024 | The maximum fetch size for per poll when read table snapshot. [...] | incremental.parallelism | Integer | No | 1 | The number of parallel readers in the incremental (binlog) phase. [...] | server-id | String | No | - | Numeric ID or numeric ID range used by this CDC reader, for example `5400` or `5400-5408`. Each ID must be unique in the MySQL cluster. When the job has multiple readers or reads multiple tables in parallel, configure an ID range large enough for the job. If this option is omitted, SeaTunnel generates a random ID, but an explicit value is recommended for production. [...] -| server-time-zone | String | No | - | The session time zone in database server. If not set, then ZoneId.systemDefault() is used to determine the server time zone. [...] +| server-time-zone | String | No | JVM system time zone | The session time zone in database server. If not set, then ZoneId.systemDefault() is used to determine the server time zone. [...] | connect.timeout.ms | Long | No | 30000 | The maximum time that the connector should wait after trying to connect to the database server before timing out. [...] | connect.max-retries | Integer | No | 3 | The max retry times that the connector should retry to build database server connection. [...] | connection.pool.size | Integer | No | 20 | The jdbc connection pool size. [...] @@ -515,6 +515,8 @@ source { Without a usable primary key (configured or physical) the connector cannot safely apply UPDATE/DELETE events. Use this mode only for append-only workloads or when downstream sink behavior does not depend on row identity. +The example above demonstrates the "logical primary key" scenario: the source table has no physical primary key, but a column is declared as a stable row identity via `table-names-config.primaryKeys`, and `exactly_once = true` makes both the snapshot phase and the binlog phase use the same logical primary key. UPDATE/DELETE events are routed correctly only if the declared column is truly unique in the source data; if the source data contains duplicate values, the behavior is no longer reliable. + NULL values in a nullable unique key column, or in a nullable column configured in `table-names-config.primaryKeys`, are emitted as NULL in the snapshot and binlog phases. A NULL cannot identify a row, so choose a NOT NULL column as the key when UPDATE/DELETE events must be applied by key. With `exactly_once = true`, rows whose key is NULL cannot be told apart either, and only one of them is kept per snapshot split. ### Start From a Specific Binlog Offset diff --git a/docs/en/connectors/source/PostgreSQL-CDC.md b/docs/en/connectors/source/PostgreSQL-CDC.md index d88b2bfede..18d5886066 100644 --- a/docs/en/connectors/source/PostgreSQL-CDC.md +++ b/docs/en/connectors/source/PostgreSQL-CDC.md @@ -110,9 +110,10 @@ ALTER TABLE your_table_name REPLICA IDENTITY FULL; | stop.mode | Enum | No | NEVER | Optional stop mode for PostgreSQL CDC consumer. The only valid enumeration is `never`: the source keeps streaming WAL changes and never stops on its own once it reaches the incremental phase. [...] | snapshot.split.size | Integer | No | 8096 | The split size (number of rows) of table snapshot, captured tables are split into multiple splits when read the snapshot of table. [...] | snapshot.fetch.size | Integer | No | 1024 | The maximum fetch size for per poll when read table snapshot. [...] +| incremental.parallelism | Integer | No | 1 | The number of parallel readers in the incremental (WAL) phase. [...] | slot.name | String | No | seatunnel | The PostgreSQL logical decoding slot name. Use a different slot name for each CDC job that reads from the same PostgreSQL instance. [...] | decoding.plugin.name | String | No | pgoutput | The name of the Postgres logical decoding plug-in installed on the server,Supported values are decoderbufs, wal2json, wal2json_rds, wal2json_streaming,wal2json_rds_streaming and pgoutput. [...] -| server-time-zone | String | No | - | The session time zone in database server. If not set, then ZoneId.systemDefault() is used to determine the server time zone. [...] +| server-time-zone | String | No | JVM system time zone | The session time zone in database server. If not set, then ZoneId.systemDefault() is used to determine the server time zone. [...] | connect.timeout.ms | Long | No | 30000 | The maximum time that the connector should wait after trying to connect to the database server before timing out. [...] | connect.max-retries | Integer | No | 3 | The max retry times that the connector should retry to build database server connection. [...] | connection.pool.size | Integer | No | 20 | The jdbc connection pool size. [...] @@ -376,7 +377,7 @@ Also set `wal_level = logical` in `postgresql.conf` and add an entry in `pg_hba. ### Which logical decoding plugins are supported? -SeaTunnel PostgreSQL CDC supports `pgoutput` (built-in since PostgreSQL 10), `wal2json`, and `decoderbufs`. The default is `pgoutput`. Use the `decoding.plugin.name` parameter to select the plugin. +The `decoding.plugin.name` option accepts `pgoutput` (default, built-in since PostgreSQL 10), `decoderbufs`, `wal2json`, `wal2json_rds`, `wal2json_streaming`, and `wal2json_rds_streaming`. Note that schema evolution (`schema-changes.enabled`) requires `pgoutput`. ### Can SeaTunnel read CDC from a PostgreSQL standby? diff --git a/docs/en/connectors/source/Shopify.md b/docs/en/connectors/source/Shopify.md index f6e1c202b4..f3bef109e8 100644 --- a/docs/en/connectors/source/Shopify.md +++ b/docs/en/connectors/source/Shopify.md @@ -65,7 +65,7 @@ The structure of the data, including field names and field types. For more detai ### format [String] -the format of upstream data, now only support `json` `text`, default `text`. +the format of upstream data, supports `json` and `text` (the underlying HTTP source also accepts `binary`), default `text`. ### params [Map] diff --git a/docs/en/connectors/source/Zendesk.md b/docs/en/connectors/source/Zendesk.md index 657a76ae7a..74a943964e 100644 --- a/docs/en/connectors/source/Zendesk.md +++ b/docs/en/connectors/source/Zendesk.md @@ -37,7 +37,7 @@ options. The differences are the required authentication options (`email` and `a | api_token | String | Yes | - | The Zendesk API token. See the [Zendesk API token docs](https://support.zendesk.com/hc/en-us/articles/4408889192858) for how to generate one. | | method | String | No | get | HTTP request method. Only `GET` and `POST` are supported. | | schema | Config | No | - | The structure of the data, including field names and field types. For more details, please refer to [Schema Feature](../../introduction/concepts/schema-feature.md). | -| format | String | No | text | The format of upstream data, only `json` and `text` are supported. | +| format | String | No | text | The format of upstream data, supports `json` and `text` (the underlying HTTP source also accepts `binary`). | | params | Map | No | - | Query parameters appended to the request URL. | | body | String | No | - | Request body sent for `POST` (or any method that accepts a body). When `format = "json"`, the body must be valid JSON. | | json_field | Config | No | - | Maps JSON paths in the response to schema fields. Must be used together with `schema`. See the [Http source](./Http.md) connector for details and examples. | @@ -76,9 +76,10 @@ The structure of the data, including field names and field types. For more detai ### format [String] -The format of upstream data. Only `json` and `text` are supported; default is `text`. Zendesk -endpoints always return JSON, so set `format = "json"` together with `content_field` to extract -the result array before mapping to rows. +The format of upstream data, supports `json` and `text`; default is `text`. Like the generic +[Http source](Http.md), the underlying connector also accepts `binary`, but Zendesk endpoints +always return JSON, so set `format = "json"` together with `content_field` to extract the result +array before mapping to rows. ### params [Map] diff --git a/docs/zh/connectors/sink/Cassandra.md b/docs/zh/connectors/sink/Cassandra.md index 4f2075621f..d76a59f24a 100644 --- a/docs/zh/connectors/sink/Cassandra.md +++ b/docs/zh/connectors/sink/Cassandra.md @@ -39,7 +39,7 @@ sink 会把数据写入已经存在的 Cassandra 表。如果不配置 `fields` | username | String | 否 | - | Cassandra 用户名,需要和 `password` 一起配置。 | | password | String | 否 | - | Cassandra 密码,需要和 `username` 一起配置。 | | datacenter | String | 否 | datacenter1 | Cassandra Java Driver 使用的本地数据中心名称。 | -| consistency_level | String | 否 | LOCAL_ONE | 写入一致性级别,例如 `LOCAL_ONE`、`ONE`、`QUORUM`、`LOCAL_QUORUM`。 | +| consistency_level | String | 否 | LOCAL_ONE | 该选项会被解析和校验,但当前 Sink 不会把它应用到写入语句;写入一致性由 Driver 默认值决定。 | | fields | Array | 否 | - | 要写入的目标字段。不配置时写入目标表的全部字段。 | | batch_size | int | 否 | 5000 | 每次 flush 前最多缓存的行数。 | | batch_type | String | 否 | UNLOGGED | Cassandra batch 类型,常用值包括 `LOGGED`、`UNLOGGED`、`COUNTER`。 | @@ -75,7 +75,8 @@ sink 会把数据写入已经存在的 Cassandra 表。如果不配置 `fields` ### consistency_level [String] -`Cassandra` 写入一致性级别, 默认为 `LOCAL_ONE`. +该选项会被解析和校验(默认 `LOCAL_ONE`),但当前 Sink 写入器不会把它应用到写入语句;写入一致性 +使用 Driver 默认级别。 ### fields [array] @@ -106,9 +107,9 @@ Sink 插件通用参数,详情请参考 [Sink 常用选项](../common-options/ - 任务启动前,目标 keyspace 和表必须已经存在。 - `fields` 适合上游数据有额外字段、只想写入其中一部分字段的场景;它不是建表配置。 -- `async_write = true` 可以提升吞吐,`batch_size` 用来控制每次 flush 前聚合的行数。 -- 连接器底层使用 Cassandra Java Driver;认证和一致性相关参数与 Driver 保持一致,需要更强一致性时请结合 - 集群的副本策略设置 `consistency_level`。 +- `async_write = true`(默认)下每行都会作为独立的异步语句发送,`batch_size` 只控制 flush 频率;`async_write = false` 时,行才会按 `batch_type` 聚合成一个批次,每次 flush 最多包含 `batch_size` 条。 +- 连接器底层使用 Cassandra Java Driver;认证相关参数与 Driver 保持一致。注意:`consistency_level` 当前 + 仅被接受、不会应用到写入语句;如需调整写入一致性级别,请调整集群/客户端的 Driver 默认配置。 - `batch_type = "UNLOGGED"` 是常见默认值;当正确性优先于吞吐时使用 `LOGGED`,counter 表使用 `COUNTER`。 ## 任务示例 @@ -192,7 +193,6 @@ sink { keyspace = "test" table = "orders" fields = ["id", "order_id", "customer", "amount", "is_deleted"] - consistency_level = "LOCAL_QUORUM" batch_size = 2000 batch_type = "UNLOGGED" async_write = true diff --git a/docs/zh/connectors/sink/Couchbase.md b/docs/zh/connectors/sink/Couchbase.md index b19e806395..d8659a88bb 100644 --- a/docs/zh/connectors/sink/Couchbase.md +++ b/docs/zh/connectors/sink/Couchbase.md @@ -75,7 +75,7 @@ sh bin/install-plugin.sh ${version} | collection | String | 是 | - | 目标 Collection 名称。 | | ready.timeout | Integer | 否 | `30` | 写入器初始化时等待目标 bucket 就绪的最长时间(秒),必须大于零。 | | primary-key | `List<String>` | 否 | - | 用于构建文档键的字段名列表(长度前缀编码:`<长度>:<值>` 分量以 `#` 分隔)。未设置时使用随机 UUID。 | -| upsert-enable | Boolean | 否 | `false` | 是否启用 Upsert(插入或替换)模式。为 `false` 时,重复键将报错。 | +| upsert-enable | Boolean | 否 | `false` | 是否启用 Upsert(插入或替换)模式。为 `false` 时,重复键将报错。必须同时配置 `primary-key`,否则作业启动时会直接报错。 | | buffer-flush.max-rows | Integer | 否 | `1000` | 触发批量写入的最大缓冲行数。设为 `-1` 禁用。 | | retry.max | Integer | 否 | `3` | 写入失败时的最大重试次数。 | | retry.interval | Long | 否 | `1000` | 线性退避基础间隔(毫秒)。第 n 次重试等待 `retry.interval × n` 毫秒。 | diff --git a/docs/zh/connectors/sink/Enterprise-WeChat.md b/docs/zh/connectors/sink/Enterprise-WeChat.md index eb1415973f..0b284c15f6 100644 --- a/docs/zh/connectors/sink/Enterprise-WeChat.md +++ b/docs/zh/connectors/sink/Enterprise-WeChat.md @@ -13,18 +13,17 @@ import ChangeLog from '../changelog/connector-http-wechat.md'; ## 描述 一个将 SeaTunnel 行数据发送到企业微信机器人 webhook 的接收器插件。作业配置中的连接器标识符为 -`WeChat`。每一行数据会按 `字段名: 字段值` 的格式序列化为多行纯文本消息,再发送到 webhook。 +`WeChat`。每一行数据会按 `字段名: 字段值` 的格式序列化为纯文本消息,字段之间以字面的两个字符 +`\n`(并非真实换行符)拼接后发送到 webhook。 -该连接器继承自 [Http sink](Http.md),因此自带标准的 HTTP 重试参数(`retry`、 -`retry_backoff_multiplier_ms`、`retry_backoff_max_ms`),并支持通用的 -`multi_table_sink_replica` 选项以调整多表写入的并行度。 +该连接器继承自 [Http sink](Http.md),并支持通用的 `multi_table_sink_replica` 选项。注意: +`retry`、`retry_backoff_multiplier_ms`、`retry_backoff_max_ms` 虽然会通过参数校验,但当前 Sink +写入器不会读取它们,因此不会进行任何重试。 -> 例如,如果上游数据为 `{"alarmStatus": "firing", "alarmTime": "2022-08-03 01:38:49", "alarmContent": "The disk usage exceeds the threshold"}`,企业微信机器人收到的内容如下: +> 例如,如果上游数据为 `{"alarmStatus": "firing", "alarmTime": "2022-08-03 01:38:49", "alarmContent": "The disk usage exceeds the threshold"}`,机器人收到的文本内容如下(其中的 `\n` 是字面的反斜杠 + `n` 两个字符,不是真实换行): > > ``` -> alarmStatus: firing -> alarmTime: 2022-08-03 01:38:49 -> alarmContent: The disk usage exceeds the threshold +> alarmStatus: firing\nalarmTime: 2022-08-03 01:38:49\nalarmContent: The disk usage exceeds the threshold > ``` ## 关键特性 @@ -35,7 +34,7 @@ import ChangeLog from '../changelog/connector-http-wechat.md'; ## 数据类型映射 连接器会把每一行渲染为一条纯文本消息,每个字段都通过 `String.valueOf(value)` 转换为字符串, -并以 `字段名: 字段值` 的格式独立成行。线上消息是纯文本,不存在按类型区分的 JSON 结构。 +并以字面 `\n` 序列把 `字段名: 字段值` 拼接在一起。线上消息是纯文本,不存在按类型区分的 JSON 结构。 | SeaTunnel 数据类型 | 企业微信消息字段 | |--------------------|-----------------| @@ -53,9 +52,9 @@ import ChangeLog from '../changelog/connector-http-wechat.md'; | url | String | 是 | - | 企业微信机器人 webhook URL,格式 `https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=XXXXXX`。 | | mentioned_list | array | 否 | - | 需要提醒的用户 ID 列表,使用 `@all` 提醒所有人。 | | mentioned_mobile_list | array | 否 | - | 需要提醒的手机号列表,使用 `@all` 提醒所有人。 | -| retry | int | 否 | - | HTTP 请求抛出 `IOException` 时的最大重试次数。默认不重试。 | -| retry_backoff_multiplier_ms | int | 否 | 100 | 重试退避倍数,单位毫秒。 | -| retry_backoff_max_ms | int | 否 | 10000 | 最大重试退避时间,单位毫秒。 | +| retry | int | 否 | - | 该参数会通过参数校验,但当前 Sink 写入器不会读取它,因此不会进行任何重试。 | +| retry_backoff_multiplier_ms | int | 否 | 100 | 该参数会通过参数校验,但当前 Sink 写入器不会应用它。 | +| retry_backoff_max_ms | int | 否 | 10000 | 该参数会通过参数校验,但当前 Sink 写入器不会应用它。 | | multi_table_sink_replica | int | 否 | 1 | 多表写入时使用的写入器副本数。 | | common-options | | 否 | - | 接收器插件通用参数,详见 [Sink 通用选项](../common-options/sink-common-options.md)。 | @@ -75,18 +74,15 @@ import ChangeLog from '../changelog/connector-http-wechat.md'; ### retry [int] -HTTP 请求抛出 `IOException` 时的最大重试次数,默认不重试。重试间隔由 -`retry_backoff_multiplier_ms` 和 `retry_backoff_max_ms` 共同决定。 +该参数会通过参数校验,但当前 Sink 写入器不会读取它,无论配置为何值都不会进行重试。 ### retry_backoff_multiplier_ms [int] -重试退避的基础单位,单位毫秒。重试之间的等待时间会在多次重试中逐渐增长,上限为 -`retry_backoff_max_ms`。增长曲线并不是每次固定的倍数关系,具体的斐波那契策略请参考 -`HttpClientProvider`(位于 `connector-http-base`)。默认 `100`。 +该参数会通过参数校验,但当前 Sink 写入器不会应用它。 ### retry_backoff_max_ms [int] -最大重试退避时间,单位毫秒。默认 `10000`。 +该参数会通过参数校验,但当前 Sink 写入器不会应用它。 ### multi_table_sink_replica [int] diff --git a/docs/zh/connectors/sink/Feishu.md b/docs/zh/connectors/sink/Feishu.md index 27a151bedf..6e4b01b926 100644 --- a/docs/zh/connectors/sink/Feishu.md +++ b/docs/zh/connectors/sink/Feishu.md @@ -59,9 +59,9 @@ import ChangeLog from '../changelog/connector-http-feishu.md'; | url | String | 是 | - | 飞书 Webhook URL,不能为空或仅包含空白字符。当前 Sink 写入器会向这个固定 URL 发送请求,不会替换表名相关占位符。 | | headers | Map | 否 | - | HTTP 请求头。Webhook 网关需要额外请求头时使用。 | | params | Map | 否 | - | 该参数会通过参数校验,但当前 Sink 写入器不会把它传入请求。如需查询参数,请把非敏感参数直接写在 `url` 中。 | -| retry | Int | 否 | - | HTTP 请求发生 `IOException` 时的最大重试次数。 | -| retry_backoff_multiplier_ms | Int | 否 | 100 | 重试退避时间倍数,单位毫秒。 | -| retry_backoff_max_ms | Int | 否 | 10000 | 最大重试退避时间,单位毫秒。 | +| retry | Int | 否 | - | 该参数会通过参数校验,但当前 Sink 写入器不会读取它,因此不会进行任何重试。 | +| retry_backoff_multiplier_ms | Int | 否 | 100 | 该参数会通过参数校验,但当前 Sink 写入器不会应用它。 | +| retry_backoff_max_ms | Int | 否 | 10000 | 该参数会通过参数校验,但当前 Sink 写入器不会应用它。 | | array_mode | Boolean | 否 | false | 为 true 时按 JSON 数组发送多条数据;为 false 时每次请求发送一个 JSON 对象。 | | batch_size | Int | 否 | 1 | 单次请求最多发送的数据条数,仅在 `array_mode` 为 true 时生效。 | | request_interval_ms | Int | 否 | 0 | 两次 HTTP 请求之间的间隔毫秒数,用于避免请求过于频繁。 | @@ -74,7 +74,7 @@ import ChangeLog from '../changelog/connector-http-feishu.md'; - 如果 Webhook URL 需要查询参数,请把非敏感参数直接写在 `url` 中。鉴权类信息建议优先放在 `headers` 里,前提是网关支持这种方式,因为完整 URL(包括查询参数)可能出现在日志或作业元数据中。 - 多表作业可以使用 `multi_table_sink_replica`,但飞书 Sink 会把所有数据发送到配置的固定 `url`,不会替换 URL 中的 `${database_name}`、`${schema_name}` 或 `${table_name}`。 - 当接收端支持 JSON 数组,且希望减少 HTTP 请求次数时,可以启用 `array_mode`。 -- 飞书 Webhook 投递不是精确一次。如果远端已经处理成功但本地收到异常并触发重试,接收端可能看到重复消息。 +- 飞书 Webhook 投递不是精确一次。如果请求被外部网关重发而远端已经处理成功,接收端可能看到重复消息。注意:本 Sink 的 `retry` 相关参数仅会被接受,当前 Sink 写入器不会实际应用。 ## 任务示例 @@ -111,7 +111,7 @@ sink { } ``` -### 配置请求头和重试 +### 配置请求头 ```hocon Feishu { @@ -119,9 +119,6 @@ Feishu { headers { Content-Type = "application/json" } - retry = 3 - retry_backoff_multiplier_ms = 200 - retry_backoff_max_ms = 5000 } ``` @@ -180,9 +177,6 @@ sink { array_mode = true batch_size = 20 request_interval_ms = 1000 - retry = 5 - retry_backoff_multiplier_ms = 200 - retry_backoff_max_ms = 10000 } } ``` diff --git a/docs/zh/connectors/sink/IoTDB.md b/docs/zh/connectors/sink/IoTDB.md index 6f5e9d4e5f..7708e8d213 100644 --- a/docs/zh/connectors/sink/IoTDB.md +++ b/docs/zh/connectors/sink/IoTDB.md @@ -53,7 +53,7 @@ import ChangeLog from '../changelog/connector-iotdb.md'; | key_timestamp | String | 否 | processing time | 在SeaTunnelRow中指定 IoTDB 时间戳的字段名。如果未指定,则使用处理时间作为时间戳 | | key_measurement_fields | Array | 否 | 排除设备和时间字段 | 在 SeaTunnelRow 中指定 IoTDB 测点字段列表。未配置时,会写入除 `key_device` 和 `key_timestamp` 对应字段外的其他字段 | | storage_group | String | 否 | - | 指定设备存储组(路径前缀) <br/> 例如: deviceId = \${storage_group} + "." + \${key_device} | -| batch_size | Integer | 否 | 1024 | 批量写入时,当缓存行数达到 `batch_size`,数据会刷新到 IoTDB 中 | +| batch_size | Integer | 否 | 1024(见说明) | 批量写入时,当缓存行数达到 `batch_size`,数据会刷新到 IoTDB 中。注意:省略该选项时内置默认值实际不会生效——不配置时仅在 checkpoint/关闭时刷新;需要按行数刷新请显式配置 `batch_size`。 | | max_retries | Integer | 否 | - | 写入失败后的最大重试次数 | | retry_backoff_multiplier_ms | Integer | 否 | - | 用作生成下一个退避延迟的乘数 | | max_retry_backoff_ms | Integer | 否 | - | 尝试重试对 IoTDB 的请求之前等待的时间量 | @@ -207,7 +207,7 @@ IoTDB> SELECT * FROM root.test_group.* align by device; ### 案例4:显式批量刷写的流式写入 -对于长时间运行的流式作业,可以调大 `batch_size` 减少单行 RPC 开销。连接器在缓冲行数达到 `batch_size` 或 checkpoint 完成时刷新缓冲。配合 `max_retries` 和 `max_retry_backoff_ms` 可以让任务在 RPC 抖动时保持稳定。 +对于长时间运行的流式作业,可以调大 `batch_size` 减少单行 RPC 开销。显式配置 `batch_size` 时,连接器在缓冲行数达到 `batch_size` 或 checkpoint 完成时刷新缓冲;若未配置 `batch_size`,则仅在 checkpoint/关闭时刷新。配合 `max_retries` 和 `max_retry_backoff_ms` 可以让任务在 RPC 抖动时保持稳定。 ```hocon env { diff --git a/docs/zh/connectors/sink/Lance.md b/docs/zh/connectors/sink/Lance.md index 83ab68cbca..8f58f39a26 100644 --- a/docs/zh/connectors/sink/Lance.md +++ b/docs/zh/connectors/sink/Lance.md @@ -131,7 +131,7 @@ Lance 使用 Apache Arrow 类型系统。sink 会根据上游 SeaTunnel 表结 | SMALLINT | int32 | | INT | int32 | | BIGINT | int64 | -| FLOAT | float32 | +| FLOAT | float64(值会被提升为 double) | | DOUBLE | float64 | | DECIMAL | decimal128 | | NULL | null | diff --git a/docs/zh/connectors/sink/Zendesk.md b/docs/zh/connectors/sink/Zendesk.md index 41417f6e50..39c481aefb 100644 --- a/docs/zh/connectors/sink/Zendesk.md +++ b/docs/zh/connectors/sink/Zendesk.md @@ -34,7 +34,7 @@ import ChangeLog from '../changelog/connector-http-zendesk.md'; | rate_limit_backoff_ms | int | 否 | 30000 | 收到 429(速率限制)响应时的基础退避时间(毫秒)。默认 30000ms。必须 `>= 0`。 | | resource_key | String | 否 | - | Zendesk API 请求体的 JSON 包装键(如 `"ticket"`、`"user"`、`"organization"`)。设置后将直接使用该值,而不从 URL 路径自动推断。适用于自动推断无法正确处理的端点。 | | rate_limit_max_retries | int | 否 | 3 | 收到 429 响应后的最大重试次数。默认 3。必须 `>= 0`。 | -| multi_table_sink_replica | int | 否 | 1 | 多表写入模式下每张表对应的 Sink 写入并发数。 | +| multi_table_sink_replica | int | 否 | 1 | 多表写入模式下每个表写入器的副本数。 | | common-options | | 否 | - | Sink 通用选项。参见 [Sink 通用选项](../common-options/sink-common-options.md)。 | ## 使用说明 diff --git a/docs/zh/connectors/source/Github.md b/docs/zh/connectors/source/Github.md index 3f5196db4b..5ae1482f83 100644 --- a/docs/zh/connectors/source/Github.md +++ b/docs/zh/connectors/source/Github.md @@ -33,9 +33,9 @@ Github 源连接器用于读取 GitHub REST API 数据。它基于 HTTP 源连 | json_field | Config | 否 | - | | content_field | String | 否 | - | | pageing | Config | 否 | - | -| page_type | String | 否 | PageNumber | -| cursor_field | String | 否 | - | -| cursor_response_field | String | 否 | - | +| pageing.page_type | String | 否 | PageNumber | +| pageing.cursor_field | String | 否 | - | +| pageing.cursor_response_field | String | 否 | - | | poll_interval_millis | int | 否 | - | | retry | int | 否 | - | | retry_backoff_multiplier_ms | int | 否 | 100 | @@ -43,10 +43,10 @@ Github 源连接器用于读取 GitHub REST API 数据。它基于 HTTP 源连 | enable_multi_lines | boolean | 否 | false | | keep_params_as_form | boolean | 否 | false | | keep_page_param_as_http_param | boolean | 否 | false | -| batch_size | int | 否 | 100 | -| start_page_number | long | 否 | 1 | -| total_page_size | long | 否 | 0 | -| use_placeholder_replacement | boolean | 否 | false | +| pageing.batch_size | int | 否 | 100 | +| pageing.start_page_number | long | 否 | 1 | +| pageing.total_page_size | long | 否 | 0 | +| pageing.use_placeholder_replacement | boolean | 否 | false | | connect_timeout_ms | int | 否 | 12000 | | socket_timeout_ms | int | 否 | 60000 | | json_filed_missed_return_null | boolean | 否 | false | @@ -94,19 +94,19 @@ HTTP 请求体。只有目标 API 接口支持请求体时才需要配置。 ### pageing [Config] -继承自 HTTP 连接器的分页配置。任务配置中请保持 `pageing` 这个拼写。必要时可配合 `page_type = "Cursor"` 使用游标分页。 +继承自 HTTP 连接器选项规则的分页设置。**这些选项不会被本连接器应用**:`GithubSource` 构建读取器时未传入分页上下文,因此无论如何配置 `pageing`,本连接器都只会读取单页数据,作业仍会正常结束。请不要依赖 `pageing`;读取更多数据需要等待连接器支持后再启用。 -### page_type [String] +### pageing.page_type [String] -分页类型,支持 `PageNumber`(默认)和 `Cursor`。对于响应中包含 `next` 游标的接口,请使用 `Cursor`。 +分页类型,支持 `PageNumber`(默认)和 `Cursor`。这些选项描述的是分页"应当如何配置",但如上所述**不会被本连接器应用**,无论取值如何本连接器都只读取单页数据。 -### cursor_field [String] +### pageing.cursor_field [String] -携带游标值的请求参数名称,与 `page_type = "Cursor"` 一起使用。 +携带游标值的请求参数名称,与 `pageing.page_type = "Cursor"` 一起使用。 -### cursor_response_field [String] +### pageing.cursor_response_field [String] -响应体中游标所在的 JSONPath,与 `page_type = "Cursor"` 一起使用。 +响应体中游标所在的 JSONPath,与 `pageing.page_type = "Cursor"` 一起使用。 ### poll_interval_millis [int] @@ -136,19 +136,19 @@ HTTP 请求因 `IOException` 失败时的最大重试次数。 分页时是否将分页参数保留在 URL 中,而不是在请求体内替换。 -### batch_size [int] +### pageing.batch_size [int] 当总页数未知时,每次请求返回的记录数。 -### start_page_number [long] +### pageing.start_page_number [long] 从哪一页开始读取。 -### total_page_size [long] +### pageing.total_page_size [long] -要读取的总页数。`0` 表示按照 `batch_size` 一直读取,直到 API 不再返回新页。 +要读取的总页数。`0` 表示按照 `pageing.batch_size` 一直读取,直到 API 不再返回新页。 -### use_placeholder_replacement [boolean] +### pageing.use_placeholder_replacement [boolean] 是否使用 `${field}` 占位符替换 headers、params 和 body 中的字段值,否则按键名替换。 @@ -174,8 +174,7 @@ HTTP 套接字超时时间(毫秒),默认 60000ms。 - 连接器始终会根据 `access_token` 添加 `Authorization: Bearer <access_token>` 请求头,请把其他自定义请求头放在 `headers` 中。 - 需要按字段读取时,把 `format` 设置为 `json` 并配置 `schema`。 - 当 GitHub 把记录嵌套在数组中时,使用 `content_field` 抽取数组元素。 -- 使用页码分页时,保持 `page_type = "PageNumber"`,并通过 `params` 配置 `page` / `per_page`。 -- 对于 GitHub Events 这类游标分页接口,请设置 `page_type = "Cursor"` 并配置 `cursor_field` 与 `cursor_response_field`。 +- 本连接器目前只读取单页数据:`pageing` 选项虽被选项规则接受,但读取器并不会应用它们(见 `pageing` 小节说明)。请不要配置 `params` 中的 `page` / `per_page` 来期望连接器自动翻页。 ## 任务示例 @@ -212,7 +211,7 @@ sink { } ``` -### 读取分页的 GitHub API 结果 +### 读取 GitHub API 结果(单页) ```hocon env { @@ -227,13 +226,7 @@ source { method = "GET" params = { per_page = "100" - page = "${page}" - } - pageing = { - page_field = "page" - total_page_size = 5 - start_page_number = 1 - use_placeholder_replacement = true + page = "1" } format = "json" schema = { diff --git a/docs/zh/connectors/source/Gitlab.md b/docs/zh/connectors/source/Gitlab.md index 187e1e8439..f64b76a429 100644 --- a/docs/zh/connectors/source/Gitlab.md +++ b/docs/zh/connectors/source/Gitlab.md @@ -33,9 +33,9 @@ Gitlab 源连接器用于读取 GitLab REST API 数据。它基于 HTTP 源连 | json_field | Config | 否 | - | | content_field | String | 否 | - | | pageing | Config | 否 | - | -| page_type | String | 否 | PageNumber | -| cursor_field | String | 否 | - | -| cursor_response_field | String | 否 | - | +| pageing.page_type | String | 否 | PageNumber | +| pageing.cursor_field | String | 否 | - | +| pageing.cursor_response_field | String | 否 | - | | poll_interval_millis | int | 否 | - | | retry | int | 否 | - | | retry_backoff_multiplier_ms | int | 否 | 100 | @@ -43,10 +43,10 @@ Gitlab 源连接器用于读取 GitLab REST API 数据。它基于 HTTP 源连 | enable_multi_lines | boolean | 否 | false | | keep_params_as_form | boolean | 否 | false | | keep_page_param_as_http_param | boolean | 否 | false | -| batch_size | int | 否 | 100 | -| start_page_number | long | 否 | 1 | -| total_page_size | long | 否 | 0 | -| use_placeholder_replacement | boolean | 否 | false | +| pageing.batch_size | int | 否 | 100 | +| pageing.start_page_number | long | 否 | 1 | +| pageing.total_page_size | long | 否 | 0 | +| pageing.use_placeholder_replacement | boolean | 否 | false | | connect_timeout_ms | int | 否 | 12000 | | socket_timeout_ms | int | 否 | 60000 | | json_filed_missed_return_null | boolean | 否 | false | @@ -94,19 +94,19 @@ HTTP 请求体。只有目标 API 接口支持请求体时才需要配置。 ### pageing [Config] -继承自 HTTP 连接器的分页配置。任务配置中请保持 `pageing` 这个拼写。 +继承自 HTTP 连接器选项规则的分页设置。**这些选项不会被本连接器应用**:`GitlabSource` 构建读取器时未传入分页上下文,因此无论如何配置 `pageing`,本连接器都只会读取单页数据,作业仍会正常结束。请不要依赖 `pageing`;读取更多数据需要等待连接器支持后再启用。 -### page_type [String] +### pageing.page_type [String] -分页类型,支持 `PageNumber`(默认)和 `Cursor`。对于响应中包含 `X-Next-Page` 游标的接口,请使用 `Cursor`。 +分页类型,支持 `PageNumber`(默认)和 `Cursor`。这些选项描述的是分页"应当如何配置",但如上所述**不会被本连接器应用**,无论取值如何本连接器都只读取单页数据。 -### cursor_field [String] +### pageing.cursor_field [String] -携带游标值的请求参数名称,与 `page_type = "Cursor"` 一起使用。 +携带游标值的请求参数名称,与 `pageing.page_type = "Cursor"` 一起使用。 -### cursor_response_field [String] +### pageing.cursor_response_field [String] -响应体中游标所在的 JSONPath,与 `page_type = "Cursor"` 一起使用。 +响应体中游标所在的 JSONPath,与 `pageing.page_type = "Cursor"` 一起使用。 ### poll_interval_millis [int] @@ -136,19 +136,19 @@ HTTP 请求因 `IOException` 失败时的最大重试次数。 分页时是否将分页参数保留在 URL 中,而不是在请求体内替换。 -### batch_size [int] +### pageing.batch_size [int] 当总页数未知时,每次请求返回的记录数。 -### start_page_number [long] +### pageing.start_page_number [long] 从哪一页开始读取。 -### total_page_size [long] +### pageing.total_page_size [long] -要读取的总页数。`0` 表示按照 `batch_size` 一直读取,直到 API 不再返回新页。 +要读取的总页数。`0` 表示按照 `pageing.batch_size` 一直读取,直到 API 不再返回新页。 -### use_placeholder_replacement [boolean] +### pageing.use_placeholder_replacement [boolean] 是否使用 `${field}` 占位符替换 headers、params 和 body 中的字段值,否则按键名替换。 @@ -174,7 +174,7 @@ HTTP 套接字超时时间(毫秒),默认 60000ms。 - 连接器始终会根据 `access_token` 添加 `PRIVATE-TOKEN` 请求头,请把其他自定义请求头放在 `headers` 中。 - 需要按字段读取时,把 `format` 设置为 `json` 并配置 `schema`。 - 当 GitLab 把记录嵌套在数组中时,使用 `content_field` 抽取数组元素。 -- 使用页码分页时,保持 `page_type = "PageNumber"`,并通过 `params` 配置 `page` / `per_page`。 +- 本连接器目前只读取单页数据:`pageing` 选项虽被选项规则接受,但读取器并不会应用它们(见 `pageing` 小节说明)。请不要配置 `params` 中的 `page` / `per_page` 来期望连接器自动翻页。 - Gitlab 源连接器当前只支持批处理模式,`poll_interval_millis` 不会启用流式行为。 ## 任务示例 @@ -212,7 +212,7 @@ sink { } ``` -### 读取分页的 GitLab API 结果 +### 读取 GitLab API 结果(单页) ```hocon env { @@ -227,13 +227,7 @@ source { method = "GET" params = { per_page = "100" - page = "${page}" - } - pageing = { - page_field = "page" - total_page_size = 5 - start_page_number = 1 - use_placeholder_replacement = true + page = "1" } format = "json" schema = { diff --git a/docs/zh/connectors/source/MongoDB-CDC.md b/docs/zh/connectors/source/MongoDB-CDC.md index 60efe7340c..42a8cebeba 100644 --- a/docs/zh/connectors/source/MongoDB-CDC.md +++ b/docs/zh/connectors/source/MongoDB-CDC.md @@ -112,6 +112,10 @@ db.grantRolesToUser("<USER_NAME>", ["<ROLE_NAME>"]) ## 源配置项 +> **注意:** MongoDB-CDC 复用 CDC 基础选项规则,因此少量面向 JDBC 的快照选项 +> (`snapshot.split.size`、`snapshot.fetch.size`、`enable_concurrent_read`)虽然能通过校验, +> 但 MongoDB-CDC 并不会使用它们。快照分块大小由 `incremental.snapshot.chunk.size.mb` 控制。 + | 名称 | 类型 | 必填 | 默认值 | 描述 | |------------------------------------|---------|------|---------|----------------------------------------------------------------------------------------------------------------------------| | hosts | String | 是 | - | MongoDB 服务器的主机名和端口列表,也可以是使用 `mongodb://` 或 `mongodb+srv://` 的标准 MongoDB 连接 URI。例如:`localhost:27017,localhost:27018` 或 `mongodb+srv://cluster.example.net`。 | diff --git a/docs/zh/connectors/source/MySQL-CDC.md b/docs/zh/connectors/source/MySQL-CDC.md index ab10c3569f..cb46b7e496 100644 --- a/docs/zh/connectors/source/MySQL-CDC.md +++ b/docs/zh/connectors/source/MySQL-CDC.md @@ -207,7 +207,7 @@ show variables where variable_name in ('log_bin', 'binlog_format', 'binlog_row_i | snapshot.fetch.size | Integer | 否 | 1024 | 每次轮询读取表快照时的最大获取大小. | | incremental.parallelism | Integer | 否 | 1 | 增量(binlog)阶段并行读取器的数量. | | server-id | String | 否 | - | 此 CDC 读取器使用的数字 ID 或数字 ID 范围,例如 `5400` 或 `5400-5408`。每个 ID 在 MySQL 集群中必须唯一。当任务有多个读取并发或并行读取多张表时,请配置足够大的 ID 范围。未配置时 SeaTunnel 会随机生成 ID,但生产环境建议显式配置。 | -| server-time-zone | String | 否 | - | 数据库服务中的会话时区. 如果没设置, 使用 ZoneId.systemDefault() 来确定服务的时区. | +| server-time-zone | String | 否 | JVM 系统时区 | 数据库服务中的会话时区. 如果没设置, 使用 ZoneId.systemDefault() 来确定服务的时区. | | connect.timeout.ms | Long | 否 | 30000 | 连接器在尝试连接数据库服务器后,在超时之前应等待的最长时间. | | connect.max-retries | Integer | 否 | 3 | 连接器在构建数据库服务器连接时应重试的最大重试次数. | | connection.pool.size | Integer | 否 | 20 | jdbc连接池大小. | @@ -505,6 +505,8 @@ source { } ``` +当没有可用主键(无论配置的还是物理的)时,连接器无法安全地应用 UPDATE/DELETE 事件。请只在仅追加(append-only)负载、或下游 Sink 行为不依赖行标识的场景中使用该模式。 + 上述示例演示的是"逻辑主键"场景:源表本身没有物理主键,但通过 `table-names-config.primaryKeys` 显式声明了一列作为稳定行标识,并启用 `exactly_once = true`,让快照阶段与 binlog 阶段都使用同一逻辑主键。只有当被声明的列在源数据中确实保持唯一时,UPDATE/DELETE 才能被正确路由;如果源数据中存在重复值,行为将不再可靠。 可空唯一键列中的 NULL 值,以及 `table-names-config.primaryKeys` 中配置的可空列中的 NULL 值,在快照阶段和 binlog 阶段都会按 NULL 输出。NULL 无法标识一行数据,因此当需要按键应用 UPDATE/DELETE 事件时,请选择 NOT NULL 的列作为键。开启 `exactly_once = true` 时,键为 NULL 的行同样无法区分,每个快照分片中只会保留其中一行。 diff --git a/docs/zh/connectors/source/PostgreSQL-CDC.md b/docs/zh/connectors/source/PostgreSQL-CDC.md index 93457719ce..ba24835a3b 100644 --- a/docs/zh/connectors/source/PostgreSQL-CDC.md +++ b/docs/zh/connectors/source/PostgreSQL-CDC.md @@ -107,9 +107,10 @@ ALTER TABLE your_table_name REPLICA IDENTITY FULL; | stop.mode | Enum | 否 | NEVER | PostgreSQL CDC 消费者的可选停止模式。唯一有效的枚举值是 `never`:一旦进入增量阶段,数据源会持续读取 WAL 变更,不会自行停止。 [...] | snapshot.split.size | Integer | 否 | 8096 | 表快照的拆分大小(行数),捕获的表在读取表快照时被拆分成多个拆分。 [...] | snapshot.fetch.size | Integer | 否 | 1024 | 读取表快照时每次轮询的最大获取大小。 [...] +| incremental.parallelism | Integer | 否 | 1 | 增量(WAL)阶段并行读取器的数量。 [...] | slot.name | String | 否 | seatunnel | PostgreSQL 逻辑解码槽名称。同一个 PostgreSQL 实例上如果有多个 CDC 任务,请为每个任务配置不同的 `slot.name`。 | | decoding.plugin.name | String | 否 | pgoutput | 安装在服务器上的 Postgres 逻辑解码插件的名称,支持的值有 decoderbufs、wal2json、wal2json_rds、wal2json_streaming、wal2json_rds_streaming 和 pgoutput。 [...] -| server-time-zone | String | 否 | - | 数据库服务器中的会话时区。如果未设置,则使用 ZoneId.systemDefault() 来确定服务器时区。 [...] +| server-time-zone | String | 否 | JVM 系统时区 | 数据库服务器中的会话时区。如果未设置,则使用 ZoneId.systemDefault() 来确定服务器时区。 [...] | connect.timeout.ms | Long | 否 | 30000 | 连接器在尝试连接到数据库服务器后应等待的最大时间,以防超时。 [...] | connect.max-retries | Integer | 否 | 3 | 连接器应重试建立数据库服务器连接的最大重试次数。 [...] | connection.pool.size | Integer | 否 | 20 | JDBC 连接池大小。 [...] @@ -366,7 +367,7 @@ GRANT SELECT ON ALL TABLES IN SCHEMA public TO replication_user; ### 支持哪些逻辑解码插件? -SeaTunnel PostgreSQL CDC 支持 `pgoutput`(PostgreSQL 10 起内置)、`wal2json` 和 `decoderbufs`,默认使用 `pgoutput`。通过 `decoding.plugin.name` 参数指定插件。 +`decoding.plugin.name` 支持 `pgoutput`(默认,PostgreSQL 10 起内置)、`decoderbufs`、`wal2json`、`wal2json_rds`、`wal2json_streaming` 和 `wal2json_rds_streaming`。注意 Schema 演进(`schema-changes.enabled`)要求使用 `pgoutput`。 ### SeaTunnel 能从 PostgreSQL 备库读取 CDC 数据吗? diff --git a/docs/zh/connectors/source/Shopify.md b/docs/zh/connectors/source/Shopify.md index f077415ccf..5413cf931c 100644 --- a/docs/zh/connectors/source/Shopify.md +++ b/docs/zh/connectors/source/Shopify.md @@ -65,7 +65,7 @@ http 请求方法,仅支持 GET、POST 方法。 ### format [String] -上游数据的格式,目前仅支持 `json` 和 `text`,默认 `text`。 +上游数据的格式,支持 `json` 和 `text`(底层 HTTP 源连接器也接受 `binary`),默认 `text`。 ### params [Map] diff --git a/docs/zh/connectors/source/Zendesk.md b/docs/zh/connectors/source/Zendesk.md index 6a63a6acbb..63f8600024 100644 --- a/docs/zh/connectors/source/Zendesk.md +++ b/docs/zh/connectors/source/Zendesk.md @@ -36,7 +36,7 @@ import ChangeLog from '../changelog/connector-http-zendesk.md'; | api_token | String | 是 | - | Zendesk API token。获取方式请参考 [Zendesk API token 文档](https://support.zendesk.com/hc/en-us/articles/4408889192858)。 | | method | String | 否 | get | HTTP 请求方法,仅支持 `GET` 和 `POST`。 | | schema | Config | 否 | - | 数据的结构,包括字段名称和字段类型。更多详情请参考 [Schema Feature](../../introduction/concepts/schema-feature.md)。 | -| format | String | 否 | text | 上游数据的格式,目前仅支持 `json` 和 `text`,默认 `text`。Zendesk 接口始终返回 JSON,因此通常需要把 `format` 设为 `json`,并配合 `content_field` 把真正的结果数组抽取出来再映射成行。 | +| format | String | 否 | text | 上游数据的格式,支持 `json` 和 `text`(底层 HTTP 源连接器也接受 `binary`)。Zendesk 接口始终返回 JSON,因此通常需要把 `format` 设为 `json`,并配合 `content_field` 把真正的结果数组抽取出来再映射成行。 | | params | Map | 否 | - | 附加到请求 URL 的查询参数,可用于过滤、分页等场景。 | | body | String | 否 | - | `POST`(或支持 body 的方法)请求体。当 `format = "json"` 时,body 必须是合法的 JSON。 | | json_field | Config | 否 | - | 该参数用于配置 schema,因此必须与 `schema` 一起使用。它将响应中的 JSON 路径映射到 schema 字段。详情和示例请参考 [Http source](./Http.md) 连接器。 | @@ -72,8 +72,9 @@ HTTP 请求方法,仅支持 `GET` 和 `POST`。`POST` 一般需要配合 `body ### format [String] -上游数据的格式,目前仅支持 `json` 和 `text`,默认 `text`。Zendesk 接口始终返回 JSON,因此需要把 -`format` 设为 `json`,并配合 `content_field` 抽取结果数组。 +上游数据的格式,支持 `json` 和 `text`,默认 `text`。与通用 [Http 源](Http.md) 一样,底层连接器也接受 +`binary`,但 Zendesk 接口始终返回 JSON,因此需要把 `format` 设为 `json`,并配合 `content_field` +抽取结果数组。 ### params [Map]
