This is an automated email from the ASF dual-hosted git repository.
davidzollo pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 122d090877 [Docs] Clarify REST API defaults and add connector
troubleshooting cookbooks (#11035)
122d090877 is described below
commit 122d090877b4a33407afc9f31c28d4d8fb9608cc
Author: Daniel <[email protected]>
AuthorDate: Sat Jun 13 17:41:00 2026 +0800
[Docs] Clarify REST API defaults and add connector troubleshooting
cookbooks (#11035)
---
docs/en/connectors/sink/Jdbc.md | 28 ++++++++-
docs/en/connectors/sink/PostgreSql.md | 42 ++++++++++++++
docs/en/connectors/source/Http.md | 106 +++++++++++++++++++++++++++++++++-
docs/en/engines/zeta/rest-api-v2.md | 38 ++++++++++--
docs/en/transforms/metadata.md | 54 +++++++++++++++++
docs/zh/connectors/sink/Jdbc.md | 28 ++++++++-
docs/zh/connectors/sink/PostgreSql.md | 41 ++++++++++++-
docs/zh/connectors/source/Http.md | 106 +++++++++++++++++++++++++++++++++-
docs/zh/engines/zeta/rest-api-v2.md | 36 ++++++++++--
docs/zh/transforms/metadata.md | 53 +++++++++++++++++
10 files changed, 516 insertions(+), 16 deletions(-)
diff --git a/docs/en/connectors/sink/Jdbc.md b/docs/en/connectors/sink/Jdbc.md
index 40f47c98ad..c5ee8ccd79 100644
--- a/docs/en/connectors/sink/Jdbc.md
+++ b/docs/en/connectors/sink/Jdbc.md
@@ -569,7 +569,9 @@ Not all databases support XA transactions. Verify that your
database and JDBC dr
### How do I configure upsert (INSERT or UPDATE) behavior?
-Specify `primary_keys` to enable upsert behavior. SeaTunnel generates an
INSERT ... ON DUPLICATE KEY UPDATE (or equivalent) statement based on the
target database dialect:
+SeaTunnel only enters the upsert / update path after it has a final key set.
That key can come from explicit `primary_keys`, or, when `primary_keys` is
omitted, from upstream catalog metadata. If no primary key is available,
SeaTunnel also tries to inherit the first unique key.
+
+When a final key set exists and `enable_upsert = true`, SeaTunnel prefers the
database-native upsert statement provided by the target dialect. For example,
PostgreSQL generates `INSERT ... ON CONFLICT (...) DO UPDATE` (or `DO NOTHING`
when every column is part of the key and there is nothing left to update):
```hocon
sink {
@@ -582,7 +584,29 @@ sink {
}
```
-Without `primary_keys`, JDBC Sink performs plain INSERTs and does not handle
duplicate key conflicts.
+When a final key set exists but `enable_upsert = false`, SeaTunnel stops using
native database upsert SQL and falls back to the row-kind-driven insert/update
path:
+
+- `INSERT` rows are written as plain INSERTs
+- CDC `UPDATE_AFTER` rows are written as UPDATEs
+- CDC `DELETE` rows are written as DELETEs
+
+As a result, `enable_upsert = false` is not appropriate for ordinary batch
imports that rely on duplicate-key overwrite behavior.
+
+### What happens if I do not configure `primary_keys`?
+
+If `primary_keys` is not configured, SeaTunnel first tries to inherit the
primary key from upstream catalog metadata. If there is no primary key, it then
tries the first unique key.
+
+JDBC Sink falls back to plain INSERT only when there is no explicit key and
nothing usable can be inherited from upstream metadata. In that keyless mode,
no database-native upsert SQL is generated, and the sink no longer uses
row-kind-aware UPDATE / DELETE executors. For CDC inputs, the write path
therefore effectively degrades to plain INSERT batching, and duplicate-key
behavior depends entirely on the target table constraints.
+
+### When should I enable `use_copy_statement`?
+
+`use_copy_statement = true` makes JDBC Sink prefer the `COPY <table> (...)
FROM STDIN WITH CSV` path instead of regular INSERT / UPSERT SQL. This happens
before the normal primary-key-based write path, so COPY is still chosen even if
`primary_keys` is configured.
+
+This option is mainly for high-volume PostgreSQL imports, and it has three
important constraints:
+
+- the JDBC driver connection must expose `getCopyAPI()`, otherwise the job
fails and tells you to switch `use_copy_statement` back to `false`
+- it is not a replacement for `ON CONFLICT`, so it does not provide
duplicate-key overwrite semantics
+- `MAP`, `ARRAY`, and `ROW` types are not supported
### How do I write to multiple tables in a single job?
diff --git a/docs/en/connectors/sink/PostgreSql.md
b/docs/en/connectors/sink/PostgreSql.md
index e1ae7339ab..6ec9fc16db 100644
--- a/docs/en/connectors/sink/PostgreSql.md
+++ b/docs/en/connectors/sink/PostgreSql.md
@@ -99,6 +99,7 @@ semantics (using XA transaction guarantee).
| data_save_mode | Enum | no | APPEND_DATA
| Before the synchronous task is turned on, different
processing schemes are selected for data existing data on the target side.
[...]
| custom_sql | String | no | -
| When data_save_mode selects CUSTOM_PROCESSING, you should
fill in the CUSTOM_SQL parameter. This parameter usually fills in a SQL that
can be executed. SQL will be executed before synchronization tasks.
[...]
| enable_upsert | Boolean | No | true
| Enable upsert by primary_keys exist, If the task has no key
duplicate data, setting this parameter to `false` can speed up data import
[...]
+| use_copy_statement | Boolean | No | false
| Use PostgreSQL `COPY <table> (...) FROM STDIN WITH CSV` for
bulk import. This option takes precedence over the regular INSERT / UPSERT
path, requires a JDBC driver connection that exposes `getCopyAPI()`, and does
not support `MAP`, `ARRAY`, or `ROW` types.
[...]
### table [string]
@@ -273,6 +274,47 @@ sink {
}
```
+## FAQ
+
+### When does PostgreSQL Sink generate `ON CONFLICT`?
+
+PostgreSQL Sink generates `INSERT ... ON CONFLICT (...) DO UPDATE` only after
it has a final
+primary-key / unique-key set and `enable_upsert = true`.
+
+That key can come from two places:
+
+- explicit `primary_keys`
+- inherited upstream catalog metadata when `primary_keys` is omitted; if no
primary key exists, SeaTunnel also tries the first unique key
+
+If every column is part of the key and there is nothing left to update,
PostgreSQL degrades this to
+`ON CONFLICT (...) DO NOTHING`.
+
+### What happens when the target table has no primary key?
+
+If there is no explicit `primary_keys` setting and no inheritable primary key
or unique key in
+upstream metadata, PostgreSQL Sink falls back to plain INSERT and does not
generate `ON CONFLICT`.
+In that keyless mode, the sink also stops using row-kind-aware UPDATE / DELETE
executors, so CDC
+inputs effectively degrade to plain INSERT batching. Duplicate-key behavior
then depends entirely on
+the target table constraints.
+
+### How should I choose between `use_copy_statement` and `enable_upsert`?
+
+`use_copy_statement = true` makes PostgreSQL Sink enter the COPY bulk-load
path before the normal
+INSERT / UPSERT path. In other words, COPY still wins even if `primary_keys`
is configured and
+`enable_upsert = true`.
+
+COPY is a good fit when:
+
+- the target is PostgreSQL
+- the goal is high-throughput bulk import
+- you do not rely on PostgreSQL native upsert semantics for duplicate-key
overwrite
+
+COPY is not a good fit when:
+
+- you want PostgreSQL native upsert conflict handling
+- your data contains `MAP`, `ARRAY`, or `ROW` types
+- the current JDBC driver connection does not expose `getCopyAPI()`
+
## Changelog
<ChangeLog />
diff --git a/docs/en/connectors/source/Http.md
b/docs/en/connectors/source/Http.md
index b0752c72a8..2d07388a86 100644
--- a/docs/en/connectors/source/Http.md
+++ b/docs/en/connectors/source/Http.md
@@ -192,7 +192,7 @@ connector will generate data as the following:
|----------------------------------------------------------|
| {"code": 200, "data": "get success", "success": true} |
-when you assign format is `binary`, the HTTP response body is treated as raw
bytes for downloading files (PDF, images, ZIP, etc.). The output schema is
fixed as `(data: bytes, relativePath: string, partIndex: long)`. Large files
are automatically split into multiple rows based on `binary_chunk_size`. Only
supports BATCH mode.
+when you assign format is `binary`, the HTTP response body is treated as raw
bytes for downloading files (PDF, images, ZIP, etc.). The output schema is
fixed as `(data: bytes, relativePath: string, partIndex: long)`. Large files
are automatically split into multiple rows based on `binary_chunk_size`. Only
supports BATCH mode and does not support pagination (`pageing`).
Example: Download a file via HTTP and write to LocalFileSink:
@@ -266,6 +266,110 @@ headers {
}
```
+### Pagination and final request shape
+
+The easiest way to troubleshoot the Http source is to reason from the final
outbound request:
+
+1. For `GET`, `params` are always appended to the URL query string.
+2. For `POST` with `keep_params_as_form = false`:
+ - `params` still go to the URL query string
+ - on the default non-form path, `body` is serialized as the JSON request
body
+ - if `body` is not configured and the request stays on that default
non-form path, the runtime sends an empty JSON object `{}` as the request body
+ - if you explicitly force `Content-Type:
application/x-www-form-urlencoded`, the runtime follows the form-body branch
instead of the default JSON branch
+3. For `POST` with `keep_params_as_form = true`:
+ - `params` are merged into the form body
+ - if `Content-Type` is not set explicitly, SeaTunnel adds
`application/x-www-form-urlencoded`
+ - if `body` and `params` contain the same key, the value from `params`
overrides the value in `body`
+4. `keep_page_param_as_http_param = true` writes the paging field directly
into `params`
+5. `keep_page_param_as_http_param = false` only updates existing keys or
placeholders in headers, params, and body; it does not invent new paging fields
automatically
+6. `pageing.use_placeholder_replacement = true` supports `${page}` and
`${cursor}` placeholders, and also prefixed/suffixed replacements such as
`"10${page}" -> "105"`; when `false`, only key-based replacement is applied
+
+Example 1: GET pagination with the page number in query parameters
+
+```hocon
+source {
+ Http {
+ url = "https://api.example.com/orders"
+ method = "GET"
+ params = {
+ page = "${page}"
+ size = "100"
+ }
+ pageing = {
+ page_field = "page"
+ page_type = "PageNumber"
+ start_page_number = 3
+ use_placeholder_replacement = true
+ }
+ }
+}
+```
+
+When the page advances to `3`, the final request is:
+
+```text
+GET https://api.example.com/orders?page=3&size=100
+```
+
+Example 2: POST JSON on the default non-form path, with URL query parameters
and the paging field inside the body
+
+```hocon
+source {
+ Http {
+ url = "https://api.example.com/orders/search"
+ method = "POST"
+ keep_params_as_form = false
+ params = {
+ tenant = "acme"
+ }
+ body = """{"page":"${page}","pageSize":100}"""
+ pageing = {
+ page_field = "page"
+ page_type = "PageNumber"
+ start_page_number = 3
+ use_placeholder_replacement = true
+ }
+ }
+}
+```
+
+When the page advances to `3`, the final request is:
+
+```text
+POST https://api.example.com/orders/search?tenant=acme
+Content-Type: application/json
+Body: {"page":"3","pageSize":100}
+```
+
+Example 3: POST form submission with paging fields merged into the form body
+
+```hocon
+source {
+ Http {
+ url = "https://api.example.com/orders/search"
+ method = "POST"
+ keep_params_as_form = true
+ keep_page_param_as_http_param = true
+ params = {
+ size = "100"
+ }
+ pageing = {
+ page_field = "page"
+ page_type = "PageNumber"
+ start_page_number = 3
+ }
+ }
+}
+```
+
+When the page advances to `3`, the final request is:
+
+```text
+POST https://api.example.com/orders/search
+Content-Type: application/x-www-form-urlencoded
+Body: size=100&page=3
+```
+
### content_field
This parameter can get some json data.If you only need the data in the 'book'
section, configure `content_field = "$.store.book.*"`.
diff --git a/docs/en/engines/zeta/rest-api-v2.md
b/docs/en/engines/zeta/rest-api-v2.md
index 672e3f962d..7afe0fc853 100644
--- a/docs/en/engines/zeta/rest-api-v2.md
+++ b/docs/en/engines/zeta/rest-api-v2.md
@@ -5,10 +5,31 @@ completed jobs. The monitoring API is a RESTful API that
accepts HTTP requests a
## Overview
-The v2 version of the api uses jetty support. It is the same as the interface
specification of v1 version
-, you can specify the port and context-path by modifying the configuration
items in `seatunnel.yaml`,
-you can configure `enable-dynamic-port` to enable dynamic ports (the default
port is accumulated starting from `port`), and the default is enabled,
-If enable-dynamic-port is true, We will use the unused port in the range
within the range of `port` and `port` + `port-range`, default range is 100
+The v2 API and the Web UI are both served by the embedded Jetty server. Jetty
starts only when
+`seatunnel.engine.http.enable-http = true` or `enable-https = true`.
+
+There are two different "default" sources that are easy to mix up:
+
+- Code defaults: `enable-http = false`, `enable-https = false`, `port = 8080`,
`context-path = ""`, `enable-dynamic-port = false`, `port-range = 100`
+- The packaged `seatunnel.yaml` example: it already sets `enable-http: true`
and `port: 8080`
+
+As a result, if you start SeaTunnel with the packaged configuration, the Web
UI and REST API usually
+listen on `http://<host>:8080/`. If you build a minimal config yourself, rely
on code defaults, or
+remove `enable-http`, Jetty will not start by default.
+
+Use the following configuration for a fixed port:
+
+```yaml
+
+seatunnel:
+ engine:
+ http:
+ enable-http: true
+ port: 8080
+```
+
+If you want Jetty to choose the first free port between `port` and `port +
port-range`, enable
+dynamic ports explicitly:
```yaml
@@ -21,7 +42,7 @@ seatunnel:
port-range: 100
```
-Context-path can also be configured as follows:
+`context-path` can also be configured as follows:
```yaml
@@ -33,6 +54,13 @@ seatunnel:
context-path: /seatunnel
```
+## Web UI and Port 8080 Troubleshooting
+
+- If `http://<host>:8080/` is unreachable, first check whether
`seatunnel.engine.http.enable-http` or `enable-https` is actually enabled. The
`network.rest-api.enabled` setting in `hazelcast.yaml` does not replace the
Jetty switch.
+- If `enable-dynamic-port = true`, the actual listening port may not be 8080.
Jetty will choose the first available port between `port` and `port +
port-range`. Use the startup log `SeaTunnel REST service will start on port
xxx` as the source of truth.
+- If `context-path = /seatunnel`, both the Web UI and REST endpoints move
under that prefix. For example, the overview endpoint becomes
`/seatunnel/overview`.
+- The Web UI static resources and REST endpoints share the same Jetty service.
If Jetty does not start, both are unavailable together.
+
## Enable HTTPS
Please refer [security](security.md)
diff --git a/docs/en/transforms/metadata.md b/docs/en/transforms/metadata.md
index 1ccba8781b..5a311465b0 100644
--- a/docs/en/transforms/metadata.md
+++ b/docs/en/transforms/metadata.md
@@ -246,3 +246,57 @@ sink {
```
Here `pt` is derived from the Kafka event time and can be used as a Hive
partition column.
+
+### Example 4: Combine Metadata and Sql to extract table suffixes and add a
load date
+
+When the upstream CDC source uses sharded tables such as monthly or daily
tables, a common pattern
+is to expose the `Table` metadata as a regular field first, then use `Sql` to
derive the shard
+suffix and a formatted load date.
+
+```hocon
+env {
+ parallelism = 1
+ job.mode = "STREAMING"
+}
+
+source {
+ MySQL-CDC {
+ plugin_output = "orders_cdc"
+ server-id = 5652
+ username = "root"
+ password = "your_password"
+ table-names = ["app.orders_202401", "app.orders_202402"]
+ url = "jdbc:mysql://localhost:3306/app"
+ }
+}
+
+transform {
+ Metadata {
+ plugin_input = "orders_cdc"
+ plugin_output = "orders_with_meta"
+ metadata_fields {
+ Table = source_table
+ EventTime = event_ts
+ }
+ }
+
+ Sql {
+ plugin_input = "orders_with_meta"
+ plugin_output = "orders_normalized"
+ query = "select id, amount, source_table, REGEXP_SUBSTR(source_table,
'[0-9]+$') as table_suffix, FROM_UNIXTIME(event_ts / 1000, 'yyyy-MM-dd
HH:mm:ss', 'Asia/Shanghai') as event_time_str,
FORMATDATETIME(CURRENT_TIMESTAMP, 'yyyyMMdd') as load_date from
orders_with_meta"
+ }
+}
+
+sink {
+ Console {
+ plugin_input = "orders_normalized"
+ }
+}
+```
+
+If the current record comes from `orders_202402`, then:
+
+- `source_table = "orders_202402"`
+- `table_suffix = "202402"`
+- `event_time_str` comes from the CDC event time
+- `load_date` is the formatted runtime date string
diff --git a/docs/zh/connectors/sink/Jdbc.md b/docs/zh/connectors/sink/Jdbc.md
index fb6f7075df..732eaa8df6 100644
--- a/docs/zh/connectors/sink/Jdbc.md
+++ b/docs/zh/connectors/sink/Jdbc.md
@@ -474,7 +474,9 @@ sink {
### 如何配置 Upsert(INSERT OR UPDATE)行为?
-指定 `primary_keys` 即可启用 upsert。SeaTunnel 会根据目标数据库方言自动生成 `INSERT ... ON
DUPLICATE KEY UPDATE`(或等效)语句:
+SeaTunnel 只有在最终拿到了主键/唯一键信息时,才会进入 upsert / update 路径。这个 key 既可以来自显式配置的
`primary_keys`,也可以在未显式配置时从上游 catalog 元数据里继承主键;如果没有主键,还会尝试继承第一组 unique key。
+
+当最终存在 key 且 `enable_upsert = true` 时,SeaTunnel 会优先使用目标数据库方言支持的原生 upsert 语句。例如
PostgreSQL 会生成 `INSERT ... ON CONFLICT (...) DO UPDATE`(如果所有字段都是主键,没有可更新列,则退化为
`DO NOTHING`):
```hocon
sink {
@@ -487,7 +489,29 @@ sink {
}
```
-不设置 `primary_keys` 时,JDBC Sink 执行普通 INSERT,不处理主键冲突。
+当最终存在 key 但 `enable_upsert = false` 时,SeaTunnel 不再生成数据库原生 upsert
语句,而是回到按行类型执行的 insert/update 路径:
+
+- `INSERT` 行执行普通 INSERT
+- CDC `UPDATE_AFTER` 行执行 UPDATE
+- CDC `DELETE` 行执行 DELETE
+
+因此,`enable_upsert = false` 不适合依赖重复键自动覆盖的普通批量导入场景。
+
+### 未显式配置 `primary_keys` 时会发生什么?
+
+如果你没有显式配置 `primary_keys`,SeaTunnel 会先尝试从上游 catalog 元数据继承主键;如果没有主键,则再尝试继承第一组
unique key。
+
+只有在“显式配置也没有、上游元数据里也没有可继承 key”时,JDBC Sink 才会退回普通 INSERT。进入这个无 key
模式后,不仅不会生成数据库原生 upsert 语句,Sink 也不会再使用按 `RowKind` 执行 UPDATE / DELETE 的写入器。对于 CDC
输入,这条写链路会实质上退化为普通 INSERT batching,重复键是否报错完全取决于目标表自身约束。
+
+### 什么时候应该开启 `use_copy_statement`?
+
+`use_copy_statement = true` 会让 JDBC Sink 直接优先走 `COPY <table> (...) FROM STDIN
WITH CSV` 路径,而不是常规的 INSERT / UPSERT 语句。即使同时配置了 `primary_keys`,也会优先进入 COPY 路径。
+
+这个选项更适合 PostgreSQL 大批量导入场景,但要同时满足下面几个前提:
+
+- JDBC 驱动连接对象必须提供 `getCopyAPI()` 能力;否则任务会直接报错,并提示把 `use_copy_statement` 改回
`false`
+- 它不是 `ON CONFLICT` 的替代品,不负责处理重复键覆盖逻辑
+- 当前不支持 `MAP`、`ARRAY`、`ROW` 类型
### 如何在一个任务中写入多张表?
diff --git a/docs/zh/connectors/sink/PostgreSql.md
b/docs/zh/connectors/sink/PostgreSql.md
index 885721eef3..947437b664 100644
--- a/docs/zh/connectors/sink/PostgreSql.md
+++ b/docs/zh/connectors/sink/PostgreSql.md
@@ -95,6 +95,7 @@ import ChangeLog from '../changelog/connector-jdbc.md';
| data_save_mode | Enum | 否 | APPEND_DATA
| 在同步任务开启之前,根据目标端现有数据选择不同处理方案。
[...]
| custom_sql | String | 否 | -
| 当 `data_save_mode` 选择 `CUSTOM_PROCESSING` 时,您应该填写 `CUSTOM_SQL` 参数。此参数通常填入可执行的
SQL。SQL 将在同步任务之前执行。
[...]
| enable_upsert | Boolean | 否 | true
| 通过主键存在启用 upsert,如果任务没有重复数据,设置此参数为 `false` 可以加快数据导入。
[...]
+| use_copy_statement | Boolean | 否 | false
| 直接使用 PostgreSQL `COPY <table> (...) FROM STDIN WITH CSV` 进行批量导入。该选项优先级高于常规
INSERT / UPSERT 路径;要求 JDBC 驱动连接对象提供 `getCopyAPI()`,且当前不支持 `MAP`、`ARRAY`、`ROW`
类型。
[...]
### table [字符串]
@@ -269,6 +270,44 @@ sink {
}
```
+## 常见问题
+
+### PostgreSQL Sink 什么时候会生成 `ON CONFLICT`?
+
+只有在最终拿到了主键/唯一键信息,并且 `enable_upsert = true` 时,PostgreSQL Sink 才会生成
+`INSERT ... ON CONFLICT (...) DO UPDATE`。
+
+这个 key 可以来自两种地方:
+
+- 你显式配置的 `primary_keys`
+- 你没有显式配置时,SeaTunnel 从上游 catalog 元数据继承到的主键;如果没有主键,还会尝试第一组 unique key
+
+如果所有字段本身都是主键,没有可更新列,PostgreSQL 会退化为
+`ON CONFLICT (...) DO NOTHING`。
+
+### 目标表没有主键时会发生什么?
+
+如果既没有显式配置 `primary_keys`,上游元数据里也没有可继承的主键或 unique key,
+PostgreSQL Sink 会退回普通 INSERT,不会生成 `ON CONFLICT`。进入这个无 key 模式后,
+Sink 也不会再使用按 `RowKind` 执行 UPDATE / DELETE 的写入器,因此 CDC 输入会实质上退化为普通
+INSERT batching。这时重复键是否报错,完全取决于目标表自身约束。
+
+### `use_copy_statement` 和 `enable_upsert` 应该怎么选?
+
+`use_copy_statement = true` 会优先进入 PostgreSQL COPY 批量导入路径,而不是常规 INSERT / UPSERT
语句。也就是说,即使同时配置了 `primary_keys` 和 `enable_upsert = true`,COPY 仍然会优先执行。
+
+适合开启 COPY 的场景:
+
+- 目标是 PostgreSQL
+- 重点是高吞吐批量导入
+- 不依赖 `ON CONFLICT` 处理重复键覆盖逻辑
+
+不适合开启 COPY 的场景:
+
+- 你希望通过 PostgreSQL 原生 upsert 语义处理冲突
+- 你的数据包含 `MAP`、`ARRAY`、`ROW` 类型
+- 当前 JDBC 驱动连接对象不提供 `getCopyAPI()`
+
## 变更日志
-<ChangeLog />
\ No newline at end of file
+<ChangeLog />
diff --git a/docs/zh/connectors/source/Http.md
b/docs/zh/connectors/source/Http.md
index a3e5dcf8ad..defc7c664f 100644
--- a/docs/zh/connectors/source/Http.md
+++ b/docs/zh/connectors/source/Http.md
@@ -181,7 +181,7 @@ schema {
|----------------------------------------------------------|
| {"code": 200, "data": "get success", "success": true} |
-当您指定 format 为 `binary` 时,HTTP 响应体作为原始字节处理,用于下载文件(PDF、图片、ZIP 等)。输出 schema 固定为
`(data: bytes, relativePath: string, partIndex: long)`。大文件会根据
`binary_chunk_size` 自动拆分为多行。仅支持 BATCH 模式。
+当您指定 format 为 `binary` 时,HTTP 响应体作为原始字节处理,用于下载文件(PDF、图片、ZIP 等)。输出 schema 固定为
`(data: bytes, relativePath: string, partIndex: long)`。大文件会根据
`binary_chunk_size` 自动拆分为多行。仅支持 BATCH 模式,且不支持分页(`pageing`)。
示例:通过 HTTP 下载文件并写入 LocalFileSink:
@@ -255,6 +255,110 @@ headers {
}
```
+### 分页与最终请求形态排查
+
+下面几条规则最容易混淆,建议先按“最终发出的 HTTP 请求长什么样”来理解:
+
+1. `GET` 请求:`params` 一定会被拼到 URL 查询串里。
+2. `POST` 且 `keep_params_as_form = false`:
+ - `params` 仍然会拼到 URL 查询串里
+ - 在默认的非 form 分支上,`body` 会作为 JSON body 发送
+ - 如果没有配置 `body`,并且请求仍然走这个默认非 form 分支,运行时会发送一个空 JSON 对象 `{}` 作为请求体
+ - 如果你显式把 `Content-Type` 设为 `application/x-www-form-urlencoded`,运行时会改走
form-body 分支,而不是默认 JSON 分支
+3. `POST` 且 `keep_params_as_form = true`:
+ - `params` 会并入表单 body
+ - 如果未显式设置 `Content-Type`,SeaTunnel 会自动补 `application/x-www-form-urlencoded`
+ - 如果 `body` 与 `params` 出现同名键,`params` 的值会覆盖 `body` 中同名键
+4. `keep_page_param_as_http_param = true`:分页字段会直接写入 `params`
+5. `keep_page_param_as_http_param = false`:SeaTunnel 只会更新 headers、params、body
里已经存在的同名键或占位符,不会凭空新增分页字段
+6. `pageing.use_placeholder_replacement = true`:支持 `${page}`、`${cursor}`
占位符,也支持 `"10${page}" -> "105"` 这种带前后缀的替换;为 `false` 时只做按 key 的整值替换
+
+示例 1:GET 分页,页码写入查询参数
+
+```hocon
+source {
+ Http {
+ url = "https://api.example.com/orders"
+ method = "GET"
+ params = {
+ page = "${page}"
+ size = "100"
+ }
+ pageing = {
+ page_field = "page"
+ page_type = "PageNumber"
+ start_page_number = 3
+ use_placeholder_replacement = true
+ }
+ }
+}
+```
+
+当页码推进到 `3` 时,最终请求为:
+
+```text
+GET https://api.example.com/orders?page=3&size=100
+```
+
+示例 2:POST JSON(默认非 form 分支),请求参数进 URL,分页字段留在 body
+
+```hocon
+source {
+ Http {
+ url = "https://api.example.com/orders/search"
+ method = "POST"
+ keep_params_as_form = false
+ params = {
+ tenant = "acme"
+ }
+ body = """{"page":"${page}","pageSize":100}"""
+ pageing = {
+ page_field = "page"
+ page_type = "PageNumber"
+ start_page_number = 3
+ use_placeholder_replacement = true
+ }
+ }
+}
+```
+
+当页码推进到 `3` 时,最终请求为:
+
+```text
+POST https://api.example.com/orders/search?tenant=acme
+Content-Type: application/json
+Body: {"page":"3","pageSize":100}
+```
+
+示例 3:POST 表单,请求参数和分页字段都写入表单 body
+
+```hocon
+source {
+ Http {
+ url = "https://api.example.com/orders/search"
+ method = "POST"
+ keep_params_as_form = true
+ keep_page_param_as_http_param = true
+ params = {
+ size = "100"
+ }
+ pageing = {
+ page_field = "page"
+ page_type = "PageNumber"
+ start_page_number = 3
+ }
+ }
+}
+```
+
+当页码推进到 `3` 时,最终请求为:
+
+```text
+POST https://api.example.com/orders/search
+Content-Type: application/x-www-form-urlencoded
+Body: size=100&page=3
+```
+
### content_field
此参数可以获取一些 json 数据。如果您只需要 'book' 部分的数据,配置 `content_field = "$.store.book.*"`。
diff --git a/docs/zh/engines/zeta/rest-api-v2.md
b/docs/zh/engines/zeta/rest-api-v2.md
index b5f3d13658..9b9676de35 100644
--- a/docs/zh/engines/zeta/rest-api-v2.md
+++ b/docs/zh/engines/zeta/rest-api-v2.md
@@ -4,9 +4,30 @@ SeaTunnel有一个用于监控的API,可用于查询运行作业的状态和
## 概述
-v2版本的api使用jetty支持,与v1版本的接口规范相同 ,可以通过修改`seatunnel.yaml`中的配置项来指定端口和context-path,
-同时可以配置 `enable-dynamic-port` 开启动态端口(默认从 `port` 开始累加),默认为开启,
-如果`enable-dynamic-port`为`true`,我们将使用`port`和`port`+`port-range`范围内未使用的端口,默认范围是100。
+v2 版本的 API 和 Web UI 都由内嵌 Jetty 提供,与 v1 版本保持相同的接口规范。只有当
+`seatunnel.engine.http.enable-http = true` 或 `enable-https = true` 时,Jetty
才会启动。
+
+这里需要区分两个容易混淆的“默认值”来源:
+
+- 代码默认值:`enable-http = false`、`enable-https = false`、`port =
8080`、`context-path = ""`、`enable-dynamic-port = false`、`port-range = 100`
+- 发行包自带的 `seatunnel.yaml` 示例:默认写入了 `enable-http: true` 和 `port: 8080`
+
+因此,直接使用发行包自带配置启动时,Web UI 和 REST API 通常会监听
+`http://<host>:8080/`。如果你是自己精简配置文件、按代码默认值装配配置,或者把
+`enable-http` 删掉了,那么 Jetty 默认不会启动。
+
+固定端口示例如下:
+
+```yaml
+
+seatunnel:
+ engine:
+ http:
+ enable-http: true
+ port: 8080
+```
+
+如需在 `port` 到 `port + port-range` 范围内自动挑选空闲端口,再显式开启动态端口:
```yaml
@@ -19,7 +40,7 @@ seatunnel:
port-range: 100
```
-同时也可以配置context-path,配置如下:
+同时也可以配置 `context-path`,配置如下:
```yaml
@@ -31,6 +52,13 @@ seatunnel:
context-path: /seatunnel
```
+## Web UI 与 8080 排查
+
+- 如果 `http://<host>:8080/` 打不开,先检查 `seatunnel.engine.http.enable-http` 或
`enable-https` 是否真的开启;仅配置 `hazelcast.yaml` 中的 `network.rest-api.enabled` 不能替代
Jetty 开关。
+- 如果开启了 `enable-dynamic-port = true`,实际监听端口可能不是 8080,而是 `port` 到 `port +
port-range` 之间的第一个空闲端口。以启动日志 `SeaTunnel REST service will start on port xxx` 为准。
+- 如果配置了 `context-path = /seatunnel`,Web UI 首页和 REST 路径都会整体前移,例如概览接口会变成
`/seatunnel/overview`。
+- Web UI 静态资源和 REST API 共用同一个 Jetty 服务。只要 Jetty 没启动,两者都会一起不可用。
+
## 开启 HTTPS
请参考 [security](security.md)
diff --git a/docs/zh/transforms/metadata.md b/docs/zh/transforms/metadata.md
index 552ea73ad9..d51c3264a7 100644
--- a/docs/zh/transforms/metadata.md
+++ b/docs/zh/transforms/metadata.md
@@ -246,3 +246,56 @@ sink {
```
上面的 `pt` 字段由 Kafka 事件时间转换而来,可在 Hive 中作为分区列使用,便于补数和校准分区。
+
+### 示例 4:结合 Metadata 和 Sql 提取分表后缀并生成装载日期
+
+当上游是按月或按天分表的 CDC 源时,常见需求是先把 `Table` 元数据暴露成普通字段,再用
+`Sql` 提取分表后缀、补充任务装载日期。
+
+```hocon
+env {
+ parallelism = 1
+ job.mode = "STREAMING"
+}
+
+source {
+ MySQL-CDC {
+ plugin_output = "orders_cdc"
+ server-id = 5652
+ username = "root"
+ password = "your_password"
+ table-names = ["app.orders_202401", "app.orders_202402"]
+ url = "jdbc:mysql://localhost:3306/app"
+ }
+}
+
+transform {
+ Metadata {
+ plugin_input = "orders_cdc"
+ plugin_output = "orders_with_meta"
+ metadata_fields {
+ Table = source_table
+ EventTime = event_ts
+ }
+ }
+
+ Sql {
+ plugin_input = "orders_with_meta"
+ plugin_output = "orders_normalized"
+ query = "select id, amount, source_table, REGEXP_SUBSTR(source_table,
'[0-9]+$') as table_suffix, FROM_UNIXTIME(event_ts / 1000, 'yyyy-MM-dd
HH:mm:ss', 'Asia/Shanghai') as event_time_str,
FORMATDATETIME(CURRENT_TIMESTAMP, 'yyyyMMdd') as load_date from
orders_with_meta"
+ }
+}
+
+sink {
+ Console {
+ plugin_input = "orders_normalized"
+ }
+}
+```
+
+如果当前记录来自 `orders_202402`,那么:
+
+- `source_table = "orders_202402"`
+- `table_suffix = "202402"`
+- `event_time_str` 来自 CDC 事件时间
+- `load_date` 是任务运行时格式化后的日期字符串