This is an automated email from the ASF dual-hosted git repository.
davidzollo pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 8f25d50b57 [Docs][Connector-V2] Improve BigQuery Firestore and GraphQL
docs (#11350)
8f25d50b57 is described below
commit 8f25d50b572741fca9e764667602f6ad891d1f33
Author: Daniel Carter <[email protected]>
AuthorDate: Mon Jul 6 22:55:38 2026 +0800
[Docs][Connector-V2] Improve BigQuery Firestore and GraphQL docs (#11350)
Co-authored-by: DanielCarter-stack
<[email protected]>
---
docs/en/connectors/sink/BigQuery.md | 16 +++++++++++++++-
docs/en/connectors/sink/GoogleFirestore.md | 18 +++++++++++++++++-
docs/en/connectors/sink/GraphQL.md | 2 ++
docs/en/connectors/source/GraphQL.md | 3 ++-
docs/zh/connectors/sink/BigQuery.md | 16 +++++++++++++++-
docs/zh/connectors/sink/GoogleFirestore.md | 17 ++++++++++++++++-
docs/zh/connectors/sink/GraphQL.md | 2 ++
docs/zh/connectors/source/GraphQL.md | 3 ++-
8 files changed, 71 insertions(+), 6 deletions(-)
diff --git a/docs/en/connectors/sink/BigQuery.md
b/docs/en/connectors/sink/BigQuery.md
index 2f92b03d83..ce8430063e 100644
--- a/docs/en/connectors/sink/BigQuery.md
+++ b/docs/en/connectors/sink/BigQuery.md
@@ -64,6 +64,8 @@ The connector writes to one configured table:
`project_id.dataset_id.table_id`.
- `batch`: uses BigQuery buffered write streams and commits data during
SeaTunnel checkpoint/commit. This is the mode covered by the exactly-once
feature mark.
- `streaming`: uses the default stream and writes CDC records with BigQuery
change fields. This mode is suitable for CDC upsert/delete records, but it is
not marked as exactly-once by this connector.
+For CDC writes in `streaming` mode, prepare the target BigQuery table with a
primary key before starting the SeaTunnel job. The connector maps SeaTunnel row
kinds to BigQuery change records: `INSERT` and `UPDATE_AFTER` are written as
`UPSERT`, while `DELETE` and `UPDATE_BEFORE` are written as `DELETE`.
+
#### sequence_number_column
`sequence_number_column` is optional.
@@ -129,6 +131,18 @@ sink {
### CDC Streaming Mode (MySQL to BigQuery)
+The target BigQuery table should already exist and should define the primary
key used by the CDC source. For example:
+
+```sql
+CREATE TABLE `my-gcp-project.cdc_dataset.orders` (
+ uuid INT64 NOT NULL,
+ name STRING,
+ score INT64,
+ PRIMARY KEY (uuid) NOT ENFORCED
+)
+OPTIONS (max_staleness = INTERVAL 0 MINUTE);
+```
+
```hocon
env {
parallelism = 1
@@ -194,7 +208,7 @@ sink {
### Testing
This connector uses the BigQuery Storage Write API. The current local BigQuery
emulator does not fully support the write path used by this connector.
-For now, the connector should be tested against a real BigQuery environment.
+Use `emulator_host` only for local or CI checks that are compatible with the
emulator. Production validation should be done against a real BigQuery
environment.
## Changelog
diff --git a/docs/en/connectors/sink/GoogleFirestore.md
b/docs/en/connectors/sink/GoogleFirestore.md
index e2f1058d72..2b653a75a7 100644
--- a/docs/en/connectors/sink/GoogleFirestore.md
+++ b/docs/en/connectors/sink/GoogleFirestore.md
@@ -4,6 +4,12 @@ import ChangeLog from
'../changelog/connector-google-firestore.md';
> Google Firestore sink connector
+## Support Those Engines
+
+> Spark<br/>
+> Flink<br/>
+> SeaTunnel Zeta<br/>
+
## Description
The GoogleFirestore sink writes SeaTunnel rows to a Google Cloud Firestore
collection.
@@ -22,6 +28,15 @@ indexes in Google Cloud before running queries that need
them.
- [ ] [stream](../../introduction/concepts/connector-v2-features.md)
- [ ] [support multiple table
write](../../introduction/concepts/connector-v2-features.md)
+## Supported DataSource Info
+
+In order to use the GoogleFirestore connector, the following dependency is
required.
+It can be downloaded via install-plugin.sh or from Maven central repository.
+
+| Datasource | Dependency |
+|-----------------|------------|
+| GoogleFirestore |
[Download](https://mvnrepository.com/artifact/org.apache.seatunnel/connector-google-firestore)
|
+
## Options
| name | type | required | default value | description |
@@ -29,7 +44,7 @@ indexes in Google Cloud before running queries that need them.
| project_id | string | yes | - | Google Cloud project ID
that owns the Firestore database. |
| collection | string | yes | - | Firestore collection
name to write to. |
| credentials | string | no | - | Base64-encoded Google
Cloud service account JSON. |
-| common-options | | no | - | Sink common options. |
+| common-options | | no | - | Sink common options.
See [Sink Common Options](../common-options/sink-common-options.md). |
### project_id [string]
@@ -84,6 +99,7 @@ Sink plugin common parameters, please refer to [Sink Common
Options](../common-o
## Notes
- The connector currently provides a sink only. There is no GoogleFirestore
source connector.
+- Each sink block writes to one configured collection. It does not switch
collections automatically for multi-table input.
- Firestore document IDs are generated automatically. Use another connector or
transform before this sink if you need deterministic document IDs.
- The sink does not interpret `UPDATE` or `DELETE` row kinds as CDC operations.
- Do not put raw service account JSON directly in `credentials`; encode it
with Base64 first.
diff --git a/docs/en/connectors/sink/GraphQL.md
b/docs/en/connectors/sink/GraphQL.md
index b0a5329b53..c199a06f13 100644
--- a/docs/en/connectors/sink/GraphQL.md
+++ b/docs/en/connectors/sink/GraphQL.md
@@ -48,6 +48,7 @@ It can be downloaded via install-plugin.sh or from Maven
central repository.
| retry_backoff_max_ms | Int | No | 10000 | Maximum HTTP retry backoff in
milliseconds. |
| connect_timeout_ms | Int | No | 12000 | HTTP connection timeout in
milliseconds. |
| socket_timeout_ms | Int | No | 60000 | HTTP socket timeout in milliseconds. |
+| multi_table_sink_replica | Int | No | - | Sink common option. It controls
sink replica count in multi-table runtime. The same GraphQL mutation is still
used for all rows routed to this sink. |
| common-options | Config | No | - | Sink common options. See [Sink Common
Options](../common-options/sink-common-options.md). |
## Notes
@@ -57,6 +58,7 @@ It can be downloaded via install-plugin.sh or from Maven
central repository.
- If `variables` already contains a key and `valueCover = true`, the
configured variable value is kept.
- If `valueCover = false`, row field values replace variables with the same
name.
- The sink sends one mutation request for each input row. If the upstream
source contains multiple tables, make sure the same mutation and variable names
can handle every table routed to this sink.
+- `multi_table_sink_replica` only affects the multi-table sink runtime replica
count. It does not choose a different GraphQL mutation per table.
## Task Example
diff --git a/docs/en/connectors/source/GraphQL.md
b/docs/en/connectors/source/GraphQL.md
index 4f5f5b61b7..9d404739ee 100644
--- a/docs/en/connectors/source/GraphQL.md
+++ b/docs/en/connectors/source/GraphQL.md
@@ -63,8 +63,9 @@ It can be downloaded via install-plugin.sh or from Maven
central repository.
- In normal query mode, `url` must start with `http://` or `https://`.
- In subscription mode, set `enable_subscription = true` and use a `ws://` or
`wss://` URL.
-- Source queries cannot be GraphQL `mutation` operations.
+- Source queries cannot be GraphQL `mutation` operations. Use the GraphQL sink
for `mutation` requests.
- `content_field` is usually needed because GraphQL responses are commonly
wrapped under `data`.
+- When `schema.fields` is configured, set `format = "json"` and point
`content_field` to the array or object that should become SeaTunnel rows.
- For streaming query mode over HTTP, configure `poll_interval_millis` to
control how often SeaTunnel sends the same query again.
- For WebSocket subscription mode, `max_retries` and `retry_delay_ms` only
control reconnect behavior after a subscription connection fails.
diff --git a/docs/zh/connectors/sink/BigQuery.md
b/docs/zh/connectors/sink/BigQuery.md
index e3e1285a0a..a99cc6288a 100644
--- a/docs/zh/connectors/sink/BigQuery.md
+++ b/docs/zh/connectors/sink/BigQuery.md
@@ -63,6 +63,8 @@ import ChangeLog from '../changelog/connector-bigquery.md';
- `batch`：使用 BigQuery buffered write stream，并在 SeaTunnel checkpoint/commit
阶段提交数据。主要特性中的精确一次能力指的是该模式。
- `streaming`：使用默认 stream，并携带 BigQuery change 字段写入 CDC 记录。该模式适合 CDC 的
upsert/delete 数据，但该连接器没有将它标记为精确一次。
+使用 `streaming` 模式写入 CDC 数据时，请先在 BigQuery 中创建好带 Primary Key 的目标表。连接器会把
SeaTunnel 的行类型转换为 BigQuery change 记录：`INSERT` 和 `UPDATE_AFTER` 会写成
`UPSERT`，`DELETE` 和 `UPDATE_BEFORE` 会写成 `DELETE`。
+
#### sequence_number_column
`sequence_number_column` 是可选配置。
@@ -128,6 +130,18 @@ sink {
### CDC 流式模式（MySQL 到 BigQuery)
+目标 BigQuery 表需要提前创建，并且应定义 CDC 源表使用的主键。例如：
+
+```sql
+CREATE TABLE `my-gcp-project.cdc_dataset.orders` (
+ uuid INT64 NOT NULL,
+ name STRING,
+ score INT64,
+ PRIMARY KEY (uuid) NOT ENFORCED
+)
+OPTIONS (max_staleness = INTERVAL 0 MINUTE);
+```
+
```hocon
env {
parallelism = 1
@@ -193,7 +207,7 @@ sink {
### 测试
该连接器使用 BigQuery Storage Write API。当前本地 BigQuery emulator 不能完整支持该连接器使用的写入路径。
-因此，目前应在真实的 BigQuery 环境中测试该连接器。
+`emulator_host` 只适合用于本地或 CI 中与 emulator 兼容的检查。生产可用性验证应在真实 BigQuery 环境中完成。
## 更新日志
diff --git a/docs/zh/connectors/sink/GoogleFirestore.md
b/docs/zh/connectors/sink/GoogleFirestore.md
index 6447393889..40133e5fb5 100644
--- a/docs/zh/connectors/sink/GoogleFirestore.md
+++ b/docs/zh/connectors/sink/GoogleFirestore.md
@@ -4,6 +4,12 @@ import ChangeLog from
'../changelog/connector-google-firestore.md';
> Google Firestore Sink 连接器
+## 支持的引擎
+
+> Spark<br/>
+> Flink<br/>
+> SeaTunnel Zeta<br/>
+
## 描述
GoogleFirestore Sink 用于将 SeaTunnel 数据写入 Google Cloud Firestore 集合。
@@ -21,6 +27,14 @@ GoogleFirestore Sink 用于将 SeaTunnel 数据写入 Google Cloud
Firestore 集
- [ ] [流处理](../../introduction/concepts/connector-v2-features.md)
- [ ] [支持多表写入](../../introduction/concepts/connector-v2-features.md)
+## 支持的数据源信息
+
+使用 GoogleFirestore 连接器需要安装下面的依赖。可以通过 install-plugin.sh 安装，也可以从 Maven 中央仓库下载。
+
+| 数据源 | 依赖 |
+|-----------------|------|
+| GoogleFirestore |
[下载](https://mvnrepository.com/artifact/org.apache.seatunnel/connector-google-firestore)
|
+
## 选项
| 名称 | 类型 | 必填 | 默认值 | 说明 |
@@ -28,7 +42,7 @@ GoogleFirestore Sink 用于将 SeaTunnel 数据写入 Google Cloud
Firestore 集
| project_id | string | 是 | - | Firestore 数据库所在的 Google Cloud 项目
ID。 |
| collection | string | 是 | - | 要写入的 Firestore 集合名称。 |
| credentials | string | 否 | - | Base64 编码后的 Google Cloud 服务账号 JSON。
|
-| common-options | | 否 | - | Sink 通用选项。 |
+| common-options | | 否 | - | Sink 通用选项，详见 [Sink
通用选项](../common-options/sink-common-options.md)。 |
### project_id [string]
@@ -83,6 +97,7 @@ Sink 插件通用参数，请参考 [Sink 通用选项](../common-options/sink-c
## 注意事项
- 当前连接器只提供 sink，不提供 GoogleFirestore source。
+- 每个 sink 配置块只写入一个固定的 collection，不会按多表输入自动切换 collection。
- Firestore 文档 ID 会自动生成。如果需要固定文档 ID，请在写入前使用其他连接器或转换处理。
- sink 不会按 `UPDATE` 或 `DELETE` 行类型执行 CDC 语义。
- `credentials` 不能直接填写服务账号 JSON 原文，需要先做 Base64 编码。
diff --git a/docs/zh/connectors/sink/GraphQL.md
b/docs/zh/connectors/sink/GraphQL.md
index 9b16ee96f0..7b3d8fed7e 100644
--- a/docs/zh/connectors/sink/GraphQL.md
+++ b/docs/zh/connectors/sink/GraphQL.md
@@ -47,6 +47,7 @@ mutation 一起发送。
| retry_backoff_max_ms | Int | 否 | 10000 | HTTP 请求失败后的最大重试退避时间，单位毫秒。 |
| connect_timeout_ms | Int | 否 | 12000 | HTTP 连接超时时间，单位毫秒。 |
| socket_timeout_ms | Int | 否 | 60000 | HTTP socket 超时时间，单位毫秒。 |
+| multi_table_sink_replica | Int | 否 | - | Sink 通用参数，用于控制多表运行时的 sink 副本数；但写入到该
sink 的所有行仍使用同一条 GraphQL mutation。 |
| common-options | Config | 否 | - | Sink 通用参数，详见 [Sink Common
Options](../common-options/sink-common-options.md)。 |
## 注意事项
@@ -56,6 +57,7 @@ mutation 一起发送。
- 如果 `variables` 已有某个 key，并且设置 `valueCover = true`，会保留配置里的变量值。
- 如果 `valueCover = false`，同名输入行字段会覆盖配置里的变量值。
- Sink 会为每一行输入数据发送一次 mutation 请求。如果上游包含多张表，需要确认同一条 mutation 和变量名可以处理所有写入到该
Sink 的表。
+- `multi_table_sink_replica` 只影响多表 sink 运行时的副本数，不会按不同表自动选择不同的 GraphQL mutation。
## 任务示例
diff --git a/docs/zh/connectors/source/GraphQL.md
b/docs/zh/connectors/source/GraphQL.md
index dbacccc1b9..5d5dfb5257 100644
--- a/docs/zh/connectors/source/GraphQL.md
+++ b/docs/zh/connectors/source/GraphQL.md
@@ -62,8 +62,9 @@ GraphQL 源连接器用于从 GraphQL 服务读取数据。它支持：
- 普通查询模式下，`url` 必须以 `http://` 或 `https://` 开头。
- 订阅模式下，需要设置 `enable_subscription = true`，并使用 `ws://` 或 `wss://` 地址。
-- Source 不支持 GraphQL `mutation` 操作。
+- Source 不支持 GraphQL `mutation` 操作。如需发送 `mutation` 请求，请使用 GraphQL sink。
- GraphQL 响应通常包在 `data` 字段下面，所以一般需要配置 `content_field`。
+- 配置 `schema.fields` 读取结构化数据时，通常需要设置 `format = "json"`，并让 `content_field`
指向要转换为 SeaTunnel 行的数组或对象。
- 使用 HTTP 流式轮询读取时，可以通过 `poll_interval_millis` 控制重复发送同一条查询的间隔。
- 使用 WebSocket 订阅读取时，`max_retries` 和 `retry_delay_ms` 只控制订阅连接失败后的重连行为。
