This is an automated email from the ASF dual-hosted git repository.
davidzollo pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 9cd939231b [Docs][Connector-V2] Improve Aerospike BigQuery and Druid
docs (#11297)
9cd939231b is described below
commit 9cd939231b242b8996af227406233daefa8270aa
Author: Daniel Carter <[email protected]>
AuthorDate: Sat Jul 4 00:17:55 2026 +0800
[Docs][Connector-V2] Improve Aerospike BigQuery and Druid docs (#11297)
Co-authored-by: DanielCarter-stack
<[email protected]>
---
docs/en/connectors/sink/Aerospike.md | 30 ++++++++++--------
docs/en/connectors/sink/BigQuery.md | 50 +++++++++++++++++++++---------
docs/en/connectors/sink/Druid.md | 16 ++++++----
docs/zh/connectors/sink/Aerospike.md | 32 +++++++++++--------
docs/zh/connectors/sink/BigQuery.md | 60 ++++++++++++++++++++++++++----------
docs/zh/connectors/sink/Druid.md | 16 ++++++----
6 files changed, 136 insertions(+), 68 deletions(-)
diff --git a/docs/en/connectors/sink/Aerospike.md
b/docs/en/connectors/sink/Aerospike.md
index 4d8eb63242..95e4dac912 100644
--- a/docs/en/connectors/sink/Aerospike.md
+++ b/docs/en/connectors/sink/Aerospike.md
@@ -23,7 +23,7 @@ When using this connector, you need to comply with AGPL 3.0
license terms.
## Description
-Sink connector for Aerospike database.
+Sink connector for Aerospike database. The connector writes records to one
Aerospike namespace and set. It uses the configured `key` field as the
Aerospike record key.
## Supported DataSource Info
@@ -54,17 +54,17 @@ Note:
| Name | Type | Required | Default | Description
|
|----------------|--------|----------|---------|-----------------------------------------------------------------------------|
-| host | string | Yes | - | Aerospike server hostname or
IP address |
-| port | int | No | 3000 | Aerospike server port
|
-| namespace | string | Yes | - | Namespace in Aerospike
|
-| set | string | Yes | - | Set name in Aerospike
|
-| username | string | No | - | Username for authentication
|
-| password | string | No | - | Password for authentication
|
-| key | string | Yes | - | Field name to use as
Aerospike primary key |
-| bin_name | string | No | - | Bin name for storing data.
Required when `data_format` is `map` or `string` |
-| data_format | string | No | string | Data storage format:
map/string/kv |
-| write_timeout | int | No | 200 | Write operation timeout in
milliseconds |
-| schema.field | map | No | {} | Field type mappings (e.g.
{"name":"STRING","age":"INTEGER"}) |
+| host | string | Yes | - | Aerospike server hostname or
IP address. |
+| port | int | No | 3000 | Aerospike server port.
|
+| namespace | string | Yes | - | Aerospike namespace.
|
+| set | string | Yes | - | Aerospike set name.
|
+| username | string | No | - | Username for authentication.
Leave unset when authentication is disabled. |
+| password | string | No | - | Password for authentication.
Leave unset when authentication is disabled. |
+| key | string | Yes | - | SeaTunnel field used as the
Aerospike record key. |
+| bin_name | string | No | - | Aerospike bin used by `map`
and `string` formats. Required for those formats. |
+| data_format | string | No | string | Storage format. Supported
values are `map`, `string`, and `kv`. |
+| write_timeout | int | No | 200 | Write operation timeout in
milliseconds. |
+| schema.field | map | No | {} | Field-to-Aerospike type
mapping. For example: `c_id = "INTEGER"`. |
### data_format Options
@@ -72,6 +72,12 @@ Note:
- **string**: Store all non-key fields as a JSON string in `bin_name`
- **kv**: Store each non-key field as a separate bin. `bin_name` is not used
+When `data_format` is `map` or `string`, configure `bin_name`; otherwise the
writer does not know which Aerospike bin should receive the packed row data.
+
+### schema.field
+
+`schema.field` is optional. Configure it when you need to control the
Aerospike type used for each field. Supported Aerospike type names include
`STRING`, `INTEGER`, `LONG`, `DOUBLE`, `BOOLEAN`, `BYTEARRAY`, and `LIST`.
+
## Task Example
### Write FakeSource Data To Aerospike
diff --git a/docs/en/connectors/sink/BigQuery.md
b/docs/en/connectors/sink/BigQuery.md
index 7018bcc09b..44ba8c02d2 100644
--- a/docs/en/connectors/sink/BigQuery.md
+++ b/docs/en/connectors/sink/BigQuery.md
@@ -39,10 +39,12 @@ Sink connector for Google Cloud BigQuery using the Storage
Write API for high-pe
| write_mode | string | No | batch | Write mode.
Supported values: `batch` and `streaming`
|
| sequence_number_column | string | No | - | Column name
used as sequence number for CDC deduplication. Only applicable when
`write_mode` is `streaming` |
| batch_size | int | No | 1000 | Number of rows
to batch before sending to BigQuery
|
+| emulator_host | string | No | - | BigQuery
emulator host, such as `localhost:9050`. This option is intended for tests
only. |
+| common-options | | No | - | Sink common
options. See [Sink Common Options](../common-options/sink-common-options.md).
|
### Authentication Options
-You must provide **one** of the following authentication methods:
+For production BigQuery jobs, provide **one** of the following authentication
methods. Authentication is skipped only when `emulator_host` is configured for
tests.
1. **service_account_key_path**: Path to service account JSON file
2. **service_account_key_json**: Inline JSON key content
@@ -53,6 +55,8 @@ You must provide **one** of the following authentication
methods:
The target BigQuery table must already exist.
The connector reads the existing table schema during writer initialization and
does not create the table automatically.
+The connector writes to one configured table:
`project_id.dataset_id.table_id`. For multi-table pipelines, configure separate
sink entries or route data before the BigQuery sink.
+
#### sequence_number_column
`sequence_number_column` is optional.
@@ -64,25 +68,42 @@ If `sequence_number_column` is not configured,
`_CHANGE_SEQUENCE_NUMBER` is not
> - The `sequence_number_column` should reference a monotonically increasing
> column in your source table (e.g., `updated_at` as epoch millis, `version`,
> or `seq_id`). The column value must be of a type convertible to `long`.
> - To enable BigQuery-side deduplication in streaming mode, the target
> BigQuery table must have a Primary Key defined. Otherwise, BigQuery will
> treat every write as an append operation, regardless of the sequence number.
+### emulator_host
+
+`emulator_host` is only for local or CI tests. When it is configured,
SeaTunnel connects to the emulator without Google credentials. Do not use this
option for production BigQuery jobs.
+
## Task Example
-### Simple Example (Using Service Account File)
+### Simple Batch Example
```hocon
env {
- parallelism = 2
+ parallelism = 1
job.mode = "BATCH"
}
source {
FakeSource {
- row.num = 1000
+ row.num = 10
+ string.fake.mode = "template"
+ string.template = ["key", "value"]
schema = {
fields {
- user_id = "bigint"
- username = "string"
- email = "string"
- created_at = "timestamp"
+ c_map = "map<string, string>"
+ c_array = "array<int>"
+ c_string = string
+ c_boolean = boolean
+ c_tinyint = tinyint
+ c_smallint = smallint
+ c_int = int
+ c_bigint = bigint
+ c_float = float
+ c_double = double
+ c_decimal = "decimal(30, 8)"
+ c_bytes = bytes
+ c_date = date
+ c_timestamp = timestamp
+ c_time = time
}
}
}
@@ -90,11 +111,11 @@ source {
sink {
BigQuery {
- project_id = "my-gcp-project"
- dataset_id = "analytics"
- table_id = "user_events"
- service_account_key_path = "/path/to/key.json"
- batch_size = 1000
+ project_id = "test-project"
+ dataset_id = "test_dataset"
+ table_id = "test_table"
+ batch_size = 2
+ emulator_host = "localhost:9050"
}
}
```
@@ -126,7 +147,6 @@ sink {
table_id = "orders"
service_account_key_path = "/path/to/key.json"
write_mode = "streaming"
- sequence_number_column = "updated_at"
batch_size = 500
}
}
@@ -171,4 +191,4 @@ For now, the connector should be tested against a real
BigQuery environment.
## Changelog
-<ChangeLog />
\ No newline at end of file
+<ChangeLog />
diff --git a/docs/en/connectors/sink/Druid.md b/docs/en/connectors/sink/Druid.md
index ce85fd4309..d6c59049db 100644
--- a/docs/en/connectors/sink/Druid.md
+++ b/docs/en/connectors/sink/Druid.md
@@ -11,7 +11,9 @@ Write data to Apache Druid through the Druid indexing task
API.
## Key features
- [ ] [exactly-once](../../introduction/concepts/connector-v2-features.md)
+- [ ] [cdc](../../introduction/concepts/connector-v2-features.md)
- [x] [support multiple table
write](../../introduction/concepts/connector-v2-features.md)
+- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md)
## Data Type Mapping
@@ -30,12 +32,12 @@ Write data to Apache Druid through the Druid indexing task
API.
## Options
-| name | type | required | default value |
-|----------------|--------|----------|---------------|
-| coordinatorUrl | string | yes | - |
-| datasource | string | yes | - |
-| batchSize | int | no | 10000 |
-| common-options | | no | - |
+| name | type | required | default value | description |
+|----------------|--------|----------|---------------|-------------|
+| coordinatorUrl | string | yes | - | Druid coordinator or
router host and port. |
+| datasource | string | yes | - | Druid datasource name.
Supports placeholders such as `${table_name}`. |
+| batchSize | int | no | 10000 | Number of rows buffered
before one indexing task is submitted. |
+| common-options | | no | - | Sink common options. |
### coordinatorUrl [string]
@@ -67,6 +69,8 @@ The connector converts each SeaTunnel row into inline CSV
data and submits it to
The connector adds a processing-time column named `timestamp` to satisfy
Druid's primary timestamp requirement. This generated timestamp is used by
Druid ingestion; source `TIMESTAMP` fields are written as string dimensions
according to the mapping above.
+Rows are flushed when the buffered row count reaches `batchSize`. Remaining
rows are also flushed when the writer closes. There is no time-based periodic
flush option.
+
## Example
### Write One Table
diff --git a/docs/zh/connectors/sink/Aerospike.md
b/docs/zh/connectors/sink/Aerospike.md
index cf5c545413..fb42b00e6a 100644
--- a/docs/zh/connectors/sink/Aerospike.md
+++ b/docs/zh/connectors/sink/Aerospike.md
@@ -19,11 +19,11 @@ import ChangeLog from '../changelog/connector-aerospike.md';
- [ ] [精确一次](../../introduction/concepts/connector-v2-features.md)
- [ ] [CDC](../../introduction/concepts/connector-v2-features.md)
-- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md)
+- [ ] [定时刷新](../../introduction/concepts/connector-v2-features.md)
## 描述
-用于向 Aerospike 数据库写入数据的连接器。
+用于向 Aerospike 数据库写入数据的连接器。该连接器会把数据写入一个指定的 Aerospike 命名空间和集合，并使用 `key` 指定的字段作为
Aerospike 记录主键。
## 支持的数据源
@@ -54,17 +54,17 @@ import ChangeLog from '../changelog/connector-aerospike.md';
| 参数名称 | 类型 | 必填 | 默认值 | 说明
|
|----------------|---------|------|---------|---------------------------------------------------------------------|
-| host | string | 是 | - | Aerospike 服务器主机名或IP地址
|
-| port | int | 否 | 3000 | Aerospike 服务器端口
|
-| namespace | string | 是 | - | Aerospike 命名空间
|
-| set | string | 是 | - | Aerospike 集合名称
|
-| username | string | 否 | - | 认证用户名
|
-| password | string | 否 | - | 认证密码
|
-| key | string | 是 | - | 用作 Aerospike 主键的字段名称
|
-| bin_name | string | 否 | - | 数据存储的 bin 名称。`data_format` 为
`map` 或 `string` 时需要配置 |
-| data_format | string | 否 | string | 数据存储格式：map/string/kv
|
-| write_timeout | int | 否 | 200 | 写入操作超时时间（毫秒）
|
-| schema.field | map | 否 | {} |
字段类型映射（示例：{"name":"STRING","age":"INTEGER"}） |
+| host | string | 是 | - | Aerospike 服务器主机名或 IP 地址。
|
+| port | int | 否 | 3000 | Aerospike 服务器端口。
|
+| namespace | string | 是 | - | Aerospike 命名空间。
|
+| set | string | 是 | - | Aerospike 集合名称。
|
+| username | string | 否 | - | 认证用户名。未开启认证时可以不配置。
|
+| password | string | 否 | - | 认证密码。未开启认证时可以不配置。
|
+| key | string | 是 | - | 用作 Aerospike 记录主键的 SeaTunnel
字段名称。 |
+| bin_name | string | 否 | - | `map` 和 `string` 格式使用的 Aerospike
bin 名称，这两种格式下必填。 |
+| data_format | string | 否 | string | 数据存储格式，支持 `map`、`string`、`kv`。
|
+| write_timeout | int | 否 | 200 | 写入操作超时时间，单位毫秒。
|
+| schema.field | map | 否 | {} | 字段到 Aerospike 类型的映射，例如：`c_id =
"INTEGER"`。 |
### data_format 选项说明
@@ -72,6 +72,12 @@ import ChangeLog from '../changelog/connector-aerospike.md';
- **string**: 将所有非主键字段作为 JSON 字符串存到 `bin_name`
- **kv**: 每个非主键字段存储为独立的 bin，此时不使用 `bin_name`
+当 `data_format` 为 `map` 或 `string` 时，必须配置 `bin_name`，否则写入器无法判断打包后的整行数据应该写入哪个
Aerospike bin。
+
+### schema.field 配置说明
+
+`schema.field` 是可选配置。需要明确控制每个字段写入 Aerospike 时的类型时再配置。支持的 Aerospike 类型名称包括
`STRING`、`INTEGER`、`LONG`、`DOUBLE`、`BOOLEAN`、`BYTEARRAY`、`LIST`。
+
## 任务示例
### 将 FakeSource 数据写入 Aerospike
diff --git a/docs/zh/connectors/sink/BigQuery.md
b/docs/zh/connectors/sink/BigQuery.md
index 70d299821a..849b022ba2 100644
--- a/docs/zh/connectors/sink/BigQuery.md
+++ b/docs/zh/connectors/sink/BigQuery.md
@@ -13,8 +13,8 @@ import ChangeLog from '../changelog/connector-bigquery.md';
## 主要特性
- [x] [精确一次](../../introduction/concepts/connector-v2-features.md) 仅适用于 batch
模式
-- [x] [cdc](../../introduction/concepts/connector-v2-features.md)
-- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md)
+- [x] [CDC](../../introduction/concepts/connector-v2-features.md)
+- [ ] [定时刷新](../../introduction/concepts/connector-v2-features.md)
## 描述
@@ -38,12 +38,24 @@ import ChangeLog from '../changelog/connector-bigquery.md';
| write_mode | string | 否 | batch | 写入模式。支持的值：`batch`
和 `streaming`
|
| sequence_number_column | string | 否 | - | 用于 CDC
去重的序列号列名。仅在 `write_mode` 为 `streaming` 时适用
|
| batch_size | int | 否 | 1000 | 发送到 BigQuery
之前批量处理的行数
|
+| emulator_host | string | 否 | - | BigQuery emulator
地址，例如 `localhost:9050`。该参数仅用于测试。
|
+| common-options | | 否 | - | Sink 通用参数，详见
[Sink Common Options](../common-options/sink-common-options.md)。
|
+
+### 认证参数
+
+生产 BigQuery 任务必须使用下面任意一种认证方式。只有配置 `emulator_host` 做测试时才会跳过认证。
+
+1. **service_account_key_path**：服务账号 JSON 密钥文件路径。
+2. **service_account_key_json**：直接填写服务账号 JSON 密钥内容。
+3. **默认凭据**：如果前两项都不配置，则使用 Google Application Default Credentials。
### 表选项
目标 BigQuery 表必须已经存在。
连接器会在 writer 初始化时读取已有的表 schema，并且不会自动创建 BigQuery 表。
+该连接器会写入一个固定的目标表：`project_id.dataset_id.table_id`。如果任务里有多张表，请配置多个 BigQuery
sink，或者在写入 BigQuery 前先完成表路由。
+
#### sequence_number_column
`sequence_number_column` 是可选配置。
@@ -55,25 +67,42 @@ import ChangeLog from '../changelog/connector-bigquery.md';
> - `sequence_number_column` 应该引用 source 表中单调递增的列，例如以 epoch millis 表示的
> `updated_at`、`version` 或 `seq_id`。该列的值必须能够转换为 `long` 类型。
> - 如果要在 streaming 模式下启用 BigQuery 侧的去重，目标 BigQuery 表必须定义 Primary Key。否则，无论是否配置
> sequence number，BigQuery 都会将每次写入视为 append 操作。
+### emulator_host
+
+`emulator_host` 只用于本地测试或 CI 测试。配置该参数后，SeaTunnel 会无凭据连接 BigQuery
emulator。生产任务不要使用该参数。
+
## 任务示例
-### 简单示例 (使用服务账号文件)
+### 简单批处理示例
```hocon
env {
- parallelism = 2
+ parallelism = 1
job.mode = "BATCH"
}
source {
FakeSource {
- row.num = 1000
+ row.num = 10
+ string.fake.mode = "template"
+ string.template = ["key", "value"]
schema = {
fields {
- user_id = "bigint"
- username = "string"
- email = "string"
- created_at = "timestamp"
+ c_map = "map<string, string>"
+ c_array = "array<int>"
+ c_string = string
+ c_boolean = boolean
+ c_tinyint = tinyint
+ c_smallint = smallint
+ c_int = int
+ c_bigint = bigint
+ c_float = float
+ c_double = double
+ c_decimal = "decimal(30, 8)"
+ c_bytes = bytes
+ c_date = date
+ c_timestamp = timestamp
+ c_time = time
}
}
}
@@ -81,11 +110,11 @@ source {
sink {
BigQuery {
- project_id = "my-gcp-project"
- dataset_id = "analytics"
- table_id = "user_events"
- service_account_key_path = "/path/to/key.json"
- batch_size = 1000
+ project_id = "test-project"
+ dataset_id = "test_dataset"
+ table_id = "test_table"
+ batch_size = 2
+ emulator_host = "localhost:9050"
}
}
```
@@ -117,7 +146,6 @@ sink {
table_id = "orders"
service_account_key_path = "/path/to/key.json"
write_mode = "streaming"
- sequence_number_column = "updated_at"
batch_size = 500
}
}
@@ -162,4 +190,4 @@ sink {
## 更新日志
-<ChangeLog />
\ No newline at end of file
+<ChangeLog />
diff --git a/docs/zh/connectors/sink/Druid.md b/docs/zh/connectors/sink/Druid.md
index baa7353d0c..08f9ec45a6 100644
--- a/docs/zh/connectors/sink/Druid.md
+++ b/docs/zh/connectors/sink/Druid.md
@@ -11,7 +11,9 @@ import ChangeLog from '../changelog/connector-druid.md';
## 主要特性
- [ ] [精确一次](../../introduction/concepts/connector-v2-features.md)
+- [ ] [CDC](../../introduction/concepts/connector-v2-features.md)
- [x] [支持多表写入](../../introduction/concepts/connector-v2-features.md)
+- [ ] [定时刷新](../../introduction/concepts/connector-v2-features.md)
## 数据类型映射
@@ -30,12 +32,12 @@ import ChangeLog from '../changelog/connector-druid.md';
## 选项
-| 名称 | 类型 | 必需 | 默认值 |
-|----------------|--------|------|--------|
-| coordinatorUrl | string | 是 | - |
-| datasource | string | 是 | - |
-| batchSize | int | 否 | 10000 |
-| common-options | | 否 | - |
+| 名称 | 类型 | 必需 | 默认值 | 说明 |
+|----------------|--------|------|--------|------|
+| coordinatorUrl | string | 是 | - | Druid 协调器或路由节点的主机和端口。 |
+| datasource | string | 是 | - | Druid datasource 名称，支持
`${table_name}` 这类占位符。 |
+| batchSize | int | 否 | 10000 | 缓存多少行后提交一次索引任务。 |
+| common-options | | 否 | - | Sink 通用参数。 |
### coordinatorUrl [string]
@@ -67,6 +69,8 @@ Sink 插件通用参数，详见 [Sink Common Options](../common-options/sink-co
Druid 写入需要主时间列。连接器会自动追加一个名为 `timestamp` 的处理时间列给 Druid 使用；上游的 `TIMESTAMP`
字段会按上面的类型映射写成字符串维度。
+当缓存行数达到 `batchSize` 时会触发一次写入；写入器关闭时，也会把剩余缓存数据提交到 Druid。当前没有按时间间隔定期刷新的配置。
+
## 示例
### 写入单表
