This is an automated email from the ASF dual-hosted git repository.
davidzollo pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new da366cc1fa [Docs][Connector-V2] Improve Google Bigtable connector docs
(#11371)
da366cc1fa is described below
commit da366cc1fa85cc2b61d014596879e2b2a2b82bf2
Author: Daniel Carter <[email protected]>
AuthorDate: Tue Jul 7 17:51:19 2026 +0800
[Docs][Connector-V2] Improve Google Bigtable connector docs (#11371)
---
docs/en/connectors/sink/GoogleBigtable.md | 46 +++++++++++
docs/en/connectors/source/GoogleBigtable.md | 36 ++++++++-
docs/zh/connectors/sink/GoogleBigtable.md | 114 ++++++++++++++++++++++++++-
docs/zh/connectors/source/GoogleBigtable.md | 118 +++++++++++++++++++++++++++-
4 files changed, 307 insertions(+), 7 deletions(-)
diff --git a/docs/en/connectors/sink/GoogleBigtable.md
b/docs/en/connectors/sink/GoogleBigtable.md
index 55b96cf398..a57905e2bc 100644
--- a/docs/en/connectors/sink/GoogleBigtable.md
+++ b/docs/en/connectors/sink/GoogleBigtable.md
@@ -12,6 +12,7 @@ Writes data to Google Cloud Bigtable using the native
Bigtable Data v2 Java clie
- [ ] [exactly-once](../../introduction/concepts/connector-v2-features.md)
- [x] [batch](../../introduction/concepts/connector-v2-features.md)
+- [x] [support multiple table
write](../../introduction/concepts/connector-v2-features.md)
## Options
@@ -27,6 +28,9 @@ Writes data to Google Cloud Bigtable using the native
Bigtable Data v2 Java clie
| version_column | string | no | - |
| null_mode | string | no | skip |
| batch_mutation_size | int | no | 100 |
+| schema_save_mode | enum | no | RECREATE_SCHEMA |
+| data_save_mode | enum | no | APPEND_DATA |
+| multi_table_sink_replica | int | no | - |
| common-options | | no | - |
### project_id [string]
@@ -91,6 +95,22 @@ How to handle `null` field values. Supported: `skip`
(default), `empty`.
Number of row mutations to accumulate before sending a BulkMutation to
Bigtable. Default is `100`. Increase for higher throughput at the cost of
higher per-task memory usage.
+### schema_save_mode [enum]
+
+Schema save mode. Only `RECREATE_SCHEMA` is supported now.
+
+The connector does not create Bigtable tables or column families. Create the
target table and all column families before the job starts.
+
+### data_save_mode [enum]
+
+Data save mode. Only `APPEND_DATA` is supported now.
+
+`DROP_DATA` and `ERROR_WHEN_DATA_EXISTS` are not implemented for this
connector. If you need a clean target, truncate or recreate the Bigtable table
before running the job.
+
+### multi_table_sink_replica [int]
+
+The number of sink replicas used for multi-table writing. For details, see
[Sink Common Options](../common-options/sink-common-options.md).
+
### common options
Sink plugin common parameters, please refer to [Sink Common
Options](../common-options/sink-common-options.md) for details.
@@ -115,6 +135,12 @@ All SeaTunnel types are supported:
| TIME | UTF-8 `HH:mm:ss` |
| TIMESTAMP | UTF-8 `yyyy-MM-dd HH:mm:ss` |
+:::tip
+
+Bigtable does not have relational columns. The sink writes every non-row-key
field as a Bigtable cell. The target column family is selected by
`column_family`; the Bigtable qualifier is the SeaTunnel field name.
+
+:::
+
## Example
### Basic — Application Default Credentials
@@ -171,6 +197,26 @@ sink {
}
```
+### Use a version column and empty null values
+
+```hocon
+sink {
+ GoogleBigtable {
+ project_id = "my-gcp-project"
+ instance_id = "my-bigtable-instance"
+ table = "events"
+ rowkey_column = ["tenant_id", "event_id"]
+ rowkey_delimiter = "#"
+ version_column = "event_ts"
+ null_mode = "empty"
+ column_family {
+ all_columns = "data"
+ event_type = "meta"
+ }
+ }
+}
+```
+
## Changelog
<ChangeLog />
diff --git a/docs/en/connectors/source/GoogleBigtable.md
b/docs/en/connectors/source/GoogleBigtable.md
index b511aa38c7..64e4d2d39c 100644
--- a/docs/en/connectors/source/GoogleBigtable.md
+++ b/docs/en/connectors/source/GoogleBigtable.md
@@ -15,6 +15,12 @@ Reads data from Google Cloud Bigtable using the native
Bigtable Data v2 Java cli
- [ ] [exactly-once](../../introduction/concepts/connector-v2-features.md)
- [ ] [parallelism](../../introduction/concepts/connector-v2-features.md)
+:::tip
+
+The source is bounded. It creates one split for the configured table or
row-key range, so increasing job parallelism does not split one Bigtable scan
into multiple tablet-range reads.
+
+:::
+
## Options
| name | type | required | default value |
@@ -50,7 +56,7 @@ Path to the Google Cloud service account JSON key file. If
omitted, Application
### rowkey_column [list]
-Optional list of field names that should receive the row key value. Declare a
field named `rowkey` in your schema to capture the raw row key bytes as a
`BYTES` or `STRING` field.
+Optional list of field names that should receive the row key value. If this
option is not set, the connector uses a schema field named `rowkey` as the
row-key field. The row-key field can be declared as `BYTES` or `STRING`.
### start_rowkey [string]
@@ -82,7 +88,7 @@ Source plugin common parameters, please refer to [Source
Common Options](../comm
## Schema Mapping
-Field names in the SeaTunnel schema must follow the pattern
`familyName:qualifier`, for example `cf:name` or `stats:age`. The special field
name `rowkey` maps to the Bigtable row key.
+Field names in the SeaTunnel schema must follow the pattern
`familyName:qualifier`, for example `cf:name` or `stats:age`. The row-key field
is controlled by `rowkey_column`; if it is not configured, the special field
name `rowkey` maps to the Bigtable row key.
| Schema field name | Mapped Bigtable cell |
|-------------------|-----------------------------|
@@ -90,6 +96,12 @@ Field names in the SeaTunnel schema must follow the pattern
`familyName:qualifie
| `cf:name` | Column family `cf`, qualifier `name` |
| `stats:age` | Column family `stats`, qualifier `age` |
+:::tip
+
+The source reads the latest returned cell for each `family:qualifier` field.
Use `start_timestamp`, `end_timestamp`, and `max_versions` to control the
Bigtable scan filter. SeaTunnel field types must match the bytes stored in
Bigtable. For example, numeric values written by this connector are binary
big-endian values, while `STRING`, `DATE`, `TIME`, `TIMESTAMP`, and `DECIMAL`
are UTF-8 text.
+
+:::
+
## Example
### Read all rows — Application Default Credentials
@@ -134,6 +146,26 @@ source {
}
```
+### Use a custom row-key field name
+
+```hocon
+source {
+ GoogleBigtable {
+ project_id = "my-gcp-project"
+ instance_id = "my-bigtable-instance"
+ table = "events"
+ rowkey_column = ["event_id"]
+ schema {
+ fields {
+ event_id = STRING
+ "cf:type" = STRING
+ "cf:data" = STRING
+ }
+ }
+ }
+}
+```
+
## Changelog
<ChangeLog />
diff --git a/docs/zh/connectors/sink/GoogleBigtable.md
b/docs/zh/connectors/sink/GoogleBigtable.md
index 65fd4f927d..4d05bc81be 100644
--- a/docs/zh/connectors/sink/GoogleBigtable.md
+++ b/docs/zh/connectors/sink/GoogleBigtable.md
@@ -11,7 +11,8 @@ import ChangeLog from
'../changelog/connector-google-bigtable.md';
## 主要特性
- [ ] [exactly-once](../../introduction/concepts/connector-v2-features.md)
-- [x] [batch](../../introduction/concepts/connector-v2-features.md)
+- [x] [批处理](../../introduction/concepts/connector-v2-features.md)
+- [x] [支持多表写入](../../introduction/concepts/connector-v2-features.md)
## 参数
@@ -27,6 +28,9 @@ import ChangeLog from
'../changelog/connector-google-bigtable.md';
| version_column | string | 否 | - |
| null_mode | string | 否 | skip |
| batch_mutation_size| int | 否 | 100 |
+| schema_save_mode | enum | 否 | RECREATE_SCHEMA |
+| data_save_mode | enum | 否 | APPEND_DATA |
+| multi_table_sink_replica | int | 否 | - |
| common-options | | 否 | - |
### project_id [string]
@@ -82,10 +86,58 @@ Google Cloud 服务账号 JSON 密钥文件路径。未设置时使用应用默
### batch_mutation_size [int]
-每次批量提交的行数，默认 `100`。
+每次批量提交的行数，默认 `100`。调大该值可以提高吞吐，但会增加每个任务本地缓存的数据量。
+
+### schema_save_mode [enum]
+
+Schema 保存模式。当前只支持 `RECREATE_SCHEMA`。
+
+连接器不会自动创建 Bigtable 表或列族。运行作业前，需要先在 Bigtable 中创建好目标表和所有会用到的列族。
+
+### data_save_mode [enum]
+
+数据保存模式。当前只支持 `APPEND_DATA`。
+
+本连接器暂不支持 `DROP_DATA` 和 `ERROR_WHEN_DATA_EXISTS`。如果需要干净的目标表，请在运行作业前手动清空或重建
Bigtable 表。
+
+### multi_table_sink_replica [int]
+
+多表写入时使用的 Sink 副本数。更多说明请参考 [Sink Common
Options](../common-options/sink-common-options.md)。
+
+### common options
+
+Sink 插件通用参数，详见 [Sink Common Options](../common-options/sink-common-options.md)。
+
+## 数据类型
+
+Bigtable 没有关系型数据库那样的列类型。连接器会按如下格式把 SeaTunnel 字段写入 Bigtable Cell：
+
+| SeaTunnel 类型 | Bigtable 中的保存格式 |
+|----------------|----------------------|
+| TINYINT | 1 字节二进制 |
+| SMALLINT | 2 字节大端二进制 |
+| INT | 4 字节大端二进制 |
+| BIGINT | 8 字节大端二进制 |
+| FLOAT | 4 字节 IEEE 754 大端二进制 |
+| DOUBLE | 8 字节 IEEE 754 大端二进制 |
+| BOOLEAN | 1 字节，`1` 表示 true，`0` 表示 false |
+| BYTES | 原始字节 |
+| STRING | UTF-8 文本 |
+| DECIMAL | UTF-8 普通数字字符串 |
+| DATE | UTF-8 `yyyy-MM-dd` |
+| TIME | UTF-8 `HH:mm:ss` |
+| TIMESTAMP | UTF-8 `yyyy-MM-dd HH:mm:ss` |
+
+:::tip
+
+Sink 会把非行键字段写成 Bigtable Cell。目标列族由 `column_family` 决定，Bigtable 的列限定符使用
SeaTunnel 字段名。
+
+:::
## 示例
+### 使用应用默认凭证写入
+
```hocon
sink {
GoogleBigtable {
@@ -100,6 +152,64 @@ sink {
}
```
+### 使用服务账号和复合行键写入
+
+```hocon
+sink {
+ GoogleBigtable {
+ project_id = "my-gcp-project"
+ instance_id = "my-bigtable-instance"
+ table = "events"
+ credentials_path = "/secrets/sa-key.json"
+ rowkey_column = ["tenant_id", "event_id"]
+ rowkey_delimiter = "#"
+ column_family {
+ all_columns = "data"
+ }
+ batch_mutation_size = 500
+ }
+}
+```
+
+### 写入多个列族
+
+```hocon
+sink {
+ GoogleBigtable {
+ project_id = "my-gcp-project"
+ instance_id = "my-bigtable-instance"
+ table = "user_profile"
+ rowkey_column = ["user_id"]
+ column_family {
+ name = "identity"
+ email = "identity"
+ age = "stats"
+ last_login = "stats"
+ }
+ }
+}
+```
+
+### 使用版本列并把空值写成空字节
+
+```hocon
+sink {
+ GoogleBigtable {
+ project_id = "my-gcp-project"
+ instance_id = "my-bigtable-instance"
+ table = "events"
+ rowkey_column = ["tenant_id", "event_id"]
+ rowkey_delimiter = "#"
+ version_column = "event_ts"
+ null_mode = "empty"
+ column_family {
+ all_columns = "data"
+ event_type = "meta"
+ }
+ }
+}
+```
+
## Changelog
<ChangeLog />
diff --git a/docs/zh/connectors/source/GoogleBigtable.md
b/docs/zh/connectors/source/GoogleBigtable.md
index ea1b4e8190..b690d4496b 100644
--- a/docs/zh/connectors/source/GoogleBigtable.md
+++ b/docs/zh/connectors/source/GoogleBigtable.md
@@ -10,9 +10,16 @@ import ChangeLog from
'../changelog/connector-google-bigtable.md';
## 主要特性
-- [x] [batch](../../introduction/concepts/connector-v2-features.md)
-- [ ] [parallelism](../../introduction/concepts/connector-v2-features.md)
+- [x] [批处理](../../introduction/concepts/connector-v2-features.md)
+- [ ] [流处理](../../introduction/concepts/connector-v2-features.md)
- [ ] [exactly-once](../../introduction/concepts/connector-v2-features.md)
+- [ ] [并行度](../../introduction/concepts/connector-v2-features.md)
+
+:::tip
+
+该 Source 是有界读取。当前只会为配置的表或行键范围生成一个切分，所以提高作业并行度不会把一次 Bigtable 扫描拆成多个 tablet
范围并发读取。
+
+:::
## 参数
@@ -31,12 +38,74 @@ import ChangeLog from
'../changelog/connector-google-bigtable.md';
| scan_row_limit | int | 否 | -1 |
| common-options | | 否 | - |
+### project_id [string]
+
+Google Cloud 项目 ID。
+
+### instance_id [string]
+
+Bigtable 实例 ID。
+
+### table [string]
+
+要读取的 Bigtable 表名。
+
+### credentials_path [string]
+
+Google Cloud 服务账号 JSON 密钥文件路径。未设置时使用应用默认凭证（ADC）。
+
+### rowkey_column [list]
+
+用于接收 Bigtable 行键的字段名列表。未设置时，连接器默认把名为 `rowkey` 的字段当作行键字段。行键字段可以声明为 `BYTES` 或
`STRING`。
+
+### start_rowkey [string]
+
+扫描起始行键，包含该行键。未设置时从表起始位置读取。
+
+### end_rowkey [string]
+
+扫描结束行键，不包含该行键。未设置时读取到表末尾。
+
+### start_timestamp [long]
+
+Cell 时间戳过滤的起始值，包含该时间戳，单位是微秒。
+
+### end_timestamp [long]
+
+Cell 时间戳过滤的结束值，不包含该时间戳，单位是微秒。
+
+### max_versions [int]
+
+每个列限定符最多返回的 Cell 版本数。默认值 `1` 表示只读取最新版本。
+
+### scan_row_limit [int]
+
+最多读取的行数。默认值 `-1` 表示不限制。
+
+### common options
+
+Source 插件通用参数，详见 [Source Common
Options](../common-options/source-common-options.md)。
+
### Schema 映射
-字段名须使用 `列族:列限定符` 格式，例如 `cf:name`、`stats:age`。特殊字段名 `rowkey` 映射到行键。
+字段名须使用 `列族:列限定符` 格式，例如 `cf:name`、`stats:age`。行键字段由 `rowkey_column`
控制；如果未配置，特殊字段名 `rowkey` 会映射到 Bigtable 行键。
+
+| SeaTunnel 字段名 | 映射到 Bigtable |
+|-----------------|----------------|
+| `rowkey` | 行键 |
+| `cf:name` | 列族 `cf`，列限定符 `name` |
+| `stats:age` | 列族 `stats`，列限定符 `age` |
+
+:::tip
+
+Source 会读取每个 `列族:列限定符` 字段返回的最新 Cell。可以通过 `start_timestamp`、`end_timestamp` 和
`max_versions` 控制 Bigtable 扫描过滤条件。SeaTunnel 字段类型需要和 Bigtable
中保存的字节格式匹配，例如本连接器写入的数字类型是大端二进制，`STRING`、`DATE`、`TIME`、`TIMESTAMP` 和 `DECIMAL` 是
UTF-8 文本。
+
+:::
## 示例
+### 使用应用默认凭证读取整张表
+
```hocon
source {
GoogleBigtable {
@@ -54,6 +123,49 @@ source {
}
```
+### 使用服务账号扫描行键范围
+
+```hocon
+source {
+ GoogleBigtable {
+ project_id = "my-gcp-project"
+ instance_id = "my-bigtable-instance"
+ table = "events"
+ credentials_path = "/secrets/sa-key.json"
+ start_rowkey = "2024-01-01#"
+ end_rowkey = "2024-02-01#"
+ max_versions = 1
+ schema {
+ fields {
+ rowkey = STRING
+ "cf:type" = STRING
+ "cf:data" = STRING
+ }
+ }
+ }
+}
+```
+
+### 使用自定义行键字段名
+
+```hocon
+source {
+ GoogleBigtable {
+ project_id = "my-gcp-project"
+ instance_id = "my-bigtable-instance"
+ table = "events"
+ rowkey_column = ["event_id"]
+ schema {
+ fields {
+ event_id = STRING
+ "cf:type" = STRING
+ "cf:data" = STRING
+ }
+ }
+ }
+}
+```
+
## Changelog
<ChangeLog />
