This is an automated email from the ASF dual-hosted git repository.
davidzollo pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 9f7f73c10d [Docs][Connector-V2] Improve TDengine Lance and Firestore
docs (#11332)
9f7f73c10d is described below
commit 9f7f73c10de4d410871ce91b41f97af97cb24bc9
Author: Daniel Carter <[email protected]>
AuthorDate: Sat Jul 4 21:58:08 2026 +0800
[Docs][Connector-V2] Improve TDengine Lance and Firestore docs (#11332)
Co-authored-by: DanielCarter-stack
<[email protected]>
Co-authored-by: David Zollo <[email protected]>
---
docs/en/connectors/sink/GoogleFirestore.md | 16 ++++++++
docs/en/connectors/sink/Lance.md | 26 ++++++++++--
docs/en/connectors/sink/TDengine.md | 63 +++++++++++++++++++++++++++++-
docs/en/connectors/source/TDengine.md | 54 ++++++++++++++++++++++---
docs/zh/connectors/sink/GoogleFirestore.md | 14 +++++++
docs/zh/connectors/sink/Lance.md | 22 ++++++++---
docs/zh/connectors/sink/TDengine.md | 57 ++++++++++++++++++++++++++-
docs/zh/connectors/source/TDengine.md | 41 +++++++++++++++++--
8 files changed, 274 insertions(+), 19 deletions(-)
diff --git a/docs/en/connectors/sink/GoogleFirestore.md
b/docs/en/connectors/sink/GoogleFirestore.md
index bb4423f68f..e2f1058d72 100644
--- a/docs/en/connectors/sink/GoogleFirestore.md
+++ b/docs/en/connectors/sink/GoogleFirestore.md
@@ -11,6 +11,8 @@ The GoogleFirestore sink writes SeaTunnel rows to a Google
Cloud Firestore colle
Each SeaTunnel row is converted to one Firestore document. The connector
generates the Firestore document ID automatically by calling Firestore `add`,
so it appends documents instead of updating a user-specified document ID.
Credentials can be passed as a Base64-encoded service account JSON string. If
`credentials` is not configured, the connector reads Google Application Default
Credentials from the runtime environment.
+The sink does not create or manage Firestore indexes; create any required
+indexes in Google Cloud before running queries that need them.
## Key Features
@@ -43,6 +45,18 @@ Base64-encoded Google Cloud service account JSON.
If this option is not set, the connector uses Google Application Default
Credentials. In that case, make sure `GOOGLE_APPLICATION_CREDENTIALS` points to
the service account JSON file or the runtime environment already provides
default credentials.
+You can generate the value with:
+
+```bash
+base64 -w 0 service-account.json
+```
+
+On macOS, use:
+
+```bash
+base64 service-account.json | tr -d '\n'
+```
+
### common options
Sink plugin common parameters, please refer to [Sink Common
Options](../common-options/sink-common-options.md) for details.
@@ -73,6 +87,8 @@ Sink plugin common parameters, please refer to [Sink Common
Options](../common-o
- Firestore document IDs are generated automatically. Use another connector or
transform before this sink if you need deterministic document IDs.
- The sink does not interpret `UPDATE` or `DELETE` row kinds as CDC operations.
- Do not put raw service account JSON directly in `credentials`; encode it
with Base64 first.
+- Field names in the upstream SeaTunnel schema become Firestore document field
+ names.
## Example
diff --git a/docs/en/connectors/sink/Lance.md b/docs/en/connectors/sink/Lance.md
index 6629cf6567..f32a6313f3 100644
--- a/docs/en/connectors/sink/Lance.md
+++ b/docs/en/connectors/sink/Lance.md
@@ -20,6 +20,7 @@ import ChangeLog from '../changelog/connector-lance.md';
The Lance sink writes SeaTunnel rows into a Lance dataset. It can create a
Lance table from the incoming SeaTunnel schema and then append or create data
according to the configured Lance write mode.
The connector currently works with directory-based Lance namespaces.
+It provides a sink only; there is no Lance source connector.
## Using Dependency
@@ -41,8 +42,8 @@ The connector currently works with directory-based Lance
namespaces.
| Name | Type | Required | Default |
Description
|
|------------------------------------|---------|----------|---------------|--------------------------------------------------------------------------------------------------|
-| dataset_path | string | yes | /test.lance |
Lance dataset path. For a directory namespace, this is the local path used by
the Lance dataset. |
-| namespace_type | string | yes | dir |
Lance namespace type. Currently only `dir` is supported.
|
+| dataset_path | string | no | /test.lance |
Lance dataset path. For a directory namespace, this is the local path used by
the Lance dataset. |
+| namespace_type | string | no | dir |
Lance namespace type. Currently only `dir` is supported.
|
| namespace_id | string | no | "" |
Lance namespace ID.
|
| namespace_ids | list | no | [] |
Lance namespace path segments used when resolving the target table namespace.
|
| root_namespace_path | string | no | /tmp |
Root path used by the Lance namespace.
|
@@ -63,13 +64,29 @@ The directory or dataset path where Lance data is stored.
In local directory mod
The Lance namespace type. Currently the connector supports `dir`.
+### namespace_id
+
+The namespace name used by the directory namespace implementation. In local
+directory mode, it can be a simple name such as `root`.
+
+### namespace_ids
+
+Additional namespace path segments used when resolving the table namespace.
+Leave it empty when writing directly under the root namespace.
+
+### root_namespace_path
+
+The root directory for the Lance namespace. The runtime user must have
permission
+to create and write files under this path.
+
### table
The target Lance table name. When it is not set, the connector uses the
upstream table name if one is available. The default option value is `test`.
### lance.write.mode
-Controls how Lance writes the dataset. The default is `CREATE`. Use a value
supported by Lance `WriteParams.WriteMode`.
+Controls how Lance writes the dataset. The default is `CREATE`. Use a value
+supported by Lance `WriteParams.WriteMode`: `CREATE`, `APPEND`, or `OVERWRITE`.
### lance.write.storage.options
@@ -95,9 +112,10 @@ Lance uses the Apache Arrow type system. The sink creates
the Lance schema from
| SMALLINT | int32 |
| INT | int32 |
| BIGINT | int32 |
-| FLOAT | float64 |
+| FLOAT | float32 |
| DOUBLE | float64 |
| DECIMAL | decimal128 |
+| NULL | null |
| BYTES | binary |
| DATE | date32 |
| TIME | time32 |
diff --git a/docs/en/connectors/sink/TDengine.md
b/docs/en/connectors/sink/TDengine.md
index e3dc45440c..428d21ace7 100644
--- a/docs/en/connectors/sink/TDengine.md
+++ b/docs/en/connectors/sink/TDengine.md
@@ -12,6 +12,11 @@ Create the target database and super table before running
the SeaTunnel job. The
sink can write one input table to one super table, or use placeholders such as
`${table_name}` in `stable` for multi-table writes.
+The input row must follow TDengine's super table write shape: the first field
is
+the target sub table name, the middle fields are normal columns, and the last
+fields are TAGS values. The connector reads the number of TAGS fields from the
+target super table metadata.
+
## Key features
- [x] [exactly-once](../../introduction/concepts/connector-v2-features.md)
@@ -64,7 +69,11 @@ The TDengine server timezone used for timestamp conversion.
The default value is
`UTC`.
### write_columns [list]
-The field names to be inserted into TDengine. If not set, all fields will be
written. The plugin will automatically append TAGS columns, so please do not
include TAGS columns in this option.
+
+The normal TDengine column names to insert. If it is not set, TDengine uses the
+column order of the target super table. Do not include the first input field
+that contains the sub table name, and do not include TAGS columns; the
connector
+adds TAGS values from the end of the input row automatically.
### common options
@@ -99,6 +108,58 @@ sink {
### Write multiple input tables to matching super tables
```hocon
+source {
+ FakeSource {
+ plugin_output = "fake"
+ tables_configs = [
+ {
+ schema = {
+ table = "meters3"
+ fields {
+ device_id = "string"
+ event_time = "timestamp"
+ metric1 = "float"
+ metric2 = "int"
+ metric3 = "float"
+ status_flag = "boolean"
+ notes = "string"
+ location_tag = "string"
+ group_tag = "int"
+ }
+ }
+ rows = [
+ {
+ kind = INSERT
+ fields = ["d2001", "2023-04-22T14:38:05", 10.3, 219, 0.31, true,
"nc", "California.SanFrancisco", 2]
+ }
+ ]
+ },
+ {
+ schema = {
+ table = "meters4"
+ fields {
+ device_id = "string"
+ event_time = "timestamp"
+ metric1 = "float"
+ metric2 = "int"
+ metric3 = "float"
+ status_flag = "boolean"
+ notes = "string"
+ location_tag = "string"
+ group_tag = "int"
+ }
+ }
+ rows = [
+ {
+ kind = INSERT
+ fields = ["d1005", "2023-04-22T14:38:05", 110.3, 219, 0.31, true,
"nc", "California.SanFrancisco", 2]
+ }
+ ]
+ }
+ ]
+ }
+}
+
sink {
TDengine {
url = "jdbc:TAOS-RS://localhost:6041/"
diff --git a/docs/en/connectors/source/TDengine.md
b/docs/en/connectors/source/TDengine.md
index 9ee6141e57..002bd28731 100644
--- a/docs/en/connectors/source/TDengine.md
+++ b/docs/en/connectors/source/TDengine.md
@@ -12,6 +12,11 @@ The source reads data in batch mode by querying a time range
from one super
table. You can read all sub tables under the super table, limit the read to
specific sub tables, and select only part of the columns.
+Each source split reads one TDengine sub table. The output schema always adds
+`subtable_name` as the first field so that downstream sinks can keep the
+original TDengine sub table name. This is also the field consumed by the
+TDengine sink when writing rows back to TDengine.
+
## Key features
- [x] [batch](../../introduction/concepts/connector-v2-features.md)
@@ -63,23 +68,29 @@ The TDengine database name.
The TDengine super table name.
### sub_tables [list]
+
A list of sub table names. If it is not configured, all sub tables under the
configured super table are read. If it is configured, only the listed sub
tables
are read.
### lower_bound [string]
-The inclusive lower bound of the query time range. Use a TDengine-compatible
-timestamp string, for example `2018-10-03 14:38:05.000`.
+The inclusive lower bound of the query time range. The connector adds
+`timestamp_column >= lower_bound` to each sub table query. Use a
+TDengine-compatible timestamp string, for example `2018-10-03 14:38:05.000`.
### upper_bound [string]
-The upper bound of the query time range. Use a TDengine-compatible timestamp
-string, for example `2018-10-03 14:38:16.801`.
+The exclusive upper bound of the query time range. The connector adds
+`timestamp_column < upper_bound` to each sub table query. Use a
+TDengine-compatible timestamp string, for example `2018-10-03 14:38:16.801`.
### read_columns [list]
+
A list of column names to read. If it is not configured, all columns are read.
-When reading from a super table, put TAGS columns at the end of the list.
+When reading from a super table, put TAGS columns at the end of the list. Do
not
+include `subtable_name`; the connector adds it automatically as the first
output
+field.
## Examples
@@ -123,6 +134,39 @@ source {
}
```
+### Read from TDengine and write to TDengine
+
+```hocon
+env {
+ parallelism = 2
+ job.mode = "BATCH"
+}
+
+source {
+ TDengine {
+ url = "jdbc:TAOS-RS://tdengine-src:6041/"
+ username = "root"
+ password = "taosdata"
+ database = "power"
+ stable = "meters"
+ lower_bound = "2018-10-03 14:38:05.000"
+ upper_bound = "2018-10-03 14:38:16.801"
+ plugin_output = "tdengine_result"
+ }
+}
+
+sink {
+ TDengine {
+ url = "jdbc:TAOS-RS://tdengine-sink:6041/"
+ username = "root"
+ password = "taosdata"
+ database = "power2"
+ stable = "meters2"
+ timezone = "UTC"
+ }
+}
+```
+
## Changelog
<ChangeLog />
diff --git a/docs/zh/connectors/sink/GoogleFirestore.md
b/docs/zh/connectors/sink/GoogleFirestore.md
index 07cd4e0b94..6447393889 100644
--- a/docs/zh/connectors/sink/GoogleFirestore.md
+++ b/docs/zh/connectors/sink/GoogleFirestore.md
@@ -11,6 +11,7 @@ GoogleFirestore Sink 用于将 SeaTunnel 数据写入 Google Cloud
Firestore 集
每一行 SeaTunnel 数据会转换为一个 Firestore 文档。连接器通过 Firestore `add` 自动生成文档
ID，因此它是追加写入，不会按用户指定的文档 ID 更新已有文档。
凭证可以通过 `credentials` 传入 Base64 编码后的服务账号 JSON；如果不配置该参数，则会从运行环境读取 Google 应用默认凭证。
+该 sink 不会创建或管理 Firestore 索引；如果后续查询需要索引，请先在 Google Cloud 中创建。
## 主要特性
@@ -43,6 +44,18 @@ Base64 编码后的 Google Cloud 服务账号 JSON。
如果不配置该参数，连接器会使用 Google 应用默认凭证。此时需要确保 `GOOGLE_APPLICATION_CREDENTIALS` 指向服务账号
JSON 文件，或者运行环境已经提供默认凭证。
+可以用下面的命令生成该配置值：
+
+```bash
+base64 -w 0 service-account.json
+```
+
+macOS 可以使用：
+
+```bash
+base64 service-account.json | tr -d '\n'
+```
+
### 通用选项
Sink 插件通用参数，请参考 [Sink 通用选项](../common-options/sink-common-options.md)。
@@ -73,6 +86,7 @@ Sink 插件通用参数，请参考 [Sink 通用选项](../common-options/sink-c
- Firestore 文档 ID 会自动生成。如果需要固定文档 ID，请在写入前使用其他连接器或转换处理。
- sink 不会按 `UPDATE` 或 `DELETE` 行类型执行 CDC 语义。
- `credentials` 不能直接填写服务账号 JSON 原文，需要先做 Base64 编码。
+- 上游 SeaTunnel schema 中的字段名会作为 Firestore 文档字段名。
## 示例
diff --git a/docs/zh/connectors/sink/Lance.md b/docs/zh/connectors/sink/Lance.md
index 664764a8b3..15350b0905 100644
--- a/docs/zh/connectors/sink/Lance.md
+++ b/docs/zh/connectors/sink/Lance.md
@@ -20,7 +20,6 @@ import ChangeLog from '../changelog/connector-lance.md';
Lance sink 用于把 SeaTunnel 数据写入 Lance 数据集。它可以根据上游 SeaTunnel 表结构创建 Lance 表，并按配置的
Lance 写入模式创建或追加数据。
当前连接器支持基于目录的 Lance namespace。
-
## 依赖
```xml
@@ -41,8 +40,8 @@ Lance sink 用于把 SeaTunnel 数据写入 Lance 数据集。它可以根据上
| 名称 | 类型 | 是否必填 | 默认值 | 说明
|
|------------------------------------|-----------|----------|----------------|------------------------------------------------------|
-| dataset_path | string | 是 | /test.lance |
Lance 数据集路径。目录 namespace 下通常是本地数据路径。 |
-| namespace_type | string | 是 | dir |
Lance namespace 类型。当前仅支持 `dir`。 |
+| dataset_path | string | 否 | /test.lance |
Lance 数据集路径。目录 namespace 下通常是本地数据路径。 |
+| namespace_type | string | 否 | dir |
Lance namespace 类型。当前仅支持 `dir`。 |
| namespace_id | string | 否 | "" |
Lance namespace ID。 |
| namespace_ids | list | 否 | [] |
解析目标表 namespace 时使用的 namespace 路径片段。 |
| root_namespace_path | string | 否 | /tmp |
Lance namespace 的根路径。 |
@@ -63,13 +62,25 @@ Lance 数据的目录或数据集路径。使用本地目录模式时，请确
Lance namespace 类型。当前连接器支持 `dir`。
+### namespace_id
+
+目录 namespace 实现使用的 namespace 名称。本地目录模式下可以填写类似 `root` 的简单名称。
+
+### namespace_ids
+
+解析目标表 namespace 时使用的额外路径片段。如果直接写入根 namespace，可以保持为空。
+
+### root_namespace_path
+
+Lance namespace 的根目录。SeaTunnel 运行用户需要有权限在该目录下创建和写入文件。
+
### table
目标 Lance 表名。不设置时，如果上游存在表名，连接器会使用上游表名；该配置本身的默认值是 `test`。
### lance.write.mode
-控制 Lance 的写入方式。默认值是 `CREATE`。该值需要是 Lance `WriteParams.WriteMode` 支持的值。
+控制 Lance 的写入方式。默认值是 `CREATE`。该值需要是 Lance `WriteParams.WriteMode`
支持的值：`CREATE`、`APPEND` 或 `OVERWRITE`。
### lance.write.storage.options
@@ -95,9 +106,10 @@ Lance 使用 Apache Arrow 类型系统。sink 会根据上游 SeaTunnel 表结
| SMALLINT | int32 |
| INT | int32 |
| BIGINT | int32 |
-| FLOAT | float64 |
+| FLOAT | float32 |
| DOUBLE | float64 |
| DECIMAL | decimal128 |
+| NULL | null |
| BYTES | binary |
| DATE | date32 |
| TIME | time32 |
diff --git a/docs/zh/connectors/sink/TDengine.md
b/docs/zh/connectors/sink/TDengine.md
index 5bfe32ee45..e9083a58aa 100644
--- a/docs/zh/connectors/sink/TDengine.md
+++ b/docs/zh/connectors/sink/TDengine.md
@@ -10,6 +10,8 @@ import ChangeLog from '../changelog/connector-tdengine.md';
运行 SeaTunnel 任务前，需要先创建目标数据库和超级表。该 Sink 支持单表写入，也支持在 `stable` 中使用
`${table_name}` 这类占位符完成多表写入。
+输入数据需要符合 TDengine 超级表写入结构：第一列是目标子表名，中间是普通列，最后几列是 TAGS 值。连接器会读取目标超级表元数据来判断末尾有几列
TAGS。
+
## 主要特性
- [x] [精确一次](../../introduction/concepts/connector-v2-features.md)
@@ -60,7 +62,8 @@ TDengine 超级表名称。多表写入时可以使用占位符，例如 `${tabl
TDengine 服务端时区，用于时间戳转换，默认值为 `UTC`。
### write_columns [list]
-要写入 TDengine 的字段列表。不配置时写入所有字段。无需包含 TAGS 字段，插件会自动处理 TAGS 字段写入。
+
+要写入 TDengine 的普通列名列表。不配置时，TDengine 会按目标超级表的列顺序写入。这里不要包含第一列子表名，也不要包含 TAGS
字段；连接器会自动从输入数据末尾取出 TAGS 值。
### 通用选项
@@ -93,6 +96,58 @@ sink {
### 多表写入匹配的超级表
```hocon
+source {
+ FakeSource {
+ plugin_output = "fake"
+ tables_configs = [
+ {
+ schema = {
+ table = "meters3"
+ fields {
+ device_id = "string"
+ event_time = "timestamp"
+ metric1 = "float"
+ metric2 = "int"
+ metric3 = "float"
+ status_flag = "boolean"
+ notes = "string"
+ location_tag = "string"
+ group_tag = "int"
+ }
+ }
+ rows = [
+ {
+ kind = INSERT
+ fields = ["d2001", "2023-04-22T14:38:05", 10.3, 219, 0.31, true,
"nc", "California.SanFrancisco", 2]
+ }
+ ]
+ },
+ {
+ schema = {
+ table = "meters4"
+ fields {
+ device_id = "string"
+ event_time = "timestamp"
+ metric1 = "float"
+ metric2 = "int"
+ metric3 = "float"
+ status_flag = "boolean"
+ notes = "string"
+ location_tag = "string"
+ group_tag = "int"
+ }
+ }
+ rows = [
+ {
+ kind = INSERT
+ fields = ["d1005", "2023-04-22T14:38:05", 110.3, 219, 0.31, true,
"nc", "California.SanFrancisco", 2]
+ }
+ ]
+ }
+ ]
+ }
+}
+
sink {
TDengine {
url = "jdbc:TAOS-RS://localhost:6041/"
diff --git a/docs/zh/connectors/source/TDengine.md
b/docs/zh/connectors/source/TDengine.md
index 85b96cd71e..9afbe14d18 100644
--- a/docs/zh/connectors/source/TDengine.md
+++ b/docs/zh/connectors/source/TDengine.md
@@ -10,6 +10,8 @@ import ChangeLog from '../changelog/connector-tdengine.md';
该 Source 以批处理方式按时间范围读取一个超级表的数据。可以读取该超级表下的所有子表，也可以只读取指定子表；同时支持只读取部分列。
+每个 source 分片会读取一个 TDengine 子表。输出字段会自动把 `subtable_name`
放在第一列，用来保留原始子表名；如果下游继续写入 TDengine，TDengine sink 也会使用这一列作为目标子表名。
+
## 主要特性
- [x] [批处理](../../introduction/concepts/connector-v2-features.md)
@@ -66,15 +68,15 @@ TDengine 子表名称列表。不配置时读取该超级表下的所有子表
### lower_bound [string]
-查询时间范围的下界，使用 TDengine 可识别的时间字符串，例如 `2018-10-03 14:38:05.000`。
+查询时间范围的下界，包含该时间点。连接器会为每个子表查询加上 `timestamp_column >= lower_bound` 条件。请使用
TDengine 可识别的时间字符串，例如 `2018-10-03 14:38:05.000`。
### upper_bound [string]
-查询时间范围的上界，使用 TDengine 可识别的时间字符串，例如 `2018-10-03 14:38:16.801`。
+查询时间范围的上界，不包含该时间点。连接器会为每个子表查询加上 `timestamp_column < upper_bound` 条件。请使用
TDengine 可识别的时间字符串，例如 `2018-10-03 14:38:16.801`。
### read_columns [list]
-要读取的列名列表。不配置时读取所有列。读取超级表时，请将 TAGS 字段放在列表末尾。
+要读取的列名列表。不配置时读取所有列。读取超级表时，请将 TAGS 字段放在列表末尾。不要在这里配置
`subtable_name`，连接器会自动把它作为第一列输出。
## 示例
@@ -100,6 +102,39 @@ source {
}
```
+### 从 TDengine 读取并写入 TDengine
+
+```hocon
+env {
+ parallelism = 2
+ job.mode = "BATCH"
+}
+
+source {
+ TDengine {
+ url = "jdbc:TAOS-RS://tdengine-src:6041/"
+ username = "root"
+ password = "taosdata"
+ database = "power"
+ stable = "meters"
+ lower_bound = "2018-10-03 14:38:05.000"
+ upper_bound = "2018-10-03 14:38:16.801"
+ plugin_output = "tdengine_result"
+ }
+}
+
+sink {
+ TDengine {
+ url = "jdbc:TAOS-RS://tdengine-sink:6041/"
+ username = "root"
+ password = "taosdata"
+ database = "power2"
+ stable = "meters2"
+ timezone = "UTC"
+ }
+}
+```
+
### 读取指定子表和指定列
```hocon
