This is an automated email from the ASF dual-hosted git repository.
zhangshenghang pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 40631a150f [Docs][Connector-V2] Improve Lance connector docs (#11262)
40631a150f is described below
commit 40631a150fb98618602189adce74233ef59ee3e6
Author: Daniel Carter <[email protected]>
AuthorDate: Fri Jul 3 08:14:48 2026 +0800
[Docs][Connector-V2] Improve Lance connector docs (#11262)
Co-authored-by: DanielCarter-stack
<[email protected]>
Co-authored-by: Shenghang <[email protected]>
---
docs/en/connectors/sink/Lance.md | 141 ++++++++++++++++++++++++---------------
docs/zh/connectors/sink/Lance.md | 137 ++++++++++++++++++++++---------------
2 files changed, 172 insertions(+), 106 deletions(-)
diff --git a/docs/en/connectors/sink/Lance.md b/docs/en/connectors/sink/Lance.md
index fc8793f836..6629cf6567 100644
--- a/docs/en/connectors/sink/Lance.md
+++ b/docs/en/connectors/sink/Lance.md
@@ -4,77 +4,116 @@ import ChangeLog from '../changelog/connector-lance.md';
> Lance sink connector
-## Support Those Engines
+## Support These Engines
-> Spark(not support version under spark 3.4, reference
https://lance.org/integrations/spark/install/#scala)<br/>
-> Flink(not support, reference
https://github.com/lance-format/lance-flink)<br/>
+> Spark 3.4 and later<br/>
+> Flink is not supported by this connector<br/>
> SeaTunnel Zeta<br/>
-## Description
+## Key Features
+
+- [ ] [exactly-once](../../introduction/concepts/connector-v2-features.md)
+- [x] [support multiple table
write](../../introduction/concepts/connector-v2-features.md)
-Sink connector for Lance format. It can support create and write dataset
、lance namespace manage schema and version.
+## Description
-## Key features
+The Lance sink writes SeaTunnel rows into a Lance dataset. It can create a
Lance table from the incoming SeaTunnel schema and then append or create data
according to the configured Lance write mode.
-- [] [exactly-once](../../introduction/concepts/connector-v2-features.md)
+The connector currently works with directory-based Lance namespaces.
## Using Dependency
- <dependency>
- <groupId>com.lancedb</groupId>
- <artifactId>lance-core</artifactId>
- <version>0.33.0</version>
- </dependency>
-
- <dependency>
- <groupId>com.lancedb</groupId>
- <artifactId>lance-namespace-core</artifactId>
- <version>0.0.14</version>
- </dependency>
+
+```xml
+<dependency>
+ <groupId>com.lancedb</groupId>
+ <artifactId>lance-core</artifactId>
+ <version>0.33.0</version>
+</dependency>
+
+<dependency>
+ <groupId>com.lancedb</groupId>
+ <artifactId>lance-namespace-core</artifactId>
+ <version>0.0.14</version>
+</dependency>
+```
## Sink Options
-| Name | Type | Required | Default | Description
|
-|-----------------|--------|----------|---------|-------------------------------------------------------------------------------------------------------------------|
-| dataset_path | string | yes | /tmp | The dataset path for the
Lance sink connection.
|
-| namespace_type | string | yes | dir | The namespace type of Lance
dataset, now only support DirectoryNamespace, the type will be set default with
"dir" |
-| table | string | yes | test | The name of Lance dataset,
If not set, the dataset name will be set default with test
|
-| namespace_id | string | no | - | The id of the lance
namespace. Please refer to https://lance.org/format/namespace/
|
+| Name | Type | Required | Default |
Description
|
+|------------------------------------|---------|----------|---------------|--------------------------------------------------------------------------------------------------|
+| dataset_path | string | yes | /test.lance |
Lance dataset path. For a directory namespace, this is the local path used by
the Lance dataset. |
+| namespace_type | string | yes | dir |
Lance namespace type. Currently only `dir` is supported.
|
+| namespace_id | string | no | "" |
Lance namespace ID.
|
+| namespace_ids | list | no | [] |
Lance namespace path segments used when resolving the target table namespace.
|
+| root_namespace_path | string | no | /tmp |
Root path used by the Lance namespace.
|
+| table | string | no | test |
Target Lance table name. If set, it overrides the upstream table name.
|
+| lance.write.max-rows-per-file | int | no | 10 |
Maximum number of rows written to one Lance file.
|
+| lance.write.max-rows-per-group | int | no | 20 |
Maximum number of rows written to one Lance row group.
|
+| lance.write.max-bytes-per-file | long | no | 20480 |
Maximum number of bytes written to one Lance file.
|
+| lance.write.mode | string | no | CREATE |
Lance write mode. The value is passed to Lance `WriteParams.WriteMode`.
|
+| lance.write.enable.stable.row.ids | boolean | no | true |
Whether to enable stable row IDs when writing Lance data.
|
+| lance.write.storage.options | map | no | {} |
Extra storage options passed to Lance.
|
+| multi_table_sink_replica | int | no | 1 |
Parallel replica count for multi-table sink writing.
|
+
+### dataset_path
+
+The directory or dataset path where Lance data is stored. In local directory
mode, make sure the SeaTunnel runtime can create and write to this path.
+
+### namespace_type
+
+The Lance namespace type. Currently the connector supports `dir`.
+
+### table
+
+The target Lance table name. When it is not set, the connector uses the
upstream table name if one is available. The default option value is `test`.
+
+### lance.write.mode
+Controls how Lance writes the dataset. The default is `CREATE`. Use a value
supported by Lance `WriteParams.WriteMode`.
+
+### lance.write.storage.options
+
+Passes extra storage parameters to Lance as key-value pairs.
+
+Example:
+
+```hocon
+lance.write.storage.options = {
+ key1 = "value1"
+ key2 = "value2"
+}
+```
## Data Type Mapping
-The data type of lance depends on the Arrow data type system
-
-| SeaTunnel Data type | Lance Data type |
-|---------------------|-----------------|
-| BOOLEAN | bool/boolean |
-| TINYINT | int8 |
-| SMALLINT | int16 |
-| INT | int32 |
-| BIGINT | int64 |
-| FLOAT | float16 |
-| DOUBLE | float32 |
-| BYTES | binary |
-| DATE | DATE |
-| TIME | TIME |
-| TIMESTAMP | TIMESTAMP |
-| STRING | string/utf8 |
+Lance uses the Apache Arrow type system. The sink creates the Lance schema
from the incoming SeaTunnel schema.
+
+| SeaTunnel Data Type | Lance / Arrow Data Type |
+|---------------------|-------------------------|
+| BOOLEAN | bool |
+| TINYINT | int32 |
+| SMALLINT | int32 |
+| INT | int32 |
+| BIGINT | int32 |
+| FLOAT | float64 |
+| DOUBLE | float64 |
+| DECIMAL | decimal128 |
+| BYTES | binary |
+| DATE | date32 |
+| TIME | time32 |
+| TIMESTAMP | timestamp |
+| STRING | utf8 |
+| ARRAY | list |
+| MAP | map |
## Task Example
-### Simple
+### Write FakeSource Data To Lance
```hocon
env {
parallelism = 1
job.mode = "BATCH"
-
- # You can set spark configuration here
- spark.app.name = "SeaTunnel"
- spark.executor.instances = 2
- spark.executor.cores = 1
- spark.executor.memory = "1g"
- spark.master = local
}
source {
@@ -100,9 +139,6 @@ source {
}
}
-transform {
-}
-
sink {
Lance {
dataset_path = "/tmp/seatunnel_mnt/lanceTest/lance_sink_table"
@@ -111,9 +147,8 @@ sink {
table = "lance_sink_table"
}
}
-
```
## Changelog
-<ChangeLog />
\ No newline at end of file
+<ChangeLog />
diff --git a/docs/zh/connectors/sink/Lance.md b/docs/zh/connectors/sink/Lance.md
index abb8caac40..664764a8b3 100644
--- a/docs/zh/connectors/sink/Lance.md
+++ b/docs/zh/connectors/sink/Lance.md
@@ -6,78 +6,114 @@ import ChangeLog from '../changelog/connector-lance.md';
## 支持的引擎
-> Spark（不支持 Spark 3.4 以下版本，参考
https://lance.org/integrations/spark/install/#scala）<br/>
-> Flink（暂不支持，参考 https://github.com/lance-format/lance-flink)<br/>
+> Spark 3.4 及以上版本<br/>
+> Flink 暂不支持<br/>
> SeaTunnel Zeta<br/>
-## 描述
+## 主要特性
-Lance 格式的 Sink 连接器。支持创建和写入数据集、Lance 命名空间管理 schema 和版本。
+- [ ] [精确一次](../../introduction/concepts/connector-v2-features.md)
+- [x] [支持多表写入](../../introduction/concepts/connector-v2-features.md)
-## 主要特性
+## 描述
-- [] [精确一次语义](../../introduction/concepts/connector-v2-features.md)
-- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md)
+Lance sink 用于把 SeaTunnel 数据写入 Lance 数据集。它可以根据上游 SeaTunnel 表结构创建 Lance 表，并按配置的
Lance 写入模式创建或追加数据。
-## 依赖
+当前连接器支持基于目录的 Lance namespace。
- <dependency>
- <groupId>com.lancedb</groupId>
- <artifactId>lance-core</artifactId>
- <version>0.33.0</version>
- </dependency>
+## 依赖
- <dependency>
- <groupId>com.lancedb</groupId>
- <artifactId>lance-namespace-core</artifactId>
- <version>0.0.14</version>
- </dependency>
+```xml
+<dependency>
+ <groupId>com.lancedb</groupId>
+ <artifactId>lance-core</artifactId>
+ <version>0.33.0</version>
+</dependency>
+
+<dependency>
+ <groupId>com.lancedb</groupId>
+ <artifactId>lance-namespace-core</artifactId>
+ <version>0.0.14</version>
+</dependency>
+```
## Sink 配置项
-| Name | Type | Required | Default | Description
|
-|-----------------|--------|----------|---------|---------------------------------------------------------|
-| dataset_path | string | yes | /tmp | Lance sink 连接的数据集路径 .
|
-| namespace_type | string | yes | dir | Lance 数据集的命名空间类型，目前仅支持
DirectoryNamespace，类型默认为 "dir" |
-| table | string | yes | test | Lance 数据集的名称，如果未设置，数据集名称默认为
test |
-| namespace_id | string | no | - | Lance 命名空间的 ID。请参考
https://lance.org/format/namespace/ |
+| 名称 | 类型 | 是否必填 | 默认值 | 说明
|
+|------------------------------------|-----------|----------|----------------|------------------------------------------------------|
+| dataset_path | string | 是 | /test.lance |
Lance 数据集路径。目录 namespace 下通常是本地数据路径。 |
+| namespace_type | string | 是 | dir |
Lance namespace 类型。当前仅支持 `dir`。 |
+| namespace_id | string | 否 | "" |
Lance namespace ID。 |
+| namespace_ids | list | 否 | [] |
解析目标表 namespace 时使用的 namespace 路径片段。 |
+| root_namespace_path | string | 否 | /tmp |
Lance namespace 的根路径。 |
+| table | string | 否 | test |
目标 Lance 表名。设置后会覆盖上游表名。 |
+| lance.write.max-rows-per-file | int | 否 | 10 |
单个 Lance 文件最多写入的行数。 |
+| lance.write.max-rows-per-group | int | 否 | 20 |
单个 Lance row group 最多写入的行数。 |
+| lance.write.max-bytes-per-file | long | 否 | 20480 |
单个 Lance 文件最多写入的字节数。 |
+| lance.write.mode | string | 否 | CREATE |
Lance 写入模式，会传给 Lance `WriteParams.WriteMode`。 |
+| lance.write.enable.stable.row.ids | boolean | 否 | true |
写入 Lance 时是否启用稳定 row ID。 |
+| lance.write.storage.options | map | 否 | {} |
传给 Lance 的额外存储参数。 |
+| multi_table_sink_replica | int | 否 | 1 |
多表写入时的 sink 并行副本数。 |
+### dataset_path
-## 数据类型映射
+Lance 数据的目录或数据集路径。使用本地目录模式时，请确保 SeaTunnel 运行环境有权限创建并写入该路径。
+
+### namespace_type
+
+Lance namespace 类型。当前连接器支持 `dir`。
+
+### table
+
+目标 Lance 表名。不设置时，如果上游存在表名，连接器会使用上游表名；该配置本身的默认值是 `test`。
+
+### lance.write.mode
-Lance 的数据类型依赖于 Arrow 数据类型系统
+控制 Lance 的写入方式。默认值是 `CREATE`。该值需要是 Lance `WriteParams.WriteMode` 支持的值。
-| Seatunnel数据类型 | Lance 数据类型 |
-|---------------|--------------|
-| BOOLEAN | bool/boolean |
-| TINYINT | int8 |
-| SMALLINT | int16 |
-| INT | int32 |
-| BIGINT | int64 |
-| FLOAT | float16 |
-| DOUBLE | float32 |
-| BYTES | binary |
-| DATE | DATE |
-| TIME | TIME |
-| TIMESTAMP | TIMESTAMP |
-| STRING | string/utf8 |
+### lance.write.storage.options
+以键值对形式传递额外的 Lance 存储参数。
+
+示例：
+
+```hocon
+lance.write.storage.options = {
+ key1 = "value1"
+ key2 = "value2"
+}
+```
+
+## 数据类型映射
+
+Lance 使用 Apache Arrow 类型系统。sink 会根据上游 SeaTunnel 表结构创建 Lance schema。
+
+| SeaTunnel 数据类型 | Lance / Arrow 数据类型 |
+|--------------------|------------------------|
+| BOOLEAN | bool |
+| TINYINT | int32 |
+| SMALLINT | int32 |
+| INT | int32 |
+| BIGINT | int32 |
+| FLOAT | float64 |
+| DOUBLE | float64 |
+| DECIMAL | decimal128 |
+| BYTES | binary |
+| DATE | date32 |
+| TIME | time32 |
+| TIMESTAMP | timestamp |
+| STRING | utf8 |
+| ARRAY | list |
+| MAP | map |
## 任务示例
-### 简单示例
+### 写入 FakeSource 数据到 Lance
```hocon
env {
parallelism = 1
job.mode = "BATCH"
-
- # 可以在这里设置 Spark 配置
- spark.app.name = "SeaTunnel"
- spark.executor.instances = 2
- spark.executor.cores = 1
- spark.executor.memory = "1g"
- spark.master = local
}
source {
@@ -103,9 +139,6 @@ source {
}
}
-transform {
-}
-
sink {
Lance {
dataset_path = "/tmp/seatunnel_mnt/lanceTest/lance_sink_table"
@@ -114,10 +147,8 @@ sink {
table = "lance_sink_table"
}
}
-
```
## 更新日志
<ChangeLog />
-
