This is an automated email from the ASF dual-hosted git repository.
zhangstar333 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 5ed2e001934 [doc](lance) update some doc about create lance catalog
(#4127)
5ed2e001934 is described below
commit 5ed2e001934d6ad7b77399f2ba59f91de07e0c87
Author: zhangstar333 <[email protected]>
AuthorDate: Fri Sep 11 10:19:43 2026 +0800
[doc](lance) update some doc about create lance catalog (#4127)
## Versions
- [ ] dev
- [ ] 4.x
- [ ] 3.x
- [ ] 2.1 or older (not covered by version/language sync gate)
## Languages
- [ ] Chinese
- [ ] English
## Docs Checklist
- [ ] Checked by AI
- [ ] Test Cases Built
- [ ] Updated required version and language counterparts, or explained
why not
- [ ] If only one language changed, confirmed whether source/translation
counterparts need sync
---
.../lakehouse/catalogs/lance-catalog.mdx | 217 ++++++++++++++++++---
.../lakehouse/catalogs/lance-catalog.mdx | 217 ++++++++++++++++++---
2 files changed, 380 insertions(+), 54 deletions(-)
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
index 72f31f51a4c..ced3c5faffa 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
@@ -99,31 +99,88 @@ Filesystem Catalog 直接从 Warehouse 目录发现 Lance Namespace 和表。
|---|---|---|
| `warehouse` | 是 | Lance Warehouse 根路径。支持本地绝对路径、`file://` URI、`s3://` URI 和
`oss://` URI。 |
-根据 Warehouse 所在的存储系统选择对应示例：
+两个参数的作用不同：
+
+- `fs.s3.support=true` 显式选择 S3 配置解析。访问 AWS S3 时通常可以通过属性自动识别；访问 MinIO、自定义
endpoint 或通过 S3 兼容接口访问 OSS 时，应显式设置，避免识别失败或根据域名选择其他存储类型。下面所有 S3 示例都设置此属性，便于直接使用。
+- `use_path_style` 控制请求地址中 Bucket 的位置，默认值为 `false`。设置为 `true` 时使用
`endpoint/bucket/对象路径`；设置为 `false` 时使用虚拟主机方式。对于当前 Lance 默认 S3 客户端，显式配置的虚拟主机
endpoint 必须包含 Bucket。该参数应与服务支持的访问方式及 endpoint 配套设置。
+
+| 访问场景 | 存储类型设置 | `use_path_style` | endpoint 写法 |
+|---|---|---|---|
+| AWS S3 | 建议设置 `fs.s3.support=true` | `false`，可省略 | 带 Bucket，如
`https://my-bucket.s3.us-east-1.amazonaws.com`。 |
+| MinIO（通过服务域名或 IP 访问） | 设置 `fs.s3.support=true` | 显式设置 `true` | 服务地址，不带
Bucket，如 `http://minio.example.com:9000`。 |
+| OSS 的 S3 兼容接口 | 设置 `fs.s3.support=true` | `false`，示例显式设置 | 带 Bucket，如
`https://my-bucket.oss-cn-beijing.aliyuncs.com`。 |
+| OSS 原生接口 | 建议设置 `fs.oss.support=true`，不设置 `fs.s3.support` | 无需设置通用
`use_path_style` | 使用 `oss.endpoint`，不带 Bucket，如
`https://oss-cn-beijing.aliyuncs.com`。 |
+
+以上 S3 endpoint 规则针对 Lance 默认 S3 访问链路。对于单一存储类型的 Filesystem Catalog，只设置对应的
`fs.*.support=true`；不要为了“开启兼容”同时设置多个存储类型。该属性选择配置解析逻辑，不会为存储服务增加协议支持。
+
+根据 Warehouse 所在的存储系统选择对应示例，替换 Bucket、路径、地址和凭证：
<Tabs groupId="lance-filesystem-storage">
- <TabItem value="s3" label="S3 / S3 兼容存储" default>
+ <TabItem value="s3" label="AWS S3" default>
```sql
CREATE CATALOG lance_fs_s3 PROPERTIES (
"type" = "lance",
"lance.catalog.type" = "filesystem",
"warehouse" = "s3://my-bucket/lance",
- "s3.endpoint" = "https://s3.us-east-1.amazonaws.com";,
+ "fs.s3.support" = "true",
+ "use_path_style" = "false",
+ "s3.endpoint" = "https://my-bucket.s3.us-east-1.amazonaws.com";,
"s3.region" = "us-east-1",
"s3.access_key" = "<ak>",
"s3.secret_key" = "<sk>"
);
```
- 访问 AWS S3 时可以省略 `s3.endpoint`。访问 MinIO 等 S3 兼容存储时，应设置对应的
`s3.endpoint`；如果服务要求 Path Style，还需设置 `"use_path_style" = "true"`。
+ 本例使用带 Bucket 的 AWS S3 endpoint，配合 `use_path_style=false`。当前 Doris 的 S3
属性校验要求 endpoint；使用这里的 `s3://my-bucket/...` 地址时，不能只设置 region 而省略 endpoint。替换
Bucket 或 region 时，请同步修改 endpoint。
+ </TabItem>
+
+ <TabItem value="minio" label="MinIO（Path Style）">
+ ```sql
+ CREATE CATALOG lance_fs_minio PROPERTIES (
+ "type" = "lance",
+ "lance.catalog.type" = "filesystem",
+ "warehouse" = "s3://my-bucket/lance",
+ "fs.s3.support" = "true",
+ "s3.endpoint" = "http://minio.example.com:9000";,
+ "s3.region" = "us-east-1",
+ "use_path_style" = "true",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>"
+ );
+ ```
+
+ 本例通过 MinIO 的服务域名访问，使用 Path Style，无需为每个 Bucket 配置子域名。endpoint 不带
Bucket；将 region 替换为服务实际配置值。启用 TLS 时，将地址替换为对应的 `https://` endpoint，并确保 FE 和 BE
都能访问该地址。
+ </TabItem>
+
+ <TabItem value="oss-s3" label="阿里云 OSS（S3 兼容访问）">
+ 通过 S3 兼容接口访问 OSS 时，使用 `s3://` Warehouse 和 `s3.*` 连接属性：
+
+ ```sql
+ CREATE CATALOG lance_fs_oss_s3 PROPERTIES (
+ "type" = "lance",
+ "lance.catalog.type" = "filesystem",
+ "warehouse" = "s3://my-bucket/lance",
+ "fs.s3.support" = "true",
+ "s3.endpoint" = "https://my-bucket.oss-cn-beijing.aliyuncs.com";,
+ "s3.region" = "cn-beijing",
+ "use_path_style" = "false",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>"
+ );
+ ```
+
+ 本例设置 `use_path_style=false`，使用 Virtual Hosted Style。当前 Lance 的默认 S3
客户端不会自动把 Bucket 拼接到显式配置的 endpoint，因此 `s3.endpoint` 必须包含 Bucket，例如
`https://my-bucket.oss-cn-beijing.aliyuncs.com`；`warehouse` 仍使用
`s3://my-bucket/lance`。使用内网访问时，可替换为
`https://my-bucket.oss-cn-beijing-internal.aliyuncs.com`，并确保 FE 和 BE 都能访问该地址。
+
+ 使用 STS 临时凭证时，增加 `"s3.session_token" = "<token>"`。该方式通过 S3 兼容接口读取 OSS
中的原有数据，无需迁移数据；使用 OSS 原生访问时，参见相邻的 OSS 原生访问示例。
</TabItem>
- <TabItem value="oss" label="阿里云 OSS">
+ <TabItem value="oss" label="阿里云 OSS（原生访问）">
```sql
CREATE CATALOG lance_fs_oss PROPERTIES (
"type" = "lance",
"lance.catalog.type" = "filesystem",
"warehouse" = "oss://my-bucket/lance",
+ "fs.oss.support" = "true",
"oss.endpoint" = "https://oss-cn-beijing.aliyuncs.com";,
"oss.region" = "cn-beijing",
"oss.access_key" = "<ak>",
@@ -161,10 +218,10 @@ REST Catalog 通过 Lance REST Namespace 获取 Namespace、表地址和存储
| `lance.rest.api-key` | 使用 API Key 认证时是 | - | API Key，通过 `x-api-key` Header
发送。 |
| `lance.rest.header.<header-name>` | 否 | - | 发送给 REST 服务的自定义 HTTP Header。认证
Header 应使用上面的专用认证属性配置。 |
-REST Namespace 返回每张表的存储地址。根据表所在的存储系统配置默认访问参数：
+REST 服务认证和对象存储认证是两套独立配置：`lance.rest.*` 用于连接 Namespace 服务，`s3.*` / `oss.*` 用于
FE 和 BE 读取表数据。实际访问链路由 Namespace 返回的表 URI 决定：`s3://` 使用 S3，`oss://` 使用 OSS
原生访问。根据返回的表地址选择默认存储配置：
<Tabs groupId="lance-rest-storage">
- <TabItem value="s3" label="S3 / S3 兼容存储" default>
+ <TabItem value="s3" label="AWS S3" default>
```sql
CREATE CATALOG lance_rest_s3 PROPERTIES (
"type" = "lance",
@@ -172,17 +229,59 @@ REST Namespace 返回每张表的存储地址。根据表所在的存储系统
"lance.rest.uri" = "https://lance.example.com";,
"lance.rest.security.type" = "bearer",
"lance.rest.bearer-token" = "<token>",
- "s3.endpoint" = "https://s3.us-east-1.amazonaws.com";,
+ "fs.s3.support" = "true",
+ "use_path_style" = "false",
+ "s3.endpoint" = "https://my-bucket.s3.us-east-1.amazonaws.com";,
"s3.region" = "us-east-1",
"s3.access_key" = "<ak>",
"s3.secret_key" = "<sk>"
);
```
- 该配置用于 REST Namespace 返回的 `s3://` Lance 表。使用 MinIO 等 S3 兼容存储时，将
`s3.endpoint` 替换为实际地址，并按需设置 `"use_path_style" = "true"`。
+ 适用于 Namespace 返回的 AWS `s3://my-bucket/...` 表。默认 endpoint 必须包含该
Bucket，并与 region 一致。不同 Bucket 的表需要各自匹配的 endpoint，见下方的多 Bucket 说明。
+ </TabItem>
+
+ <TabItem value="minio" label="MinIO（Path Style）">
+ ```sql
+ CREATE CATALOG lance_rest_minio PROPERTIES (
+ "type" = "lance",
+ "lance.catalog.type" = "rest",
+ "lance.rest.uri" = "https://lance.example.com";,
+ "lance.rest.security.type" = "bearer",
+ "lance.rest.bearer-token" = "<token>",
+ "fs.s3.support" = "true",
+ "s3.endpoint" = "http://minio.example.com:9000";,
+ "s3.region" = "us-east-1",
+ "use_path_style" = "true",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>"
+ );
+ ```
+
+ 适用于 Namespace 返回的 MinIO `s3://` 表。endpoint 使用 FE 和 BE 都能访问的服务地址，不带
Bucket；`use_path_style=true` 将各表的 Bucket 放入请求路径。region 应与 MinIO 服务配置一致。
+ </TabItem>
+
+ <TabItem value="oss-s3" label="阿里云 OSS（S3 兼容访问）">
+ ```sql
+ CREATE CATALOG lance_rest_oss_s3 PROPERTIES (
+ "type" = "lance",
+ "lance.catalog.type" = "rest",
+ "lance.rest.uri" = "https://lance.example.com";,
+ "lance.rest.security.type" = "bearer",
+ "lance.rest.bearer-token" = "<token>",
+ "fs.s3.support" = "true",
+ "s3.endpoint" = "https://my-bucket.oss-cn-beijing.aliyuncs.com";,
+ "s3.region" = "cn-beijing",
+ "use_path_style" = "false",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>"
+ );
+ ```
+
+ 适用于 Namespace 返回的 `s3://my-bucket/...` 表，通过 S3 兼容接口访问 OSS。endpoint
必须包含同一个 Bucket；内网地址可使用
`https://my-bucket.oss-cn-beijing-internal.aliyuncs.com`。Namespace 必须返回 `s3://`
地址，不能仅设置 `fs.s3.support=true` 就将返回的 `oss://` 地址切换成 S3 访问。
</TabItem>
- <TabItem value="oss" label="阿里云 OSS">
+ <TabItem value="oss" label="阿里云 OSS（原生访问）">
```sql
CREATE CATALOG lance_rest_oss PROPERTIES (
"type" = "lance",
@@ -190,6 +289,7 @@ REST Namespace 返回每张表的存储地址。根据表所在的存储系统
"lance.rest.uri" = "https://lance.example.com";,
"lance.rest.security.type" = "bearer",
"lance.rest.bearer-token" = "<token>",
+ "fs.oss.support" = "true",
"oss.endpoint" = "https://oss-cn-beijing.aliyuncs.com";,
"oss.region" = "cn-beijing",
"oss.access_key" = "<ak>",
@@ -197,13 +297,19 @@ REST Namespace 返回每张表的存储地址。根据表所在的存储系统
);
```
- 该配置用于 REST Namespace 返回的 `oss://` Lance 表。使用 STS 临时凭证时，增加
`"oss.session_token" = "<token>"`。
+ 适用于 Namespace 返回的 `oss://` 表，使用 OSS 原生访问。`oss.endpoint` 不带 Bucket。使用
STS 临时凭证时，增加 `"oss.session_token" = "<token>"`。
</TabItem>
</Tabs>
无认证时省略 `lance.rest.security.type` 和认证属性。使用 API Key 时，将认证配置替换为
`"lance.rest.security.type" = "api_key"` 和 `"lance.rest.api-key" = "<api-key>"`。
-如果 REST Namespace 为表下发临时存储凭证，Doris 会优先使用下发的凭证，此时可以省略 Catalog 中的 S3 或 OSS
访问密钥。对于 OSS，Namespace 可以下发
`oss_endpoint`、`oss_access_key_id`、`oss_secret_access_key`、`oss_region`、`oss_security_token`，也可以使用对应的
OSS 原生名称
`endpoint`、`access_key_id`、`access_key_secret`、`region`、`security_token`。
+如果 REST Namespace 在每张表的 `storage_options` 中下发凭证或 endpoint，Doris 会按选项覆盖 Catalog
的对应默认值，并将合并结果用于 FE 和 BE。只下发凭证时，Catalog 中的 endpoint 和访问方式仍然生效；只有 Namespace
提供了完整访问配置时，才可以省略全部 Catalog 存储属性。
+
+对于 S3，Namespace 可以使用
`aws_endpoint`、`aws_region`、`aws_access_key_id`、`aws_secret_access_key`、`aws_session_token`
和 `aws_virtual_hosted_style_request`。注意最后一个参数与 Doris 的 `use_path_style`
取值相反：虚拟主机方式为 `aws_virtual_hosted_style_request=true`，Path Style 为
`false`。`fs.s3.support` 属于 Doris Catalog 属性，不是 Lance 的 `storage_options` 参数。
+
+对于 OSS 原生访问，Namespace 可以下发
`oss_endpoint`、`oss_access_key_id`、`oss_secret_access_key`、`oss_region`、`oss_security_token`，也可以使用对应的
OSS 原生名称
`endpoint`、`access_key_id`、`access_key_secret`、`region`、`security_token`。
+
+如果不同表位于不同 Bucket，不要让它们共用上例中固定为 `my-bucket` 的 S3 虚拟主机 endpoint。应由 Namespace
为每张表下发匹配的 endpoint 和访问方式，或为不同 Bucket 分别配置 Catalog。对于支持 Path Style 的服务，也可以使用不带
Bucket 的服务 endpoint，配合 `use_path_style=true`，由客户端从各表 URI 读取 Bucket。
:::caution
当前 BE Reader 不支持由 REST Namespace 管理版本的 Lance 表（Managed Versioning）。
@@ -236,7 +342,12 @@ CREATE CATALOG lance_analytics PROPERTIES (
"lance.catalog.type" = "filesystem",
"warehouse" = "s3://my-bucket/lance",
"lance.namespace.parent" = "production$analytics",
- "s3.region" = "us-east-1"
+ "fs.s3.support" = "true",
+ "s3.endpoint" = "https://my-bucket.s3.us-east-1.amazonaws.com";,
+ "use_path_style" = "false",
+ "s3.region" = "us-east-1",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>"
);
```
@@ -439,21 +550,73 @@ Runtime Filter 下推目前只适用于普通 Lance 表扫描，不用于 `vecto
### S3 TVF
-```sql
-SELECT user_id, name
-FROM s3(
- "uri" = "s3://my-bucket/lance/user_profiles.lance",
- "s3.endpoint" = "http://127.0.0.1:9000";,
- "s3.access_key" = "admin",
- "s3.secret_key" = "password",
- "s3.region" = "us-east-1",
- "use_path_style" = "true",
- "format" = "lance"
-)
-WHERE user_id > 100;
-```
+`s3()` TVF 使用 S3 兼容接口。访问 OSS 时，也应使用 `s3.*` 参数和带 Bucket 的虚拟主机 endpoint；需要 OSS
原生访问时，使用前面的 OSS Catalog 示例。下面的 SQL 读取同一个示例数据集的前 10 行，替换地址、凭证和列名后执行：
+
+<Tabs groupId="lance-s3-tvf-storage">
+ <TabItem value="s3" label="AWS S3" default>
+ ```sql
+ SELECT user_id, name
+ FROM s3(
+ "uri" = "s3://my-bucket/lance/user_profiles.lance",
+ "format" = "lance",
+ "fs.s3.support" = "true",
+ "use_path_style" = "false",
+ "s3.endpoint" = "https://my-bucket.s3.us-east-1.amazonaws.com";,
+ "s3.region" = "us-east-1",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>"
+ )
+ ORDER BY user_id
+ LIMIT 10;
+ ```
+
+ AWS S3 同样显式设置 endpoint。本例使用虚拟主机方式，endpoint 中的 Bucket 和 region 必须与 URI
及 `s3.region` 一致。
+ </TabItem>
+
+ <TabItem value="minio" label="MinIO（Path Style）">
+ ```sql
+ SELECT user_id, name
+ FROM s3(
+ "uri" = "s3://my-bucket/lance/user_profiles.lance",
+ "format" = "lance",
+ "fs.s3.support" = "true",
+ "s3.endpoint" = "http://minio.example.com:9000";,
+ "s3.region" = "us-east-1",
+ "use_path_style" = "true",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>"
+ )
+ ORDER BY user_id
+ LIMIT 10;
+ ```
+
+ endpoint 指向 MinIO 服务地址，不带 Bucket，配合 `use_path_style=true`。不要使用仅 FE
本机可访问的地址；FE 和执行查询的 BE 都需要访问该服务。
+ </TabItem>
+
+ <TabItem value="oss-s3" label="阿里云 OSS（S3 兼容访问）">
+ ```sql
+ SELECT user_id, name
+ FROM s3(
+ "uri" = "s3://my-bucket/lance/user_profiles.lance",
+ "format" = "lance",
+ "fs.s3.support" = "true",
+ "s3.endpoint" = "https://my-bucket.oss-cn-beijing.aliyuncs.com";,
+ "s3.region" = "cn-beijing",
+ "use_path_style" = "false",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>"
+ )
+ ORDER BY user_id
+ LIMIT 10;
+ ```
+
+ 本例与 Filesystem Catalog 的 OSS S3 兼容配置一致，但 `uri` 指向具体的 `.lance`
数据集，`warehouse` 则指向包含多个数据集的目录。保持
`fs.s3.support=true`、`use_path_style=false`，并确保 endpoint 中的 Bucket 与 URI 一致。
+ </TabItem>
+</Tabs>
+
+对于 OSS，上例的 `uri` 也可以写为 `oss://my-bucket/lance/user_profiles.lance`，但仍需保留相同的
`s3.*` 参数和 `fs.s3.support=true`；TVF 会将 URI 规范化为 `s3://`，不会因此使用 OSS 原生
Provider。不要直接照搬原生 OSS Catalog 的 `oss.*` 配置：不带 Bucket 的 endpoint
与虚拟主机方式组合可能返回数据集不存在；改成 Path Style 则可能被 OSS 以 `SecondLevelDomainForbidden` 拒绝。
-S3 TVF 在 FE 获取 Schema、当前版本和 Fragment 列表，并固定该版本后按 Fragment 并行扫描。
+S3 TVF 在 FE 获取 Schema、当前版本和 Fragment 列表，并固定该版本后按 Fragment 并行扫描。`EXPLAIN` 中会显示
`VTVF_SCAN_NODE`；多个 split 可以共享同一个数据集 URI，通过各自的 Fragment ID 区分。`totalFileSize=0`
或 `length=0` 是 Lance split 的占位值，不表示数据集为空，也不表示每个 split 都会重复扫描整张表。
### Local TVF
diff --git a/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
b/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
index 7585a961649..77800271e98 100644
--- a/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
+++ b/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
@@ -99,31 +99,88 @@ A Filesystem Catalog discovers Lance Namespaces and tables
directly from a wareh
|---|---|---|
| `warehouse` | Yes | Root path of the Lance warehouse. Local absolute paths,
`file://` URIs, `s3://` URIs, and `oss://` URIs are supported. |
-Select the example that matches the storage system hosting the warehouse:
+The two properties serve different purposes:
+
+- `fs.s3.support=true` explicitly selects S3 configuration parsing. AWS S3 can
usually be detected from its properties. For MinIO, custom endpoints, or OSS
accessed through its S3-compatible API, set it explicitly to avoid failed
detection or selection of another storage type based on the domain. All S3
examples below include this property for direct use.
+- `use_path_style` controls where the bucket appears in request URLs and
defaults to `false`. Setting it to `true` uses `endpoint/bucket/object-path`;
setting it to `false` uses virtual-hosted-style access. With the current
default S3 client in Lance, an explicitly configured virtual-hosted-style
endpoint must include the bucket. Set this property together with the endpoint
according to the access style supported by the service.
+
+| Access scenario | Storage type setting | `use_path_style` | Endpoint format |
+|---|---|---|---|
+| AWS S3 | Recommended: `fs.s3.support=true` | `false`; can be omitted |
Include the bucket, such as `https://my-bucket.s3.us-east-1.amazonaws.com`. |
+| MinIO (access through a service domain or IP) | Set `fs.s3.support=true` |
Explicitly set `true` | Service address without the bucket, such as
`http://minio.example.com:9000`. |
+| OSS S3-compatible API | Set `fs.s3.support=true` | `false`; explicitly set
in the example | Include the bucket, such as
`https://my-bucket.oss-cn-beijing.aliyuncs.com`. |
+| Native OSS API | Recommended: `fs.oss.support=true`; do not set
`fs.s3.support` | No need to set the generic `use_path_style` | Use
`oss.endpoint` without the bucket, such as
`https://oss-cn-beijing.aliyuncs.com`. |
+
+These S3 endpoint rules apply to the default S3 access path in Lance. For a
Filesystem Catalog using a single storage type, set only the corresponding
`fs.*.support=true`; do not enable multiple storage types to “enable
compatibility.” This property selects configuration parsing and does not add
protocol support to the storage service.
+
+Select the example that matches the storage system hosting the warehouse, and
replace the bucket, path, address, and credentials:
<Tabs groupId="lance-filesystem-storage">
- <TabItem value="s3" label="S3 / S3-Compatible" default>
+ <TabItem value="s3" label="AWS S3" default>
```sql
CREATE CATALOG lance_fs_s3 PROPERTIES (
"type" = "lance",
"lance.catalog.type" = "filesystem",
"warehouse" = "s3://my-bucket/lance",
- "s3.endpoint" = "https://s3.us-east-1.amazonaws.com";,
+ "fs.s3.support" = "true",
+ "use_path_style" = "false",
+ "s3.endpoint" = "https://my-bucket.s3.us-east-1.amazonaws.com";,
"s3.region" = "us-east-1",
"s3.access_key" = "<ak>",
"s3.secret_key" = "<sk>"
);
```
- You can omit `s3.endpoint` for AWS S3. For S3-compatible storage such
as MinIO, set the service endpoint and add `"use_path_style" = "true"` when the
service requires path-style access.
+ This example uses an AWS S3 endpoint containing the bucket with
`use_path_style=false`. Current Doris S3 property validation requires an
endpoint; when using the `s3://my-bucket/...` URI shown here, setting only the
region is insufficient. Update the endpoint whenever you change the bucket or
region.
+ </TabItem>
+
+ <TabItem value="minio" label="MinIO (Path Style)">
+ ```sql
+ CREATE CATALOG lance_fs_minio PROPERTIES (
+ "type" = "lance",
+ "lance.catalog.type" = "filesystem",
+ "warehouse" = "s3://my-bucket/lance",
+ "fs.s3.support" = "true",
+ "s3.endpoint" = "http://minio.example.com:9000";,
+ "s3.region" = "us-east-1",
+ "use_path_style" = "true",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>"
+ );
+ ```
+
+ This example accesses MinIO through its service domain using
path-style requests, without requiring a subdomain for each bucket. Keep the
bucket out of the endpoint and replace the region with the value configured for
the service. If TLS is enabled, use the corresponding `https://` endpoint and
ensure that both the FE and BEs can reach it.
+ </TabItem>
+
+ <TabItem value="oss-s3" label="Alibaba Cloud OSS (S3-Compatible Access)">
+ To access OSS through its S3-compatible API, use a `s3://` warehouse
and `s3.*` connection properties:
+
+ ```sql
+ CREATE CATALOG lance_fs_oss_s3 PROPERTIES (
+ "type" = "lance",
+ "lance.catalog.type" = "filesystem",
+ "warehouse" = "s3://my-bucket/lance",
+ "fs.s3.support" = "true",
+ "s3.endpoint" = "https://my-bucket.oss-cn-beijing.aliyuncs.com";,
+ "s3.region" = "cn-beijing",
+ "use_path_style" = "false",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>"
+ );
+ ```
+
+ This example sets `use_path_style=false` for virtual-hosted-style
access. The current default S3 client in Lance does not automatically add the
bucket to an explicitly configured endpoint, so `s3.endpoint` must include the
bucket, for example `https://my-bucket.oss-cn-beijing.aliyuncs.com`; keep
`warehouse` as `s3://my-bucket/lance`. For internal network access, use
`https://my-bucket.oss-cn-beijing-internal.aliyuncs.com` and ensure that both
the FE and BEs can reach it.
+
+ For temporary STS credentials, add `"s3.session_token" = "<token>"`.
This mode reads existing OSS data through the S3-compatible API without moving
the data. For native OSS access, see the adjacent native OSS example.
</TabItem>
- <TabItem value="oss" label="Alibaba Cloud OSS">
+ <TabItem value="oss" label="Alibaba Cloud OSS (Native Access)">
```sql
CREATE CATALOG lance_fs_oss PROPERTIES (
"type" = "lance",
"lance.catalog.type" = "filesystem",
"warehouse" = "oss://my-bucket/lance",
+ "fs.oss.support" = "true",
"oss.endpoint" = "https://oss-cn-beijing.aliyuncs.com";,
"oss.region" = "cn-beijing",
"oss.access_key" = "<ak>",
@@ -161,10 +218,10 @@ A REST Catalog obtains Namespaces, table locations, and
storage access parameter
| `lance.rest.api-key` | Yes for API Key authentication | - | API Key sent in
the `x-api-key` header. |
| `lance.rest.header.<header-name>` | No | - | Custom HTTP header sent to the
REST service. Use the dedicated authentication properties above for
authentication headers. |
-The REST Namespace returns the storage location of each table. Configure
default access properties for the storage system hosting those tables:
+REST service authentication and object storage authentication are separate:
`lance.rest.*` connects to the Namespace service, while `s3.*` / `oss.*` lets
the FE and BEs read table data. The table URI returned by the Namespace
determines the access path: `s3://` uses S3, and `oss://` uses native OSS
access. Select default storage properties according to the returned table URIs:
<Tabs groupId="lance-rest-storage">
- <TabItem value="s3" label="S3 / S3-Compatible" default>
+ <TabItem value="s3" label="AWS S3" default>
```sql
CREATE CATALOG lance_rest_s3 PROPERTIES (
"type" = "lance",
@@ -172,17 +229,59 @@ The REST Namespace returns the storage location of each
table. Configure default
"lance.rest.uri" = "https://lance.example.com";,
"lance.rest.security.type" = "bearer",
"lance.rest.bearer-token" = "<token>",
- "s3.endpoint" = "https://s3.us-east-1.amazonaws.com";,
+ "fs.s3.support" = "true",
+ "use_path_style" = "false",
+ "s3.endpoint" = "https://my-bucket.s3.us-east-1.amazonaws.com";,
"s3.region" = "us-east-1",
"s3.access_key" = "<ak>",
"s3.secret_key" = "<sk>"
);
```
- This configuration applies to `s3://` Lance tables returned by the
REST Namespace. For S3-compatible storage such as MinIO, replace `s3.endpoint`
with the service endpoint and add `"use_path_style" = "true"` when required.
+ For AWS `s3://my-bucket/...` tables returned by the Namespace. The
default endpoint must include that bucket and match the region. Tables in
different buckets need matching endpoints, as explained below.
+ </TabItem>
+
+ <TabItem value="minio" label="MinIO (Path Style)">
+ ```sql
+ CREATE CATALOG lance_rest_minio PROPERTIES (
+ "type" = "lance",
+ "lance.catalog.type" = "rest",
+ "lance.rest.uri" = "https://lance.example.com";,
+ "lance.rest.security.type" = "bearer",
+ "lance.rest.bearer-token" = "<token>",
+ "fs.s3.support" = "true",
+ "s3.endpoint" = "http://minio.example.com:9000";,
+ "s3.region" = "us-east-1",
+ "use_path_style" = "true",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>"
+ );
+ ```
+
+ For MinIO `s3://` tables returned by the Namespace. Use a service
endpoint reachable by both the FE and BEs, without a bucket.
`use_path_style=true` puts each table's bucket in the request path. The region
must match the MinIO service configuration.
+ </TabItem>
+
+ <TabItem value="oss-s3" label="Alibaba Cloud OSS (S3-Compatible Access)">
+ ```sql
+ CREATE CATALOG lance_rest_oss_s3 PROPERTIES (
+ "type" = "lance",
+ "lance.catalog.type" = "rest",
+ "lance.rest.uri" = "https://lance.example.com";,
+ "lance.rest.security.type" = "bearer",
+ "lance.rest.bearer-token" = "<token>",
+ "fs.s3.support" = "true",
+ "s3.endpoint" = "https://my-bucket.oss-cn-beijing.aliyuncs.com";,
+ "s3.region" = "cn-beijing",
+ "use_path_style" = "false",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>"
+ );
+ ```
+
+ For `s3://my-bucket/...` tables returned by the Namespace, accessing
OSS through its S3-compatible API. The endpoint must include the same bucket;
for internal access, use
`https://my-bucket.oss-cn-beijing-internal.aliyuncs.com`. The Namespace must
return a `s3://` URI: setting `fs.s3.support=true` does not switch a returned
`oss://` URI to S3 access.
</TabItem>
- <TabItem value="oss" label="Alibaba Cloud OSS">
+ <TabItem value="oss" label="Alibaba Cloud OSS (Native Access)">
```sql
CREATE CATALOG lance_rest_oss PROPERTIES (
"type" = "lance",
@@ -190,6 +289,7 @@ The REST Namespace returns the storage location of each
table. Configure default
"lance.rest.uri" = "https://lance.example.com";,
"lance.rest.security.type" = "bearer",
"lance.rest.bearer-token" = "<token>",
+ "fs.oss.support" = "true",
"oss.endpoint" = "https://oss-cn-beijing.aliyuncs.com";,
"oss.region" = "cn-beijing",
"oss.access_key" = "<ak>",
@@ -197,13 +297,19 @@ The REST Namespace returns the storage location of each
table. Configure default
);
```
- This configuration applies to `oss://` Lance tables returned by the
REST Namespace. For temporary STS credentials, add `"oss.session_token" =
"<token>"`.
+ For `oss://` tables returned by the Namespace, using native OSS
access. Keep the bucket out of `oss.endpoint`. For temporary STS credentials,
add `"oss.session_token" = "<token>"`.
</TabItem>
</Tabs>
For unauthenticated REST access, omit `lance.rest.security.type` and the
authentication property. For API Key authentication, replace them with
`"lance.rest.security.type" = "api_key"` and `"lance.rest.api-key" =
"<api-key>"`.
-If the REST Namespace vends temporary storage credentials for a table, Doris
gives them precedence over the Catalog credentials, allowing the S3 or OSS
access keys to be omitted from the Catalog. For OSS, the Namespace may vend
`oss_endpoint`, `oss_access_key_id`, `oss_secret_access_key`, `oss_region`, and
`oss_security_token`, or the corresponding native names `endpoint`,
`access_key_id`, `access_key_secret`, `region`, and `security_token`.
+If the REST Namespace vends credentials or an endpoint in each table's
`storage_options`, Doris overrides the corresponding Catalog defaults option by
option and uses the merged configuration in both the FE and BE. If only
credentials are vended, the Catalog endpoint and access style still apply. All
Catalog storage properties can be omitted only when the Namespace provides the
complete access configuration.
+
+For S3, the Namespace can use `aws_endpoint`, `aws_region`,
`aws_access_key_id`, `aws_secret_access_key`, `aws_session_token`, and
`aws_virtual_hosted_style_request`. The last option is the inverse of Doris's
`use_path_style`: virtual-hosted-style access uses
`aws_virtual_hosted_style_request=true`, while path-style access uses `false`.
`fs.s3.support` is a Doris Catalog property, not a Lance `storage_options`
parameter.
+
+For native OSS access, the Namespace may vend `oss_endpoint`,
`oss_access_key_id`, `oss_secret_access_key`, `oss_region`, and
`oss_security_token`, or the corresponding native names `endpoint`,
`access_key_id`, `access_key_secret`, `region`, and `security_token`.
+
+If tables reside in different buckets, do not share the S3
virtual-hosted-style endpoint fixed to `my-bucket` in the example. Have the
Namespace vend a matching endpoint and access style for each table, or
configure separate Catalogs for different buckets. For services supporting
path-style access, you can also use a service endpoint without a bucket and set
`use_path_style=true`, allowing the client to read the bucket from each table
URI.
:::caution
The current BE Reader does not support Lance tables whose versions are managed
by REST Namespace (Managed Versioning).
@@ -236,7 +342,12 @@ CREATE CATALOG lance_analytics PROPERTIES (
"lance.catalog.type" = "filesystem",
"warehouse" = "s3://my-bucket/lance",
"lance.namespace.parent" = "production$analytics",
- "s3.region" = "us-east-1"
+ "fs.s3.support" = "true",
+ "s3.endpoint" = "https://my-bucket.s3.us-east-1.amazonaws.com";,
+ "use_path_style" = "false",
+ "s3.region" = "us-east-1",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>"
);
```
@@ -439,21 +550,73 @@ If you only need to read a Lance dataset at a known path,
you can use the `s3()`
### S3 TVF
-```sql
-SELECT user_id, name
-FROM s3(
- "uri" = "s3://my-bucket/lance/user_profiles.lance",
- "s3.endpoint" = "http://127.0.0.1:9000";,
- "s3.access_key" = "admin",
- "s3.secret_key" = "password",
- "s3.region" = "us-east-1",
- "use_path_style" = "true",
- "format" = "lance"
-)
-WHERE user_id > 100;
-```
+The `s3()` TVF uses the S3-compatible API. For OSS, use `s3.*` properties and
a virtual-hosted-style endpoint containing the bucket. For native OSS access,
use the OSS Catalog example above. The following SQL reads the first 10 rows of
the example dataset; replace the address, credentials, and column names before
running it:
+
+<Tabs groupId="lance-s3-tvf-storage">
+ <TabItem value="s3" label="AWS S3" default>
+ ```sql
+ SELECT user_id, name
+ FROM s3(
+ "uri" = "s3://my-bucket/lance/user_profiles.lance",
+ "format" = "lance",
+ "fs.s3.support" = "true",
+ "use_path_style" = "false",
+ "s3.endpoint" = "https://my-bucket.s3.us-east-1.amazonaws.com";,
+ "s3.region" = "us-east-1",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>"
+ )
+ ORDER BY user_id
+ LIMIT 10;
+ ```
+
+ Explicitly set the endpoint for AWS S3 as well. This example uses
virtual-hosted-style access; the bucket and region in the endpoint must match
the URI and `s3.region`.
+ </TabItem>
+
+ <TabItem value="minio" label="MinIO (Path Style)">
+ ```sql
+ SELECT user_id, name
+ FROM s3(
+ "uri" = "s3://my-bucket/lance/user_profiles.lance",
+ "format" = "lance",
+ "fs.s3.support" = "true",
+ "s3.endpoint" = "http://minio.example.com:9000";,
+ "s3.region" = "us-east-1",
+ "use_path_style" = "true",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>"
+ )
+ ORDER BY user_id
+ LIMIT 10;
+ ```
+
+ Use the MinIO service endpoint without a bucket and set
`use_path_style=true`. Both the FE and the BEs executing the query must be able
to reach the service; avoid an address accessible only from the FE host.
+ </TabItem>
+
+ <TabItem value="oss-s3" label="Alibaba Cloud OSS (S3-Compatible Access)">
+ ```sql
+ SELECT user_id, name
+ FROM s3(
+ "uri" = "s3://my-bucket/lance/user_profiles.lance",
+ "format" = "lance",
+ "fs.s3.support" = "true",
+ "s3.endpoint" = "https://my-bucket.oss-cn-beijing.aliyuncs.com";,
+ "s3.region" = "cn-beijing",
+ "use_path_style" = "false",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>"
+ )
+ ORDER BY user_id
+ LIMIT 10;
+ ```
+
+ This example uses the same OSS S3-compatible settings as the
Filesystem Catalog, but `uri` points to a specific `.lance` dataset, whereas
`warehouse` points to a directory containing datasets. Keep
`fs.s3.support=true` and `use_path_style=false`, and ensure the endpoint bucket
matches the URI.
+ </TabItem>
+</Tabs>
+
+For OSS, the example's `uri` can also be written as
`oss://my-bucket/lance/user_profiles.lance`, while retaining the same `s3.*`
properties and `fs.s3.support=true`. The TVF normalizes the URI to `s3://`; it
does not switch to the native OSS provider. Do not directly copy the `oss.*`
configuration from a native OSS Catalog: a bucketless endpoint combined with
virtual-hosted-style access can report that the dataset does not exist, while
switching to path-style access can be rejected by OS [...]
-For an S3 TVF, the FE obtains the Schema, current version, and Fragment list.
Doris pins that version and scans its Fragments in parallel.
+For an S3 TVF, the FE obtains the Schema, current version, and Fragment list.
Doris pins that version and scans its Fragments in parallel. `EXPLAIN` shows
`VTVF_SCAN_NODE`; multiple splits can share the dataset URI and are
distinguished by their Fragment IDs. `totalFileSize=0` or `length=0` is a
placeholder for Lance splits, not an indication of an empty dataset or repeated
whole-table scans by each split.
### Local TVF
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
