This is an automated email from the ASF dual-hosted git repository.
zhangstar333 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new bc523f4fb2c [doc](lance) refactor the lance catalog doc (#4109)
bc523f4fb2c is described below
commit bc523f4fb2c399790028e72ab3cdc7660647d24c
Author: zhangstar333 <[email protected]>
AuthorDate: Wed Sep 2 19:27:54 2026 +0800
[doc](lance) refactor the lance catalog doc (#4109)
refactor the lance catalog doc
---
.../lakehouse/catalogs/lance-catalog.mdx | 336 +++++++++++----------
.../lakehouse/catalogs/lance-catalog.mdx | 334 ++++++++++----------
2 files changed, 345 insertions(+), 325 deletions(-)
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
index 2eb959c6a16..a318f0d2058 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
@@ -6,21 +6,33 @@
}
---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
:::note
这是一个实验性功能。
Lance Catalog 自 Apache Doris 4.2 版本开始支持。
:::
-Lance 是面向分析和 AI 场景的列式数据格式。Doris 可以通过 Lance Catalog 发现 Lance Namespace
中的数据库和表，并直接查询存储在本地文件系统或 S3 兼容对象存储中的 Lance 数据集。
+Lance 是面向分析和 AI 场景的列式数据格式。Doris 可以通过 Lance Catalog 发现 Lance Namespace
中的数据库和表，并直接查询存储在本地文件系统、S3 兼容对象存储或阿里云 OSS 中的 Lance 数据集。
当前 Doris 对 Lance 提供只读能力，不支持创建、写入、更新或删除 Lance 表。
+## 适用场景
+
+| 场景 | 说明 |
+|---|---|
+| 直接分析 Lance 数据 | 无需迁移或导入数据，使用 Doris SQL 查询已有的 Lance 数据集。 |
+| AI 向量检索 | 复用 Lance 中已有的向量索引，通过 `vector_search()` 执行向量检索，并与标量过滤和 Doris SQL
分析结合。 |
+| 数据集成 | 从 Lance 读取数据并写入 Doris 内表，用于后续加工、关联分析或长期存储。 |
+| Namespace 管理 | 简单目录结构可以使用 Filesystem Catalog；需要集中管理 Namespace、表地址或临时存储凭证时使用
REST Catalog。 |
+
## 功能概览
| 功能 | 支持情况 |
|---|---|
-| Filesystem Catalog | 支持本地文件系统、`file://` 和 `s3://` Warehouse |
+| Filesystem Catalog | 支持本地文件系统、`file://`、`s3://` 和 `oss://` Warehouse |
| REST Catalog | 支持 Lance REST Namespace，以及无认证、Bearer Token、API Key 和自定义 HTTP
Header |
| 元数据访问 | 支持 `SHOW DATABASES`、`SHOW TABLES`、`DESC` 和 `SHOW INDEX`（仅 Filesystem
Catalog） |
| 系统表（`table$...`） | 暂不支持；索引元数据请使用 `SHOW INDEX` |
@@ -85,54 +97,55 @@ Filesystem Catalog 直接从 Warehouse 目录发现 Lance Namespace 和表。
|---|---|---|
| `warehouse` | 是 | Lance Warehouse 根路径。支持本地绝对路径、`file://` URI、`s3://` URI 和
`oss://` URI。 |
-#### 使用 S3 兼容对象存储
-
-下面以 MinIO 为例创建 Catalog：
-
-```sql
-CREATE CATALOG lance_catalog PROPERTIES (
- "type" = "lance",
- "lance.catalog.type" = "filesystem",
- "warehouse" = "s3://my-bucket/lance",
- "s3.endpoint" = "http://127.0.0.1:9000";,
- "s3.access_key" = "admin",
- "s3.secret_key" = "password",
- "s3.region" = "us-east-1",
- "use_path_style" = "true"
-);
-```
-
-访问 AWS S3 时，可以省略 `s3.endpoint`，并按实际环境配置访问密钥、Region 和 Path Style。
-
-#### 使用阿里云 OSS
-
-```sql
-CREATE CATALOG lance_oss PROPERTIES (
- "type" = "lance",
- "lance.catalog.type" = "filesystem",
- "warehouse" = "oss://my-bucket/lance",
- "oss.endpoint" = "oss-cn-beijing.aliyuncs.com",
- "oss.region" = "cn-beijing",
- "oss.access_key" = "<ak>",
- "oss.secret_key" = "<sk>"
-);
-```
-
-`warehouse` 也支持带 Endpoint 的形式
`oss://my-bucket.oss-cn-beijing.aliyuncs.com/lance`，Doris 会在打开 Namespace 前将其还原为
Bucket。`warehouse` 必须指定 Bucket，因此 `oss:/lance` 这类根路径会在创建 Catalog 时被拒绝。
-
-使用 STS 临时凭证访问 OSS 时，可以增加 `"oss.session_token" = "<token>"`。
-
-#### 使用本地文件系统
-
-```sql
-CREATE CATALOG lance_local PROPERTIES (
- "type" = "lance",
- "lance.catalog.type" = "filesystem",
- "warehouse" = "/data/lance"
-);
-```
-
-使用本地文件系统时，`warehouse` 必须是绝对路径。FE 需要通过该路径读取 Namespace 和表元数据，执行查询的 BE
也需要能够通过相同路径访问数据。因此在多节点环境中，应将相同的共享目录挂载到所有相关 FE 和 BE 节点。
+根据 Warehouse 所在的存储系统选择对应示例：
+
+<Tabs groupId="lance-filesystem-storage">
+ <TabItem value="s3" label="S3 / S3 兼容存储" default>
+ ```sql
+ CREATE CATALOG lance_fs_s3 PROPERTIES (
+ "type" = "lance",
+ "lance.catalog.type" = "filesystem",
+ "warehouse" = "s3://my-bucket/lance",
+ "s3.endpoint" = "https://s3.us-east-1.amazonaws.com";,
+ "s3.region" = "us-east-1",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>"
+ );
+ ```
+
+ 访问 AWS S3 时可以省略 `s3.endpoint`。访问 MinIO 等 S3 兼容存储时，应设置对应的
`s3.endpoint`；如果服务要求 Path Style，还需设置 `"use_path_style" = "true"`。
+ </TabItem>
+
+ <TabItem value="oss" label="阿里云 OSS">
+ ```sql
+ CREATE CATALOG lance_fs_oss PROPERTIES (
+ "type" = "lance",
+ "lance.catalog.type" = "filesystem",
+ "warehouse" = "oss://my-bucket/lance",
+ "oss.endpoint" = "https://oss-cn-beijing.aliyuncs.com";,
+ "oss.region" = "cn-beijing",
+ "oss.access_key" = "<ak>",
+ "oss.secret_key" = "<sk>"
+ );
+ ```
+
+ 使用 STS 临时凭证时，增加 `"oss.session_token" = "<token>"`。`warehouse` 也支持
`oss://my-bucket.oss-cn-beijing.aliyuncs.com/lance`，Doris 会将其规范化为 Bucket
路径。OSS-HDFS 当前不受支持。
+ </TabItem>
+
+ <TabItem value="local" label="本地文件系统">
+ ```sql
+ CREATE CATALOG lance_fs_local PROPERTIES (
+ "type" = "lance",
+ "lance.catalog.type" = "filesystem",
+ "warehouse" = "/data/lance"
+ );
+ ```
+
+ `warehouse` 必须是绝对路径。FE 和执行查询的 BE
都必须能够通过相同路径访问数据；多节点环境应在所有相关节点挂载同一个共享目录。
+ </TabItem>
+</Tabs>
+
+对于 S3 和 OSS，`warehouse` 必须包含 Bucket，例如 `s3://bucket/path` 或
`oss://bucket/path`。
### REST Catalog
@@ -146,38 +159,49 @@ REST Catalog 通过 Lance REST Namespace 获取 Namespace、表地址和存储
| `lance.rest.api-key` | 使用 API Key 认证时是 | - | API Key，通过 `x-api-key` Header
发送。 |
| `lance.rest.header.<header-name>` | 否 | - | 发送给 REST 服务的自定义 HTTP Header。认证
Header 应使用上面的专用认证属性配置。 |
-使用 Bearer Token 创建 REST Catalog：
-
-```sql
-CREATE CATALOG lance_rest PROPERTIES (
- "type" = "lance",
- "lance.catalog.type" = "rest",
- "lance.rest.uri" = "https://lance.example.com";,
- "lance.rest.security.type" = "bearer",
- "lance.rest.bearer-token" = "your-token"
-);
-```
-
-使用 API Key 时，将认证配置替换为：
-
-```sql
-"lance.rest.security.type" = "api_key",
-"lance.rest.api-key" = "your-api-key"
-```
-
-如果 REST 服务返回临时存储凭证，Doris 会使用这些凭证访问对应的 Lance 表。也可以在 Catalog 中配置默认的对象存储访问参数：S3
兼容存储使用 `s3.endpoint`、`s3.access_key`、`s3.secret_key`、`s3.region` 和
`use_path_style`，OSS 使用
`oss.endpoint`、`oss.access_key`、`oss.secret_key`、`oss.region` 和
`oss.session_token`。
-
-对于同一张表，Namespace 下发的凭证优先于 Catalog 中配置的凭证。因此，当 Namespace 能够下发完整凭证时，创建 Catalog
时无需保存任何访问密钥：
-
-```sql
-CREATE CATALOG lance_rest_oss PROPERTIES (
- "type" = "lance",
- "lance.catalog.type" = "rest",
- "lance.rest.uri" = "https://lance.example.com";
-);
-```
-
-对于 OSS 表，Namespace 下发的存储选项可以写成
`oss_endpoint`、`oss_access_key_id`、`oss_secret_access_key`、`oss_region`、`oss_security_token`，也可以写成
OSS 原生的
`endpoint`、`access_key_id`、`access_key_secret`、`region`、`security_token`。Doris
同时接受这两种写法，并将其视为同一个选项。如果同一个选项以两种写法下发且取值不同，则会报错。
+REST Namespace 返回每张表的存储地址。根据表所在的存储系统配置默认访问参数：
+
+<Tabs groupId="lance-rest-storage">
+ <TabItem value="s3" label="S3 / S3 兼容存储" default>
+ ```sql
+ CREATE CATALOG lance_rest_s3 PROPERTIES (
+ "type" = "lance",
+ "lance.catalog.type" = "rest",
+ "lance.rest.uri" = "https://lance.example.com";,
+ "lance.rest.security.type" = "bearer",
+ "lance.rest.bearer-token" = "<token>",
+ "s3.endpoint" = "https://s3.us-east-1.amazonaws.com";,
+ "s3.region" = "us-east-1",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>"
+ );
+ ```
+
+ 该配置用于 REST Namespace 返回的 `s3://` Lance 表。使用 MinIO 等 S3 兼容存储时，将
`s3.endpoint` 替换为实际地址，并按需设置 `"use_path_style" = "true"`。
+ </TabItem>
+
+ <TabItem value="oss" label="阿里云 OSS">
+ ```sql
+ CREATE CATALOG lance_rest_oss PROPERTIES (
+ "type" = "lance",
+ "lance.catalog.type" = "rest",
+ "lance.rest.uri" = "https://lance.example.com";,
+ "lance.rest.security.type" = "bearer",
+ "lance.rest.bearer-token" = "<token>",
+ "oss.endpoint" = "https://oss-cn-beijing.aliyuncs.com";,
+ "oss.region" = "cn-beijing",
+ "oss.access_key" = "<ak>",
+ "oss.secret_key" = "<sk>"
+ );
+ ```
+
+ 该配置用于 REST Namespace 返回的 `oss://` Lance 表。使用 STS 临时凭证时，增加
`"oss.session_token" = "<token>"`。
+ </TabItem>
+</Tabs>
+
+无认证时省略 `lance.rest.security.type` 和认证属性。使用 API Key 时，将认证配置替换为
`"lance.rest.security.type" = "api_key"` 和 `"lance.rest.api-key" = "<api-key>"`。
+
+如果 REST Namespace 为表下发临时存储凭证，Doris 会优先使用下发的凭证，此时可以省略 Catalog 中的 S3 或 OSS
访问密钥。对于 OSS，Namespace 可以下发
`oss_endpoint`、`oss_access_key_id`、`oss_secret_access_key`、`oss_region`、`oss_security_token`，也可以使用对应的
OSS 原生名称
`endpoint`、`access_key_id`、`access_key_secret`、`region`、`security_token`。
:::caution
当前 BE Reader 不支持由 REST Namespace 管理版本的 Lance 表（Managed Versioning）。
@@ -470,9 +494,9 @@ ORDER BY _distance ASC, user_id;
| `metric` | 否 | `uint8` 为 `hamming`，其他支持类型为 `l2` | 距离类型：`l2`、`cosine`、`dot` 或
`hamming`。`dot_product` 是 `dot` 的别名。`uint8` 向量仅支持 `hamming`；其他当前支持的向量元素类型支持
`l2`、`cosine` 和 `dot`。Doris 选择向量索引时会把未设置的 `metric` 视为 `l2`，因此查询使用 `cosine` 或
`dot` 创建的索引时必须显式设置 `metric`。 |
| `filter` | 否 | - | Lance SQL 条件，在生成候选向量之前执行，即 Prefilter。 |
| `nprobes` | 否 | 最少 `1`，不限制最大值 | IVF 索引探测的分区数量，必须为正整数。不设置时从 1 个分区开始；使用
Prefilter 且候选不足时，Lance 可以继续探测更多分区。显式设置为 `N` 时，最少和最多探测数都会固定为 `N`。 |
-| `refine_factor` | 否 | 不启用精排 | 候选集精排倍数，必须为正整数。不设置时不基于原始向量重新计算距离，量化索引返回的
`_distance` 可能是近似距离；设置为 `N` 后，Lance 先获取 `(top_k + offset) × N`
个候选，再用原始向量计算真实距离并重新排序。即使设置为 `1` 也会执行精排，因此与不设置不同。 |
+| `refine_factor` | 否 | 不启用精排 | 候选集精排倍数，必须为正整数。不设置时不基于原始向量重新计算距离，量化索引返回的
`_distance` 可能是近似距离；设置为 `N` 后，Lance 先获取 `(top_k + offset) × N`
个候选，再用原始向量计算真实距离并重新排序。**精排会读取这些候选的原始向量数据；`N` 越大，读取和计算的候选越多，可能显著增加 I/O
并降低查询性能。** 即使设置为 `1` 也会执行精排，因此与不设置不同。 |
| `ef` | 否 | `floor(1.5 × (top_k + offset))` | HNSW
图索引搜索时保留的候选宽度，必须为正整数。如果同时设置了 `refine_factor`，默认值为 `floor(1.5 × (top_k + offset)
× refine_factor)`。对非 HNSW 索引无效。 |
-| `use_index` | 否 | `true` | `true` 表示将兼容的物理 Lance Index Segment 规划为索引
Split，并将未覆盖的 Fragment 保留为 Flat Search Split；如果没有可用的兼容索引元数据，则退回按 Fragment
拆分。`false` 表示每个可见 Fragment 生成一个 Split，并强制执行 Flat Search。 |
+| `use_index` | 否 | `true` | `true` 表示优先使用与向量列和距离类型兼容的 Lance 向量索引；没有可用索引时自动使用
Flat Search。`false` 表示禁用向量索引，对数据执行 Flat Search。 |
以上默认值对应 Doris 当前集成的 Lance Scanner 行为。`metric` 未指定时，Doris 在选择向量索引时按 `l2`
处理，因此不会选中使用 `cosine` 或 `dot` 创建的索引。未选中索引或 `"use_index" = "false"` 时，`uint8`
向量使用 `hamming`，其他当前支持的向量元素类型使用 `l2`。
@@ -489,43 +513,39 @@ ORDER BY _distance ASC, user_id;
| `IVF_HNSW_SQ` | IVF、HNSW 与 Scalar Quantization |
`nprobes`、`ef`、`refine_factor` |
| `IVF_HNSW_PQ` | IVF、HNSW 与 Product Quantization |
`nprobes`、`ef`、`refine_factor` |
-`vector_search()` 只负责查询，不负责在 Doris 中创建索引，也不提供指定索引类型或索引名称的参数。当 `use_index=true`
时，FE 从固定的数据集快照读取向量索引元数据，并选择与向量列和 Metric 兼容的一个逻辑索引；随后将该逻辑索引中仍覆盖可见数据的每个物理 Segment
分配给一个索引 Scan Split。每个索引 Split 都携带 Segment UUID 以及该 Segment 覆盖且在当前快照中可见的
Fragment，因此 BE 会检索指定的物理 Segment，而不是再次让 Lance 自行选择索引。
-
-一个 Lance 逻辑索引可以包含多个物理 Index Segment，一个物理 Segment 也可以覆盖多个 Fragment。未被所选索引覆盖的
Fragment 不会被遗漏：Doris 会为每个这样的 Fragment 增加一个执行 Flat Search 的回退 Split。如果 FE
无法生成可用的 Index Segment 计划，则退回按 Fragment 拆分。当 `use_index=false` 时，Doris
跳过索引元数据规划，并对每个可见 Fragment 强制执行 Flat Search。Flat Search 不是一种 ANN 索引，它需要在 Lance
内直接读取并比较向量。
+`vector_search()` 只使用 Lance 中已有的向量索引，不负责创建索引，也不能指定索引类型或索引名称。当 `use_index=true`
时，Doris 自动选择与向量列和距离类型兼容的索引；没有可用索引或索引未覆盖的数据会自动使用 Flat Search，不会被遗漏。当
`use_index=false` 时，所有数据都使用 Flat Search。
### 支持的向量元素类型和距离类型
-索引能否用于 `vector_search()`，取决于向量元素类型以及创建索引时使用的距离类型，因为内置的 Lance
版本并未实现全部组合。下表描述使用索引检索的支持情况，其内容来自实测而非推断：元素类型 x 距离类型 x 索引类型矩阵的每个组合都在内置 Lance
版本上实际构建过，标记为可用的正是那些构建成功并能返回检索结果的组合。
-
-每个可用组合都由回归测试覆盖，其中有一项检查对全部组合生效：把检索限制到单个 IVF 分区必须改变结果。Flat Search
没有分区概念，不可能因此给出不同结果，所以这一条能区分真正走索引和静默回退。
-
-大部分组合还额外断言两件事：Doris 规划出索引分片且没有任何 Fragment
退化为无索引扫描；以及索引检索在对候选做精确距离重排后，返回的行与穷举扫描完全一致。按算法、元素类型和距离类型各取的一张代表表则是**换了一种方式**覆盖，而非更强：有提交在库中的查询结果作为基线，数据形状有闭式解时还对照可手工推导的距离值。对量化类和图类算法，这些表只**记录**它们与穷举扫描的一致性而不断言
—— 因为那种一致性来自这份冻结的 fixture，并不是算法本身的保证。所有层次都不测量召回率。
+向量索引支持的距离类型取决于向量元素类型。请选择下表中支持的组合；不支持的组合无法使用向量索引。
| 向量元素类型 | `l2` | `cosine` | `dot` | `hamming` |
|---|---|---|---|---|
-| `float16` | 支持，但受数值范围限制 [1] | 支持 | 支持 | 不支持 |
+| `float16` | 支持 [1] | 支持 | 支持 | 不支持 |
| `float32` | 支持 | 支持 | 支持 | 不支持 |
| `float64` | 支持 | 支持 | 支持 | 不支持 |
| `uint8` | 不支持 | 不支持 | 不支持 | 仅 `IVF_FLAT` 和 `IVF_HNSW_FLAT` [2] |
| `int8` | 仅支持 Flat Search [3] | 仅支持 Flat Search [3] | 仅支持 Flat Search [3] |
不支持 |
-除脚注另有说明外，表中可用的组合适用于上文列出的全部六种索引类型。
+除脚注另有说明外，“支持”表示该组合适用于上文列出的全部六种索引类型；“仅支持 Flat Search”表示可以查询，但不会使用向量索引。
-1. 使用 `l2` 创建 `float16` 索引，随着数据数值增大会变得不可靠，且没有一个干净的阈值可以给出。在 16
维共线向量上实测（最大平方距离随行数增长）：2.6e5 和 1.0e6 时约一秒建成，1.7e7 时始终无法完成，而 4.2e6
时对同一份输入四次尝试只成功一次 —— 说明聚类过程的随机性也参与其中，不只取决于数据。在数值有界的向量上（例如单位量级的
embedding），六种索引类型全部可以正常构建和检索。此处并未确定成因，所以上述数字应视为观测结果而非规则。`cosine` 和 `dot`
在该范围内的每一组数据上都能成功构建，包括 `l2` 无法完成的那些规模；`dot` 还额外在同样形状放大 32 倍（坐标约
33000）的数据上构建成功，因此并非只是在 `l2` 失败的量级上未经测试。这就是已验证的范围；更接近 `float16` 上限 65504
时的行为不在其中，那时存储值本身已无
法表示。如果 `float16` 的 `l2` 索引无法构建完成，可改用 `cosine`，或使用 Flat Search —— 后者对 `float16
[...]
-2. Lance 将 `uint8` 向量视为二进制向量，因此只接受 `hamming` 距离。其 Product Quantization 和
Scalar Quantization 构建器不接受 `uint8`，所以该元素类型只能使用 `IVF_FLAT` 和 `IVF_HNSW_FLAT`。
-3. Lance 目前没有完整的 `int8` 索引构建路径，因此 `int8` 向量只能使用 Flat Search 检索。此外，在当前内置的 Lance
版本下，检索**可为 NULL** 的 `int8` 向量列会导致 BE 进程退出。在 Doris
升级到包含该修复（[lance#7498](https://github.com/lance-format/lance/pull/7498)）的
`lance-c` 版本之前，请避免检索可为 NULL 的 `int8` 向量列。
-
-Lance 在创建索引时就会拒绝的组合不会影响 Doris，例如使用 `hamming` 创建浮点类型索引，或使用 `l2`、`cosine`、`dot`
创建 `uint8` 索引。
+1. `float16` 数据数值较大时，使用 `l2` 创建索引可能耗时过长或失败。建议使用数值范围受控的向量；如果建索引失败，可使用 Flat
Search，或在业务语义允许时改用 `cosine`。
+2. Lance 将 `uint8` 向量视为二进制向量，只支持 `hamming` 距离。`uint8` 不支持 Product Quantization
或 Scalar Quantization，因此只能使用 `IVF_FLAT` 和 `IVF_HNSW_FLAT`。
+3. `int8` 向量目前不能使用向量索引，只能执行 Flat Search。当前内置的 Lance 版本还存在一个已知问题：检索**可为 NULL**
的 `int8` 向量列可能导致 BE 进程退出。在 Doris 升级到包含
[lance#7498](https://github.com/lance-format/lance/pull/7498) 修复的 `lance-c`
版本前，请避免此类查询。
#### 查询距离类型必须与索引一致
-只有当查询请求的 `metric` 与创建索引时使用的距离类型一致时，Doris 才会选择该向量索引；否则 Doris 不使用索引，改为执行 Flat
Search——结果仍然正确，但需要直接扫描向量。`EXPLAIN` 可以区分两者：使用索引的计划中 `lanceSearchIndexSegments`
不为 0，未选中索引时显示 `lanceSearchIndexSegments=0`。由于未设置的 `metric` 会按 `l2` 处理，凡是不以 `l2`
创建的索引都必须显式设置 `metric`。这包括所有 `uint8` 列：它的索引只能使用 `hamming`，因此不设置 `metric`
时**必然**选不中索引、退回 Flat Search。返回的行仍然正确（flat 路径对 `uint8` 应用 `hamming`），但索引没有被使用。
+查询指定的 `metric` 必须与索引创建时使用的距离类型一致，否则 Doris 会改用 Flat
Search。结果仍然正确，但由于需要直接扫描向量，性能通常更低。
+
+不设置 `metric` 时，Doris 在选择索引时按 `l2` 处理。因此，使用 `cosine`、`dot` 或 `hamming`
索引时必须显式设置 `metric`。例如，查询 `uint8` 向量索引时必须设置 `"metric" = "hamming"`，否则不会使用索引。
-此外，Doris 对每个向量列只会考虑一个索引。如果同一个向量列上存在多个使用不同距离类型创建的向量索引，Doris
选中的可能并不是与查询距离类型匹配的那一个，查询会退回 Flat Search。建议每个向量列最多创建一个向量索引，使索引选择结果明确。
+可以通过 `EXPLAIN` 确认是否使用索引：`lanceSearchIndexSegments` 大于 `0` 表示使用了索引，等于 `0` 表示使用
Flat Search。
+
+Doris 对每个向量列只考虑一个向量索引。建议每个向量列最多创建一个向量索引，避免多个不同距离类型的索引导致选择不明确。
### Prefilter 和 Post-filter
-TVF 的 `filter` 参数是 **Prefilter**。Doris 将该字符串传给每个搜索 Split 的 Lance Scanner，Lance
在 ANN 或 Flat Search 生成候选之前执行过滤：
+Prefilter 和 Post-filter 可以在同一个查询中使用：
```sql
SELECT user_id, category, _distance
@@ -536,71 +556,42 @@ FROM vector_search(
"top_k" = "10",
"filter" = "category = 'book'"
)
+WHERE user_id > 100
ORDER BY _distance ASC, user_id;
```
-`filter` 中引用的列由 Lance 内部读取并计算；如果该列没有被 `SELECT` 或其他 Doris 表达式引用，它不需要作为列返回给
Doris。
-
-外层 `WHERE` 是 **Post-filter**。优化器会将它下移到 Doris 的 Lance Scan 中，但不会把它转换成 Lance 的
Prefilter。它的执行位置是：Lance 为每个搜索 Split 生成候选之后、Doris 执行局部和全局 TopN 之前。
-
-```sql
-SELECT user_id, category, _distance
-FROM vector_search(
- "table" = "lance_catalog.default.items",
- "column" = "embedding",
- "query_vector" = "[0.1, 0.2, 0.3, 0.4]",
- "top_k" = "10"
-)
-WHERE category = 'book'
-ORDER BY _distance ASC, user_id;
-```
-
-因此，外层 `WHERE` 只过滤已经生成的候选，不会触发 Lance 补充候选，最终结果可能少于
`top_k`。如果过滤条件应该缩小向量候选的搜索空间并保证在过滤后的数据中选择最近邻，应使用 TVF 的 `filter` 参数。
+| 过滤方式 | 示例条件 | 执行时机 | 对结果的影响 |
+|---|---|---|---|
+| Prefilter | TVF 参数 `"filter" = "category = 'book'"` | Lance 生成向量候选之前 | 只在
`category = 'book'` 的数据中搜索最近邻。 |
+| Post-filter | 外层 `WHERE user_id > 100` | Lance 生成候选之后、Doris 执行最终 TopN 之前 |
从已生成的候选中删除不满足条件的行，不会补充新的候选，因此最终结果可能少于 `top_k`。 |
-### 当前执行方式
+如果 `user_id > 100` 也必须参与最近邻候选生成，应将它合并到 `filter` 中，例如 `"filter" = "category =
'book' AND user_id > 100"`，而不是使用外层 `WHERE`。即使优化器将外层 `WHERE` 下移到 Doris 的 Lance
Scan，它仍然是 Post-filter，不会转换成 Lance Prefilter。
-`vector_search()` 使用分布式候选搜索，而不是由一个 Scanner 扫描整个数据集。Split 的边界取决于索引覆盖范围：
+`filter` 中引用的列由 Lance 内部读取和计算；如果该列未被 `SELECT` 或其他 Doris 表达式引用，则不需要返回给 Doris。
-1. FE 在规划阶段固定一个正数版本的 Lance 数据集快照，并读取该快照中可见的 Fragment；当 `use_index=true`
时，还会读取向量索引元数据。
-2. 如果存在具有可用 Segment 覆盖信息的兼容逻辑向量索引，每个仍覆盖可见 Fragment 的物理 Index Segment 都会生成一个索引
Scan Split。该 Split 包含 Segment UUID，以及其 Fragment Bitmap 与固定快照中可见 Fragment
的交集，因此一个 Split 可以包含多个 Fragment ID。
-3. 没有被这些索引 Split 覆盖的每个可见 Fragment，都会生成一个独立的回退 Fragment
Split。这样，即使数据是在索引创建后追加的、尚未执行索引优化，也仍然可以被检索。如果不存在可用的 Index Segment 计划，所有可见
Fragment 都按 Fragment 拆分；当 `use_index=false` 时，所有可见 Fragment 直接使用 Flat Search
Split。
-4. 假设查询参数为 `top_k=K`、`offset=n`，每个索引或回退 Split 都请求最多 `K+n` 个候选，并且不在 Split 内应用
offset。索引 Split 只检索为其分配的物理 Index Segment，回退 Fragment Split 对自身 Fragment 执行 Flat
Search。TVF 的 `filter` 在候选生成前由 Lance 执行，外层 `WHERE` 则在候选生成后由 Doris Scan 执行。
-5. Doris 对所有 Split 返回的候选执行局部 TopN、Exchange 和全局 TopN，按 `_distance ASC` 合并；只有全局
TopN 应用 `offset=n`，跳过前 `n` 行后返回 `K` 行。
+### TopN 两阶段读取和延迟物化
-因此，Split 级候选集只用于向全局合并提供候选，不能直接视为最终结果。索引 Segment Split、回退 Fragment Split 以及后续按
Row ID 取列都使用同一个固定快照。刷新索引覆盖会改变新追加 Fragment 的检索方式，但未被索引覆盖的 Fragment 仍会通过 Flat
Search 进入检索范围。
+向量搜索通常会生成多于最终 `top_k` 的候选行。如果在搜索阶段就读取 `title`、`payload` 等较宽的输出列，大部分数据会在 TopN
排序后被丢弃。两阶段读取会延迟读取这些列：第一阶段只读取过滤、排序和 TopN 所需的列；全局 TopN 完成后，第二阶段仅为最终保留的行读取其他输出列。
-执行顺序可以概括为：
+这种方式可以减少存储 I/O、网络传输和内存占用。当 `top_k` 较小、候选较多，或者查询包含较宽的字符串、JSON 等输出列时，收益通常更明显。
-```text
-固定数据集快照
- -> FE Split 规划
- -> 索引覆盖：每个物理 Index Segment 一个 Split -> ANN Search
- -> 未覆盖或无索引数据：每个 Fragment 一个 Split -> Flat Search
- -> 每个 Split：Lance Prefilter -> ANN/Flat Search -> 最多 K+n 个候选
- -> Doris Scan Post-filter
- -> Doris 局部 TopN
- -> Exchange
- -> Doris 全局 TopN（应用 offset=n 和 limit=K）
- -> 可选的延迟物化 Fetch
-```
-
-### TopN 两阶段读取和延迟物化
+| 阶段 | 读取内容 |
+|---|---|
+| 第一阶段 | 向量搜索内部需要的列、`_distance`、Post-filter 和排序使用的列，以及内部 Row Location。 |
+| 第二阶段 | 只为全局 TopN 保留的行读取仅用于最终输出的顶层列。 |
-当 `experimental_topn_lazy_materialization_threshold` 大于 `0`、`top_k`
不超过该阈值，并且存在可以延迟读取的顶层列时，`vector_search()` 可以使用两阶段读取。默认阈值为 `1024`。第一阶段只传递完成候选过滤和
TopN 所必需的列以及内部 Row Location；全局 TopN 完成后，第二阶段只为最终保留的行读取其他输出列。
+#### 触发条件
-例如，源表包含以下列：
+满足以下条件时，Doris 才会使用两阶段读取：
-| 列 | 用途 |
-|---|---|
-| `user_id` | 最终输出列 |
-| `category` | 外层 `WHERE` 的 Post-filter 列 |
-| `title`、`payload` | 最终输出列 |
-| `embedding` | Lance 向量搜索列 |
+1. `topn_lazy_materialization_threshold` 大于 `0`，默认值为 `1024`。
+2. `top_k` 不超过该阈值。
+3. 查询至少包含一个可以延迟读取的顶层列。仅用于最终 `SELECT` 输出的列可以延迟；Post-filter、`ORDER BY` 或其他 TopN
前表达式使用的列必须在第一阶段读取。嵌套子列当前不能延迟读取。
-执行以下查询，其中 `K=10`、`n=3`：
+例如，以下查询中的 `category` 必须在第一阶段用于 Post-filter，而 `user_id`、`title` 和 `payload` 可以在
TopN 完成后读取：
```sql
-SET experimental_topn_lazy_materialization_threshold = 1024;
+SET topn_lazy_materialization_threshold = 1024;
SELECT user_id, title, payload, _distance
FROM vector_search(
@@ -613,21 +604,40 @@ FROM vector_search(
WHERE category = 'book';
```
-典型的两阶段列流转如下：
+TVF `filter` 中引用的 Prefilter 列由 Lance 在搜索内部使用，不会仅因为出现在 `filter`
字符串中就作为第一阶段结果列返回给 Doris。
-| 阶段或算子 | 读取或输出的列 | 说明 |
-|---|---|---|
-| Lance Split Search | 内部使用 `embedding`；向 Doris 返回 `_distance`、`category` 和内部
Lance Row ID | `embedding` 用于 ANN/Flat Search，但没有被 SQL 投影时不作为结果列返回。每个 Index
Segment 或回退 Fragment Split 最多产生 `K+n` 个候选。 |
-| Doris Scan Post-filter | `_distance`、`category`、内部 Row Location | 执行
`category = 'book'`。外层 `WHERE` 的列必须留在第一阶段。Doris 将 Lance Row ID 和数据集映射编码为内部 Row
Location，Fetch 再通过该映射解析到同一个固定快照。 |
-| 局部和全局 TopN | 第一阶段必需列和内部 Row Location | 全局 TopN 按 `_distance` 合并，应用
`offset=n` 和 `limit=K`。 |
-| Row ID Fetch | 使用内部 Row Location 读取 `user_id`、`title`、`payload` | 对全局 TopN
保留的行，在同一个 Lance 数据集快照上调用 Row-ID 随机读取，不重新扫描 Fragment。 |
-| 最终 Materialize | `user_id`、`title`、`payload`、`_distance` |
将延迟列与第一阶段保留的列合并，形成 SQL 最终输出。 |
+#### 关闭两阶段读取
+
+设置以下会话变量可以关闭两阶段读取：
+
+```sql
+SET topn_lazy_materialization_threshold = -1;
+```
+
+当 `top_k` 大于阈值或没有可延迟列时，Doris 会自动使用单阶段读取，通常不需要手动关闭。以下情况可以考虑关闭并进行性能对比：
+
+- `top_k` 很大，第二阶段仍需读取大部分候选行。
+- 查询只返回少量窄列，两阶段能够减少的 I/O 很少。
+- 底层存储的随机 Row-ID 读取延迟较高，第二阶段 Fetch 的额外开销超过延迟读取带来的收益。
-第一阶段的必需列不只包括 `_distance` 和 Post-filter 列。凡是在全局 TopN 完成前被 Doris
表达式或算子引用的列，都属于第一阶段列。例如，如果查询增加 `ORDER BY _distance, user_id`，`user_id`
也需要提前读取，不能再等到第二阶段 Fetch。嵌套子列投影当前也不会延迟到 Row-ID Fetch。相反，只被最终投影使用的顶层列可以在第二阶段读取。
+关闭后只会改变输出列的读取时机，不会禁用向量索引或改变 Prefilter、Post-filter 和 TopN
的语义。建议根据实际查询和存储环境进行开启与关闭的性能对比。
-TVF `filter` 中引用的 Prefilter 列与外层 `WHERE` 列不同：前者由 Lance 在搜索内部使用，并不因为出现在
`filter` 字符串中就必须返回到 Doris；后者由 Doris Scan 执行，所以必须进入第一阶段。
+## 当前执行方式
-将 `experimental_topn_lazy_materialization_threshold` 设置为 `-1` 会关闭两阶段读取。如果
`top_k` 大于阈值，或者没有可延迟的列，也会使用单阶段读取。单阶段模式会在 Scan 阶段返回查询所需的全部输出列，但向量搜索仍然按 Index
Segment 或回退 Fragment Split 并行生成候选，并由 Doris 合并全局 TopN；它不会因此退化为 Doris
对整张表做普通全列扫描。索引 Split 使用为其分配的物理 Index Segment，强制或回退到 Flat Search 时才直接比较向量。
+`vector_search()` 的执行顺序如下：
+
+```text
+固定数据集快照
+ -> FE Split 规划
+ -> 索引覆盖：每个物理 Index Segment 一个 Split -> ANN Search
+ -> 未覆盖或无索引数据：每个 Fragment 一个 Split -> Flat Search
+ -> 每个 Split：Lance Prefilter -> ANN/Flat Search -> 最多 K+n 个候选
+ -> Doris Scan Post-filter
+ -> Doris 局部 TopN
+ -> Exchange
+ -> Doris 全局 TopN（应用 offset=n 和 limit=K）
+ -> 可选的延迟物化 Fetch
+```
## 当前限制和建议
diff --git a/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
b/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
index 69d16d8d90f..562773b0ef4 100644
--- a/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
+++ b/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
@@ -6,21 +6,33 @@
}
---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
:::note
This is an experimental feature.
Lance Catalog is supported starting from Apache Doris 4.2.
:::
-Lance is a columnar data format designed for analytics and AI workloads. Doris
can use a Lance Catalog to discover databases and tables in a Lance Namespace
and directly query Lance datasets stored on a local file system or
S3-compatible object storage.
+Lance is a columnar data format designed for analytics and AI workloads. Doris
can use a Lance Catalog to discover databases and tables in a Lance Namespace
and directly query Lance datasets stored on a local file system, S3-compatible
object storage, or Alibaba Cloud OSS.
Doris currently provides read-only access to Lance. Creating, writing,
updating, or deleting Lance tables is not supported.
+## Use Cases
+
+| Use case | Description |
+|---|---|
+| Query Lance data directly | Analyze existing Lance datasets with Doris SQL
without migrating or importing the data first. |
+| AI vector search | Reuse vector indexes already present in Lance, run
searches through `vector_search()`, and combine them with scalar filtering and
Doris SQL analytics. |
+| Data integration | Read Lance data into Doris internal tables for further
processing, joins, or long-term storage. |
+| Namespace management | Use a Filesystem Catalog for a simple directory-based
layout. Use a REST Catalog when Namespaces, table locations, or temporary
storage credentials need centralized management. |
+
## Feature Overview
| Feature | Support |
|---|---|
-| Filesystem Catalog | Supports warehouses on a local file system, `file://`,
or `s3://` |
+| Filesystem Catalog | Supports warehouses on a local file system, `file://`,
`s3://`, or `oss://` |
| REST Catalog | Supports Lance REST Namespace with no authentication, Bearer
Token, API Key, or custom HTTP headers |
| Metadata access | Supports `SHOW DATABASES`, `SHOW TABLES`, `DESC`, and
`SHOW INDEX` (Filesystem Catalogs only) |
| System tables (`table$...`) | Not supported; use `SHOW INDEX` for index
metadata |
@@ -85,54 +97,55 @@ A Filesystem Catalog discovers Lance Namespaces and tables
directly from a wareh
|---|---|---|
| `warehouse` | Yes | Root path of the Lance warehouse. Local absolute paths,
`file://` URIs, `s3://` URIs, and `oss://` URIs are supported. |
-#### Use S3-Compatible Object Storage
-
-The following example creates a Catalog for MinIO:
-
-```sql
-CREATE CATALOG lance_catalog PROPERTIES (
- "type" = "lance",
- "lance.catalog.type" = "filesystem",
- "warehouse" = "s3://my-bucket/lance",
- "s3.endpoint" = "http://127.0.0.1:9000";,
- "s3.access_key" = "admin",
- "s3.secret_key" = "password",
- "s3.region" = "us-east-1",
- "use_path_style" = "true"
-);
-```
-
-When accessing AWS S3, you can omit `s3.endpoint` and configure credentials,
Region, and Path Style for your environment.
-
-#### Use Alibaba Cloud OSS
-
-```sql
-CREATE CATALOG lance_oss PROPERTIES (
- "type" = "lance",
- "lance.catalog.type" = "filesystem",
- "warehouse" = "oss://my-bucket/lance",
- "oss.endpoint" = "oss-cn-beijing.aliyuncs.com",
- "oss.region" = "cn-beijing",
- "oss.access_key" = "<ak>",
- "oss.secret_key" = "<sk>"
-);
-```
-
-`warehouse` also accepts the qualified form
`oss://my-bucket.oss-cn-beijing.aliyuncs.com/lance`, which Doris reduces to the
bucket before opening the Namespace. It must always name a bucket, so a root
such as `oss:/lance` is rejected when the Catalog is created.
-
-To access OSS with a temporary STS credential, add `"oss.session_token" =
"<token>"`.
-
-#### Use a Local File System
-
-```sql
-CREATE CATALOG lance_local PROPERTIES (
- "type" = "lance",
- "lance.catalog.type" = "filesystem",
- "warehouse" = "/data/lance"
-);
-```
-
-For a local file system, `warehouse` must be an absolute path. The FE must be
able to read Namespace and table metadata through this path, and each BE that
executes a query must be able to access the data through the same path. In a
multi-node deployment, mount the same shared directory on all relevant FE and
BE nodes.
+Select the example that matches the storage system hosting the warehouse:
+
+<Tabs groupId="lance-filesystem-storage">
+ <TabItem value="s3" label="S3 / S3-Compatible" default>
+ ```sql
+ CREATE CATALOG lance_fs_s3 PROPERTIES (
+ "type" = "lance",
+ "lance.catalog.type" = "filesystem",
+ "warehouse" = "s3://my-bucket/lance",
+ "s3.endpoint" = "https://s3.us-east-1.amazonaws.com";,
+ "s3.region" = "us-east-1",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>"
+ );
+ ```
+
+ You can omit `s3.endpoint` for AWS S3. For S3-compatible storage such
as MinIO, set the service endpoint and add `"use_path_style" = "true"` when the
service requires path-style access.
+ </TabItem>
+
+ <TabItem value="oss" label="Alibaba Cloud OSS">
+ ```sql
+ CREATE CATALOG lance_fs_oss PROPERTIES (
+ "type" = "lance",
+ "lance.catalog.type" = "filesystem",
+ "warehouse" = "oss://my-bucket/lance",
+ "oss.endpoint" = "https://oss-cn-beijing.aliyuncs.com";,
+ "oss.region" = "cn-beijing",
+ "oss.access_key" = "<ak>",
+ "oss.secret_key" = "<sk>"
+ );
+ ```
+
+ For temporary STS credentials, add `"oss.session_token" = "<token>"`.
The qualified form `oss://my-bucket.oss-cn-beijing.aliyuncs.com/lance` is also
accepted and normalized to the bucket path. OSS-HDFS is not currently supported.
+ </TabItem>
+
+ <TabItem value="local" label="Local File System">
+ ```sql
+ CREATE CATALOG lance_fs_local PROPERTIES (
+ "type" = "lance",
+ "lance.catalog.type" = "filesystem",
+ "warehouse" = "/data/lance"
+ );
+ ```
+
+ `warehouse` must be an absolute path. The FE and every BE executing
the query must access the same path. In a multi-node deployment, mount the same
shared directory on all relevant nodes.
+ </TabItem>
+</Tabs>
+
+For S3 and OSS, `warehouse` must include a bucket, such as `s3://bucket/path`
or `oss://bucket/path`.
### REST Catalog
@@ -146,38 +159,49 @@ A REST Catalog obtains Namespaces, table locations, and
storage access parameter
| `lance.rest.api-key` | Yes for API Key authentication | - | API Key sent in
the `x-api-key` header. |
| `lance.rest.header.<header-name>` | No | - | Custom HTTP header sent to the
REST service. Use the dedicated authentication properties above for
authentication headers. |
-The following example creates a REST Catalog using a Bearer Token:
-
-```sql
-CREATE CATALOG lance_rest PROPERTIES (
- "type" = "lance",
- "lance.catalog.type" = "rest",
- "lance.rest.uri" = "https://lance.example.com";,
- "lance.rest.security.type" = "bearer",
- "lance.rest.bearer-token" = "your-token"
-);
-```
-
-For API Key authentication, replace the authentication properties with:
-
-```sql
-"lance.rest.security.type" = "api_key",
-"lance.rest.api-key" = "your-api-key"
-```
-
-If the REST service returns temporary storage credentials, Doris uses those
credentials to access the corresponding Lance table. You can also configure
default object storage access parameters in the Catalog: `s3.endpoint`,
`s3.access_key`, `s3.secret_key`, `s3.region`, and `use_path_style` for
S3-compatible storage, or `oss.endpoint`, `oss.access_key`, `oss.secret_key`,
`oss.region`, and `oss.session_token` for OSS.
-
-Credentials vended by the Namespace take precedence over the ones configured
in the Catalog, per table. A Namespace that vends complete credentials
therefore lets you create the Catalog without storing any access key in it:
-
-```sql
-CREATE CATALOG lance_rest_oss PROPERTIES (
- "type" = "lance",
- "lance.catalog.type" = "rest",
- "lance.rest.uri" = "https://lance.example.com";
-);
-```
-
-For an OSS table, the Namespace may spell the vended options either as
`oss_endpoint`, `oss_access_key_id`, `oss_secret_access_key`, `oss_region`, and
`oss_security_token`, or with the bare OSS-native names `endpoint`,
`access_key_id`, `access_key_secret`, `region`, and `security_token`. Doris
accepts both and treats them as the same option. Vending the same option under
both spellings with different values is an error.
+The REST Namespace returns the storage location of each table. Configure
default access properties for the storage system hosting those tables:
+
+<Tabs groupId="lance-rest-storage">
+ <TabItem value="s3" label="S3 / S3-Compatible" default>
+ ```sql
+ CREATE CATALOG lance_rest_s3 PROPERTIES (
+ "type" = "lance",
+ "lance.catalog.type" = "rest",
+ "lance.rest.uri" = "https://lance.example.com";,
+ "lance.rest.security.type" = "bearer",
+ "lance.rest.bearer-token" = "<token>",
+ "s3.endpoint" = "https://s3.us-east-1.amazonaws.com";,
+ "s3.region" = "us-east-1",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>"
+ );
+ ```
+
+ This configuration applies to `s3://` Lance tables returned by the
REST Namespace. For S3-compatible storage such as MinIO, replace `s3.endpoint`
with the service endpoint and add `"use_path_style" = "true"` when required.
+ </TabItem>
+
+ <TabItem value="oss" label="Alibaba Cloud OSS">
+ ```sql
+ CREATE CATALOG lance_rest_oss PROPERTIES (
+ "type" = "lance",
+ "lance.catalog.type" = "rest",
+ "lance.rest.uri" = "https://lance.example.com";,
+ "lance.rest.security.type" = "bearer",
+ "lance.rest.bearer-token" = "<token>",
+ "oss.endpoint" = "https://oss-cn-beijing.aliyuncs.com";,
+ "oss.region" = "cn-beijing",
+ "oss.access_key" = "<ak>",
+ "oss.secret_key" = "<sk>"
+ );
+ ```
+
+ This configuration applies to `oss://` Lance tables returned by the
REST Namespace. For temporary STS credentials, add `"oss.session_token" =
"<token>"`.
+ </TabItem>
+</Tabs>
+
+For unauthenticated REST access, omit `lance.rest.security.type` and the
authentication property. For API Key authentication, replace them with
`"lance.rest.security.type" = "api_key"` and `"lance.rest.api-key" =
"<api-key>"`.
+
+If the REST Namespace vends temporary storage credentials for a table, Doris
gives them precedence over the Catalog credentials, allowing the S3 or OSS
access keys to be omitted from the Catalog. For OSS, the Namespace may vend
`oss_endpoint`, `oss_access_key_id`, `oss_secret_access_key`, `oss_region`, and
`oss_security_token`, or the corresponding native names `endpoint`,
`access_key_id`, `access_key_secret`, `region`, and `security_token`.
:::caution
The current BE Reader does not support Lance tables whose versions are managed
by REST Namespace (Managed Versioning).
@@ -470,9 +494,9 @@ Do not use the unquoted form
`lance_catalog.doris.analytics.items`; it parses as
| `metric` | No | `hamming` for `uint8`, `l2` for the other supported types |
Distance metric: `l2`, `cosine`, `dot`, or `hamming`. `dot_product` is an alias
for `dot`. `uint8` vectors support only `hamming`; the other currently
supported vector element types support `l2`, `cosine`, and `dot`. An unset
`metric` is treated as `l2` when Doris selects a vector index, so querying an
index built with `cosine` or `dot` requires setting `metric` explicitly. |
| `filter` | No | - | Lance SQL condition evaluated before vector candidates
are generated; that is, a Prefilter. |
| `nprobes` | No | Minimum `1`, with no maximum | Number of IVF index
partitions to probe. It must be a positive integer. When unset, Lance starts
with one partition and can probe additional partitions when a Prefilter leaves
too few candidates. Setting it explicitly to `N` fixes both the minimum and
maximum number of probes to `N`. |
-| `refine_factor` | No | Refinement disabled | Candidate refinement
multiplier. It must be a positive integer. When unset, Lance does not recompute
distances from the original vectors, so `_distance` from a quantized index may
be approximate. When set to `N`, Lance first retrieves `(top_k + offset) × N`
candidates, recomputes their exact distances from the original vectors, and
reorders them. Setting it to `1` still enables refinement and therefore differs
from leaving it unset. |
+| `refine_factor` | No | Refinement disabled | Candidate refinement
multiplier. It must be a positive integer. When unset, Lance does not recompute
distances from the original vectors, so `_distance` from a quantized index may
be approximate. When set to `N`, Lance first retrieves `(top_k + offset) × N`
candidates, recomputes their exact distances from the original vectors, and
reorders them. **Refinement reads the original vector data for these
candidates. As `N` increases, more candida [...]
| `ef` | No | `floor(1.5 × (top_k + offset))` | Candidate width retained
during HNSW graph search. It must be a positive integer. If `refine_factor` is
also set, the default is `floor(1.5 × (top_k + offset) × refine_factor)`. It
has no effect on non-HNSW indexes. |
-| `use_index` | No | `true` | When `true`, Doris plans compatible physical
Lance index segments as indexed splits and keeps uncovered Fragments as Flat
Search splits. If no usable compatible index metadata is available, Doris falls
back to Fragment splits. When `false`, Doris creates one split per visible
Fragment and forces Flat Search. |
+| `use_index` | No | `true` | When `true`, Doris prefers a Lance vector index
compatible with the vector column and distance metric, and automatically uses
Flat Search if no usable index is available. When `false`, Doris disables
vector indexes and performs Flat Search over the data. |
These defaults correspond to the Lance Scanner behavior currently integrated
with Doris. When `metric` is omitted, Doris treats the query as `l2` while
selecting a vector index, so an index built with `cosine` or `dot` is not
selected. When no index is selected, or if `"use_index" = "false"`, `uint8`
vectors use `hamming`, while the other currently supported vector element types
use `l2`.
@@ -489,43 +513,39 @@ The embedded `lance-c v0.1.6` explicitly supports the
following Lance vector ind
| `IVF_HNSW_SQ` | IVF and HNSW with Scalar Quantization | `nprobes`, `ef`,
`refine_factor` |
| `IVF_HNSW_PQ` | IVF and HNSW with Product Quantization | `nprobes`, `ef`,
`refine_factor` |
-`vector_search()` only queries indexes. It does not create an index in Doris
and does not expose an index-type or index-name parameter. With
`use_index=true`, the FE reads vector-index metadata from the pinned dataset
snapshot and selects a logical index compatible with the vector column and
metric. It then assigns each physical segment of that logical index that still
covers visible data to an indexed Scan Split. Each indexed Split carries the
segment UUID and the currently visible Frag [...]
-
-A logical Lance index can contain multiple physical index segments, and one
physical segment can cover multiple Fragments. Fragments not covered by the
selected index are not omitted: Doris adds one fallback Split for each such
Fragment, which uses Flat Search. If the FE cannot construct a usable
index-segment plan, it falls back to Fragment-level splits. With
`use_index=false`, Doris skips index metadata planning and forces Flat Search
for every visible Fragment. Flat Search is not an A [...]
+`vector_search()` only uses vector indexes that already exist in Lance. It
does not create indexes or let users specify an index type or name. With
`use_index=true`, Doris automatically selects an index compatible with the
vector column and distance metric. Data without a usable index, including data
not covered by the selected index, automatically uses Flat Search and is not
omitted. With `use_index=false`, all data uses Flat Search.
### Supported Vector Element Types and Distance Metrics
-Whether an index can serve `vector_search()` depends on the vector element
type and the distance metric it was built with, because the embedded Lance
version does not implement every combination. The following matrix describes
index-backed search, and it is measured rather than inferred: every cell of the
element type x metric x index type matrix was built against the embedded Lance
version, and the combinations marked supported are exactly those that built and
answered a search.
-
-Every supported combination is covered by the regression suite, and one check
runs on all of them: restricting the search to a single IVF partition must
change the answer. A Flat Search has no partitions and cannot answer
differently, so this is what distinguishes a real indexed search from a silent
fallback.
-
-Most combinations are additionally asserted to plan an indexed split with no
fragment left scanned unindexed, and to return the same rows as an exhaustive
scan once their candidates are reranked with exact distances. A representative
table per algorithm, element type and metric is covered differently rather than
more: committed query results, and hand-derivable distances where the data
shape has a closed form. For the quantizing and graph algorithms those tables
record their agreement wi [...]
+The distance metrics supported by a vector index depend on the vector element
type. Choose a supported combination from the following table; unsupported
combinations cannot use a vector index.
| Vector element type | `l2` | `cosine` | `dot` | `hamming` |
|---|---|---|---|---|
-| `float16` | Supported, with a range caveat [1] | Supported | Supported | Not
supported |
+| `float16` | Supported [1] | Supported | Supported | Not supported |
| `float32` | Supported | Supported | Supported | Not supported |
| `float64` | Supported | Supported | Supported | Not supported |
| `uint8` | Not supported | Not supported | Not supported | `IVF_FLAT` and
`IVF_HNSW_FLAT` only [2] |
| `int8` | Flat Search only [3] | Flat Search only [3] | Flat Search only [3]
| Not supported |
-Unless a footnote states otherwise, a supported combination applies to all six
index types listed above.
+Unless a footnote states otherwise, "Supported" means that the combination
works with all six index types listed above. "Flat Search only" means that the
vectors can be queried, but no vector index is used.
-1. Building a `float16` index with `l2` becomes unreliable as the magnitude of
the data grows, and there is no clean threshold to quote. Measured on
16-dimensional collinear vectors, where the largest squared distance grows with
the row count: at 2.6e5 and 1.0e6 the build finishes in about a second, at
1.7e7 it never finishes, and at 4.2e6 it finished once in four attempts over
identical input - so run-to-run variation in the clustering is part of it, not
the data alone. Over bounded vec [...]
-2. Lance treats `uint8` vectors as binary vectors, so `hamming` is the only
distance it accepts for them. Its Product Quantization and Scalar Quantization
builders reject `uint8`, which leaves `IVF_FLAT` and `IVF_HNSW_FLAT` as the
only index types available for this element type.
-3. Lance does not currently provide a complete `int8` index build path, so
`int8` vectors can only be searched with Flat Search. In addition, with the
embedded Lance version, searching a **nullable** `int8` vector column
terminates the BE process. Avoid searching nullable `int8` vector columns until
Doris upgrades to a `lance-c` release that contains the fix,
[lance#7498](https://github.com/lance-format/lance/pull/7498).
-
-Combinations that Lance rejects when the index is created never reach Doris.
Building a floating-point index with `hamming`, or a `uint8` index with `l2`,
`cosine`, or `dot`, falls into this category.
+1. Building an `l2` index over `float16` data with large values may take too
long or fail. Keep vector values within a controlled range. If index creation
fails, use Flat Search or, when appropriate for the application, use `cosine`
instead.
+2. Lance treats `uint8` vectors as binary vectors and supports only the
`hamming` metric. Product Quantization and Scalar Quantization do not support
`uint8`, so only `IVF_FLAT` and `IVF_HNSW_FLAT` are available.
+3. `int8` vectors cannot currently use a vector index and are limited to Flat
Search. The embedded Lance version also has a known issue where searching a
**nullable** `int8` vector column may terminate the BE process. Avoid these
queries until Doris upgrades to a `lance-c` release containing the
[lance#7498](https://github.com/lance-format/lance/pull/7498) fix.
#### The Query Metric Must Match the Index Metric
-Doris selects a vector index only when the requested `metric` equals the
metric that index was built with. Otherwise Doris does not use the index and
runs Flat Search, which still returns correct results but scans the vectors
directly. `EXPLAIN` reports the difference: an indexed plan shows a non-zero
`lanceSearchIndexSegments`, while a declined index shows
`lanceSearchIndexSegments=0`. Since an unset `metric` is treated as `l2`,
remember to set `metric` explicitly for any index not buil [...]
+The query `metric` must match the metric used to build the index. Otherwise,
Doris uses Flat Search. The results remain correct, but performance is usually
lower because the vectors must be scanned directly.
+
+When `metric` is omitted, Doris treats it as `l2` while selecting an index.
Therefore, explicitly set `metric` when using an index built with `cosine`,
`dot`, or `hamming`. For example, a `uint8` vector index requires `"metric" =
"hamming"`; otherwise, the index is not used.
-Doris also considers only one index per vector column. If a vector column
carries several vector indexes built with different metrics, the one Doris
selects may not be the one matching the query metric, and the query then falls
back to Flat Search. Keep at most one vector index per vector column to make
index selection unambiguous.
+Use `EXPLAIN` to confirm index usage. A `lanceSearchIndexSegments` value
greater than `0` indicates indexed search, while `0` indicates Flat Search.
+
+Doris considers only one vector index per vector column. Keep at most one
vector index on each vector column to avoid ambiguous selection among indexes
built with different metrics.
### Prefilter and Post-Filter
-The TVF `filter` parameter is a **Prefilter**. Doris passes the string to the
Lance Scanner for each search Split, and Lance evaluates it before ANN or Flat
Search generates candidates:
+Prefilter and Post-filter can be used in the same query:
```sql
SELECT user_id, category, _distance
@@ -536,40 +556,75 @@ FROM vector_search(
"top_k" = "10",
"filter" = "category = 'book'"
)
+WHERE user_id > 100
ORDER BY _distance ASC, user_id;
```
+| Filter type | Example condition | When it runs | Effect on results |
+|---|---|---|---|
+| Prefilter | TVF parameter `"filter" = "category = 'book'"` | Before Lance
generates vector candidates | Searches for nearest neighbors only among rows
where `category = 'book'`. |
+| Post-filter | Outer `WHERE user_id > 100` | After Lance generates candidates
and before Doris applies the final TopN | Removes non-matching rows from the
generated candidates. It does not replenish candidates, so the final result may
contain fewer than `top_k` rows. |
+
+If `user_id > 100` must also participate in candidate generation, include it
in the TVF parameter, for example `"filter" = "category = 'book' AND user_id >
100"`, instead of using an outer `WHERE`. Even if the optimizer moves the outer
`WHERE` into the Doris Lance Scan, it remains a Post-filter and is not
converted into a Lance Prefilter.
+
Lance reads and evaluates columns referenced only by `filter` internally. If
such a column is not referenced by `SELECT` or another Doris expression, it
does not have to be returned to Doris.
-An outer `WHERE` is a **Post-filter**. The optimizer moves it into the Doris
Lance Scan, but does not convert it into a Lance Prefilter. It runs after Lance
generates candidates for each search Split and before Doris performs its local
and global TopN operations.
+### Two-Phase TopN Read and Lazy Materialization
+
+Vector search commonly produces more candidate rows than the final `top_k`.
Reading wide output columns such as `title` or `payload` during the search
wastes work because most candidate rows are discarded by TopN. Two-phase
reading defers these columns: Phase 1 reads only the columns required for
filtering, ordering, and TopN; after global TopN, Phase 2 reads the remaining
output columns only for the retained rows.
+
+This reduces storage I/O, network transfer, and memory usage. The benefit is
usually greatest when `top_k` is small, the search produces many candidates, or
the query returns wide string, JSON, or similar columns.
+
+| Phase | Data read |
+|---|---|
+| Phase 1 | Columns used internally by vector search, `_distance`, columns
required by Post-filter or ordering, and an internal Row Location. |
+| Phase 2 | Top-level columns used only by the final projection, read only for
rows retained by global TopN. |
+
+#### Trigger Conditions
+
+Doris uses two-phase reading only when all of the following conditions are met:
+
+1. `topn_lazy_materialization_threshold` is greater than `0`. Its default
value is `1024`.
+2. `top_k` does not exceed the threshold.
+3. At least one top-level column can be deferred. A column used only by the
final `SELECT` projection can be deferred. A column used by a Post-filter,
`ORDER BY`, or another expression before TopN must be read in Phase 1. Nested
subcolumns cannot currently be deferred.
+
+In the following query, `category` must be read in Phase 1 for the
Post-filter, while `user_id`, `title`, and `payload` can be fetched after TopN:
```sql
-SELECT user_id, category, _distance
+SET topn_lazy_materialization_threshold = 1024;
+
+SELECT user_id, title, payload, _distance
FROM vector_search(
"table" = "lance_catalog.default.items",
"column" = "embedding",
"query_vector" = "[0.1, 0.2, 0.3, 0.4]",
- "top_k" = "10"
+ "top_k" = "10",
+ "offset" = "3"
)
-WHERE category = 'book'
-ORDER BY _distance ASC, user_id;
+WHERE category = 'book';
```
-Consequently, an outer `WHERE` only filters candidates that have already been
generated and does not cause Lance to replenish them. The final result may
contain fewer than `top_k` rows. If the filter must reduce the vector search
space and nearest neighbors must be selected from the filtered rows, use the
TVF `filter` parameter.
+A Prefilter column referenced in the TVF `filter` is used internally by Lance.
Its presence in the `filter` string alone does not make it a Phase-1 result
column returned to Doris.
+
+#### Disabling Two-Phase Reading
+
+Use the following session setting to disable two-phase reading:
+
+```sql
+SET topn_lazy_materialization_threshold = -1;
+```
-### Current Execution Model
+Doris automatically uses a single-phase read when `top_k` exceeds the
threshold or no column can be deferred, so manual disabling is usually
unnecessary. Consider disabling it and comparing performance in the following
cases:
-`vector_search()` uses distributed candidate search instead of one Scanner for
the entire dataset. Its Split boundary depends on index coverage:
+- `top_k` is large, so Phase 2 still reads most candidate rows.
+- The query returns only a few narrow columns, leaving little I/O for
two-phase reading to save.
+- Random Row-ID reads have high latency on the underlying storage, and the
extra Phase-2 Fetch costs more than deferred reading saves.
-1. During planning, the FE pins a positive Lance dataset snapshot version and
reads the visible Fragments in that snapshot. When `use_index=true`, it also
reads vector-index metadata.
-2. If a compatible logical vector index has usable segment coverage, each
physical index segment that still covers visible Fragments becomes one indexed
Scan Split. The Split contains that segment's UUID and the intersection of its
Fragment bitmap with the visible Fragments in the pinned snapshot. A Split can
therefore contain multiple Fragment IDs.
-3. Every visible Fragment not covered by those indexed Splits becomes an
independent fallback Fragment Split. This keeps data appended after index
creation searchable without requiring the index to be optimized first. If no
usable index-segment plan exists, all visible Fragments use Fragment splits.
With `use_index=false`, all visible Fragments use Flat Search splits directly.
-4. For query parameters `top_k=K` and `offset=n`, every indexed or fallback
Split requests at most `K+n` candidates and does not apply the offset locally.
An indexed Split searches only its assigned physical index segment; a fallback
Fragment Split performs Flat Search for its Fragment. Lance evaluates the TVF
`filter` before generating candidates, while Doris Scan evaluates an outer
`WHERE` afterward.
-5. Doris performs local TopN, Exchange, and global TopN over candidates from
all Splits, merging by `_distance ASC`. Only the global TopN applies
`offset=n`: it skips the first `n` rows and returns `K` rows.
+Disabling this optimization changes only when output columns are read. It does
not disable vector indexes or change Prefilter, Post-filter, or TopN semantics.
Compare both settings with representative queries and storage conditions.
-A Split-level candidate set therefore only supplies candidates for global
merging and is not the final result. Index-segment splits, fallback Fragment
splits, and any later Row-ID fetches all use the same pinned snapshot
throughout the query. Refreshing index coverage changes how newly appended
Fragments are searched, but uncovered Fragments remain part of the result space
through Flat Search.
+## Current Execution Model
-The execution order is:
+The execution order of `vector_search()` is:
```text
Pinned dataset snapshot
@@ -584,51 +639,6 @@ Pinned dataset snapshot
-> Optional lazy-materialization Fetch
```
-### Two-Phase TopN Read and Lazy Materialization
-
-`vector_search()` can use a two-phase read when
`experimental_topn_lazy_materialization_threshold` is greater than `0`, `top_k`
does not exceed the threshold, and at least one top-level column is eligible
for deferred reading. The default threshold is `1024`. Phase 1 carries only the
columns required for candidate filtering and TopN, plus an internal Row
Location. After global TopN, Phase 2 reads the other output columns only for
the retained rows.
-
-For example, assume the source table has these columns:
-
-| Column | Purpose |
-|---|---|
-| `user_id` | Final output column |
-| `category` | Post-filter column used by the outer `WHERE` |
-| `title`, `payload` | Final output columns |
-| `embedding` | Lance vector-search column |
-
-Run this query with `K=10` and `n=3`:
-
-```sql
-SET experimental_topn_lazy_materialization_threshold = 1024;
-
-SELECT user_id, title, payload, _distance
-FROM vector_search(
- "table" = "lance_catalog.default.items",
- "column" = "embedding",
- "query_vector" = "[0.1, 0.2, 0.3, 0.4]",
- "top_k" = "10",
- "offset" = "3"
-)
-WHERE category = 'book';
-```
-
-A typical two-phase column flow is:
-
-| Stage or operator | Columns read or output | Description |
-|---|---|---|
-| Lance Split Search | Uses `embedding` internally; returns `_distance`,
`category`, and the internal Lance Row ID to Doris | `embedding` participates
in ANN/Flat Search but is not returned as a result column unless SQL projects
it. Each Index Segment or fallback Fragment Split produces at most `K+n`
candidates. |
-| Doris Scan Post-filter | `_distance`, `category`, and the internal Row
Location | Evaluates `category = 'book'`. A column used by an outer `WHERE`
must remain in Phase 1. Doris encodes the Lance Row ID and dataset mapping into
an internal Row Location, which Fetch resolves to the same pinned snapshot. |
-| Local and global TopN | Phase-1 required columns and the internal Row
Location | Global TopN merges by `_distance` and applies `offset=n` and
`limit=K`. |
-| Row ID Fetch | Uses the internal Row Location to read `user_id`, `title`,
and `payload` | Reads deferred columns for rows retained by global TopN from
the same Lance dataset snapshot without rescanning Fragments. |
-| Final Materialize | `user_id`, `title`, `payload`, and `_distance` |
Combines deferred columns with columns retained from Phase 1 to produce the
final SQL output. |
-
-Phase-1 required columns are not limited to `_distance` and Post-filter
columns. Any column referenced by a Doris expression or operator before global
TopN is operative and must be read in Phase 1. For example, adding `ORDER BY
_distance, user_id` makes `user_id` a Phase-1 column, so it cannot be deferred
to the Row-ID Fetch. Nested subcolumn projections are also not currently
deferred. A top-level column used only by the final projection can be fetched
in Phase 2.
-
-A Prefilter column referenced in the TVF `filter` differs from an outer
`WHERE` column. Lance uses the former internally during search, so appearing in
the `filter` string alone does not require the column to be returned to Doris.
Doris Scan evaluates the latter, so it must be present in Phase 1.
-
-Setting `experimental_topn_lazy_materialization_threshold` to `-1` disables
the two-phase read. A single-phase read is also used when `top_k` exceeds the
threshold or no column can be deferred. Single-phase mode returns all
query-required output columns from Scan, but vector search still generates
candidates in parallel per Index Segment or fallback Fragment Split and Doris
still merges a global TopN. It does not become an ordinary Doris full-column
table scan. Indexed splits use their a [...]
-
## Current Limitations and Recommendations
- Lance Catalogs and Lance TVFs are read-only. `CREATE TABLE`, `INSERT`,
`UPDATE`, `DELETE`, `TRUNCATE TABLE`, and writing data back to Lance are not
supported.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
