This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new e6d0ac62c74 [docs] Explain Lance array predicate pushdown and scalar
index verification (#4184)
e6d0ac62c74 is described below
commit e6d0ac62c741dd5355f1c34ba386b44fc0c97718
Author: Gabriel <[email protected]>
AuthorDate: Sat Oct 17 05:38:42 2026 +0800
[docs] Explain Lance array predicate pushdown and scalar index verification
(#4184)
## Summary
Document array-label predicate pushdown and scalar-index verification
for Lance Catalog in English and Chinese 4.x docs. Covers built-in
array_contains forms, NULL and ordered-subsequence restrictions, boolean
combinations, single-index scope, uncovered fragments and the
EXPLAIN/Profile evidence for actual index use.
## Verified versions
The implementation apache/doris#68687 has merged into branch-4.1; its
commit is included in 4.1.5-release. The page marks array-label pushdown
from 4.1.5 and corrects Lance Catalog's overall availability to 4.1.4.
No current mirror is added because master has no Lance catalog
implementation.
## Validation
Rebased on website master and compared the predicate converter,
scalar-index planner, Profile counter names and source release tags.
Both languages match. `git diff --check` passes; no source build or live
Lance query was run in this audit.
---------
Co-authored-by: morningman <[email protected]>
---
.../lakehouse/catalogs/lance-catalog.mdx | 46 +++++++++++++++++++++-
.../lakehouse/catalogs/lance-catalog.mdx | 46 +++++++++++++++++++++-
2 files changed, 88 insertions(+), 4 deletions(-)
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
index 0ecad36aa7d..3f802eb678a 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
@@ -582,8 +582,9 @@ Doris 会将语义兼容的谓词转换为 Substrait 表达式，并交给 Lance
| `utf8`、`large_utf8` | 支持 |
| `date32(day)` | 支持 |
| 无时区 `timestamp(s/ms/us)` | 支持 |
+| `list<utf8>` | 内置 `array_contains`，标签为非 `NULL` 字符串字面量；详见[数组标签谓词](#数组标签谓词) |
-其他可读取类型，例如 `float16`、`decimal256`、Binary、`date64`、Time、纳秒 Timestamp、带时区
Timestamp 和复杂类型，当前保留为 Doris 侧谓词。
+其他可读取类型，例如 `float16`、`decimal256`、Binary、`date64`、Time、纳秒 Timestamp、带时区
Timestamp，以及上述数组成员判断以外的复杂类型，当前保留为 Doris 侧谓词。
### 支持下推的操作符
@@ -596,13 +597,14 @@ Doris 会将语义兼容的谓词转换为 Substrait 表达式，并交给 Lance
| 布尔列、`NOT` 布尔列 | 直接引用布尔列 |
| `LIKE`、`NOT LIKE` | 直接引用 `utf8` 或 `large_utf8` 列，模式为字符串字面量，且不包含反斜杠转义、显式
`ESCAPE` 子句或 NUL 字符 |
| `starts_with`、`ends_with` | 使用内置函数，第一个参数直接引用 `utf8` 或 `large_utf8`
列，第二个参数为字符串字面量 |
+| `array_contains(labels, 'red')` | 直接引用 `list<utf8>` 列，标签为非 `NULL` 字符串字面量 |
| `AND` | 顶层 Conjunct 可以分别下推，不能下推的部分保留在 Doris |
| `OR` | 两个分支都能完整转换时下推 |
| `NOT` | 操作数能完整转换时下推 |
以下形式通常不会下推：
-- 除上述内置字符串函数之外的函数，或列上的算术表达式。名称为 `like`、`starts_with` 或 `ends_with`
的用户自定义函数不会按同名内置函数下推。
+- 除上述内置字符串函数和数组成员判断函数之外的函数，或列上的算术表达式。名称为 `like`、`starts_with`、`ends_with` 或
`array_contains` 的用户自定义函数不会按同名内置函数下推。
- 模式包含 NUL 字符的字符串谓词。使用反斜杠转义或显式 `ESCAPE` 子句的 `LIKE`、`NOT LIKE` 也保留在 Doris。
- `IN` 列表为空或包含 `NULL`。
- `OR` 或 `NOT` 中只有部分表达式可转换。
@@ -619,6 +621,46 @@ WHERE active
AND country LIKE 'C%';
```
+### 数组标签谓词
+
+从 Apache Doris 4.1.5 开始，数组标签下推支持对直接引用的 Lance `list<utf8>` 列执行
`array_contains`，标签必须为非 `NULL` 字符串字面量。较早的构建版本可能仍将这些条件保留在 Doris，请先通过 `EXPLAIN`
确认。`NULL` 数组仍返回 `NULL`，空数组不匹配，重复标签不会导致结果行重复。
+
+对于包含 `id` 列和 `list<utf8>` 类型 `labels` 列的 Lance 表：
+
+```sql
+EXPLAIN
+SELECT id
+FROM lance_catalog.default.items
+WHERE array_contains(labels, 'red')
+ AND array_contains(labels, 'blue');
+```
+
+该条件匹配同时包含两个标签的数组，不要求标签顺序或相邻位置。将 `AND` 改为 `OR` 表示匹配任一标签。`NOT
(array_contains(labels, 'red'))` 也可以转换，并保留 SQL 的 `NULL` 语义。这些示例适用于普通表扫描，不改变搜索
TVF 的 `filter` 参数与外层 `WHERE` 条件的区别。
+
+以下条件仍保留在 Doris 求值：
+
+- `array_contains(labels, NULL)`：Doris 可以匹配数组中的 `NULL` 元素，Lance 成员判断函数对 `NULL`
标签的语义不同。
+- `array_contains_all(labels, ['red', 'blue'])`：Doris 判断有序、连续的子序列，不能替换为 Lance
的集合包含函数或两个独立的成员判断条件。
+- 动态标签、数组列上的表达式或类型转换，以及其他数组元素或容器类型，包括 `large_list`、定长列表和 `list<large_utf8>`。
+
+### 谓词下推与标量索引使用
+
+谓词下推表示由 Lance 求值，不保证使用索引。普通扫描中，Doris 可以为每个扫描任务指定一个物理 BTree、Bitmap 或 LabelList
索引段。对于同列的 `IN`、`AND`、`OR`、`NOT` 组合，Lance 支持相应完整表达式时可以使用选中的索引。`OR`
的两个分支都必须提供安全的候选集，`NOT` 不能对仅求值一部分的子树取反。不支持的索引表达式会回退为分配的 Fragment 范围内的扫描。
+
+每个任务选择**一个逻辑标量索引**。同时包含标签、类别和数值条件的查询，并不表示 Doris 会对三个索引求交集。下推的完整条件会在扫描 LIMIT
生效前再次检查，索引未覆盖的 Fragment 仍会读取。没有可用索引时，兼容的谓词也可以下推，由 Lance 扫描求值。
+
+应结合以下信息验证：
+
+| 验证信息 | 含义 |
+|---|---|
+| `EXPLAIN` 中的 `lancePushdownPredicate` | 已发送给 Lance 的条件；剩余的 `predicates:` 条件由
Doris 求值。 |
+| `EXPLAIN` 中的 `lanceScalarIndexScan=SEGMENT` 和 `lanceGroupingIndex` |
已规划标量索引段扫描，并显示选中的逻辑索引；不能单独证明运行时使用了索引。 |
+| Profile 中的 `LanceScalarIndexSegmentsSearched` | 已埋点的标量索引段路径实际执行的搜索次数。 |
+| Profile 中的 `LanceScalarIndexCandidateRows` | 该路径报告的候选行数，不一定等于最终结果行数。部分 `NOT`
表达式等补集掩码可能不报告候选集基数。 |
+| Profile 中的 `LanceScalarIndexSegmentFallbacks` | 从规划的标量索引段路径回退的任务数。 |
+
+开启 `enable_profile` 后查看这些计数器。衡量优化效果时，应保持
SQL、投影列、数据集版本和缓存状态一致。这些计数器描述普通标量索引段扫描，不能单独用于判断 ANN 或全文搜索预过滤内部是否使用了标量索引。
+
### Runtime Filter 下推
普通 Lance 表参与 Join 时，Doris 可以将 Join 构建端生成的部分 Runtime Filter 转换为 Lance SQL 条件，并在
Lance 读取数据时提前过滤。这可以减少返回给 Doris 的数据量，尤其适合大表与过滤结果较小的表进行 Join 的场景。
diff --git a/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
b/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
index a335affdc17..1e20a20b4a4 100644
--- a/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
+++ b/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
@@ -582,8 +582,9 @@ Doris converts semantically compatible predicates into
Substrait expressions and
| `utf8`, `large_utf8` | Supported |
| `date32(day)` | Supported |
| Timezone-naive `timestamp(s/ms/us)` | Supported |
+| `list<utf8>` | Built-in `array_contains` with a non-`NULL` string literal;
see [Array Label Predicates](#array-label-predicates) |
-Predicates on other readable types, including `float16`, `decimal256`, Binary,
`date64`, Time, nanosecond Timestamp, timezone-aware Timestamp, and complex
types, currently remain in Doris.
+Predicates on other readable types, including `float16`, `decimal256`, Binary,
`date64`, Time, nanosecond Timestamp, timezone-aware Timestamp, and complex
types other than the array membership case above, currently remain in Doris.
### Operators Supported for Pushdown
@@ -596,13 +597,14 @@ Predicates on other readable types, including `float16`,
`decimal256`, Binary, `
| Boolean column, `NOT` Boolean column | Direct Boolean column reference |
| `LIKE`, `NOT LIKE` | Direct `utf8` or `large_utf8` column and a string
literal pattern without backslash escapes, an explicit `ESCAPE` clause, or an
embedded NUL character |
| `starts_with`, `ends_with` | Built-in function with a direct `utf8` or
`large_utf8` column and a string literal |
+| `array_contains(labels, 'red')` | Direct `list<utf8>` column and a
non-`NULL` string literal |
| `AND` | Top-level conjuncts can be pushed down independently, with
unsupported conjuncts retained in Doris |
| `OR` | Both branches must be fully convertible |
| `NOT` | The operand must be fully convertible |
The following forms are generally not pushed down:
-- Functions other than the supported built-in string functions, or arithmetic
expressions applied to a column. A user-defined function named `like`,
`starts_with`, or `ends_with` is not treated as the corresponding built-in
function.
+- Functions other than the supported built-in string and array membership
functions, or arithmetic expressions applied to a column. A user-defined
function named `like`, `starts_with`, `ends_with`, or `array_contains` is not
treated as the corresponding built-in function.
- String predicates whose pattern contains an embedded NUL character. `LIKE`
and `NOT LIKE` patterns that use a backslash escape or an explicit `ESCAPE`
clause also remain in Doris.
- An empty `IN` list or an `IN` list containing `NULL`.
- An `OR` or `NOT` expression in which only part of the expression can be
converted.
@@ -619,6 +621,46 @@ WHERE active
AND country LIKE 'C%';
```
+### Array Label Predicates
+
+Starting from Apache Doris 4.1.5, array-label pushdown supports
`array_contains` on a direct Lance `list<utf8>` column with a non-`NULL` string
literal. Older builds may keep these conditions in Doris; check `EXPLAIN`
before relying on this optimization. A `NULL` array still produces `NULL`, an
empty array does not match, and duplicate labels do not duplicate result rows.
+
+For a Lance table with an `id` column and a `labels` column of type
`list<utf8>`:
+
+```sql
+EXPLAIN
+SELECT id
+FROM lance_catalog.default.items
+WHERE array_contains(labels, 'red')
+ AND array_contains(labels, 'blue');
+```
+
+This matches arrays containing both labels, regardless of their order or
adjacency. Replacing `AND` with `OR` matches either label. `NOT
(array_contains(labels, 'red'))` is also convertible and preserves SQL `NULL`
semantics. These examples apply to ordinary table scans; they do not change the
distinction between a search TVF's `filter` parameter and an outer `WHERE`
condition.
+
+The following remain Doris residual predicates:
+
+- `array_contains(labels, NULL)`: Doris can match a `NULL` element; Lance's
membership function has different `NULL`-needle semantics.
+- `array_contains_all(labels, ['red', 'blue'])`: Doris tests an ordered,
contiguous subsequence. It cannot be replaced with Lance's set-containment
function or two independent membership predicates.
+- Dynamic needles, expressions or casts around the array column, and other
array element/container types, including `large_list`, fixed-size lists, and
`list<large_utf8>`.
+
+### Predicate Pushdown and Scalar Index Use
+
+Predicate pushdown means Lance evaluates a condition; it does not guarantee
index use. For ordinary scans, Doris can assign a physical BTree, Bitmap, or
LabelList segment to each scan task. Same-column `IN`, `AND`, `OR`, and `NOT`
combinations can use the selected index when Lance supports the complete
expression. `OR` requires safe candidates from both branches; `NOT` cannot
negate an incompletely evaluated subtree. Unsupported index expressions fall
back to a scan within the assigned f [...]
+
+Each task selects **one logical scalar index**. A query combining label,
category, and numeric predicates does not imply that Doris intersects all three
indexes. The complete pushed condition is rechecked before applying the scan
limit, and fragments outside index coverage are still read. Without a usable
index, compatible predicates can still be pushed down and evaluated by a Lance
scan.
+
+Use the following evidence together:
+
+| Evidence | Meaning |
+|---|---|
+| `lancePushdownPredicate` in `EXPLAIN` | Conditions sent to Lance. Remaining
`predicates:` conditions are evaluated by Doris. |
+| `lanceScalarIndexScan=SEGMENT` and `lanceGroupingIndex` in `EXPLAIN` | A
scalar segment scan was planned and identifies the selected logical index. This
alone does not prove runtime index use. |
+| `LanceScalarIndexSegmentsSearched` in Profile | Actual searches on the
instrumented scalar segment path. |
+| `LanceScalarIndexCandidateRows` in Profile | Candidate rows reported by that
path, not necessarily final result rows. Complement masks, such as some `NOT`
expressions, may not report a candidate cardinality. |
+| `LanceScalarIndexSegmentFallbacks` in Profile | Tasks that fell back from
the planned scalar segment path. |
+
+Enable `enable_profile` to inspect these counters. Compare the same SQL,
projection, dataset version, and cache state when measuring an optimization.
These counters describe ordinary scalar segment scans and must not be used
alone to infer scalar-index use inside ANN or full-text prefiltering.
+
### Runtime Filter Pushdown
When a regular Lance table participates in a join, Doris can convert some
Runtime Filters produced by the join build side into Lance SQL conditions and
apply them while Lance reads the data. This can reduce the amount of data
returned to Doris, especially when a large table is joined with a highly
selective build-side table.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
