This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 91f18c17dc3 vector-search (#2918)
91f18c17dc3 is described below
commit 91f18c17dc3903dba8797fa5e8ce84fefc071214
Author: zhiqiang <[email protected]>
AuthorDate: Tue Sep 30 11:46:15 2025 +0800
vector-search (#2918)
## Versions
- [X] dev
- [ ] 3.0
- [ ] 2.1
- [ ] 2.0
## Languages
- [X] Chinese
- [X] English
## Docs Checklist
- [ ] Checked by AI
- [ ] Test Cases Built
---
docs/ai/vector-search.md | 355 +++++++++++++++++++++
.../current/ai/vector-search.md | 326 +++++++++++++++++++
static/images/ann-sq-build-time.png | Bin 0 -> 50748 bytes
3 files changed, 681 insertions(+)
diff --git a/docs/ai/vector-search.md b/docs/ai/vector-search.md
new file mode 100644
index 00000000000..d2b85105ef6
--- /dev/null
+++ b/docs/ai/vector-search.md
@@ -0,0 +1,355 @@
+---
+{
+ "title": "Vector Search",
+ "language": "en"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In generative AI applications, relying solely on a large model's internal
parameter “memory” has clear limitations: (1) the model’s knowledge becomes
outdated and cannot cover the latest information; (2) directly asking the model
to “generate” answers increases the risk of hallucinations. This gives rise to
RAG (Retrieval-Augmented Generation). The key task of RAG is not to have the
model fabricate answers from nothing, but to retrieve the Top-K most relevant
information chunks from an e [...]
+
+To achieve this, we need a mechanism to measure semantic relatedness between a
user query and documents in the knowledge base. Vector representations are a
standard tool: by encoding both queries and documents into semantic vectors, we
can use vector similarity to measure relevance. With the advancement of
pretrained language models, generating high-quality embeddings has become
mainstream. Thus, the retrieval stage of RAG becomes a typical vector
similarity search problem: from a large [...]
+
+Vector retrieval in RAG is not limited to text; it naturally extends to
multimodal scenarios. In a multimodal RAG system, images, audio, video, and
other data types can also be encoded into vectors for retrieval and then
supplied to the generative model as context. For example, if a user uploads an
image, the system can first retrieve related descriptions or knowledge
snippets, then generate explanatory content. In medical QA, RAG can retrieve
patient records and literature to support mo [...]
+
+## Brute-Force Search
+
+Starting from version 2.0, Apache Doris supports nearest-neighbor search based
on vector distance. Performing vector search with SQL is natural and simple:
+
+```
+SELECT id,
+ l2_distance(embedding, [1.0, 2.0, xxx, 10.0]) AS distance
+FROM vector_table
+ORDER BY distance
+LIMIT 10;
+```
+
+When the dataset is small (under ~1 million rows), Doris’s exact K-Nearest
Neighbor search performance is sufficient, providing 100% recall and precision.
As the dataset grows, however, most users are willing to trade a small amount
of recall/accuracy for significantly lower latency. The problem then becomes
Approximate Nearest Neighbor (ANN) search.
+
+## Approximate Nearest Neighbor Search
+
+From version 4.0, Apache Doris officially supports ANN search. No additional
data type is introduced: vectors are stored as fixed-length arrays. For
distance-based indexing a new index type, ANN, is implemented based on Faiss.
+
+Using the common [SIFT](http://corpus-texmex.irisa.fr/) dataset as an example,
you can create a table like this:
+
+```
+CREATE TABLE sift_1M (
+ id int NOT NULL,
+ embedding array<float> NOT NULL COMMENT "",
+ INDEX ann_index (embedding) USING ANN PROPERTIES(
+ "index_type"="hnsw",
+ "metric_type"="l2_distance",
+ "dim"="128",
+ "quant"="flat"
+ )
+) ENGINE=OLAP
+DUPLICATE KEY(id) COMMENT "OLAP"
+DISTRIBUTED BY HASH(id) BUCKETS 1
+PROPERTIES (
+ "replication_num" = "1"
+);
+```
+
+- index_type: `hnsw` means using the [Hierarchical Navigable Small World
algorithm](https://en.wikipedia.org/wiki/Hierarchical_navigable_small_world)
+- metric_type: `l2_distance` means using L2 distance as the distance function
+- dim: `128` means the vector dimension is 128
+- quant: `flat` means each vector dimension is stored as original float32
+
+| Parameter | Required | Supported/Options | Default | Description |
+|-----------|----------|-------------------|---------|-------------|
+| `index_type` | Yes | hnsw only | (none) | ANN index algorithm. Currently
only HNSW supported. |
+| `metric_type` | Yes | `l2_distance`, `inner_product` | (none) | Vector
similarity/distance metric. L2 = Euclidean; inner_product can approximate
cosine if vectors are normalized. |
+| `dim` | Yes | Positive integer (> 0) | (none) | Vector dimension. All
imported vectors must match or an error is raised. |
+| `max_degree` | No | Positive integer | `32` | HNSW M (max neighbors per
node). Affects index memory and search performance. |
+| `ef_construction` | No | Positive integer | `40` | HNSW efConstruction
(candidate queue size during build). Larger gives better quality but slower
build. |
+| `quantizer` | No | `flat`, `sq8`, `sq4` | `flat` | Vector
encoding/quantization: `flat` = raw; `sq8`/`sq4` = symmetric quantization (8/4
bit) to reduce memory. |
+
+Import via S3 TVF:
+
+```sql
+INSERT INTO sift_1M
+SELECT *
+FROM S3("uri" =
+"https://selectdb-customers-tools-bj.oss-cn-beijing.aliyuncs.com/sift_database.tsv";,
"format" = "csv");
+
+select count(*) from sift_1M
+
++----------+
+| count(*) |
++----------+
+| 1000000 |
++----------+
+```
+
+The SIFT dataset ships with a ground-truth set for result validation. Pick one
query vector and first run an exact Top-N using the precise distance:
+
+```
+SELECT id, l2_distance(embedding,
[0,11,77,24,3,0,0,0,28,70,125,8,0,0,0,0,44,35,50,45,9,0,0,0,4,0,4,56,18,0,3,9,16,17,59,10,10,8,57,57,100,105,125,41,1,0,6,92,8,14,73,125,29,7,0,5,0,0,8,124,66,6,3,1,63,5,0,1,49,32,17,35,125,21,0,3,2,12,6,109,21,0,0,35,74,125,14,23,0,0,6,50,25,70,64,7,59,18,7,16,22,5,0,1,125,23,1,0,7,30,14,32,4,0,2,2,59,125,19,4,0,0,2,1,6,53,33,2])
as distance FROM sift_1M ORDER BY distance limit 10
+--------------
+
++--------+----------+
+| id | distance |
++--------+----------+
+| 178811 | 210.1595 |
+| 177646 | 217.0161 |
+| 181997 | 218.5406 |
+| 181605 | 219.2989 |
+| 821938 | 221.7228 |
+| 807785 | 226.7135 |
+| 716433 | 227.3148 |
+| 358802 | 230.7314 |
+| 803100 | 230.9112 |
+| 866737 | 231.6441 |
++--------+----------+
+10 rows in set (0.29 sec)
+```
+
+When using `l2_distance` or `inner_product`, Doris computes the distance
between the query vector and all 1,000,000 candidate vectors, then applies a
TopN operator globally. Using `l2_distance_approximate` /
`inner_product_approximate` triggers the index path:
+
+```
+SELECT id, l2_distance_approximate(embedding,
[0,11,77,24,3,0,0,0,28,70,125,8,0,0,0,0,44,35,50,45,9,0,0,0,4,0,4,56,18,0,3,9,16,17,59,10,10,8,57,57,100,105,125,41,1,0,6,92,8,14,73,125,29,7,0,5,0,0,8,124,66,6,3,1,63,5,0,1,49,32,17,35,125,21,0,3,2,12,6,109,21,0,0,35,74,125,14,23,0,0,6,50,25,70,64,7,59,18,7,16,22,5,0,1,125,23,1,0,7,30,14,32,4,0,2,2,59,125,19,4,0,0,2,1,6,53,33,2])
as distance FROM sift_1M ORDER BY distance limit 10
+--------------
+
++--------+----------+
+| id | distance |
++--------+----------+
+| 178811 | 210.1595 |
+| 177646 | 217.0161 |
+| 181997 | 218.5406 |
+| 181605 | 219.2989 |
+| 821938 | 221.7228 |
+| 807785 | 226.7135 |
+| 716433 | 227.3148 |
+| 358802 | 230.7314 |
+| 803100 | 230.9112 |
+| 866737 | 231.6441 |
++--------+----------+
+10 rows in set (0.02 sec)
+```
+
+With the ANN index, query latency in this example drops from about 290 ms to
20 ms.
+
+ANN indexes are built at the segment granularity. Because tables are
distributed, after each segment returns its local TopN, the TopN operator
merges results across tablets and segments to produce the global TopN.
+
+Note: When `metric_type = l2_distance`, a smaller distance means closer
vectors. For `inner_product`, a larger value means closer vectors. Therefore,
if using `inner_product`, you must use `ORDER BY dist DESC` to obtain TopN via
the index.
+
+## Approximate Range Search
+
+Beyond the common TopN nearest neighbor search (returning the closest N
records), another typical pattern is threshold-based range search. Instead of
returning a fixed number of results, it returns all points whose distance to
the target vector satisfies a predicate (>, >=, <, <=). For example, you might
want vectors whose distance is greater than or less than a threshold. This is
useful when you need candidates that are “sufficiently similar” or
“sufficiently dissimilar.” In recommendat [...]
+
+Example SQL:
+
+```
+SELECT count(*) FROM sift_1M WHERE l2_distance_approximate(embedding,
[0,11,77,24,3,0,0,0,28,70,125,8,0,0,0,0,44,35,50,45,9,0,0,0,4,0,4,56,18,0,3,9,16,17,59,10,10,8,57,57,100,105,125,41,1,0,6,92,8,14,73,125,29,7,0,5,0,0,8,124,66,6,3,1,63,5,0,1,49,32,17,35,125,21,0,3,2,12,6,109,21,0,0,35,74,125,14,23,0,0,6,50,25,70,64,7,59,18,7,16,22,5,0,1,125,23,1,0,7,30,14,32,4,0,2,2,59,125,19,4,0,0,2,1,6,53,33,2])
> 300
+--------------
+
++----------+
+| count(*) |
++----------+
+| 999271 |
++----------+
+1 row in set (0.19 sec)
+```
+
+These range-based vector searches are also accelerated by the ANN index: the
index first narrows candidates, then approximate distances are computed,
reducing cost and improving latency. Supported predicates: `>`, `>=`, `<`, `<=`.
+
+## Compound Search
+
+Compound Search combines an ANN TopN search with a range predicate in the same
SQL statement, returning the TopN results that also satisfy a distance
constraint.
+
+```
+SELECT id, l2_distance_approximate(embedding,
[0,11,77,24,3,0,0,0,28,70,125,8,0,0,0,0,44,35,50,45,9,0,0,0,4,0,4,56,18,0,3,9,16,17,59,10,10,8,57,57,100,105,125,41,1,0,6,92,8,14,73,125,29,7,0,5,0,0,8,124,66,6,3,1,63,5,0,1,49,32,17,35,125,21,0,3,2,12,6,109,21,0,0,35,74,125,14,23,0,0,6,50,25,70,64,7,59,18,7,16,22,5,0,1,125,23,1,0,7,30,14,32,4,0,2,2,59,125,19,4,0,0,2,1,6,53,33,2])
as dist FROM sift_1M WHERE l2_distance_approximate(embedding,
[0,11,77,24,3,0,0,0,28,70,125,8,0,0,0,0,44,35,50,4 [...]
+--------------
+
++--------+----------+
+| id | dist |
++--------+----------+
+| 243590 | 300.005 |
+| 549298 | 300.0317 |
+| 429685 | 300.0533 |
+| 690172 | 300.0916 |
+| 123410 | 300.1333 |
+| 232540 | 300.1649 |
+| 547696 | 300.2066 |
+| 855437 | 300.2782 |
+| 589017 | 300.3048 |
+| 930696 | 300.3381 |
++--------+----------+
+10 rows in set (0.12 sec)
+```
+
+A key question is whether predicate filtering happens before or after TopN. If
predicates filter first and TopN is applied on the reduced set, it’s
pre-filtering; otherwise, it’s post-filtering. Post-filtering can be faster but
may dramatically reduce recall. Doris uses pre-filtering to preserve recall.
+
+Doris can accelerate both phases with the index. However, if the first phase
(range filter) is too selective, indexing both phases can hurt recall. Doris
adaptively decides whether to use the index twice based on predicate
selectivity and index type.
+
+## ANN Search with Additional Filters
+
+This refers to applying other predicates before the ANN TopN and returning the
TopN under those constraints.
+
+Example with a small 8-D vector and a text filter:
+
+```
+create table ann_with_fulltext (
+ id int not null,
+ embedding array<float> not null,
+ comment String not null,
+ value int null,
+ INDEX idx_comment(`comment`) USING INVERTED PROPERTIES("parser" = "english")
COMMENT 'inverted index for comment',
+ INDEX ann_embedding(`embedding`) USING ANN
PROPERTIES("index_type"="hnsw","metric_type"="l2_distance","dim"="8")
+) duplicate key (`id`)
+distributed by hash(`id`) buckets 1
+properties("replication_num"="1");
+```
+
+Insert sample data and search only within rows where `comment` contains
“music”:
+```
+INSERT INTO ann_with_fulltext VALUES
+(1, [0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8], 'this is about music', 10),
+(2, [0.2,0.1,0.5,0.3,0.9,0.4,0.7,0.1], 'sports news today', 20),
+(3, [0.9,0.8,0.7,0.6,0.5,0.4,0.3,0.2], 'latest music trend', 30),
+(4, [0.05,0.06,0.07,0.08,0.09,0.1,0.2,0.3], 'politics update',40);
+
+SELECT id, comment,
+ l2_distance_approximate(embedding, [0.1,0.1,0.2,0.2,0.3,0.3,0.4,0.4])
AS dist
+FROM ann_with_fulltext
+WHERE comment MATCH_ANY 'music' -- Filter using inverted index
+ORDER BY dist ASC -- Ann topn calculation after predicates
evaluate.
+LIMIT 2;
+
++------+---------------------+----------+
+| id | comment | dist |
++------+---------------------+----------+
+| 1 | this is about music | 0.663325 |
+| 3 | latest music trend | 1.280625 |
++------+---------------------+----------+
+2 rows in set (0.04 sec)
+```
+
+To ensure TopN can be accelerated via the vector index, all predicate columns
should have appropriate secondary indexes (e.g., an inverted index).
+
+## Session Variables Related to ANN Search
+
+Beyond build-time parameters for HNSW, you can pass search-time parameters via
session variables:
+
+- hnsw_ef_search: EF search parameter. Controls max length of the candidate
queue; larger = higher accuracy, higher latency. Default 32.
+- hnsw_check_relative_distance: Whether to enable relative distance checking
to improve accuracy. Default true.
+- hnsw_bounded_queue: Whether to use a bounded priority queue to optimize
performance. Default true.
+
+## Vector Quantization
+
+With FLAT encoding, an HNSW index (raw vectors plus graph structure) may
consume large amounts of memory. HNSW must be fully resident in memory to
function, so memory can become a bottleneck at large scale.
+
+Vector quantization compresses float32 storage to reduce memory. Doris
currently supports two scalar quantization schemes: INT8 and INT4 (SQ8 / SQ4).
Example using SQ8:
+
+```
+CREATE TABLE sift_1M (
+ id int NOT NULL,
+ embedding array<float> NOT NULL COMMENT "",
+ INDEX ann_index (embedding) USING ANN PROPERTIES(
+ "index_type"="hnsw",
+ "metric_type"="l2_distance",
+ "dim"="128",
+ "quant"="sq8"
+ )
+) ENGINE=OLAP
+DUPLICATE KEY(id) COMMENT "OLAP"
+DISTRIBUTED BY HASH(id) BUCKETS 1
+PROPERTIES (
+ "replication_num" = "1"
+);
+```
+
+On 768-D Cohere-MEDIUM-1M and Cohere-LARGE-10M datasets, SQ8 reduces index
size to roughly one third compared to FLAT.
+
+| Dataset | Dim | Storage/Index Scheme | Total Disk | Data Part | Index Part |
Notes |
+|---------|-----|----------------------|------------|-----------|------------|-------|
+| Cohere-MEDIUM-1M | 768D | Doris (FLAT) | 5.647 GB (2.533 + 3.114) | 2.533 GB
| 3.114 GB | 1M vectors |
+| Cohere-MEDIUM-1M | 768D | Doris SQ INT8 | 3.501 GB (2.533 + 0.992) | 2.533
GB | 0.992 GB | INT8 symmetric quantization |
+| Cohere-LARGE-10M | 768D | Doris (FLAT) | 56.472 GB (25.328 + 31.145) |
25.328 GB | 31.145 GB | 10M vectors |
+| Cohere-LARGE-10M | 768D | Doris SQ INT8 | 35.016 GB (25.329 + 9.687) |
25.329 GB | 9.687 GB | INT8 quantization |
+
+Quantization introduces extra build-time overhead because each distance
computation must decode quantized values. For 128-D vectors, build time
increases with row count; SQ vs. FLAT can be up to ~10× slower to build.
+
+
+
+## Performance Tuning
+
+Vector search is a typical secondary-index point lookup scenario. For high QPS
and low latency, consider the following:
+
+With tuning, on hardware FE 32C 64GB + BE 32C 64GB, Doris can reach 3000+ QPS
(dataset: Cohere-MEDIUM-1M).
+
+### Query Performance
+
+| Concurrency | Scheme | QPS | Avg Latency (s) | P99 (s) | CPU Usage | Recall |
+|-------------|--------|-----|-----------------|---------|-----------|--------|
+| 240 | Doris | 3340.4399 | 0.071368168 | 0.163399825 | 40% | 91.00% |
+| 240 | Doris SQ INT8 | 3188.6359 | 0.074728852 | 0.160370195 | 40% | 88.26% |
+| 240 | Doris SQ INT4 | 2818.2291 | 0.084663868 | 0.174826815 | 43% | 80.38% |
+| 240 | Doris brute force | 3.6787 | 25.554878826 | 29.363227973 | 100% |
100.00% |
+| 480 | Doris | 4155.7220 | 0.113387271 | 0.261086075 | 60% | 91.00% |
+| 480 | Doris SQ INT8 | 3833.1130 | 0.123040214 | 0.276912867 | 50% | 88.26% |
+| 480 | Doris SQ INT4 | 3431.0538 | 0.137636995 | 0.281631249 | 57% | 80.38% |
+| 480 | Doris brute force | 3.6787 | 25.554878826 | 29.363227973 | 100% |
100.00% |
+
+### Use Prepared Statements
+
+Modern embedding models often output 768-D or higher vectors. If you inline a
768-D literal into SQL, parsing time can exceed execution time. Use prepared
statements. Currently Doris does not support MySQL client prepare commands
directly; use JDBC:
+
+1. Enable server-side prepared statements in the JDBC URL:
+ `jdbc:mysql://127.0.0.1:9030/demo?useServerPrepStmts=true`
+2. Use PreparedStatement with placeholders (`?`) and reuse it.
+
+### Reduce Segment Count
+
+ANN indexes are built per segment. Too many segments cause overhead. Ideally
each tablet should have no more than ~5 segments for an ANN-indexed table.
Adjust `write_buffer_size` and `vertical_compaction_max_segment_size` in
`be.conf` (e.g., both to 10737418240).
+
+### Reduce Rowset Count
+
+Same motivation as reducing segments: minimize scheduling overhead. Each load
creates a rowset, so prefer stream load or `INSERT INTO SELECT` for batched
ingestion.
+
+### Keep ANN Index in Memory
+
+Current ANN algorithms are memory-based. If a segment’s index is not in
memory, a disk I/O occurs. Set `enable_segment_cache_prune=false` in `be.conf`
to keep ANN indexes resident.
+
+### parallel_pipeline_task_num = 1
+
+ANN TopN queries return very few rows from each scanner, so high pipeline task
parallelism is unnecessary. Set `parallel_pipeline_task_num = 1`.
+
+### enable_profile = false
+
+Disable query profiling for ultra latency-sensitive queries.
+
+## Usage Limitations
+
+1. The ANN index column must be a NOT NULL `Array<Float>`, and every imported
vector must match the declared `dim`, otherwise an error is thrown.
+2. ANN index is only supported on DuplicateKey table model.
+3. Doris uses pre-filter semantics (predicates applied before ANN TopN). If
predicates include columns without secondary indexes that can precisely locate
rows (e.g., no inverted index), Doris falls back to brute force to preserve
correctness.
+ Example:
+ ```
+ SELECT id, l2_distance_approximate(embedding, [xxx]) AS distance
+ FROM sift_1M
+ WHERE round(id) > 100
+ ORDER BY distance LIMIT 10;
+ ```
+ Although `id` is a key, without a secondary index (such as an inverted
index), its predicate is applied after index analysis, so Doris falls back to
brute force to honor pre-filter semantics.
+4. If the distance function in SQL does not match the metric type defined in
the index DDL, Doris cannot use the ANN index for TopN—even if you call
`l2_distance_approximate` / `inner_product_approximate`.
+5. For metric type `inner_product`, only `ORDER BY
inner_product_approximate(...) DESC LIMIT N` (DESC required) can be accelerated
by the ANN index.
+6. The first parameter of `xxx_approximate()` must be a ColumnArray, and the
second must be a CAST or ArrayLiteral. Reversing them triggers brute-force
search.
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/ai/vector-search.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/ai/vector-search.md
new file mode 100644
index 00000000000..6e533d476d7
--- /dev/null
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/ai/vector-search.md
@@ -0,0 +1,326 @@
+---
+{
+ "title": "向量搜索",
+ "language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+在生成式 AI
的应用中，单纯依赖大模型自身的参数“记忆”存在明显局限：一方面，模型知识具有时效性，无法覆盖最新信息；另一方面，完全依赖模型直接“生成”容易产生幻觉（Hallucination）。因此，RAG（检索增强生成）应运而生。其核心目标不是让模型凭空构造答案，而是从外部知识库中检索出与用户查询最相关的
Top-K
信息片段，作为生成依据。为实现这一点，需要一种机制衡量“用户查询”与“知识库文档”之间的语义相关性。向量表示正是常用手段：将查询与文档统一编码为语义向量后，可通过向量相似度衡量相关程度。随着预训练模型的发展，生成高质量语义向量已成主流，RAG
的检索阶段也演化为一个标准的向量相似度搜索问题——从大规模向量集合中找出与查询最相似的 K 个向量（候选知识片段）。需要注意，RAG 的向量检索不�
�文本，也可扩展到多模态：图片、语音、视频等数据同样可以编码为向量供生成模型使用。例如，用户上传图片后，系统先检索相关描述或知识片段，再辅助生成解释性内容；在医学问答中，可检索病例资料与医学文献，生成更准确的诊断建议。
+## 暴力搜索
+Apache Doris 自 2.0 版本起支持基于向量距离的最近邻搜索，通过 SQL 实现向量搜索是一个自然且简单的过程。
+```
+SELECT id,
+ l2_distance(embedding, [1.0, 2.0, xxx, 10.0]) AS distance
+FROM vector_table
+ORDER BY distance
+LIMIT 10;
+```
+
+当数据量不大（小于 100 万行）时，Apache Doris 的精确最近邻（K-Nearest Neighbor）搜索性能足以满足需求，可获得 100%
召回与 100% 精确。但随着数据进一步增长，用户通常愿意牺牲少量召回与精度以换取显著的查询加速，此时问题就转化为向量近似最近邻搜索（Approximate
Nearest Neighbor，ANN）。
+
+## 近似最近邻搜索
+
+Apache Doris 自 4.0 版本开始正式支持 ANN 搜索。系统未引入额外数据类型，向量仍以定长数组存储；针对向量距离检索，我们基于 Faiss
实现了新的 ANN 索引类型。
+以下以常见的 [SIFT](http://corpus-texmex.irisa.fr/) 数据集为例，建表示例如下：
+```
+CREATE TABLE sift_1M (
+ id int NOT NULL,
+ embedding array<float> NOT NULL COMMENT "",
+ INDEX ann_index (embedding) USING ANN PROPERTIES(
+ "index_type"="hnsw",
+ "metric_type"="l2_distance",
+ "dim"="128",
+ "quant"="flat"
+ )
+) ENGINE=OLAP
+DUPLICATE KEY(id) COMMENT "OLAP"
+DISTRIBUTED BY HASH(id) BUCKETS 1
+PROPERTIES (
+ "replication_num" = "1"
+);
+```
+- index_type: hnsw 表示使用 [Hierarchical Navigable Small World
算法](https://en.wikipedia.org/wiki/Hierarchical_navigable_small_world)
+- metric: l2_distance 表示使用 L2 距离作为距离函数
+- dim: 128 表示向量维度为 128
+- quantizer: flat 表示按原始 float32 存储各维度
+
+
+| 参数 | 是否必填 | 支持/可选值 | 默认值 | 说明 |
+|------|----------|-------------|--------|------|
+| `index_type` | 是 | 仅支持：hnsw | （无） | 指定所使用的 ANN 索引算法。当前只支持 HNSW。 |
+| `metric_type` | 是 | `l2_distance`，`inner_product` | （无） | 指定向量相似度/距离度量方式。L2
为欧氏距离，inner_product 可用于余弦相似时需先归一化向量。 |
+| `dim` | 是 | 正整数 (> 0) | （无） | 指定向量维度，后续导入的所有向量的维度必须与此一致，否则报错。 |
+| `max_degree` | 否 | 正整数 | `32` | HNSW 图中单个节点的最大邻居数（M），影响索引内存与搜索性能。 |
+| `ef_construction` | 否 | 正整数 | `40` | HNSW
构建阶段的候选队列大小（efConstruction），越大构图质量越好但构建更慢。 |
+| `quantizer` | 否 | `flat`，`sq8`，`sq4` | `flat` | 指定向量编码/量化方式：`flat`
为原始存储，`sq8`/`sq4` 为对称量化（8/4 bit）以降低内存占用。 |
+
+通过 S3 TVF 导入数据：
+```sql
+INSERT INTO sift_1M
+SELECT *
+FROM S3("uri" =
+"https://selectdb-customers-tools-bj.oss-cn-beijing.aliyuncs.com/sift_database.tsv";,
"format" = "csv");
+
+select count(*) from sift_1M
+--------------
+
++----------+
+| count(*) |
++----------+
+| 1000000 |
++----------+
+```
+SIFT 数据集同时发布了一组 ground truth，用于校验结果。下面选取一组向量，先使用精确距离函数进行 TopN 召回：
+```
+SELECT id, l2_distance(embedding,
[0,11,77,24,3,0,0,0,28,70,125,8,0,0,0,0,44,35,50,45,9,0,0,0,4,0,4,56,18,0,3,9,16,17,59,10,10,8,57,57,100,105,125,41,1,0,6,92,8,14,73,125,29,7,0,5,0,0,8,124,66,6,3,1,63,5,0,1,49,32,17,35,125,21,0,3,2,12,6,109,21,0,0,35,74,125,14,23,0,0,6,50,25,70,64,7,59,18,7,16,22,5,0,1,125,23,1,0,7,30,14,32,4,0,2,2,59,125,19,4,0,0,2,1,6,53,33,2])
as distance FROM sift_1M ORDER BY distance limit 10
+--------------
+
++--------+----------+
+| id | distance |
++--------+----------+
+| 178811 | 210.1595 |
+| 177646 | 217.0161 |
+| 181997 | 218.5406 |
+| 181605 | 219.2989 |
+| 821938 | 221.7228 |
+| 807785 | 226.7135 |
+| 716433 | 227.3148 |
+| 358802 | 230.7314 |
+| 803100 | 230.9112 |
+| 866737 | 231.6441 |
++--------+----------+
+10 rows in set (0.29 sec)
+```
+当使用 `l2_distance` 或 `inner_product` 时，Doris 需要计算查询向量与 1,000,000 个候选向量之间的距离，再通过
TopN 算子得到全局结果。使用 `l2_distance_approximate` / `inner_product_approximate`
可触发索引执行路径：
+```
+SELECT id, l2_distance_approximate(embedding,
[0,11,77,24,3,0,0,0,28,70,125,8,0,0,0,0,44,35,50,45,9,0,0,0,4,0,4,56,18,0,3,9,16,17,59,10,10,8,57,57,100,105,125,41,1,0,6,92,8,14,73,125,29,7,0,5,0,0,8,124,66,6,3,1,63,5,0,1,49,32,17,35,125,21,0,3,2,12,6,109,21,0,0,35,74,125,14,23,0,0,6,50,25,70,64,7,59,18,7,16,22,5,0,1,125,23,1,0,7,30,14,32,4,0,2,2,59,125,19,4,0,0,2,1,6,53,33,2])
as distance FROM sift_1M ORDER BY distance limit 10
+--------------
+
++--------+----------+
+| id | distance |
++--------+----------+
+| 178811 | 210.1595 |
+| 177646 | 217.0161 |
+| 181997 | 218.5406 |
+| 181605 | 219.2989 |
+| 821938 | 221.7228 |
+| 807785 | 226.7135 |
+| 716433 | 227.3148 |
+| 358802 | 230.7314 |
+| 803100 | 230.9112 |
+| 866737 | 231.6441 |
++--------+----------+
+10 rows in set (0.02 sec)
+```
+可以看到使用 ANN 索引后，查询耗时从约 290 ms 降至约 20 ms。
+Doris 中，ANN 索引建立在 segment 粒度；由于表是分布式的，各 segment 返回局部 TopN 后，TopN 算子会将多个 tablet
的结果归并生成全局 TopN。
+
+需要注意：当 `l2_distance` 作为索引 metric 时，distance 越小表示越接近；`inner_product`
则相反，值越大越接近。因此若使用 `inner_product`，必须 `ORDER BY dist DESC` 才能通过索引获得 TopN。
+## 近似范围搜索
+
+除了常见的 TopN 最近邻搜索（即返回与目标向量最近的前 N 条记录）之外，向量检索中还有一类常见的查询方式是 基于距离阈值的范围搜索。
+这类查询不返回固定数量，而是找出所有与目标向量距离满足条件的数据点。例如：查找距离大于或小于某阈值的向量。范围搜索在需要“足够相似”或“足够不相似”候选集的场景中很有用：推荐系统中可获取“接近但不完全相同”内容以增加多样性；异常检测中可定位远离正常模式的数据点。
+一个典型的 SQL 为：
+```
+SELECT count(*) FROM sift_1M WHERE l2_distance_approximate(embedding,
[0,11,77,24,3,0,0,0,28,70,125,8,0,0,0,0,44,35,50,45,9,0,0,0,4,0,4,56,18,0,3,9,16,17,59,10,10,8,57,57,100,105,125,41,1,0,6,92,8,14,73,125,29,7,0,5,0,0,8,124,66,6,3,1,63,5,0,1,49,32,17,35,125,21,0,3,2,12,6,109,21,0,0,35,74,125,14,23,0,0,6,50,25,70,64,7,59,18,7,16,22,5,0,1,125,23,1,0,7,30,14,32,4,0,2,2,59,125,19,4,0,0,2,1,6,53,33,2])
> 300
+--------------
+
++----------+
+| count(*) |
++----------+
+| 999271 |
++----------+
+1 row in set (0.19 sec)
+```
+在 Doris 中，这类基于范围的向量搜索同样通过 ANN 索引 来加速执行。通过 ANN
索引，系统能够快速筛选出候选向量集合，然后再计算精确的近似距离，从而显著降低计算开销、提升查询效率。目前支持的范围查询条件包括 `>, >=, <, <=`。
+## 组合搜索
+Compound Search 指在同一条 SQL 中同时进行 ANN TopN 与 Range 条件过滤，返回满足范围约束的 TopN。
+```
+SELECT id, l2_distance_approximate(embedding,
[0,11,77,24,3,0,0,0,28,70,125,8,0,0,0,0,44,35,50,45,9,0,0,0,4,0,4,56,18,0,3,9,16,17,59,10,10,8,57,57,100,105,125,41,1,0,6,92,8,14,73,125,29,7,0,5,0,0,8,124,66,6,3,1,63,5,0,1,49,32,17,35,125,21,0,3,2,12,6,109,21,0,0,35,74,125,14,23,0,0,6,50,25,70,64,7,59,18,7,16,22,5,0,1,125,23,1,0,7,30,14,32,4,0,2,2,59,125,19,4,0,0,2,1,6,53,33,2])
as dist FROM sift_1M WHERE l2_distance_approximate(embedding,
[0,11,77,24,3,0,0,0,28,70,125,8,0,0,0,0,44,35,50,4 [...]
+--------------
+
++--------+----------+
+| id | dist |
++--------+----------+
+| 243590 | 300.005 |
+| 549298 | 300.0317 |
+| 429685 | 300.0533 |
+| 690172 | 300.0916 |
+| 123410 | 300.1333 |
+| 232540 | 300.1649 |
+| 547696 | 300.2066 |
+| 855437 | 300.2782 |
+| 589017 | 300.3048 |
+| 930696 | 300.3381 |
++--------+----------+
+10 rows in set (0.12 sec)
+```
+对于 Compound Search，一个关键点是谓词过滤与 TopN 的执行顺序：若先做谓词过滤再在剩余集合上取
TopN，称为“前过滤”；反之为“后过滤”。后过滤通常更快，但可能显著降低召回，因此 Doris 采用前过滤策略。
+在 Doris 中，Compound Search 的两个阶段均可通过索引加速。但在某些场景（如第一阶段 Range
过滤率极高）双阶段同时使用索引可能导致召回下降。Doris 会自适应判断是否对两阶段均使用索引，依据谓词过滤率与索引类型综合决策。
+## 带过滤条件的 ANN 搜索
+带过滤条件的 ANN 搜索是指在执行 ANN TopN 之前先应用其他谓词过滤，返回满足条件的 TopN。
+下面用一个 8 维示例说明混合搜索流程。
+```
+create table ann_with_fulltext (
+ id int not null,
+ embedding array<float> not null,
+ comment String not null,
+ value int null,
+ INDEX idx_comment(`comment`) USING INVERTED PROPERTIES("parser" =
"english") COMMENT 'inverted index for comment',
+ INDEX ann_embedding(`embedding`) USING ANN
PROPERTIES("index_type"="hnsw","metric_type"="l2_distance","dim"="8")
+ ) duplicate key (`id`)
+ distributed by hash(`id`) buckets 1
+ properties("replication_num"="1");
+
+INSERT INTO ann_with_fulltext VALUES
+(1, [0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8], 'this is about music', 10),
+(2, [0.2,0.1,0.5,0.3,0.9,0.4,0.7,0.1], 'sports news today', 20),
+(3, [0.9,0.8,0.7,0.6,0.5,0.4,0.3,0.2], 'latest music trend', 30),
+(4, [0.05,0.06,0.07,0.08,0.09,0.1,0.2,0.3], 'politics update',40)
+```
+假设用户输入查询向量 `[0.1,0.1,0.2,0.2,0.3,0.3,0.4,0.4]`，只在 comment 含 “music”
的文档中检索最相似的前 2 条：
+```
+SELECT id, comment,
+ l2_distance_approximate(embedding, [0.1,0.1,0.2,0.2,0.3,0.3,0.4,0.4])
AS dist
+FROM ann_with_fulltext
+WHERE comment MATCH_ANY 'music' -- 先用倒排索引过滤
+ORDER BY dist ASC -- 在过滤后的结果集上做 ANN TopN
+LIMIT 2;
+
++------+---------------------+----------+
+| id | comment | dist |
++------+---------------------+----------+
+| 1 | this is about music | 0.663325 |
+| 3 | latest music trend | 1.280625 |
++------+---------------------+----------+
+2 rows in set (0.04 sec)
+```
+带过滤条件的 ANN 搜索要想利用向量索引加速 TopN，需要确保涉及的过滤列具备倒排等二级索引。
+## 查询参数
+
+除了在构建 HNSW 索引时可指定参数外，查询阶段也可通过会话变量调节行为。
+
+- hnsw_ef_search：HNSW索引的EF搜索参数。ef_search 用来控制搜索阶段时 candidates
队列的最大长度，ef_search 越大则搜索的精度越高，代价是搜索的耗时越高。默认值为 32。
+
+- hnsw_check_relative_distance：是否启用相对距离检查机制，以提升HNSW搜索的准确性。默认为 true。
+
+- hnsw_bounded_queue： 是否使用有界优先队列来优化HNSW的搜索性能。默认为 true。
+## 向量量化
+采用 FLAT 编码时，HNSW 索引（原始向量 + 图结构）可能占用大量内存。HNSW 必须全量驻留内存才能工作，因此在超大规模数据集上易成瓶颈。
+向量量化通过压缩 FLOAT32 减少内存开销。Doris 当前支持两种标量量化：INT8 与 INT4（SQ8 / SQ4）。以 SQ8 为例：
+
+```
+CREATE TABLE sift_1M (
+ id int NOT NULL,
+ embedding array<float> NOT NULL COMMENT "",
+ INDEX ann_index (embedding) USING ANN PROPERTIES(
+ "index_type"="hnsw",
+ "metric_type"="l2_distance",
+ "dim"="128",
+ "quant"="sq8" -- 指定使用 INT8 进行量化
+ )
+) ENGINE=OLAP
+DUPLICATE KEY(id) COMMENT "OLAP"
+DISTRIBUTED BY HASH(id) BUCKETS 1
+PROPERTIES (
+ "replication_num" = "1"
+);
+```
+在 768 维的 Cohere-MEDIUM-1M 与 Cohere-LARGE-10M 数据集测试中，SQ8 可将索引大小压缩至 FLAT 的约 1/3。
+数据集
+
+| 数据集 | 向量维度 | 存储/索引方案 | 总磁盘占用 | 数据部分 | 索引部分 | 备注 |
+|--------|----------|---------------|------------|----------|----------|------|
+| Cohere-MEDIUM-1M | 768D | Doris (FLAT) | 5.647 GB (2.533 + 3.114) | 2.533
GB | 3.114 GB | 1M 向量，原始 + HNSW FLAT 索引 |
+| Cohere-MEDIUM-1M | 768D | Doris SQ INT8 | 3.501 GB (2.533 + 0.992) | 2.533
GB | 0.992 GB | INT8 对称量化 |
+| Cohere-LARGE-10M | 768D | Doris (FLAT) | 56.472 GB (25.328 + 31.145) |
25.328 GB | 31.145 GB | 10M 向量 |
+| Cohere-LARGE-10M | 768D | Doris SQ INT8 | 35.016 GB (25.329 + 9.687) |
25.329 GB | 9.687 GB | INT8 量化，索引显著减小 |
+
+量化会带来额外构建开销，原因是构建阶段需要大量距离计算，且每次计算需对量化值解码。以 128 维向量为例，随行数增长构建时间上升，SQ 相比 FLAT
可能引入约 10 倍构建成本。
+
+
+
+
+## 性能调优
+向量搜索是典型的二级索引点查场景。若对 QPS 与延迟要求较高，可参考以下建议。经调优，在 FE 32C 64GB + BE 32C 64GB
机器上，Doris 可达到 3000+ QPS（数据集：Cohere-MEDIUM-1M）。
+### 查询性能
+| 并发 | 方案 | QPS | 平均延迟 (s) | P99 延迟 (s) | CPU 使用率 | 召回率 |
+|------|------|------|---------------|--------------|------------|--------|
+| 240 | Doris | 3340.4399 | 0.071368168 | 0.163399825 | 40% | 91.00% |
+| 240 | Doris SQ INT8 | 3188.6359 | 0.074728852 | 0.160370195 | 40% | 88.26% |
+| 240 | Doris SQ INT4 | 2818.2291 | 0.084663868 | 0.174826815 | 43% | 80.38% |
+| 240 | Doris 暴力计算 | 3.6787 | 25.554878826 | 29.363227973 | 100% | 100.00% |
+| 480 | Doris | 4155.7220 | 0.113387271 | 0.261086075 | 60% | 91.00% |
+| 480 | Doris SQ INT8 | 3833.1130 | 0.123040214 | 0.276912867 | 50% | 88.26% |
+| 480 | Doris SQ INT4 | 3431.0538 | 0.137636995 | 0.281631249 | 57% | 80.38% |
+| 480 | Doris 暴力计算 | 3.6787 | 25.554878826 | 29.363227973 | 100% | 100.00% |
+
+### 使用 prepared statement
+常见 embedding 模型输出通常为 768 维或更高。如果将该向量作为字面量直接写入 SQL，解析耗时可能超过实际执行时间，因此建议使用
Prepared Statement。当前 Doris 不支持通过 mysql client 直接执行相关命令，需要通过 JDBC 调用。
+```
+1. 在 jdbc url 里面开启服务端 prepared statement
+url = jdbc:mysql://127.0.0.1:9030/demo?useServerPrepStmts=true
+2. 使用 prepared statement
+// use `?` for placement holders, readStatement should be reused
+
+PreparedStatement readStatement = conn.prepareStatement("SELECT id,
l2_distance_approximate(embedding, cast (? as ARRAY<FLOAT>)) AS distance
+ FROM l2_distance_approximate
+ ORDER BY distance
+ LIMIT 10");
+
+...
+
+readStatement.setString("[0,11,77,24,3,0,0,0,28,70,125,8,0,0,0,0,44,35,50,45,9,0,0,0,4,0,4,56,18,0,3,9,16,17,59,10,10,8,57,57,100,105,125,41,1,0,6,92,8,14,73,125,29,7,0,5,0,0,8,124,66,6,3,1,63,5,0,1,49,32,17,35,125,21,0,3,2,12,6,109,21,0,0,35,74,125,14,23,0,0,6,50,25,70,64,7,59,18,7,16,22,5,0,1,125,23,1,0,7,30,14,32,4,0,2,2,59,125,19,4,0,0,2,1,6,53,33,2]");
+
+ResultSet resultSet = readStatement.executeQuery();
+```
+### 减少 segment 数量
+Doris 的 ANN 索引建立在 segment 上，segment 过多会引入额外开销。理想情况下，带 ANN 索引的表每个 tablet 下
segment 数不应超过 5 个。可通过调整 be.conf 中 `write_buffer_size` 与
`vertical_compaction_max_segment_size` 增大单 segment 大小以减少数量；建议两者设置为
10737418240（10GB）。
+### 减少 rowset 数量
+减少 rowset 数量与减少 segment 的目的相同：降低调度开销。每次导入都会生成一个 rowset，建议使用 stream load 或
`INSERT INTO SELECT` 做批量导入。
+### Ann 索引常驻内存
+当前 ANN 索引算法基于内存，若查询到的 segment 索引未驻留内存会触发磁盘 I/O。为性能考虑建议常驻，可在 be.conf 中设置
`enable_segment_cache_prune=false`。
+### parallel_pipeine_task_num = 1
+ANN TopN 查询返回行数很少，无需高并行度，建议 `SET parallel_pipeline_task_num = 1`。
+### enable_profile = false
+若对延迟极其敏感，建议关闭 query profile（`enable_profile=false`）。
+## 使用限制
+1. Doris 要求 ANN Index 对应的列必须是 NOT NULLABLE 的
`Array<Float>`，并且在后续的导入过程中，需要确保该列的每一个向量的长度均等于索引属性中指定的维度（dim），否则会报错。
+
+2. ANN Index 只能在 DuplicateKey 的表模型上使用。
+
+3. Doris 使用前过滤语意（谓词计算在AnnTopN 计算之前）当 SQL 中的谓词涉及到的列有非二级索引列时，为了保证结果的正确性，此时 Doris
会回退到暴力计算。
+比如
+```
+SELECT id, l2_distance_approximate(embedding, [xxx]) AS distance
+ FROM sift_1M
+ WHERE round(id) > 100
+ ORDER BY distance limit 10;
+```
+
+虽然 id 是主键，但未在该列上构建倒排等可精确定位行号的二级索引，此类谓词在 Doris 中会在索引分析之后执行。为保证 ANN TopN
的前过滤语义，系统会回退为暴力计算。
+
+4. 如果 SQL 中指定的距离函数与 DDL 中索引的 metric 类型不匹配，那么此时 doris 无法通过 ANN 索引进行 TOPN
的计算，哪怕你是用的是 l2_distance_approximate/inner_product_approximate。
+
+5. 如果 metric 类型是 inner_product，那么只有 ORDER BY inner_product_approximate() DESC
LIMIT N（这里的 DESC 不能省略）才能通过 ANN 索引加速。
+
+6. xxx_approximate() 函数的第一个参数为 ColumnArray，第二个参数为 CAST 或者 ArrayLiteral
时，才能触发索引分析，交换位置会回退暴力搜索。
\ No newline at end of file
diff --git a/static/images/ann-sq-build-time.png
b/static/images/ann-sq-build-time.png
new file mode 100644
index 00000000000..7588709acf2
Binary files /dev/null and b/static/images/ann-sq-build-time.png differ
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
