This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 7872ff4f23b [feature](lance) Support approx_mode in vector_search
(#69852)
7872ff4f23b is described below
commit 7872ff4f23bdaba75578fbb0e63e4d67db903ef9
Author: Gabriel <[email protected]>
AuthorDate: Mon Nov 2 09:55:30 2026 +0800
[feature](lance) Support approx_mode in vector_search (#69852)
### What problem does this PR solve?
`vector_search` cannot currently select Lance's approximation policy.
This PR adds `"approx_mode"="fast|normal|accurate"`, validates the value
before metadata access, transports it as an optional typed Thrift field,
and forwards it to `lance_scanner_set_approx_mode` on each BE scanner.
`EXPLAIN` shows an explicitly requested mode.
Omitting the property preserves the existing native default (`normal` in
the pinned Lance revision). The setting applies through the shared
single-vector and multi-vector scanner setup. Existing refinement, probe
counts, search width, concurrency, split coverage, and distributed
merging remain unchanged. The existing lance-c dependency already
provides this API, so no third-party update or local patch is needed.
Website documentation is provided in
[apache/doris-website](https://github.com/apache/doris-website/pull/4554),
updating the English and Chinese 4.x Lance Catalog pages with mode
semantics, selection guidance, ACORN activation and fallback rules, SQL
usage, and compatibility notes. The temporary guide in the Doris
repository has been removed. The local experiment remains recorded
below. In particular, `accurate` still selects approximate candidates
and does not guarantee exact Top-K or original-vector distances on a
quantized index. During a smooth upgrade, FE rejects explicit `fast` or
`accurate` at scan initialization if any backend in the query's
candidate pool is marked as an old smooth-upgrade source. Older BEs
would ignore the optional field and silently use NORMAL, so rejecting
prevents a global TopN from merging split candidates generated under
different policies. The error names the backend and requested mode and
advises completing the upgrade or using `normal`. Omission and explicit
`normal` remain compatible. This checks only the selected backend
policy, not unrelated backends, and does not detect arbitrary older
builds without the smooth-upgrade-source marker.
### Mode semantics and recommended selection
Use `normal` as the baseline; choose another mode only after measuring
representative queries against an exact reference. These are
recommendations based on the pinned implementation and the local
experiment, not fixed speed/recall guarantees.
| Search path | `fast` | `normal` | `accurate` |
|---|---|---|---|
| RabitQ raw-query distance estimation | Uses one-bit scores even for a
multi-bit index, skipping extension-bit computation and its lower-bound
gate. | Uses the standard 8-bit quantized distance lookup table and
extension-bit computation when available. | Uses a higher-precision
16-bit distance lookup table and extension-bit computation when
available. |
| HNSW with prefiltering | Enables ACORN traversal when the runtime
conditions below permit it. | Uses regular filtered graph traversal. |
Uses the same traversal policy as `normal`; quantized distance
computation can still differ. |
| IVF_FLAT or exhaustive Flat Search | No mode-specific quantization
change. | Same scoring policy. | Same scoring policy. |
The 8-bit and 16-bit values describe the **query-time distance lookup
table**, not the number of bits stored per dimension in the index. For
example, on a 5-bit RabitQ index, `fast` uses one-bit scores on the
raw-query path; the other modes can use the additional four bits. Other
RabitQ paths, including residual-query scoring, can behave differently.
A mode description does not expand the supported vector-type or metric
combinations.
`accurate` is still approximate search. It does not guarantee exact
Top-K, exhaustive candidate evaluation, or original-vector `_distance`
on a quantized index. `refine_factor` controls reranking with original
vectors; increasing it can improve scores and recall but cannot recover
every candidate missed earlier. Multi-vector queries retain their
mandatory original-vector refinement.
| Workload | Starting choice | What to verify |
|---|---|---|
| General workloads or an unknown bottleneck | `normal` | Establish
recall and latency before changing the policy. |
| HNSW_FLAT with an effective prefilter, where latency matters | Compare
`fast` with `normal` | ACORN can reduce wasted distance calculations.
Check recall, result count, median latency, and p95; a speedup is not
guaranteed. |
| Multi-bit RabitQ where recall matters | `normal` | `fast` can discard
useful encoding information and substantially reduce recall. Use it only
if representative queries still meet the quality target. |
| RabitQ without original-vector reranking, where approximate score
error matters | Compare `accurate` with `normal` | Measure score error
and ranking quality against original-vector distances, as well as extra
latency. Better estimation does not guarantee higher recall. |
| RabitQ already using original-vector reranking | `normal` | The final
score benefit of `accurate` may be smaller; candidate selection can
still differ, so validate recall separately. |
| IVF_FLAT or HNSW_FLAT without an effective prefilter | Keep `normal` |
The current implementation offers no corresponding quantization or ACORN
benefit from changing the mode. IVF_FLAT recall still depends on the
probed partitions. |
For a compatible **HNSW index backed by RabitQ**, `fast` can both enable
ACORN traversal and simplify RabitQ distance estimation on the
applicable raw-query path. These effects are coupled by one option. Do
not infer its recall from an HNSW_FLAT test; validate the combined index
independently. `normal` and `accurate` have the same HNSW traversal
policy, but their RabitQ scoring can differ.
Recommended tuning order:
1. Pin the same dataset snapshot, query set, metric, filters, and index
coverage. Use `use_index=false` to obtain exact Top-K ground truth over
the same eligible rows; put eligibility conditions in the TVF `filter`
with `prefilter=true`.
2. Establish a `normal` baseline. Measure Recall@K, result count, score
error where relevant, and median/p95 latency.
3. If recall is insufficient, tune IVF `nprobes` and HNSW `ef`. If
original-vector scores or more reranked candidates are required,
configure `refine_factor`, accounting for the extra vector reads and
computation.
4. With the other settings fixed, compare `fast` for filtered HNSW or
`accurate` for RabitQ estimation precision. Select a mode only if it
meets the quality target and provides a worthwhile measured benefit.
Measure cold and warm cache behavior separately.
### ACORN activation and fallback
The selected compatible index must contain an HNSW graph. Request
`approx_mode="fast"` and a TVF `filter` with `prefilter="true"`; an
outer Doris `WHERE` does not supply this prefilter. `use_index=true`
alone does not select or force HNSW. The website documentation includes
a complete SQL example.
The current native implementation dispatches separately inside each HNSW
subindex (an IVF partition), using the number of nodes passing the
prefilter:
| Runtime condition | Path |
|---|---|
| No effective prefilter, or all nodes pass | Ordinary HNSW traversal. |
| Fewer than roughly 10% of nodes pass | Flat scoring over the remaining
nodes. |
| A partial filter remains and the sparse-filter rule does not apply |
`fast` uses ACORN; `normal` and `accurate` use regular filtered graph
traversal. |
| ACORN returns too few required candidates and the query has no
distance bounds | Retry with regular filtered graph traversal. |
The threshold applies within each subindex, not to the whole table.
There is no separate force-ACORN property in this interface. In Doris,
distance-range safety rules can also select Flat Search before native
graph dispatch.
`EXPLAIN` showing `lanceApproxMode=FAST` confirms the requested policy,
not that every subindex actually ran ACORN. HNSW_RabitQ behavior is
inferred from the coupled traversal/scoring implementation; the local
HNSW measurements below used HNSW_FLAT and do not validate the combined
index.
### Local validation of behavior and effect
These are **native lance-c scanner measurements, not end-to-end Doris
SQL timings**. The library was freshly built from pinned lance-c
`a51ab47a9c814a746cb1714c99fad93dd3a188b8`, using Lance
`68c12dfd7efe02f90ad2d7f3a239a7eb64884e57`. The unoptimized development
build and Python/Arrow result conversion make the timings illustrative,
not production performance claims.
Procedure:
1. Generate 16,384 Gaussian Float32 vectors of dimension 128, then 128
independent query vectors, using xorshift64/Box-Muller with seed
`20261029`. Assign `row_id=0..16383` and `category=row_id%10`.
2. Build fully covering IVF_RQ (32 partitions, 5 bits), IVF_FLAT (32
partitions), and IVF_HNSW_FLAT (32 partitions, 16 edges, construction
ef=100) indexes. Select each index explicitly by its segment UUIDs.
3. Search squared L2 with k=10, nprobes=32, and query_parallelism=1.
Repeat IVF_RQ with refine_factor=4. For HNSW, set ef=64 and prefilter
`category < 5` (about 50% of rows).
4. Compute exact Top-10 independently with NumPy. Warm up each mode with
eight queries; run three rounds of 128 queries per mode, rotating mode
order per query and round. The table below reports mean Recall@10,
median latency, and p95 latency, from 384 searches per mode.
5. Compare omitted mode with explicit normal over all 128 queries per
case, checking ordered IDs and Float32 distances. Independently
recompute original-vector distances for returned IVF_RQ rows to measure
score error.
| Index / refinement | Mode | Recall@10 | Median (ms) | p95 (ms) |
| --- | --- | ---: | ---: | ---: |
| IVF_RQ / none | fast | 26.80% | 14.57 | 16.03 |
| IVF_RQ / none | normal | 91.72% | 15.90 | 18.12 |
| IVF_RQ / none | accurate | 91.72% | 20.52 | 22.91 |
| IVF_RQ / 4 | fast | 54.30% | 20.50 | 22.07 |
| IVF_RQ / 4 | normal | 99.92% | 23.44 | 25.64 |
| IVF_RQ / 4 | accurate | 99.92% | 28.19 | 30.58 |
| IVF_FLAT / none | fast | 100% | 25.71 | 29.62 |
| IVF_FLAT / none | normal | 100% | 25.78 | 29.82 |
| IVF_FLAT / none | accurate | 100% | 25.71 | 30.74 |
| IVF_HNSW_FLAT / 50% prefilter | fast | 100% | 40.45 | 44.04 |
| IVF_HNSW_FLAT / 50% prefilter | normal | 100% | 44.51 | 49.86 |
| IVF_HNSW_FLAT / 50% prefilter | accurate | 100% | 44.48 | 50.05 |
All searches returned ten rows. Omitted mode matched explicit normal
exactly in every case. IVF_FLAT and filtered HNSW returned identical
ordered IDs and distances across modes. The data seed fixes vectors; ANN
index training/layout can vary across rebuilds.
Without refinement, normal and accurate had equal mean recall but
different ordered IDs on 89/128 queries. Mean absolute squared-L2 score
error was 30.7325 for fast, 0.7986 for normal, and 0.7783 for accurate.
Thus accurate slightly improved distance estimation on this fixture
without improving mean recall. With refinement factor 4, normal and
accurate returned identical results, while fast still missed many
neighbors. The mode is not interchangeable with refinement, and its
effect depends on the index/path and data.
### Release note
Add the optional `approx_mode` property to Lance `vector_search`,
accepting `fast`, `normal`, and `accurate`, with the existing default
preserved when omitted.
### Check List (For Author)
- Test
- [x] Unit tests: focused source compilation and JUnit execution of
`VectorSearchTableValuedFunctionTest` and `LanceScanNodeTest`: **59
tests passed**. Coverage includes accepted/invalid modes, typed Thrift
round trips, default omission, EXPLAIN, and smooth-upgrade scan
initialization.
- [x] Review-fix regression: the new real scan-initialization test
failed on the previous code (the expected upgrade rejection was
missing), then passed with the gate. Coverage includes FAST/ACCURATE
with old-only and mixed candidate pools, single/multi-vector requests,
all modes on current BEs, omitted/explicit NORMAL on mixed BEs, and
unaffected FTS initialization.
- [x] Manual test: native mode comparison described above, plus
independent distance-error checks.
- [x] FE `mvn -B clean checkstyle:check` passed; clang-format 16 checks
passed for both affected C++ files; generated Thrift sources rebuilt
successfully.
- [x] Fresh BE reader and test source objects compiled successfully
against the new headers and pinned C API.
- [ ] Full BE unit-test execution: not run locally because the available
prebuilt BE libraries have an incompatible ABI. New request-validation
coverage is included for CI.
- [ ] SQL regression execution: added mode pass-through, EXPLAIN, and
invalid-value coverage to `test_lance_vector_search`; the full suite
parsed successfully with Groovy 4.0.19, but execution is pending CI.
- Behavior changed:
- [x] Yes. Explicit `approx_mode` now selects the native policy;
omission preserves existing behavior.
- Does this need documentation?
- [x] Yes. English and Chinese 4.x documentation: [website
PR](https://github.com/apache/doris-website/pull/4554).
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---
be/src/format_v2/table/lance_reader.cpp | 31 ++++++
be/test/format_v2/table/lance_reader_test.cpp | 20 ++++
.../datasource/lance/source/LanceScanNode.java | 28 ++++++
.../VectorSearchTableValuedFunction.java | 26 ++++-
.../datasource/lance/source/LanceScanNodeTest.java | 106 ++++++++++++++++++++-
.../VectorSearchTableValuedFunctionTest.java | 43 +++++++++
gensrc/thrift/PlanNodes.thrift | 8 ++
.../lance/test_lance_vector_search.groovy | 21 ++++
8 files changed, 280 insertions(+), 3 deletions(-)
diff --git a/be/src/format_v2/table/lance_reader.cpp
b/be/src/format_v2/table/lance_reader.cpp
index 752ce7460c2..2eb9bbc1ced 100644
--- a/be/src/format_v2/table/lance_reader.cpp
+++ b/be/src/format_v2/table/lance_reader.cpp
@@ -76,6 +76,25 @@ Status LanceTableReader::fetch_schema(const TFileRangeDesc&
range,
}
namespace {
+Status to_lance_approx_mode(TVectorApproxMode::type mode, LanceApproxMode*
native_mode) {
+ switch (mode) {
+ case TVectorApproxMode::FAST:
+ *native_mode = LANCE_APPROX_MODE_FAST;
+ break;
+ case TVectorApproxMode::NORMAL:
+ *native_mode = LANCE_APPROX_MODE_NORMAL;
+ break;
+ case TVectorApproxMode::ACCURATE:
+ *native_mode = LANCE_APPROX_MODE_ACCURATE;
+ break;
+ default:
+ return Status::InvalidArgument(
+ "Lance approx_mode must be FAST, NORMAL, or ACCURATE, got {}",
+ static_cast<int>(mode));
+ }
+ return Status::OK();
+}
+
// A dataset URI without its query string or a password in its user
information, either of which
// may carry credentials, for errors and profiles. A user part without a
password stays: Azure
// URIs name the container there (abfss://container@account...).
@@ -619,6 +638,10 @@ Status
LanceTableReader::_validate_external_search_request() const {
if (_search_kind == SearchKind::VECTOR &&
request.__isset.vector_search_options) {
const auto& options = request.vector_search_options;
+ if (options.__isset.approx_mode) {
+ LanceApproxMode native_mode;
+ RETURN_IF_ERROR(to_lance_approx_mode(options.approx_mode,
&native_mode));
+ }
if (options.__isset.query_parallelism && options.query_parallelism <
-1) {
return Status::InvalidArgument("Lance query_parallelism must be
-1, 0, or positive");
}
@@ -1131,6 +1154,14 @@ Status
LanceTableReader::_configure_vector_search(LanceScanner* scanner,
}
if (request.__isset.vector_search_options) {
const auto& options = request.vector_search_options;
+ // Leave an omitted mode unset so Lance retains its default search
policy.
+ if (options.__isset.approx_mode) {
+ LanceApproxMode native_mode;
+ RETURN_IF_ERROR(to_lance_approx_mode(options.approx_mode,
&native_mode));
+ if (lance_scanner_set_approx_mode(scanner, native_mode) != 0) {
+ return lance_error("set Lance vector approx_mode");
+ }
+ }
if (options.__isset.query_parallelism &&
lance_scanner_set_query_parallelism(scanner,
options.query_parallelism) != 0) {
return lance_error("set Lance vector query parallelism");
diff --git a/be/test/format_v2/table/lance_reader_test.cpp
b/be/test/format_v2/table/lance_reader_test.cpp
index c27190b8fe7..93ee0d26ea6 100644
--- a/be/test/format_v2/table/lance_reader_test.cpp
+++ b/be/test/format_v2/table/lance_reader_test.cpp
@@ -1654,6 +1654,26 @@ TEST(LanceTableReaderVectorSearchTest,
RejectsInvalidQueryParallelismBeforeOpeni
EXPECT_NE(std::string::npos, status.to_string().find("query_parallelism"));
}
+TEST(LanceTableReaderVectorSearchTest,
ValidatesApproxModeBeforeOpeningDataset) {
+ TQueryGlobals globals;
+ RuntimeState state(globals);
+ const Columns columns {projected_column("_distance", TYPE_FLOAT, true)};
+ for (const int value : {0, 1, 2, 3}) {
+ SCOPED_TRACE(value);
+ RuntimeProfile profile("lance_approx_mode_validation");
+ auto params = make_float32_vector_search_params({0.0F, 0.0F, 0.0F}, 2,
0);
+
params.lance_scan_params.external_search_request.vector_search_options.__set_approx_mode(
+ static_cast<TVectorApproxMode::type>(value));
+ LanceTableReader reader;
+ const auto status = init_reader(&reader, columns, &state, &profile,
¶ms);
+ const bool valid = value >= 0 && value <= 2;
+ EXPECT_EQ(valid, status.ok()) << status.to_string();
+ if (!valid) {
+ EXPECT_NE(std::string::npos,
status.to_string().find("approx_mode"));
+ }
+ }
+}
+
TEST(LanceTableReaderVectorSearchTest,
ValidatesDistanceRangeBeforeDatasetAccess) {
TQueryGlobals globals;
RuntimeState state(globals);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanNode.java
index ebd18437d20..ce2c49a2d96 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanNode.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanNode.java
@@ -49,7 +49,9 @@ import org.apache.doris.thrift.TFullTextSearchParams;
import org.apache.doris.thrift.TLanceFileDesc;
import org.apache.doris.thrift.TLanceScanParams;
import org.apache.doris.thrift.TTableFormatFileDesc;
+import org.apache.doris.thrift.TVectorApproxMode;
import org.apache.doris.thrift.TVectorMetric;
+import org.apache.doris.thrift.TVectorSearchOptions;
import org.apache.doris.thrift.TVectorSearchParams;
import com.google.common.annotations.VisibleForTesting;
@@ -143,6 +145,7 @@ public class LanceScanNode extends FileQueryScanNode {
super.doInitialize();
checkAdditionalTypeBackendCompatibility(
projectsCurrentReaderType(), backendPolicy.getBackends());
+ checkApproxModeBackendCompatibility();
ExternalUtil.initSchemaInfo(params, -1L, sourceColumns);
if (searchKind.isExternalSearch()) {
@@ -157,6 +160,26 @@ public class LanceScanNode extends FileQueryScanNode {
lazyMaterializedColumns.add(columnName.toLowerCase(Locale.ROOT));
}
+ private void checkApproxModeBackendCompatibility() throws UserException {
+ if (searchKind != SearchKind.VECTOR ||
!externalSearchRequest.isSetVectorSearchOptions()) {
+ return;
+ }
+ TVectorSearchOptions options =
externalSearchRequest.getVectorSearchOptions();
+ if (!options.isSetApproxMode() || options.getApproxMode() ==
TVectorApproxMode.NORMAL) {
+ return;
+ }
+ // Old BEs skip the optional field, which would mix NORMAL with
FAST/ACCURATE split candidates.
+ for (Backend backend : backendPolicy.getBackends()) {
+ if (backend.isSmoothUpgradeSrc()) {
+ throw new UserException("Lance approx_mode="
+ +
options.getApproxMode().name().toLowerCase(Locale.ROOT)
+ + " is unavailable while backend " + backend.getId()
+ + " is a smooth upgrade source; finish upgrading the
query backends"
+ + " or use approx_mode=normal");
+ }
+ }
+ }
+
/** Checks columns read in either phase of a Lance scan. */
private boolean projectsCurrentReaderType() {
// Global row IDs route the second-phase take back to the first-phase
BE, so lazy pruning
@@ -354,6 +377,11 @@ public class LanceScanNode extends FileQueryScanNode {
.append(scanPlan.vectorIndexStatus).append("\n");
result.append(prefix).append("lanceVectorColumn=")
.append(vector.getColumn()).append("\n");
+ if (externalSearchRequest.isSetVectorSearchOptions()
+ &&
externalSearchRequest.getVectorSearchOptions().isSetApproxMode()) {
+ result.append(prefix).append("lanceApproxMode=")
+
.append(externalSearchRequest.getVectorSearchOptions().getApproxMode()).append("\n");
+ }
if (externalSearchRequest.isSetVectorSearchOptions()
&&
externalSearchRequest.getVectorSearchOptions().isSetQueryParallelism()) {
result.append(prefix).append("lanceQueryParallelism=")
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunction.java
b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunction.java
index fbb42ed7c0c..20a0ed19d8f 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunction.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunction.java
@@ -23,6 +23,7 @@ import
org.apache.doris.datasource.lance.metadata.LanceTableMetadata;
import org.apache.doris.thrift.TExternalSearchQuery;
import org.apache.doris.thrift.TExternalSearchRequest;
import org.apache.doris.thrift.TSearchVector;
+import org.apache.doris.thrift.TVectorApproxMode;
import org.apache.doris.thrift.TVectorElementType;
import org.apache.doris.thrift.TVectorMetric;
import org.apache.doris.thrift.TVectorSearchOptions;
@@ -61,6 +62,7 @@ public class VectorSearchTableValuedFunction extends
LanceExternalSearchTableVal
private static final String QUERY_VECTOR = "query_vector";
private static final String METRIC = "metric";
private static final String QUERY_PARALLELISM = "query_parallelism";
+ private static final String APPROX_MODE = "approx_mode";
private static final String NPROBES = "nprobes";
private static final String REFINE_FACTOR = "refine_factor";
private static final String EF = "ef";
@@ -69,7 +71,7 @@ public class VectorSearchTableValuedFunction extends
LanceExternalSearchTableVal
private static final String USE_INDEX = "use_index";
private static final Set<String> PROPERTIES = ImmutableSet.of(
TABLE, COLUMN, QUERY_VECTOR, TOP_K, OFFSET, METRIC, FILTER,
PREFILTER,
- NPROBES, REFINE_FACTOR, EF, USE_INDEX, QUERY_PARALLELISM,
+ NPROBES, REFINE_FACTOR, EF, USE_INDEX, QUERY_PARALLELISM,
APPROX_MODE,
VERSION, TIMESTAMP, TAG, BRANCH, DISTANCE_LOWER_BOUND,
DISTANCE_UPPER_BOUND);
public VectorSearchTableValuedFunction(Map<String, String> properties)
@@ -89,6 +91,7 @@ public class VectorSearchTableValuedFunction extends
LanceExternalSearchTableVal
boolean hasDistanceRange = vectorParams.isSetDistanceLowerBound() ||
vectorParams.isSetDistanceUpperBound();
boolean useIndex = !params.containsKey(USE_INDEX)
|| parseBoolean(params.get(USE_INDEX), USE_INDEX);
+ TVectorSearchOptions vectorSearchOptions =
buildVectorSearchOptions(params, useIndex);
CommonSearch common = prepareCommon(params, NAME,
"VectorSearchTableValuedFunction", "vector search", useIndex);
@@ -126,7 +129,6 @@ public class VectorSearchTableValuedFunction extends
LanceExternalSearchTableVal
TExternalSearchRequest searchRequest = new TExternalSearchRequest()
.setSchemaVersion(1)
.setSearchQuery(TExternalSearchQuery.vector_search(vectorParams));
- TVectorSearchOptions vectorSearchOptions =
buildVectorSearchOptions(params, useIndex);
if (vectorSearchOptions != null) {
searchRequest.setVectorSearchOptions(vectorSearchOptions);
}
@@ -169,6 +171,10 @@ public class VectorSearchTableValuedFunction extends
LanceExternalSearchTableVal
Map<String, String> params, boolean useIndex) throws
AnalysisException {
TVectorSearchOptions options = new TVectorSearchOptions();
boolean configured = false;
+ if (params.containsKey(APPROX_MODE)) {
+ options.setApproxMode(parseApproxMode(params.get(APPROX_MODE)));
+ configured = true;
+ }
if (params.containsKey(QUERY_PARALLELISM)) {
options.setQueryParallelism((int) parseLong(
params.get(QUERY_PARALLELISM), QUERY_PARALLELISM, -1,
Integer.MAX_VALUE));
@@ -194,6 +200,22 @@ public class VectorSearchTableValuedFunction extends
LanceExternalSearchTableVal
return configured ? options : null;
}
+ private static TVectorApproxMode parseApproxMode(String value) throws
AnalysisException {
+ if (value != null) {
+ switch (value.trim().toLowerCase(Locale.ROOT)) {
+ case "fast":
+ return TVectorApproxMode.FAST;
+ case "normal":
+ return TVectorApproxMode.NORMAL;
+ case "accurate":
+ return TVectorApproxMode.ACCURATE;
+ default:
+ break;
+ }
+ }
+ throw new AnalysisException("Unsupported 'approx_mode': expected fast,
normal, or accurate");
+ }
+
@VisibleForTesting
static List<Column> buildOutputColumns(LanceTableMetadata metadata)
throws AnalysisException {
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/source/LanceScanNodeTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/source/LanceScanNodeTest.java
index 32173603c8f..d7cd2881cf8 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/source/LanceScanNodeTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/source/LanceScanNodeTest.java
@@ -29,6 +29,7 @@ import org.apache.doris.catalog.TableIf;
import org.apache.doris.common.UserException;
import org.apache.doris.common.profile.RuntimeProfile;
import org.apache.doris.common.profile.SummaryProfile;
+import org.apache.doris.datasource.ExternalScanNode;
import org.apache.doris.datasource.FederationBackendPolicy;
import org.apache.doris.datasource.lance.LanceExternalCatalog;
import org.apache.doris.datasource.lance.LanceExternalTable;
@@ -54,6 +55,7 @@ import org.apache.doris.thrift.TFtsQueryType;
import org.apache.doris.thrift.TFullTextSearchParams;
import org.apache.doris.thrift.TLanceFileDesc;
import org.apache.doris.thrift.TPushAggOp;
+import org.apache.doris.thrift.TVectorApproxMode;
import org.apache.doris.thrift.TVectorMetric;
import org.apache.doris.thrift.TVectorSearchOptions;
import org.apache.doris.thrift.TVectorSearchParams;
@@ -80,6 +82,90 @@ import java.util.UUID;
public class LanceScanNodeTest {
+ @Test
+ public void
testNonDefaultApproxModeRejectsSmoothUpgradeSourcesDuringInitialization()
throws Exception {
+ Backend current = new Backend(1, "127.0.0.1", 9050);
+ Backend old = new Backend(2, "127.0.0.2", 9050);
+ old.setSmoothUpgradeSrc(true);
+ for (TVectorApproxMode mode : Arrays.asList(TVectorApproxMode.FAST,
TVectorApproxMode.ACCURATE)) {
+ for (boolean multiVector : Arrays.asList(false, true)) {
+ for (List<Backend> backends :
Arrays.asList(Collections.singletonList(old),
+ Arrays.asList(current, old))) {
+ TExternalSearchRequest request = vectorSearchRequest(5, 0);
+ request.setVectorSearchOptions(new
TVectorSearchOptions().setApproxMode(mode));
+ if (multiVector) {
+
request.getSearchQuery().getVectorSearch().setQueryVector(
+ new org.apache.doris.thrift.TSearchVector()
+
.setElementType(org.apache.doris.thrift.TVectorElementType.FLOAT32)
+
.setDimension(2).setNumVectors(2).setValues(new byte[16]));
+ }
+ LanceScanNode node =
newSearchNode(basicSearchMetadata(vectorSchema()), request);
+ setBackendCandidates(node, backends);
+
+ UserException error =
Assert.assertThrows(UserException.class, node::doInitialize);
+
+ Assert.assertTrue(error.getMessage(),
error.getMessage().contains("approx_mode="
+ + mode.name().toLowerCase(java.util.Locale.ROOT)));
+ Assert.assertTrue(error.getMessage(),
error.getMessage().contains("backend 2"));
+ Assert.assertTrue(error.getMessage(),
error.getMessage().contains("smooth upgrade source"));
+ }
+ }
+ }
+ }
+
+ @Test
+ public void testAllApproxModesInitializeWithCurrentBackends() throws
Exception {
+ List<Backend> backends = Arrays.asList(new Backend(1, "127.0.0.1",
9050),
+ new Backend(2, "127.0.0.2", 9050));
+ for (TVectorApproxMode mode : TVectorApproxMode.values()) {
+ TExternalSearchRequest request = vectorSearchRequest(5, 0);
+ request.setVectorSearchOptions(new
TVectorSearchOptions().setApproxMode(mode));
+ LanceScanNode node =
newSearchNode(basicSearchMetadata(vectorSchema()), request);
+ setBackendCandidates(node, backends);
+
+ node.doInitialize();
+
+ Assert.assertEquals(mode,
node.createSplitSearchRequest().getVectorSearchOptions().getApproxMode());
+ }
+ }
+
+ @Test
+ public void
testDefaultApproxModeRemainsCompatibleWithSmoothUpgradeSources() throws
Exception {
+ Backend old = new Backend(2, "127.0.0.2", 9050);
+ old.setSmoothUpgradeSrc(true);
+ List<Backend> backends = Arrays.asList(new Backend(1, "127.0.0.1",
9050), old);
+ for (TVectorSearchOptions options : Arrays.asList(null, new
TVectorSearchOptions().setNprobes(4),
+ new
TVectorSearchOptions().setApproxMode(TVectorApproxMode.NORMAL))) {
+ TExternalSearchRequest request = vectorSearchRequest(5, 0);
+ if (options != null) {
+ request.setVectorSearchOptions(options);
+ }
+ LanceScanNode node =
newSearchNode(basicSearchMetadata(vectorSchema()), request);
+ setBackendCandidates(node, backends);
+
+ node.doInitialize();
+
+ TExternalSearchRequest split = node.createSplitSearchRequest();
+ Assert.assertEquals(request.isSetVectorSearchOptions(),
split.isSetVectorSearchOptions());
+ if (options != null) {
+ Assert.assertEquals(options, split.getVectorSearchOptions());
+ }
+ }
+ }
+
+ @Test
+ public void testApproxModeCompatibilityDoesNotRejectFullTextSearch()
throws Exception {
+ Backend old = new Backend(2, "127.0.0.2", 9050);
+ old.setSmoothUpgradeSrc(true);
+ TExternalSearchRequest request = fullTextSearchRequest(5, 0,
TFtsCoverageMode.STRICT);
+ LanceScanNode node =
newSearchNode(basicSearchMetadata(fullTextSchema()), request);
+ setBackendCandidates(node, Collections.singletonList(old));
+
+ node.doInitialize();
+
+ Assert.assertEquals(request, node.createSplitSearchRequest());
+ }
+
@Test
public void testSplitPlanningReportsSuccessAndFailureToQueryProfile()
throws Exception {
ConnectContext previous = ConnectContext.get();
@@ -669,13 +755,16 @@ public class LanceScanNodeTest {
Collections.singletonList(9),
Arrays.asList(3L, 4L),
IndexType.VECTOR, "L2")));
TExternalSearchRequest request = vectorSearchRequest(5, 0);
- request.setVectorSearchOptions(new
TVectorSearchOptions().setQueryParallelism(4));
+ request.setVectorSearchOptions(new TVectorSearchOptions()
+
.setQueryParallelism(4).setApproxMode(TVectorApproxMode.ACCURATE));
LanceScanNode node = newSearchNode(metadata, request);
List<Split> splits = node.getSplits(3);
Assert.assertTrue(node.getNodeExplainString("", TExplainLevel.NORMAL)
.contains("lanceQueryParallelism=4"));
+ Assert.assertTrue(node.getNodeExplainString("", TExplainLevel.NORMAL)
+ .contains("lanceApproxMode=ACCURATE"));
Assert.assertTrue(node.getNodeExplainString("", TExplainLevel.NORMAL)
.contains("lanceVectorIndexStatus=USED"));
@@ -1110,6 +1199,21 @@ public class LanceScanNodeTest {
return newNode(new SessionVariable());
}
+ private static LanceTableMetadata basicSearchMetadata(Schema schema) {
+ return LanceTableMetadata.createBasicSnapshot(
+ new LanceTableAccess("s3://bucket/table.lance",
Collections.emptyMap()), 42, schema,
+ Collections.singletonList(new LanceFragmentInfo(1, 8, 8)));
+ }
+
+ private static void setBackendCandidates(LanceScanNode node, List<Backend>
backends) throws Exception {
+ FederationBackendPolicy policy =
Mockito.mock(FederationBackendPolicy.class);
+ Mockito.when(policy.getBackends()).thenReturn(backends);
+ Mockito.when(policy.numBackends()).thenReturn(backends.size());
+ java.lang.reflect.Field field =
ExternalScanNode.class.getDeclaredField("backendPolicy");
+ field.setAccessible(true);
+ field.set(node, policy);
+ }
+
private static LanceScanNode newNode(SessionVariable sessionVariable) {
return new LanceScanNode(
new PlanNodeId(0),
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunctionTest.java
b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunctionTest.java
index 181abca8dc9..cfdbd558693 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunctionTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunctionTest.java
@@ -22,6 +22,7 @@ import org.apache.doris.catalog.Column;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.datasource.lance.metadata.LanceTableAccess;
import org.apache.doris.datasource.lance.metadata.LanceTableMetadata;
+import org.apache.doris.thrift.TVectorApproxMode;
import org.apache.doris.thrift.TVectorSearchOptions;
import org.apache.doris.thrift.TVectorSearchParams;
@@ -44,6 +45,48 @@ public class VectorSearchTableValuedFunctionTest {
return
VectorSearchTableValuedFunction.buildVectorSearchOptions(params, true);
}
+ @Test
+ public void testApproxModeIsIncludedInSearchOptions() throws Exception {
+ for (String value : new String[] {"fast", "normal", "accurate", " FAST
", "Normal", "ACCURATE"}) {
+ TVectorSearchOptions options =
parseOptions(Collections.singletonMap("approx_mode", value));
+ Assert.assertNotNull("approx_mode must be serialized: " + value,
options);
+ TVectorSearchOptions decoded = new TVectorSearchOptions();
+ new TDeserializer().deserialize(decoded, new
TSerializer().serialize(options));
+ Assert.assertTrue(decoded.isSetApproxMode());
+
Assert.assertEquals(TVectorApproxMode.valueOf(value.trim().toUpperCase(java.util.Locale.ROOT)),
+ decoded.getApproxMode());
+ Assert.assertFalse(decoded.isSetNprobes());
+ }
+ }
+
+ @Test
+ public void testOmittedApproxModePreservesNativeDefault() throws Exception
{
+ Assert.assertNull(parseOptions(Collections.emptyMap()));
+ TVectorSearchOptions options = parseOptions(ImmutableMap.of(
+ "nprobes", "4", "refine_factor", "2", "query_parallelism",
"1"));
+ Assert.assertFalse(options.isSetApproxMode());
+ Assert.assertEquals(4, options.getNprobes());
+ Assert.assertEquals(2, options.getRefineFactor());
+ Assert.assertEquals(1, options.getQueryParallelism());
+ }
+
+ @Test
+ public void testRejectInvalidApproxMode() {
+ for (String value : new String[] {null, "", " ", "invalid", "0", "1",
"2", "fastest"}) {
+ AnalysisException error =
Assert.assertThrows(AnalysisException.class,
+ () -> parseOptions(Collections.singletonMap("approx_mode",
value)));
+ Assert.assertTrue(error.getMessage(),
error.getMessage().contains("approx_mode"));
+ }
+ }
+
+ @Test
+ public void testRejectInvalidApproxModeBeforeMetadataAccess() {
+ AnalysisException error = Assert.assertThrows(AnalysisException.class,
+ () -> new VectorSearchTableValuedFunction(
+ Collections.singletonMap("approx_mode", "invalid")));
+ Assert.assertTrue(error.getMessage(),
error.getMessage().contains("approx_mode"));
+ }
+
@Test
public void testRejectInvalidDistanceBoundsBeforeMetadataAccess() {
for (String value : new String[] {"NaN", "Infinity", "-Infinity",
"1e100", "invalid", ""}) {
diff --git a/gensrc/thrift/PlanNodes.thrift b/gensrc/thrift/PlanNodes.thrift
index 9d9a042c469..fb336e14dc1 100644
--- a/gensrc/thrift/PlanNodes.thrift
+++ b/gensrc/thrift/PlanNodes.thrift
@@ -471,6 +471,12 @@ enum TVectorMetric {
HAMMING
}
+enum TVectorApproxMode {
+ FAST = 0,
+ NORMAL = 1,
+ ACCURATE = 2
+}
+
struct TSearchVector {
1: optional TVectorElementType element_type
2: optional i32 dimension
@@ -558,6 +564,8 @@ struct TVectorSearchOptions {
4: optional bool use_index
// Lance: -1 uses available CPU parallelism, 0 selects automatically,
positive values cap it.
5: optional i32 query_parallelism
+ // Unset preserves the vector engine's default speed/recall tradeoff.
+ 6: optional TVectorApproxMode approx_mode
}
// The active union field identifies the logical search kind. A future hybrid
field can contain both
diff --git
a/regression-test/suites/external_table_p0/lance/test_lance_vector_search.groovy
b/regression-test/suites/external_table_p0/lance/test_lance_vector_search.groovy
index efb2ce302d1..98f437e9bfd 100644
---
a/regression-test/suites/external_table_p0/lance/test_lance_vector_search.groovy
+++
b/regression-test/suites/external_table_p0/lance/test_lance_vector_search.groovy
@@ -57,6 +57,7 @@ suite("test_lance_vector_search", "p0,external") {
* refine_factor Positive candidate refinement/reranking factor.
* ef Positive graph-search candidate width.
* use_index true allows index search; false explicitly requests a
flat search.
+ * approx_mode fast, normal, or accurate; defaults to Lance's normal
policy.
*
* An outer WHERE is deliberately different from the TVF filter property:
it is evaluated by
* Doris after Lance returns Top-K and can therefore reduce the final
result below top_k.
@@ -152,6 +153,26 @@ suite("test_lance_vector_search", "p0,external") {
"""
def baselineRows = sql "SELECT row_id, label, _distance FROM
${indexedTopFive} ORDER BY _distance, row_id"
+ // This refined IVF_PQ fixture is a pass-through control, not a
guarantee of equal
+ // recall for algorithms whose candidate selection depends on
approx_mode.
+ ["fast", "normal", "accurate", " FAST "].each { mode ->
+ String modeSearch = indexedTopFive.substring(0,
indexedTopFive.length() - 1) +
+ ", \"approx_mode\"=\"${mode}\")"
+ explain {
+ sql("SELECT row_id, label, _distance FROM ${modeSearch}")
+ contains
"lanceApproxMode=${mode.trim().toUpperCase(java.util.Locale.ROOT)}"
+ }
+ assertEquals(baselineRows,
+ sql("SELECT row_id, label, _distance FROM ${modeSearch}
ORDER BY _distance, row_id"))
+ }
+ ["", "invalid", "0", "1", "2"].each { mode ->
+ String modeSearch = indexedTopFive.substring(0,
indexedTopFive.length() - 1) +
+ ", \"approx_mode\"=\"${mode}\")"
+ test {
+ sql "SELECT row_id FROM ${modeSearch}"
+ exception "approx_mode"
+ }
+ }
[-1, 0, 1, 2, 4].each { parallelism ->
String parallelSearch = indexedTopFive.substring(0,
indexedTopFive.length() - 1) +
", \"query_parallelism\"=\"${parallelism}\")"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
