This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 0011652b1dc [test](lance) Build the missing IVF_PQ regression fixture
(#66779)
0011652b1dc is described below
commit 0011652b1dcdc247b0c0bf51ea6d566a71633079
Author: FANNG <[email protected]>
AuthorDate: Sun Aug 16 07:51:05 2026 +0800
[test](lance) Build the missing IVF_PQ regression fixture (#66779)
### What problem does this PR solve?
Issue Number: Part of #66495
Problem Summary: test_lance_vector_search documents its
doris.vector_search
fixture as carrying an IVF_PQ index, but the fixture SQL
(run07_create_vector_types.sql) delegated index creation to a companion
create_vector_search_index.py that was never committed, and
lance-spark-bundle
0.4.0 cannot create vector indexes through SQL. The table therefore had
no
index at all, so every use_index / nprobes / refine_factor query in the
suite
silently executed a flat KNN scan while the goldens still looked
correct.
Nothing in the repository could observe the difference.
Reproduction: build the table the old fixture built and probe it with
nprobes=1. Lance ignores nprobes on an unindexed dataset and returns
exactly
the flat top-10 (rows 256,255,257,254,258,253,259,252,260,251 for the
boundary
query) - identical to the flat baseline, which is why the defect was
invisible.
Fix: replace the Spark-created table with an offline-generated Directory
Namespace V2 catalog that carries a real IVF_PQ index, and add the
evidence
that the index is actually used.
- lance_build_preinstalled_catalog.py rebuilds the committed fixture and
self-checks it: exactly one IVF_PQ index named embedding_ivf_pq_f32
covering
every fragment, ANNSubIndex and ANNIvfPartition present in the indexed
plan,
KNNVectorDistance and no ANN node in the flat plan, and the exact
16 * (n - r)^2 distance ladder that every golden and comment encodes, so
a
change to the data shape fails here instead of surfacing as an opaque
golden
diff. Index creation goes through the physical dataset because
DirectoryNamespace.create_table_index raises UnsupportedOperationError.
- doris.vs_ivf_pq_f32 replaces doris.vector_search: 1024 rows in two
fragments, 16-dimensional Float32 embedding[j] = (row_id - 1) + j, so
the
exact squared L2 distance between rows r and n is 16 * (n - r)^2 and the
head/tail queries have no distance ties. Columns are declared NOT NULL
to
match the fixture being replaced, keeping the only non-nullable Lance
column
mapping recorded by any Lance suite's DESC golden. The
vs_<algorithm>_<element type>
name encodes one cell of the algorithm x element type matrix, so a
missing
combination is visible from the table list alone.
- The suite gains a silent-fallback discriminator. Row 256 sits on the
first
IVF partition boundary, so a genuine single-partition probe must miss
true
neighbours from the next partition. The suite asserts that the nprobes=1
distance sequence differs from flat search; on the previous unindexed
fixture the two are identical and the assertion fails. Distances are
compared rather than row ids because the boundary query is symmetric and
rows r-d and r+d tie. top_k is 9 there, the last cut that lands on a
complete tie pair: at 10 the pair at distance 400 is split, so the
golden
would pin an arbitrary winner that any change to Lance's top-k selection
could flip. Which partition edge row 256 lands next to changes on every
retrain, so no measured range is hardcoded; --check prints it instead.
- IVF_PQ is lossy, so every indexed query uses refine_factor and the
suite
documents indexed/flat agreement as an observed property of this frozen
fixture and pinned Lance version, not an algorithm guarantee.
The fixture is generated with the pins in
lance_fixture_requirements.txt.
Its readers do not all run the same Lance version - a BE built from
source uses
lance-c v0.1.2 (lance-rs 4.0.1) per thirdparty/vars.sh, the BE in CI
comes from
the prebuilt doris-thirdparty package and is already on lance-c v0.1.6
(lance-rs 7.0.0-beta), and Spark writes into the same __manifest through
lance-java 4.0.0. The writer is therefore pinned to the oldest Lance in
that
set, which every reader can read. Verified that this does not make the
goldens
version-dependent: pylance 7.0.0 reads the committed fixture with
results
identical to pylance 4.0.1 - same index, same refined top-5, same
nprobes=1
boundary rows, same IVF partition ranges.
Index training is not bit-reproducible, so regenerating the fixture
changes
the binary output; the reproducible properties are asserted by the
generator
self-check instead. IVF_FLAT, IVF_SQ, IVF_HNSW_* and the other vector
element
types are follow-up work for #66495.
### Release note
None
### Check List (For Author)
- Test: Regression test
- Fixture generator self-check with the pinned dependencies
- test_lance_vector_search regenerated with -forceGenOut, then passed
the
normal golden comparison
- The whole external_table_p0/lance directory passed (6 suites, 0
failed),
covering the pre-existing suites that share the regenerated __manifest
- Cross-checked that the nprobes=1 golden row order matches what pylance
records probing the same physical index directly
- Behavior changed: No, test fixture and regression coverage only
- Does this need documentation: No
---
.../lance/run07_create_vector_types.sql | 46 ---
.../scripts/lance_build_preinstalled_catalog.py | 398 +++++++++++++++++++++
.../iceberg/scripts/lance_fixture_requirements.txt | 37 ++
.../bitmap_page_lookup.lance | Bin 0 -> 703 bytes
.../page_data.lance | Bin 0 -> 576 bytes
.../page_lookup.lance | Bin 0 -> 1151 bytes
.../page_data.lance | Bin 635 -> 0 bytes
.../page_lookup.lance | Bin 1269 -> 0 bytes
.../bitmap_page_lookup.lance | Bin 393 -> 0 bytes
.../bitmap_page_lookup.lance | Bin 0 -> 393 bytes
.../bitmap_page_lookup.lance | Bin 683 -> 0 bytes
.../0-aabf9667-c2e9-49d9-a16d-59e616c77195.txn | 2 -
.../14-2b2b0055-53cb-4e2e-a2d8-c17bb410ce79.txn | Bin 0 -> 940 bytes
.../_versions/18446744073709551600.manifest | Bin 0 -> 1949 bytes
.../_versions/18446744073709551613.manifest | Bin 1257 -> 0 bytes
.../_versions/18446744073709551614.manifest | Bin 693 -> 0 bytes
.../__manifest/_versions/latest_version_hint.json | 2 +-
...1101100100100100a8e404e57ae0aa8a08e98d5ec.lance | Bin 0 -> 1667 bytes
...0101001100101009117684cbe99ac14edc8e6fcd2.lance | Bin 1456 -> 0 bytes
.../auxiliary.idx | Bin 0 -> 12017 bytes
.../4231299b-169a-4694-b440-7fb7396b1188/index.idx | Bin 0 -> 592 bytes
.../0-c618efe7-b762-45a3-9600-02e54c17cf87.txn | Bin 0 -> 352 bytes
.../1-050f2e5b-6487-4532-b15a-8200a82dc87b.txn | Bin 0 -> 126 bytes
.../2-8b309f5c-51bc-414c-98cf-dcc867178c25.txn | Bin 0 -> 196 bytes
.../_versions/18446744073709551612.manifest | Bin 0 -> 873 bytes
.../_versions/18446744073709551613.manifest | Bin 0 -> 644 bytes
.../_versions/18446744073709551614.manifest | Bin 0 -> 785 bytes
...1101111011010118622b246be93248675bfdaadfa.lance | Bin 0 -> 46994 bytes
...011100011000111e29e464c90b09bf330ca3b02c2.lance | Bin 0 -> 46994 bytes
.../lance/test_lance_vector_search.out | 61 ++--
.../lance/test_lance_vector_search.groovy | 109 ++++--
31 files changed, 561 insertions(+), 94 deletions(-)
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/create_preinstalled_scripts/lance/run07_create_vector_types.sql
b/docker/thirdparties/docker-compose/iceberg/scripts/create_preinstalled_scripts/lance/run07_create_vector_types.sql
deleted file mode 100644
index 6a6a99c572a..00000000000
---
a/docker/thirdparties/docker-compose/iceberg/scripts/create_preinstalled_scripts/lance/run07_create_vector_types.sql
+++ /dev/null
@@ -1,46 +0,0 @@
--- Dedicated fixture for Doris Lance vector_search() regression tests.
--- Two INSERT statements intentionally create multiple Lance fragments. The
--- companion create_vector_search_index.py builds the vector index because
--- lance-spark-bundle 0.4.0 does not expose vector index creation through SQL.
-CREATE NAMESPACE IF NOT EXISTS lance.doris;
-
-DROP TABLE IF EXISTS lance.doris.vector_search;
-
-CREATE TABLE lance.doris.vector_search (
- row_id BIGINT NOT NULL,
- category STRING NOT NULL,
- label STRING NOT NULL,
- embedding ARRAY<FLOAT> NOT NULL
-) USING lance
-TBLPROPERTIES ('embedding.arrow.fixed-size-list.size' = '4');
-
--- row_id 1 is the origin. For query [0, 0, 0, 0], the exact squared L2
--- distance of row_id n is 30 * (n - 1)^2.
-INSERT INTO lance.doris.vector_search
-SELECT
- id + 1 AS row_id,
- CASE WHEN id % 2 = 0 THEN 'even' ELSE 'odd' END AS category,
- concat('item-', lpad(CAST(id + 1 AS STRING), 4, '0')) AS label,
- array(
- CAST(id AS FLOAT),
- CAST(id * 2 AS FLOAT),
- CAST(id * 3 AS FLOAT),
- CAST(id * 4 AS FLOAT)
- ) AS embedding
-FROM range(0, 512);
-
-INSERT INTO lance.doris.vector_search
-SELECT
- id + 1 AS row_id,
- CASE WHEN id % 2 = 0 THEN 'even' ELSE 'odd' END AS category,
- concat('item-', lpad(CAST(id + 1 AS STRING), 4, '0')) AS label,
- array(
- CAST(id AS FLOAT),
- CAST(id * 2 AS FLOAT),
- CAST(id * 3 AS FLOAT),
- CAST(id * 4 AS FLOAT)
- ) AS embedding
-FROM range(512, 1024);
-
-SELECT count(*) AS row_count, min(row_id) AS min_id, max(row_id) AS max_id
-FROM lance.doris.vector_search;
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/lance_build_preinstalled_catalog.py
b/docker/thirdparties/docker-compose/iceberg/scripts/lance_build_preinstalled_catalog.py
new file mode 100644
index 00000000000..0d5a408d1b9
--- /dev/null
+++
b/docker/thirdparties/docker-compose/iceberg/scripts/lance_build_preinstalled_catalog.py
@@ -0,0 +1,398 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Rebuild the preinstalled Lance Directory (V2) catalog fixture.
+
+This offline generator produces
docker-compose/iceberg/scripts/preinstalled_data/lance,
+the fixture that the iceberg docker environment copies into MinIO at
s3://warehouse/lance.
+It exists because lance-spark-bundle does not expose vector index creation
through SQL, so
+indexed vector-search fixtures cannot be built by the Spark preinstall
scripts. That gap is
+why the previous fixture had no vector index at all:
run07_create_vector_types.sql created
+doris.vector_search and deferred index creation to a companion script that
never existed, so
+every "indexed" query in test_lance_vector_search silently ran a flat KNN scan.
+
+The generated catalog contains:
+ - __manifest Directory Namespace V2 manifest table (with its
scalar indexes).
+ - all_types.lance The pre-existing compatibility-mode root table,
re-registered as-is.
+ - The `doris` namespace with one indexed vector table per ANN algorithm
(hash-prefixed
+ directories), listed in VECTOR_TABLES below.
+
+Every vector table holds identical deterministic data: 1024 rows in two
512-row fragments,
+16-dim Float32 `embedding` where embedding[j] = (row_id - 1) + j. For a query
equal to the
+vector of row r, the exact squared L2 distance of row n is 16 * (n - r)^2, so
head/tail
+queries have no distance ties. All values are integers below 2^11 and
therefore exact in
+Float32.
+
+Environment: python3 with the exact pins in lance_fixture_requirements.txt.
Index training
+(IVF kmeans) is not bit-reproducible across runs, so regenerating this fixture
changes the
+binary output and requires regenerating the dependent regression .out files.
Reproducible
+properties are asserted by the self-check below instead: logical metadata, the
exact L2
+distance ladder the regression goldens encode, plan shape, and the
partition-boundary
+discriminator. Indexed-vs-flat agreement is lossy for IVF_PQ, so it is only
recorded.
+
+The self-check is the entire contract for a fixture whose bytes cannot be
reproduced, so
+this script refuses to run under python -O, where assert statements are
stripped.
+
+Usage:
+ python3 lance_build_preinstalled_catalog.py # rebuild in place
+ python3 lance_build_preinstalled_catalog.py --check # self-check the
existing fixture
+"""
+
+import argparse
+import io
+import json
+import shutil
+import sys
+import tempfile
+from datetime import timedelta
+from pathlib import Path
+
+import lance
+import lance_namespace
+import pyarrow as pa
+import pyarrow.ipc as ipc
+from lance_namespace_urllib3_client.models import (
+ CreateNamespaceRequest,
+ CreateTableRequest,
+ DescribeTableRequest,
+ ListTablesRequest,
+ RegisterTableRequest,
+)
+
+DIM = 16
+ROWS = 1024
+FRAGMENT_ROWS = 512
+NUM_PARTITIONS = 4
+NAMESPACE = "doris"
+ALL_TYPES_DIR = "all_types.lance"
+MANIFEST_DIR = "__manifest"
+
+# 4-bit PQ keeps codebook training comfortable on 1024 rows. This only serves
fixture
+# stability and is not a Doris compatibility statement about PQ parameters.
+PQ_BUILD_PARAMS = {"num_sub_vectors": 4, "num_bits": 4}
+
+# One table per ANN algorithm and element type, identical data, exactly one
index named
+# embedding_<the table name without its vs_ prefix>. Naming is
vs_<algorithm>_<element
+# type>, so one table is exactly one cell of the algorithm x element type
matrix and a
+# missing combination is visible from the table list alone. The build loop and
the
+# self-check are driven entirely by these specs; follow-up work for #66495
adds the
+# remaining algorithms (IVF_FLAT, IVF_SQ, IVF_HNSW_*) and element types here.
+VECTOR_TABLES = {
+ "vs_ivf_pq_f32": {"index_type": "IVF_PQ", "params": PQ_BUILD_PARAMS},
+}
+
+# The head query is exactly row 1's vector; the tail query is row 1024's. Only
endpoint
+# vectors are used so that 16 * (n - r)^2 never ties between two different
rows n.
+HEAD_QUERY = [float(j) for j in range(DIM)]
+TAIL_QUERY = [float(ROWS - 1 + j) for j in range(DIM)]
+# On this collinear data IVF kmeans yields four contiguous row ranges, and row
256 lands near
+# one of the internal edges - which side, and at exactly which row, changes
every time the
+# index is retrained, so nothing here hardcodes it (--check prints the
measured range). What
+# matters is only that part of row 256's true neighbourhood falls in an
adjacent partition.
+# The regression suites query this row with nprobes=1 as their silent-fallback
discriminator:
+# a real single-partition probe must miss those neighbours (result != flat),
while a silent
+# flat fallback returns exactly the flat result - verified directly: on an
unindexed copy of
+# this data Lance ignores nprobes entirely and returns the flat rows. The
self-check pins
+# this property for every vector table, so regenerating the fixture with
partition edges that
+# no longer split row 256's neighbourhood fails here instead of in the suites.
Each table
+# trains its own IVF clustering, so this is checked per table.
+BOUNDARY_ROW = 256
+BOUNDARY_QUERY = [float(BOUNDARY_ROW - 1 + j) for j in range(DIM)]
+
+
+def make_fragment_table(row_offset_start: int, row_offset_end: int) ->
pa.Table:
+ offsets = list(range(row_offset_start, row_offset_end))
+ embedding = pa.FixedSizeListArray.from_arrays(
+ pa.array(
+ [float(offset + j) for offset in offsets for j in range(DIM)],
+ type=pa.float32(),
+ ),
+ DIM,
+ )
+ table = pa.table(
+ {
+ "row_id": pa.array([offset + 1 for offset in offsets],
type=pa.int64()),
+ "category": pa.array(
+ ["even" if offset % 2 == 0 else "odd" for offset in offsets]
+ ),
+ "label": pa.array([f"item-{offset + 1:04d}" for offset in
offsets]),
+ "embedding": embedding,
+ }
+ )
+ # The Spark fixture this replaces declared every column NOT NULL, and its
DESC golden is
+ # the only place in the Lance suites that records a non-nullable Lance
column mapping to
+ # Doris 'No'. pyarrow defaults to nullable, so restate it to keep that
coverage.
+ return table.cast(
+ pa.schema([pa.field(f.name, f.type, nullable=False) for f in
table.schema])
+ )
+
+
+def index_name_of(table_name: str) -> str:
+ # vs_ivf_pq_f32 -> embedding_ivf_pq_f32
+ return "embedding_" + table_name.removeprefix("vs_")
+
+
+def create_vector_table(namespace, table_name: str) -> str:
+ first = make_fragment_table(0, FRAGMENT_ROWS)
+ buffer = io.BytesIO()
+ with ipc.new_stream(buffer, first.schema) as writer:
+ writer.write_table(first)
+ response = namespace.create_table(
+ CreateTableRequest(id=[NAMESPACE, table_name]), buffer.getvalue()
+ )
+ # Never predict the hashed storage path; always use the location the
namespace returns.
+ location = response.location
+ lance.write_dataset(make_fragment_table(FRAGMENT_ROWS, ROWS), location,
mode="append")
+ return location
+
+
+def compact_manifest(root: Path) -> None:
+ # Every namespace mutation above leaves a manifest fragment, index delta,
and version
+ # behind. Fold them together so the committed fixture stays small and
reviewable. Only
+ # the manifest is compacted: the vector tables must keep exactly two
fragments.
+ manifest = lance.dataset(str(root / MANIFEST_DIR))
+ manifest.optimize.compact_files()
+
manifest.optimize.optimize_indices(num_indices_to_merge=len(manifest.list_indices()))
+ manifest.cleanup_old_versions(older_than=timedelta(0),
delete_unverified=True)
+ # cleanup_old_versions does not reclaim superseded index deltas; drop
every index
+ # directory the resulting manifest version no longer references.
+ manifest = lance.dataset(str(root / MANIFEST_DIR))
+ referenced = {index["uuid"] for index in manifest.list_indices()}
+ for index_dir in (root / MANIFEST_DIR / "_indices").iterdir():
+ if index_dir.name not in referenced:
+ shutil.rmtree(index_dir)
+ # pylance 4.0.1 does not write the optional latest-version hint. Write it
to keep the
+ # fixture shape identical to the previous one for every consuming reader.
+ hint = root / MANIFEST_DIR / "_versions" / "latest_version_hint.json"
+ hint.write_text(f'{{"version":{manifest.version}}}')
+
+
+def build(root: Path, all_types_source: Path) -> None:
+ shutil.copytree(all_types_source, root / ALL_TYPES_DIR)
+ namespace = lance_namespace.connect("dir", {"root": str(root)})
+ namespace.register_table(
+ RegisterTableRequest(id=["all_types"], location=ALL_TYPES_DIR)
+ )
+ namespace.create_namespace(CreateNamespaceRequest(id=[NAMESPACE]))
+ for table_name, spec in VECTOR_TABLES.items():
+ location = create_vector_table(namespace, table_name)
+ # DirectoryNamespace.create_table_index exists but raises
UnsupportedOperationError,
+ # so open the physical dataset at the location the namespace returned
and index it
+ # there. index_file_version V3 is what the Doris BE reads through
lance-c; the
+ # default would produce an index the backend cannot open.
+ lance.dataset(location).create_index(
+ "embedding",
+ spec["index_type"],
+ name=index_name_of(table_name),
+ metric="L2",
+ num_partitions=NUM_PARTITIONS,
+ sample_rate=256,
+ index_file_version="V3",
+ **spec["params"],
+ )
+ compact_manifest(root)
+
+
+def topk(dataset, query, k: int, use_index: bool, **nearest_kwargs):
+ nearest = {"column": "embedding", "q": query, "k": k, "use_index":
use_index}
+ if use_index:
+ nearest.setdefault("nprobes", NUM_PARTITIONS)
+ nearest.update(nearest_kwargs)
+ table = dataset.scanner(nearest=nearest).to_table()
+ return list(zip(table["row_id"].to_pylist(),
table["_distance"].to_pylist()))
+
+
+def check_vector_dataset(name: str, location: str, index_type: str):
+ dataset = lance.dataset(location)
+ assert dataset.count_rows() == ROWS, f"{name}: expected {ROWS} rows"
+ fragments = dataset.get_fragments()
+ assert len(fragments) == 2, f"{name}: expected 2 fragments"
+ embedding_type = dataset.schema.field("embedding").type
+ assert pa.types.is_fixed_size_list(embedding_type), f"{name}: embedding
type"
+ assert embedding_type.list_size == DIM, f"{name}: embedding dimension"
+ assert embedding_type.value_type == pa.float32(), f"{name}: embedding
element type"
+ for field in dataset.schema:
+ assert not field.nullable, f"{name}: column {field.name} must be NOT
NULL"
+
+ # The regression goldens are hand-checkable only because embedding[j] =
(row_id-1)+j,
+ # which makes the exact squared L2 distance between rows r and n equal
16*(n-r)^2. Every
+ # distance in the .out files and every comment in the suites encodes that
ladder, so
+ # assert it against a flat scan rather than trusting the row/dimension
counts above: a
+ # change to the data shape would otherwise leave the self-check green and
surface only
+ # as an opaque golden diff after a full docker regression run.
+ ladder = [(row, 16.0 * step * step) for step, row in enumerate(range(1,
11))]
+ assert topk(dataset, HEAD_QUERY, 10, use_index=False) == ladder, (
+ f"{name}: flat top-10 from row 1 is not the 16*(n-r)^2 ladder; the
fixture data "
+ "shape changed and every dependent golden and comment is now stale"
+ )
+
+ indices = dataset.list_indices()
+ assert len(indices) == 1, f"{name}: expected exactly one index"
+ index = indices[0]
+ assert index["name"] == index_name_of(name), f"{name}: index name
{index['name']}"
+ assert index["type"] == index_type, f"{name}: index type {index['type']}"
+ indexed_fragments = set(index["fragment_ids"])
+ all_fragments = {fragment.fragment_id for fragment in fragments}
+ assert indexed_fragments == all_fragments, f"{name}: index does not cover
all fragments"
+
+ for query in (HEAD_QUERY, TAIL_QUERY):
+ indexed_plan = dataset.scanner(
+ nearest={"column": "embedding", "q": query, "k": 5, "nprobes":
NUM_PARTITIONS}
+ ).explain_plan(True)
+ assert "ANNSubIndex" in indexed_plan, f"{name}: indexed plan lacks
ANNSubIndex"
+ assert "ANNIvfPartition" in indexed_plan, f"{name}: plan lacks
ANNIvfPartition"
+ flat_plan = dataset.scanner(
+ nearest={"column": "embedding", "q": query, "k": 5, "use_index":
False}
+ ).explain_plan(True)
+ assert "ANNSubIndex" not in flat_plan, f"{name}: flat plan uses ANN"
+ assert "KNNVectorDistance" in flat_plan, f"{name}: flat plan lacks KNN
node"
+ return dataset
+
+
+def check_lossy_results(name: str, dataset) -> None:
+ # IVF_PQ stores quantized codes, so agreement with flat search is an
observed property
+ # of this frozen fixture and the pinned Lance version, never a guarantee.
The
+ # regression suite therefore queries it with refine_factor, which reranks
candidates
+ # with exact distances; record what that suite will observe.
+ raw = topk(dataset, HEAD_QUERY, 5, use_index=True)
+ flat = topk(dataset, HEAD_QUERY, 5, use_index=False)
+ assert len(raw) == 5, f"{name}: indexed search must return k rows"
+ raw_agreement = "matches" if raw == flat else "differs from"
+ print(f"record: {name} full-probe top-5 {raw_agreement} flat search:
{raw}")
+ refined = topk(dataset, HEAD_QUERY, 5, use_index=True, refine_factor=10)
+ refined_agreement = "matches" if refined == flat else "differs from"
+ print(f"record: {name} refined top-5 {refined_agreement} flat search:
{refined}")
+
+
+def check_boundary_discriminator(name: str, dataset) -> None:
+ # See BOUNDARY_ROW above. The lossy index uses refine_factor so the
comparison against
+ # flat runs on exact distances, exactly like the regression suite does.
+ single_rows = topk(
+ dataset, BOUNDARY_QUERY, 10, use_index=True, nprobes=1,
refine_factor=10)
+ flat_rows = topk(dataset, BOUNDARY_QUERY, 10, use_index=False)
+ # Compare distances, not row ids: the boundary query is symmetric, so rows
r-d and r+d
+ # tie and either may fill the last slot. Only a missed neighbour changes
the distances.
+ single = [distance for _, distance in single_rows]
+ flat = [distance for _, distance in flat_rows]
+ assert len(single) == 10, f"{name}: boundary nprobes=1 must still return k
rows"
+ assert single != flat, (
+ f"{name}: row {BOUNDARY_ROW} no longer discriminates nprobes=1 from
flat search; "
+ "the IVF partition boundaries moved. Update BOUNDARY_ROW here and the
boundary "
+ "queries in the regression suites together."
+ )
+ print(f"record: {name} boundary nprobes=1 top-10 rows: "
+ f"{[row for row, _ in single_rows]}")
+ # The partition edge moves on every retrain, so report where it actually
landed. This is
+ # the first thing to look at when a boundary golden shifts or this check
starts failing.
+ probed = sorted(row for row, _ in topk(
+ dataset, BOUNDARY_QUERY, ROWS, use_index=True, nprobes=1,
refine_factor=1))
+ contiguous = probed == list(range(probed[0], probed[-1] + 1))
+ print(f"record: {name} partition holding row {BOUNDARY_ROW}: rows "
+ f"{probed[0]}-{probed[-1]} ({len(probed)} rows,
contiguous={contiguous})")
+
+
+def check_catalog(root: Path) -> None:
+ namespace = lance_namespace.connect("dir", {"root": str(root)})
+ tables = namespace.list_tables(ListTablesRequest(id=[NAMESPACE]))
+ assert sorted(tables.tables) == sorted(VECTOR_TABLES), (
+ f"unexpected {NAMESPACE} tables: {tables.tables}"
+ )
+ root_tables = namespace.list_tables(ListTablesRequest(id=[]))
+ assert "all_types" in root_tables.tables, "all_types is not registered at
the root"
+ all_types =
namespace.describe_table(DescribeTableRequest(id=["all_types"]))
+ all_types_path = Path(all_types.location.removeprefix("file://"))
+ assert all_types_path.is_dir(), f"all_types location missing:
{all_types.location}"
+ # all_types.lance is copied through verbatim and cannot be regenerated by
this script, so
+ # open it rather than only stat it: the rebuild path below deletes the
previous fixture,
+ # which is its only copy. 12 rows is what test_lance_catalog_all_types.out
records.
+ assert lance.dataset(str(all_types_path)).count_rows() == 12, (
+ "all_types.lance did not survive the copy intact"
+ )
+
+ manifest = lance.dataset(str(root / MANIFEST_DIR))
+ manifest_indices = {index["name"] for index in manifest.list_indices()}
+ for required in ("object_id_btree", "object_type_bitmap",
"base_objects_label_list"):
+ assert required in manifest_indices, f"__manifest lacks index
{required}"
+ hint = json.loads(
+ (root / MANIFEST_DIR / "_versions" /
"latest_version_hint.json").read_text()
+ )
+ assert hint["version"] == manifest.version, (
+ f"latest_version_hint {hint['version']} does not match manifest
version "
+ f"{manifest.version}"
+ )
+
+ for table_name, spec in VECTOR_TABLES.items():
+ described = namespace.describe_table(
+ DescribeTableRequest(id=[NAMESPACE, table_name])
+ )
+ path = Path(described.location.removeprefix("file://"))
+ assert path.is_dir(), f"{table_name} location missing:
{described.location}"
+ dataset = check_vector_dataset(table_name, described.location,
spec["index_type"])
+ check_lossy_results(table_name, dataset)
+ check_boundary_discriminator(table_name, dataset)
+ print(f"self-check OK: {root}")
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument(
+ "--output",
+ type=Path,
+ default=Path(__file__).resolve().parent / "preinstalled_data" /
"lance",
+ help="fixture directory to rebuild (default:
scripts/preinstalled_data/lance)",
+ )
+ parser.add_argument(
+ "--check",
+ action="store_true",
+ help="only run the self-check against the existing fixture",
+ )
+ args = parser.parse_args()
+ output: Path = args.output
+
+ # Every verification in this script is an assert, and the self-check is
the whole
+ # contract for a fixture whose bytes are not reproducible. Under -O the
rebuild
+ # below would replace the committed fixture having verified nothing at all.
+ if not __debug__:
+ print("refusing to run with assertions disabled (python -O)",
file=sys.stderr)
+ return 1
+
+ if args.check:
+ check_catalog(output)
+ return 0
+
+ all_types_source = output / ALL_TYPES_DIR
+ if not all_types_source.is_dir():
+ print(f"missing all_types source: {all_types_source}", file=sys.stderr)
+ return 1
+
+ with tempfile.TemporaryDirectory(prefix="lance_fixture_") as staging_name:
+ staging = Path(staging_name) / "lance"
+ staging.mkdir()
+ build(staging, all_types_source)
+ check_catalog(staging)
+ backup = output.with_name(output.name + ".old")
+ if backup.exists():
+ shutil.rmtree(backup)
+ output.rename(backup)
+ shutil.move(str(staging), str(output))
+ shutil.rmtree(backup)
+ check_catalog(output)
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/lance_fixture_requirements.txt
b/docker/thirdparties/docker-compose/iceberg/scripts/lance_fixture_requirements.txt
new file mode 100644
index 00000000000..58c29376cc1
--- /dev/null
+++
b/docker/thirdparties/docker-compose/iceberg/scripts/lance_fixture_requirements.txt
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Exact pins for lance_build_preinstalled_catalog.py.
+#
+# The fixture has several readers and they do not all run the same Lance
version:
+#
+# lance-c v0.1.2 (lance-rs 4.0.1) BE built from source -
thirdparty/vars.sh
+# lance-c v0.1.6 (lance-rs 7.0.0-beta) BE in CI - the prebuilt
doris-thirdparty package
+# lance-java 4.0.0 Spark, via lance-spark-bundle 0.4.0;
it registers
+# runtime tables into the same
__manifest
+# lance-java (FE) Doris FE Directory Namespace client
+#
+# So pin the writer to the oldest Lance in that set rather than to whichever
one the BE
+# happens to use: an older writer is readable by every reader above, while a
newer one
+# would leave the source-build and Spark cells unverified. Verified for the
committed
+# fixture: pylance 7.0.0 reads it with results identical to pylance 4.0.1 -
same index,
+# same refined top-5, same nprobes=1 boundary rows, same IVF partition ranges
- so the
+# regression goldens are stable across the lance-rs 4 to 7 gap and do not
depend on which
+# lance-c the backend was built with.
+pylance==4.0.1
+lance-namespace==0.6.1
+pyarrow==25.0.0
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/1f436c06-a392-4c38-af67-d10bef96f82f/bitmap_page_lookup.lance
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/1f436c06-a392-4c38-af67-d10bef96f82f/bitmap_page_lookup.lance
new file mode 100644
index 00000000000..950f84fb016
Binary files /dev/null and
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/1f436c06-a392-4c38-af67-d10bef96f82f/bitmap_page_lookup.lance
differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/6ac57aba-0a7a-4303-ba43-c61610e205b9/page_data.lance
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/6ac57aba-0a7a-4303-ba43-c61610e205b9/page_data.lance
new file mode 100644
index 00000000000..a28abb8fd6b
Binary files /dev/null and
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/6ac57aba-0a7a-4303-ba43-c61610e205b9/page_data.lance
differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/6ac57aba-0a7a-4303-ba43-c61610e205b9/page_lookup.lance
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/6ac57aba-0a7a-4303-ba43-c61610e205b9/page_lookup.lance
new file mode 100644
index 00000000000..c9e326171ed
Binary files /dev/null and
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/6ac57aba-0a7a-4303-ba43-c61610e205b9/page_lookup.lance
differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/859ed2a5-7560-4701-a5be-e93d512df107/page_data.lance
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/859ed2a5-7560-4701-a5be-e93d512df107/page_data.lance
deleted file mode 100644
index c4501c74cc3..00000000000
Binary files
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/859ed2a5-7560-4701-a5be-e93d512df107/page_data.lance
and /dev/null differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/859ed2a5-7560-4701-a5be-e93d512df107/page_lookup.lance
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/859ed2a5-7560-4701-a5be-e93d512df107/page_lookup.lance
deleted file mode 100644
index 12e258c1a40..00000000000
Binary files
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/859ed2a5-7560-4701-a5be-e93d512df107/page_lookup.lance
and /dev/null differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/a72dde56-017f-4679-a8c8-f2898fa14c02/bitmap_page_lookup.lance
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/a72dde56-017f-4679-a8c8-f2898fa14c02/bitmap_page_lookup.lance
deleted file mode 100644
index 1f85d1ccab3..00000000000
Binary files
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/a72dde56-017f-4679-a8c8-f2898fa14c02/bitmap_page_lookup.lance
and /dev/null differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/be88a0b2-a11c-4537-b8eb-120ad3c24380/bitmap_page_lookup.lance
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/be88a0b2-a11c-4537-b8eb-120ad3c24380/bitmap_page_lookup.lance
new file mode 100644
index 00000000000..46d15e405e1
Binary files /dev/null and
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/be88a0b2-a11c-4537-b8eb-120ad3c24380/bitmap_page_lookup.lance
differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/cd4c2b88-a91b-4ad4-9957-c98473f1e258/bitmap_page_lookup.lance
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/cd4c2b88-a91b-4ad4-9957-c98473f1e258/bitmap_page_lookup.lance
deleted file mode 100644
index 2cf940fadf8..00000000000
Binary files
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_indices/cd4c2b88-a91b-4ad4-9957-c98473f1e258/bitmap_page_lookup.lance
and /dev/null differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_transactions/0-aabf9667-c2e9-49d9-a16d-59e616c77195.txn
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_transactions/0-aabf9667-c2e9-49d9-a16d-59e616c77195.txn
deleted file mode 100644
index 514d2181575..00000000000
---
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_transactions/0-aabf9667-c2e9-49d9-a16d-59e616c77195.txn
+++ /dev/null
@@ -1,2 +0,0 @@
-$aabf9667-c2e9-49d9-a16d-59e616c77195��U object_id ���������*string8R1
-,lance-schema:unenforced-primary-key:position0`$ object_type
���������*string8#location ���������*string08#metadata
���������*string08% base_objects ���������*list08 object_id
*string08
\ No newline at end of file
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_transactions/14-2b2b0055-53cb-4e2e-a2d8-c17bb410ce79.txn
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_transactions/14-2b2b0055-53cb-4e2e-a2d8-c17bb410ce79.txn
new file mode 100644
index 00000000000..e6e6a29a6ef
Binary files /dev/null and
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_transactions/14-2b2b0055-53cb-4e2e-a2d8-c17bb410ce79.txn
differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_versions/18446744073709551600.manifest
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_versions/18446744073709551600.manifest
new file mode 100644
index 00000000000..bb11a8c6aa1
Binary files /dev/null and
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_versions/18446744073709551600.manifest
differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_versions/18446744073709551613.manifest
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_versions/18446744073709551613.manifest
deleted file mode 100644
index 021e641ad20..00000000000
Binary files
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_versions/18446744073709551613.manifest
and /dev/null differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_versions/18446744073709551614.manifest
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_versions/18446744073709551614.manifest
deleted file mode 100644
index e5a5389182f..00000000000
Binary files
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_versions/18446744073709551614.manifest
and /dev/null differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_versions/latest_version_hint.json
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_versions/latest_version_hint.json
index 218abba1699..e64e1a4a8d6 100644
---
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_versions/latest_version_hint.json
+++
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/_versions/latest_version_hint.json
@@ -1 +1 @@
-{"version":2}
\ No newline at end of file
+{"version":15}
\ No newline at end of file
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/data/1011101111101100100100100a8e404e57ae0aa8a08e98d5ec.lance
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/data/1011101111101100100100100a8e404e57ae0aa8a08e98d5ec.lance
new file mode 100644
index 00000000000..eb5fe14bc4d
Binary files /dev/null and
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/data/1011101111101100100100100a8e404e57ae0aa8a08e98d5ec.lance
differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/data/1111110000101001100101009117684cbe99ac14edc8e6fcd2.lance
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/data/1111110000101001100101009117684cbe99ac14edc8e6fcd2.lance
deleted file mode 100644
index e223dd4c0a4..00000000000
Binary files
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/__manifest/data/1111110000101001100101009117684cbe99ac14edc8e6fcd2.lance
and /dev/null differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_indices/4231299b-169a-4694-b440-7fb7396b1188/auxiliary.idx
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_indices/4231299b-169a-4694-b440-7fb7396b1188/auxiliary.idx
new file mode 100644
index 00000000000..ceadc9bfb3f
Binary files /dev/null and
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_indices/4231299b-169a-4694-b440-7fb7396b1188/auxiliary.idx
differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_indices/4231299b-169a-4694-b440-7fb7396b1188/index.idx
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_indices/4231299b-169a-4694-b440-7fb7396b1188/index.idx
new file mode 100644
index 00000000000..034746fb2a5
Binary files /dev/null and
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_indices/4231299b-169a-4694-b440-7fb7396b1188/index.idx
differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_transactions/0-c618efe7-b762-45a3-9600-02e54c17cf87.txn
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_transactions/0-c618efe7-b762-45a3-9600-02e54c17cf87.txn
new file mode 100644
index 00000000000..e88a51dc6af
Binary files /dev/null and
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_transactions/0-c618efe7-b762-45a3-9600-02e54c17cf87.txn
differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_transactions/1-050f2e5b-6487-4532-b15a-8200a82dc87b.txn
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_transactions/1-050f2e5b-6487-4532-b15a-8200a82dc87b.txn
new file mode 100644
index 00000000000..2f70cfa4346
Binary files /dev/null and
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_transactions/1-050f2e5b-6487-4532-b15a-8200a82dc87b.txn
differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_transactions/2-8b309f5c-51bc-414c-98cf-dcc867178c25.txn
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_transactions/2-8b309f5c-51bc-414c-98cf-dcc867178c25.txn
new file mode 100644
index 00000000000..dc51913682e
Binary files /dev/null and
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_transactions/2-8b309f5c-51bc-414c-98cf-dcc867178c25.txn
differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_versions/18446744073709551612.manifest
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_versions/18446744073709551612.manifest
new file mode 100644
index 00000000000..06e94bcc45d
Binary files /dev/null and
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_versions/18446744073709551612.manifest
differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_versions/18446744073709551613.manifest
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_versions/18446744073709551613.manifest
new file mode 100644
index 00000000000..1206a247848
Binary files /dev/null and
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_versions/18446744073709551613.manifest
differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_versions/18446744073709551614.manifest
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_versions/18446744073709551614.manifest
new file mode 100644
index 00000000000..1adf04ca4f5
Binary files /dev/null and
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/_versions/18446744073709551614.manifest
differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/data/1110011011101111011010118622b246be93248675bfdaadfa.lance
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/data/1110011011101111011010118622b246be93248675bfdaadfa.lance
new file mode 100644
index 00000000000..116f0b1c496
Binary files /dev/null and
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/data/1110011011101111011010118622b246be93248675bfdaadfa.lance
differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/data/111110100011100011000111e29e464c90b09bf330ca3b02c2.lance
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/data/111110100011100011000111e29e464c90b09bf330ca3b02c2.lance
new file mode 100644
index 00000000000..41bf64a968c
Binary files /dev/null and
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/c325fd55_doris$vs_ivf_pq_f32/data/111110100011100011000111e29e464c90b09bf330ca3b02c2.lance
differ
diff --git
a/regression-test/data/external_table_p0/lance/test_lance_vector_search.out
b/regression-test/data/external_table_p0/lance/test_lance_vector_search.out
index 1f4c38cdff7..0a98ba5b3a7 100644
--- a/regression-test/data/external_table_p0/lance/test_lance_vector_search.out
+++ b/regression-test/data/external_table_p0/lance/test_lance_vector_search.out
@@ -10,39 +10,54 @@ row_id bigint No false \N
-- !indexed_l2_topk --
1 item-0001 0.0
-2 item-0002 30.0
-3 item-0003 120.0
-4 item-0004 270.0
-5 item-0005 480.0
+2 item-0002 16.0
+3 item-0003 64.0
+4 item-0004 144.0
+5 item-0005 256.0
-- !flat_l2_topk --
1 item-0001 0.0
-2 item-0002 30.0
-3 item-0003 120.0
-4 item-0004 270.0
-5 item-0005 480.0
+2 item-0002 16.0
+3 item-0003 64.0
+4 item-0004 144.0
+5 item-0005 256.0
-- !indexed_tail --
1024 item-1024 0.0
-1023 item-1023 30.0
-1022 item-1022 120.0
+1023 item-1023 16.0
+1022 item-1022 64.0
-- !indexed_offset --
-2 item-0002 30.0
-3 item-0003 120.0
+2 item-0002 16.0
+3 item-0003 64.0
-- !indexed_pre_search_filter --
-2 odd 30.0
-4 odd 270.0
-6 odd 750.0
+2 odd 16.0
+4 odd 144.0
+6 odd 400.0
-- !post_search_filter --
-2 odd 30.0
-
--- !pruned_vector_column --
-1 item-0001 0.0
-2 item-0002 30.0
-3 item-0003 120.0
-4 item-0004 270.0
-5 item-0005 480.0
+2 odd 16.0
+
+-- !boundary_single_probe --
+256 item-0256 0.0
+255 item-0255 16.0
+257 item-0257 16.0
+254 item-0254 64.0
+258 item-0258 64.0
+253 item-0253 144.0
+259 item-0259 144.0
+252 item-0252 256.0
+251 item-0251 400.0
+
+-- !boundary_flat --
+256 item-0256 0.0
+255 item-0255 16.0
+257 item-0257 16.0
+254 item-0254 64.0
+258 item-0258 64.0
+253 item-0253 144.0
+259 item-0259 144.0
+252 item-0252 256.0
+260 item-0260 256.0
diff --git
a/regression-test/suites/external_table_p0/lance/test_lance_vector_search.groovy
b/regression-test/suites/external_table_p0/lance/test_lance_vector_search.groovy
index 5858c0d4a04..bde006738be 100644
---
a/regression-test/suites/external_table_p0/lance/test_lance_vector_search.groovy
+++
b/regression-test/suites/external_table_p0/lance/test_lance_vector_search.groovy
@@ -59,6 +59,19 @@ suite("test_lance_vector_search", "p0,external") {
* Doris after Lance returns Top-K and can therefore reduce the final
result below top_k.
* The current implementation pins one Lance dataset version and searches
its entire snapshot
* with one scanner. Multi-scanner search plus global Top-K merging
remains future work.
+ *
+ * Fixture: doris.vs_ivf_pq_f32 is generated offline by
+ *
docker/thirdparties/docker-compose/iceberg/scripts/lance_build_preinstalled_catalog.py.
+ * It holds 1024 rows in two fragments; embedding[j] = (row_id - 1) + j
with dimension 16,
+ * covered by a real IVF_PQ index (4 partitions, 4-bit PQ) whose creation
is verified by the
+ * generator self-check. For a query equal to the vector of row r, the
exact squared L2
+ * distance of row n is 16 * (n - r)^2.
+ *
+ * IVF_PQ is lossy: raw PQ distances are approximations, so every indexed
query here uses
+ * refine_factor to rerank candidates with exact distances. The agreement
between indexed and
+ * flat results below is an observed property of this frozen fixture and
the pinned Lance
+ * version, not an IVF_PQ algorithm guarantee. The remaining algorithms
(IVF_FLAT, IVF_SQ,
+ * IVF_HNSW_*) and the other vector element types are follow-up work for
#66495.
*/
String enabled = context.config.otherConfigs.get("enableIcebergTest")
if (enabled == null || !enabled.equalsIgnoreCase("true")) {
@@ -69,13 +82,21 @@ suite("test_lance_vector_search", "p0,external") {
String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
String minioPort = context.config.otherConfigs.get("iceberg_minio_port")
String catalogName = "test_lance_vector_search"
- String tableName = "${catalogName}.doris.vector_search"
- String indexedTopFive = """vector_search("table"="${tableName}",
"column"="embedding", "query_vector"="[0,0,0,0]", "top_k"="5", "metric"="l2",
"nprobes"="4", "refine_factor"="10", "use_index"="true")"""
- String flatTopFive = """vector_search("table"="${tableName}",
"column"="embedding", "query_vector"="[0,0,0,0]", "top_k"="5", "metric"="l2",
"use_index"="false")"""
- String indexedTopTwo = """vector_search("table"="${tableName}",
"column"="embedding", "query_vector"="[0,0,0,0]", "top_k"="2", "metric"="l2",
"nprobes"="4", "refine_factor"="10", "use_index"="true")"""
- String indexedOffset = """vector_search("table"="${tableName}",
"column"="embedding", "query_vector"="[0,0,0,0]", "top_k"="2", "offset"="1",
"metric"="l2", "nprobes"="4", "refine_factor"="10", "use_index"="true")"""
- String indexedPrefilter = """vector_search("table"="${tableName}",
"column"="embedding", "query_vector"="[0,0,0,0]", "top_k"="3",
"filter"="category = 'odd'", "metric"="l2", "nprobes"="4",
"refine_factor"="10", "use_index"="true")"""
- String indexedTail = """vector_search("table"="${tableName}",
"column"="embedding", "query_vector"="[1023,2046,3069,4092]", "top_k"="3",
"metric"="l2", "nprobes"="4", "refine_factor"="10", "use_index"="true")"""
+ String tableName = "${catalogName}.doris.vs_ivf_pq_f32"
+ // headQuery is exactly row 1's vector and tailQuery is row 1024's, so
distances are the
+ // deterministic ladder 0, 16, 64, 144, ... with no ties.
+ String headQuery = "[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15]"
+ String tailQuery =
"[1023,1024,1025,1026,1027,1028,1029,1030,1031,1032,1033,1034,1035,1036,1037,1038]"
+ // boundaryQuery is row 256's vector. On this frozen index row 256 lies
next to an IVF
+ // partition edge, so part of its true neighbourhood sits in an adjacent
partition
+ // (pinned by the generator self-check); see the discriminator block below.
+ String boundaryQuery =
"[255,256,257,258,259,260,261,262,263,264,265,266,267,268,269,270]"
+ String indexedTopFive = """vector_search("table"="${tableName}",
"column"="embedding", "query_vector"="${headQuery}", "top_k"="5",
"metric"="l2", "nprobes"="4", "refine_factor"="10", "use_index"="true")"""
+ String flatTopFive = """vector_search("table"="${tableName}",
"column"="embedding", "query_vector"="${headQuery}", "top_k"="5",
"metric"="l2", "use_index"="false")"""
+ String indexedTopTwo = """vector_search("table"="${tableName}",
"column"="embedding", "query_vector"="${headQuery}", "top_k"="2",
"metric"="l2", "nprobes"="4", "refine_factor"="10", "use_index"="true")"""
+ String indexedOffset = """vector_search("table"="${tableName}",
"column"="embedding", "query_vector"="${headQuery}", "top_k"="2", "offset"="1",
"metric"="l2", "nprobes"="4", "refine_factor"="10", "use_index"="true")"""
+ String indexedPrefilter = """vector_search("table"="${tableName}",
"column"="embedding", "query_vector"="${headQuery}", "top_k"="3",
"filter"="category = 'odd'", "metric"="l2", "nprobes"="4",
"refine_factor"="10", "use_index"="true")"""
+ String indexedTail = """vector_search("table"="${tableName}",
"column"="embedding", "query_vector"="${tailQuery}", "top_k"="3",
"metric"="l2", "nprobes"="4", "refine_factor"="10", "use_index"="true")"""
sql """DROP CATALOG IF EXISTS `${catalogName}`"""
try {
@@ -93,12 +114,16 @@ suite("test_lance_vector_search", "p0,external") {
"""
sql """SET enable_file_scanner_v2 = true"""
- order_qt_vector_table_desc """DESC
`${catalogName}`.`doris`.`vector_search`"""
+ order_qt_vector_table_desc """DESC
`${catalogName}`.`doris`.`vs_ivf_pq_f32`"""
qt_vector_table_rows """
SELECT count(*), count(DISTINCT row_id), min(row_id), max(row_id)
- FROM `${catalogName}`.`doris`.`vector_search`
+ FROM `${catalogName}`.`doris`.`vs_ivf_pq_f32`
"""
+ // EXPLAIN asserts the logical search parameters Doris sends to the
backend. It does not
+ // prove which physical Lance index served the query; that proof lives
in the fixture
+ // generator's plan self-check plus the nprobes=1 discriminator below,
until lance-c
+ // exposes the selected index at runtime.
explain {
sql("""SELECT row_id, label, _distance FROM ${indexedTopFive}
ORDER BY _distance, row_id""")
contains "externalSearchType=VECTOR"
@@ -107,19 +132,19 @@ suite("test_lance_vector_search", "p0,external") {
contains "lanceOffset=0"
contains "lanceMetric=l2"
contains "lanceSearchScanners=1"
- notContains "[0,0,0,0]"
+ // The raw query vector must not be echoed into the plan output.
+ notContains "[0,1,2,3"
}
- // The fixture has an IVF_PQ index over embedding. nprobes covers all
- // four IVF partitions, and refine_factor reranks physical candidates.
+ // nprobes=4 covers all four IVF partitions and refine_factor reranks
candidates with
+ // exact distances. On this frozen fixture the result equals the flat
search.
qt_indexed_l2_topk """
SELECT row_id, label, _distance
FROM ${indexedTopFive}
ORDER BY _distance, row_id
"""
- // Disable the index explicitly. Indexed and flat search must agree for
- // these deterministic nearest rows.
+ // Disable the index explicitly for the exact flat baseline.
qt_flat_l2_topk """
SELECT row_id, label, _distance
FROM ${flatTopFive}
@@ -142,16 +167,16 @@ suite("test_lance_vector_search", "p0,external") {
ORDER BY _distance, row_id
"""
- // The TVF filter is evaluated before Lance chooses Top-K. The nearest
- // eligible rows are row_id 2, 4 and 6.
+ // The TVF filter is evaluated before Lance chooses Top-K.
Odd-category rows are the
+ // even row ids, so the nearest eligible rows are row_id 2, 4 and 6.
qt_indexed_pre_search_filter """
SELECT row_id, category, _distance
FROM ${indexedPrefilter}
ORDER BY _distance, row_id
"""
- // An outer WHERE remains a Doris post-search predicate. Search first
- // selects row_id 1 and 2; filtering for odd retains only row_id 2.
+ // An outer WHERE remains a Doris post-search predicate. Search first
selects row_id 1
+ // and 2; filtering for odd retains only row_id 2.
qt_post_search_filter """
SELECT row_id, category, _distance
FROM ${indexedTopTwo}
@@ -159,20 +184,60 @@ suite("test_lance_vector_search", "p0,external") {
ORDER BY _distance, row_id
"""
- // Lance can search embedding even when Doris does not project it.
- qt_pruned_vector_column """
+ // Silent-fallback discriminator. Row 256's true nearest neighbours
straddle an IVF
+ // partition edge, so a genuine single-partition probe must miss the
ones on the far
+ // side and differ from flat search even after exact reranking. A
pipeline that ignores use_index/nprobes and silently scans
+ // flat fails this assertion: on an unindexed table Lance ignores
nprobes and
+ // returns exactly the flat rows.
+ // top_k is 9, not 10: distances here come in symmetric pairs (rows
256-d and
+ // 256+d tie), and 9 is the last cut that lands on a complete pair. At
10 the tie
+ // group at distance 400 (rows 251 and 261) is split and only one of
them fits, so
+ // the golden would pin an arbitrary choice and could flip on any
change to Lance's
+ // top-k selection.
+ def boundarySingleProbe = sql """
+ SELECT row_id, _distance
+ FROM vector_search("table"="${tableName}", "column"="embedding",
"query_vector"="${boundaryQuery}", "top_k"="9", "metric"="l2", "nprobes"="1",
"refine_factor"="10", "use_index"="true")
+ ORDER BY _distance, row_id
+ """
+ def boundaryFlat = sql """
+ SELECT row_id, _distance
+ FROM vector_search("table"="${tableName}", "column"="embedding",
"query_vector"="${boundaryQuery}", "top_k"="9", "metric"="l2",
"use_index"="false")
+ ORDER BY _distance, row_id
+ """
+ assertEquals(9, boundarySingleProbe.size())
+ assertEquals(9, boundaryFlat.size())
+ // Compare distance sequences rather than row ids: the boundary query
is symmetric, so
+ // rows r-d and r+d tie at the same distance and either may fill the
last slot. Only a
+ // genuinely missed neighbour changes the distances.
+ def singleProbeDistances = boundarySingleProbe.collect { it[1] }
+ def flatDistances = boundaryFlat.collect { it[1] }
+ assertFalse(singleProbeDistances.equals(flatDistances),
+ "nprobes=1 produced the same distance sequence as the flat
search, so the "
+ + "single-partition restriction had no effect: the IVF_PQ
index was not used "
+ + "(silent flat fallback or ignored nprobes). "
+ + "nprobes=1 distances=" + singleProbeDistances + " flat=" +
flatDistances)
+
+ qt_boundary_single_probe """
SELECT row_id, label, _distance
- FROM ${indexedTopFive}
+ FROM vector_search("table"="${tableName}", "column"="embedding",
"query_vector"="${boundaryQuery}", "top_k"="9", "metric"="l2", "nprobes"="1",
"refine_factor"="10", "use_index"="true")
+ ORDER BY _distance, row_id
+ """
+ qt_boundary_flat """
+ SELECT row_id, label, _distance
+ FROM vector_search("table"="${tableName}", "column"="embedding",
"query_vector"="${boundaryQuery}", "top_k"="9", "metric"="l2",
"use_index"="false")
ORDER BY _distance, row_id
"""
+ // Note on vector-column pruning: no query in this suite projects
embedding, so Lance
+ // searching an unprojected vector column is exercised by every block
above.
+
test {
sql("""SELECT row_id FROM vector_search("table"="${tableName}",
"column"="embedding", "query_vector"="[0,0,0]", "top_k"="1")""")
exception "dimension"
}
test {
- sql("""SELECT row_id FROM vector_search("table"="${tableName}",
"column"="embedding", "query_vector"="[0,0,0,0]", "top_k"="0")""")
+ sql("""SELECT row_id FROM vector_search("table"="${tableName}",
"column"="embedding", "query_vector"="${headQuery}", "top_k"="0")""")
exception "top_k"
}
} finally {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
