Gabriel39 opened a new issue, #68625:
URL: https://github.com/apache/doris/issues/68625
## Background
Follow-up to #66491 and #68453; tracked by the Lance roadmap #66340.
Lance catalog scans support explicit versions, timestamps, tags, and
branches. `vector_search()` and `full_text_search()` currently accept only a
fully qualified table name and resolve the latest version of the main chain
during planning. They pin that version for execution, but callers cannot choose
a historical or branch snapshot.
This prevents reproducible searches against a released tag, comparisons
between historical snapshots, and searches against an experimental branch.
## Scope
- [ ] Support explicit versions, timestamps, tags, and branches in both
search TVFs, for filesystem and REST catalogs, including namespace-managed
versions within the supported catalog read contract.
- [ ] Resolve the selector through the existing `LanceRefSelector` and
catalog snapshot resolver.
- [ ] Bind the output schema, search column and field ID, vector
type/dimension, fragments, and index metadata to the selected snapshot.
- [ ] Keep candidate generation, filtering, Top-K, and two-phase
materialization on the same resolved dataset URI and version.
- [ ] Preserve existing behavior when no selector is supplied: select and
pin the latest main version for that execution.
## Proposed interface
Add optional `version`, `timestamp`, `tag`, and `branch` properties to both
TVFs. Keep `table` as the fully qualified catalog/database/table name.
- `version`: a positive integer; mutually exclusive with `timestamp` and
`tag`.
- `timestamp`: parsed in the session time zone with the same semantics as
catalog time travel; mutually exclusive with `version` and `tag`.
- `tag`: mutually exclusive with all other selectors; the tag determines its
branch and version.
- `branch`: may be used alone or with `version` or `timestamp`.
Proposed syntax, not currently supported:
```sql
SELECT * FROM vector_search(
"table" = "lance_catalog.db.documents",
"version" = "3",
"column" = "embedding",
"query_vector" = "[0.1, 0.2, 0.3]",
"top_k" = "10"
);
SELECT * FROM full_text_search(
"table" = "lance_catalog.db.documents",
"branch" = "dev",
"timestamp" = "2026-09-20 12:00:00",
"column" = "content",
"query" = "database"
);
```
## Execution semantics
- Resolve each search relation's snapshot before binding its schema and
planning index coverage. Do not combine the latest schema or indexes with
historical data.
- Vector search retains its existing flat-scan fallback where applicable,
using only the selected snapshot's data.
- Full-text search requires an appropriate committed FTS index in the
selected snapshot. Apply existing `strict` / `index_only` coverage semantics to
that snapshot; do not borrow a newer index or silently switch versions.
- Missing, expired, or inaccessible snapshots and missing required
historical index files must produce clear errors, with no fallback to latest.
- Resolve moving selectors for each new execution, including prepared
statements and cached plans; retain the resolved snapshot throughout that
execution.
- Expose the resolved version and branch in EXPLAIN.
## Implementation direction
Extend the shared search TVF parameter parser and selector-aware overloads
of `loadMetadataForSearch()` / `loadBasicMetadata()`. Reuse catalog snapshot
resolution and the existing fixed-URI/version BE search path. Audit
index-segment planning and two-phase reads for any reopening of latest.
Determine any necessary BE changes from this audit rather than assuming that
accepting the parameters alone completes support.
## Acceptance criteria
- [ ] Both TVFs support every valid selector combination and reject
conflicts and invalid inputs.
- [ ] Tests cover versions before index creation, partially indexed appends,
deletions, index rebuilds, schema changes, and branches whose version numbers
overlap main while their data and indexes differ.
- [ ] Planning followed by a new commit or tag movement cannot change the
snapshot used by candidate generation or materialization.
- [ ] Prepared statements and plan reuse preserve the documented
per-execution behavior.
- [ ] Filesystem and REST-managed regression tests verify actual rows and
search results, including two-phase reads, not only metadata or EXPLAIN.
- [ ] Documentation covers syntax, selector combinations, historical index
behavior, and cleanup-related errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
