u70b3 commented on issue #66497:
URL: https://github.com/apache/doris/issues/66497#issuecomment-5204040291
## Revised Design Proposal v2 — Asynchronous Lance Index Lifecycle
This revision supersedes the previous proposal and incorporates the review
feedback from @Gabriel39.
### Major changes from v1
- Heavy index creation, replacement, and incremental maintenance no longer
run synchronously in FE. Production execution is modeled as a durable
asynchronous job, with FE orchestration and BE or external-service execution.
- The SQL path introduces a neutral parsed index specification. Doris
resolves the target table and catalog before choosing internal-index
serialization or Lance-specific validation and dispatch.
- Vector syntax reuses the Doris category-level surface: **USING ANN** with
**index_type=IVF_PQ**.
- **BUILD INDEX** is added to the Phase-1 lifecycle as the incremental
operation for newly appended or otherwise uncovered fragments.
- **SHOW INDEX** returns only bounded logical summaries. Physical UUIDs,
dataset versions, and fragment membership are exposed through structured,
filterable metadata surfaces.
- REST behavior is capability-based. REST mutations go through Lance
Namespace APIs and never bypass the metadata service with vended credentials.
- The initial type set is narrowed to IVF_PQ, BTREE, and BITMAP. Lance
INVERTED/FTS and IVF_FLAT are deferred until the Doris query path is verified
end to end.
- Index-name resolution, post-commit reconciliation, metadata refresh
retries, and local-filesystem restrictions are defined explicitly.
## 1. Goals and Phase-1 boundary
Phase 1 provides a production-oriented lifecycle for user-visible Lance
indexes:
- create IVF_PQ vector indexes and BTREE/BITMAP scalar indexes;
- inspect logical index definitions, aggregate coverage, and logical index
count;
- inspect physical index segments through a separate metadata surface;
- incrementally index newly appended or uncovered fragments;
- atomically replace/rebuild an index by logical name;
- drop an index by logical name;
- provide durable status, cancellation, concurrency limits,
retry/reconciliation, and metadata refresh;
- support Directory Catalog operations and capability-based REST Namespace
operations.
Lance dataset manifests or the Namespace service remain the authoritative
index metadata. Doris does not persist a second copy of Lance index definitions
in its internal table/index metadata. Doris persists only its own asynchronous
job state and the information needed to reconcile an external operation.
Historical-version tables remain read-only for index mutation. Inspection
uses the selected snapshot; mutation targets only the latest writable table
state.
An FE-side synchronous native build is not part of the production
implementation. If retained temporarily for development, it must be
experimental, disabled by default, and protected by strict dataset-size,
fragment-count, and concurrency limits. It does not satisfy Phase-1 completion.
## 2. SQL surface
### Vector creation
~~~sql
CREATE INDEX [IF NOT EXISTS] idx
ON lance_ctl.db.tbl (vec_col)
USING ANN
PROPERTIES (
"index_type" = "IVF_PQ",
"metric" = "l2",
"num_partitions" = "256",
"num_sub_vectors" = "16"
);
~~~
### Scalar creation
~~~sql
CREATE INDEX idx
ON lance_ctl.db.tbl (scalar_col)
USING BTREE
PROPERTIES (
"zone_size" = "4096"
);
CREATE INDEX idx_bitmap
ON lance_ctl.db.tbl (category_col)
USING BITMAP;
~~~
### Full replacement and incremental maintenance
~~~sql
CREATE OR REPLACE INDEX idx
ON lance_ctl.db.tbl (vec_col)
USING ANN
PROPERTIES (
"index_type" = "IVF_PQ",
"metric" = "l2",
"num_partitions" = "256",
"num_sub_vectors" = "16"
);
BUILD INDEX idx ON lance_ctl.db.tbl;
~~~
**CREATE OR REPLACE INDEX** performs a full rebuild and atomically replaces
the logical index. **BUILD INDEX** indexes only fragments not covered by the
current logical index and maps to the Lance incremental optimization path.
### Inspection, status, cancellation, and drop
~~~sql
SHOW INDEX FROM lance_ctl.db.tbl;
SHOW BUILD INDEX FROM db
WHERE TableName = "tbl";
CANCEL BUILD INDEX ON lance_ctl.db.tbl (job_id);
DROP INDEX [IF EXISTS] idx ON lance_ctl.db.tbl;
~~~
The existing Doris **SHOW BUILD INDEX** and **CANCEL BUILD INDEX** surfaces
are extended to Lance external-index jobs instead of introducing a second
job-control vocabulary.
### Statement semantics
- **USING** is required for Lance tables.
- **IF NOT EXISTS** and **OR REPLACE** are mutually exclusive.
- Plain **CREATE INDEX** fails if the logical name exists.
- **IF NOT EXISTS** verifies the visible logical definition and returns a
reconciled no-op when it already matches. A conflicting definition with the
same name is an error rather than a silent no-op.
- **CREATE OR REPLACE INDEX** performs a full rebuild. It is convergent by
name but not strictly idempotent: a retry can produce new segment UUIDs and a
new dataset version.
- **BUILD INDEX** is incremental and does not change the logical definition.
- Phase 1 supports one indexed column per statement. Composite scalar
indexes are deferred.
- Unknown index types, unknown properties, unsupported column types, and
invalid property values fail during analysis.
- Create, replace, and build return after a durable job is accepted, not
after native index construction completes.
- Drop is a metadata operation. It may complete synchronously when the
selected provider exposes an atomic drop, but it still uses the same
post-commit reconciliation and refresh rules.
## 3. Neutral command model and catalog dispatch
The current internal path must not translate every parsed definition into
persisted **catalog.Index** before the target catalog is known.
The revised analysis flow is:
~~~text
parsed SQL
-> ParsedIndexSpec
-> resolve catalog / database / table
-> select InternalIndexProvider or LanceIndexProvider
-> provider-specific validation
-> internal schema-change operation or Lance external-index job
~~~
**ParsedIndexSpec** contains only SQL-level information:
- logical name;
- target columns;
- category from USING;
- properties;
- IF NOT EXISTS / OR REPLACE;
- operation kind: create, replace, incremental build, or drop.
For an internal OLAP table, the existing provider validates and translates
the specification into Doris **catalog.Index** and the legacy schema-change
path.
For a Lance external table, **LanceIndexProvider** produces a Lance-specific
immutable definition and operation request. Lance types and physical metadata
are never added to Doris internal **IndexType** serialization.
The provider boundary is also used by SHOW, BUILD, DROP, reconciliation, and
capability checks so lifecycle behavior does not drift across separate command
implementations.
## 4. Asynchronous execution model
### Durable job
Create, replace, and incremental build create a persistent **LanceIndexJob**
before heavy work starts. The minimum states are:
~~~text
PENDING
-> PREPARING
-> BUILDING
-> COMMITTING
-> REFRESHING
-> FINISHED
Terminal or recovery states:
FAILED
CANCELLING
CANCELLED
COMMITTED_REFRESH_PENDING
OUTCOME_UNKNOWN
~~~
The persisted job records:
- catalog, table identity, dataset URI or Namespace table ID;
- logical index name and normalized definition;
- operation kind and starting dataset version;
- old segment UUIDs for replace/reconciliation;
- provider and executor identity;
- worker task IDs or remote transaction ID;
- current state, progress, timestamps, and sanitized failure information.
FE failover reconstructs unfinished jobs from the edit log and resumes
polling, reconciliation, commit, or refresh. It does not blindly restart native
training.
### Responsibility split
FE performs:
- SQL analysis and table-level privilege checks;
- durable job creation and state transitions;
- executor selection and concurrency admission;
- optimistic commit/finalization when the selected execution model separates
build from commit;
- outcome reconciliation;
- metadata invalidation, refresh edit log, and retry of post-commit refresh.
BE workers or an external service perform:
- column scans;
- vector training;
- scalar/vector segment construction;
- native CPU, memory, and object-store I/O work;
- periodic progress reporting and cooperative cancellation.
FE metadata locks are not held during native training. Doris serializes
conflicting jobs for the same dataset and logical index name, while Lance
transaction rules remain authoritative for conflicts with external writers.
### Resource and cancellation behavior
- Jobs are admitted through explicit per-catalog and cluster-wide
concurrency limits.
- BE execution is associated with workload-management accounting where the
executor supports it.
- Cancellation stops unscheduled work and signals active workers.
- A production executor must define whether cancellation is cooperative or
hard. Doris must not report **CANCELLED** while native work is still able to
commit.
- If an executor cannot reliably cancel an in-flight native call, the job
remains **CANCELLING** until the operation ends and is reconciled.
- Native exception strings are not parsed to implement automatic conflict
retry. Typed conflict results are used where the provider exposes them.
- Same-name concurrent create/replace conflicts are surfaced as retryable
failures unless a later reconciliation proves that this job's requested
postcondition was committed.
## 5. Provider and catalog behavior
### Directory Catalog
Directory-backed datasets use direct Dataset operations. Heavy builds must
execute outside FE.
For object-store-backed datasets, workers use the catalog credentials and
the dataset URI resolved from the latest table metadata.
A local/file dataset is mutable only when the deployment guarantees that the
path is visible with identical contents to the selected executor and to every
FE that may reconcile or finalize the operation. Otherwise mutation is
rejected. Local filesystem support is intended primarily for single-node
development and tests; it must not silently depend on the current master FE's
local disk.
### REST Namespace Catalog
REST behavior is capability-based:
- **SHOW INDEX** uses Namespace list/stats operations.
- Create, replace, and drop use the Namespace index operations when
supported by the server.
- Managed-versioning and service-side transaction policy are preserved.
- Doris never performs a direct Dataset mutation with vended credentials as
a fallback for an unsupported Namespace mutation.
- An unsupported Namespace operation returns a clear capability error.
- If the Namespace service returns an asynchronous transaction or operation
ID, the Doris job tracks and polls it.
- Physical-segment inspection is exposed only when the Namespace capability
supplies the required metadata; Doris does not bypass the service to obtain it.
## 6. Initial type and property matrix
Only types verified by both creation and the Doris query path are enabled.
### IVF_PQ
- SQL category: **ANN**
- Required property: **index_type=IVF_PQ**
- Column count: exactly one
- Column shape: fixed-size list
- Initial element types: FLOAT16, FLOAT32, UINT8, and INT8; FLOAT64 is
deferred because the pinned lance-c creation surface does not advertise it
- Nullability: the vector field and its elements must be non-null in Phase 1
- Metrics: L2, COSINE, and DOT
- **num_partitions**: optional positive integer; SDK default is used when
absent
- **num_sub_vectors**: optional positive integer; must divide the vector
dimension
- **num_bits**: fixed to 8 in Phase 1
- Advanced centroids, codebooks, sample rate, training iterations, and
HNSW/RQ/SQ properties are not user-visible in Phase 1
- Build/query compatibility is verified against the exact Lance Java SDK and
lance-c versions pinned by Doris
### BTREE
- Column count: exactly one
- Initial column types: integral numeric, floating numeric, DECIMAL, STRING,
DATE, DATETIME, and TIMESTAMPTZ types already supported by the Lance predicate
pushdown path
- Nullable columns are supported only with end-to-end coverage for equality,
range, IS NULL, and IS NOT NULL over indexed and unindexed fragments
- Optional **zone_size** must be a positive integer
- Distributed-only/internal fields such as range identifiers are not exposed
as user properties
### BITMAP
- Column count: exactly one
- Initial column types: BOOLEAN, integral numeric, STRING, and DATE
- Nullable columns are supported with explicit NULL predicate tests
- No user-visible properties in Phase 1
- Distributed shard identifiers remain executor-internal
### Deferred types
- **IVF_FLAT** remains disabled until an index built by the lifecycle path
is consumed by the Doris vector query path in an end-to-end test.
- Lance **INVERTED** is not exposed as Doris INVERTED. It is an FTS/BM25
index with different query and tokenizer semantics. A later design should
expose it as **FTS** together with Doris FTS query support.
- Other Lance scalar and vector types are rejected rather than passed
through.
## 7. Logical and physical metadata surfaces
Lance has a logical index identified by name, columns, type, definition, and
aggregate coverage. It may contain multiple physical segments with independent
UUIDs, dataset versions, and fragment sets.
### SHOW INDEX
The existing 13-column Doris schema is retained. It returns one row per
logical index/column:
- **Key_name**: logical name;
- **Column_name**: indexed column;
- **Index_type**: ANN, BTREE, or BITMAP;
- **Properties**: bounded logical details only, such as Lance physical type,
metric, indexed/unindexed row and fragment counts, and supported index-specific
parameters.
The Properties field never contains a segment array or fragment-ID list.
Logical index count is the number of user-visible logical descriptions,
excluding system indexes. It is not derived from physical manifest entries. No
separate count statement is added.
### Physical metadata
A filterable **lance_index_segments** system table or TVF exposes one row
per physical segment:
- catalog, database, table;
- logical index name;
- segment UUID;
- physical index type and version;
- dataset version;
- indexed row and fragment counts;
- creation timestamp when available.
Fragment membership is exposed through a streaming/filterable
**lance_index_segment_fragments** surface with one row per segment UUID and
fragment ID, instead of embedding an unbounded list in one MySQL result field.
Both surfaces read live metadata from the selected Lance snapshot. They do
not persist a copy in Doris catalog metadata.
## 8. Authorization and identifier semantics
- CREATE, CREATE OR REPLACE, BUILD, DROP, and CANCEL require table-level
ALTER.
- SHOW INDEX, SHOW BUILD INDEX, and physical metadata inspection require
table-level SHOW.
- Remote storage still uses catalog credentials; Doris table privilege
remains the user-facing authorization boundary.
Doris preserves the display case returned by Lance but resolves
user-supplied index names case-insensitively, matching existing Doris/MySQL
identifier behavior.
- New indexes whose names differ only by case are rejected.
- Create, build, replace, and drop first resolve a unique case-insensitive
logical match, then pass the exact physical name to Lance.
- If a pre-existing external dataset contains case-only collisions, SHOW
displays all of them, while mutation by that ambiguous name fails with an
explicit ambiguity error.
- Phase 1 does not claim quoted exact-name resolution because the current
parsed identifier path does not reliably preserve a distinct quoted-name
matching mode.
## 9. Versioning, concurrency, and visibility
- A successful create, replace, build, or drop commits a new table version
atomically.
- Queries already pinned to an older version continue to use that version.
- Mutation of a historical-version table is rejected.
- Ordinary mutation and live inspection target the latest version.
- Concurrent append is compatible with a build; newly appended fragments may
remain uncovered and are reported as unindexed.
- BUILD INDEX later adds coverage for uncovered fragments.
- Overwrite, restore, fragment rewrite, or data replacement follows Lance
transaction-conflict rules.
- Doris does not hold table metadata locks for the lifetime of a build.
- Doris prevents two active jobs for the same dataset and case-insensitive
logical name, but it does not assume it controls external writers.
- Queries over a partially indexed table must combine indexed results with
scans of unindexed fragments. End-to-end tests must prove result equivalence
with index use disabled.
## 10. Failure recovery and metadata refresh
The implementation distinguishes three outcomes.
### Confirmed pre-commit failure
The external manifest or Namespace transaction was not committed. The job
becomes FAILED. Doris does not run post-DDL refresh. Unreferenced artifacts are
reclaimed later by Lance cleanup according to retention policy.
### Confirmed commit
The requested postcondition is visible in the latest snapshot:
- create: a matching logical definition is present;
- replace: the definition matches and the old segment identity has been
replaced;
- incremental build: coverage has advanced or is already complete;
- drop: the logical name is absent.
Doris invalidates and refreshes table metadata and emits the normal
external-DDL refresh edit log. A refresh failure does not turn the job into an
ordinary build failure; the job becomes **COMMITTED_REFRESH_PENDING** and
retries refresh.
### Unknown or post-commit failure
After FE failover, timeout, lost worker response, or an ambiguous commit
result, Doris reopens the latest snapshot or queries the Namespace transaction
and verifies the operation-specific postcondition.
- If the postcondition holds, Doris records the commit, runs refresh, and
finishes the job.
- If it does not hold and the provider proves no commit occurred, the job
fails or is safely retried according to the operation policy.
- If the outcome remains unknowable, the job remains **OUTCOME_UNKNOWN** and
exposes a diagnostic rather than silently rebuilding.
A reconciled **IF NOT EXISTS** or **IF EXISTS** no-op still
invalidates/refreshes Doris metadata when it observes external state that may
be newer than the local cache.
SHOW INDEX is the user-visible reconciliation tool, but it is not a
substitute for repairing Doris metadata caches.
## 11. Test plan
Tests cover:
- parser and to-SQL behavior for ANN/index_type, BTREE, BITMAP, OR REPLACE,
BUILD, SHOW, CANCEL, and DROP;
- internal-table behavior remaining unchanged after neutral command
refactoring;
- ALTER/SHOW privileges;
- the complete type/property matrix and rejection of unknown values;
- duplicate, IF NOT EXISTS, case-only duplicate, ambiguous pre-existing
name, replace, incremental build, and drop semantics;
- durable job state transitions, FE restart/failover recovery, concurrency
limits, cancellation, and retryable conflicts;
- successful commit followed by edit-log/refresh failure and subsequent
reconciliation;
- logical versus physical metadata and multi-segment logical counts;
- append after create, coverage decay, incremental BUILD, and query
correctness across indexed plus unindexed fragments;
- historical-version mutation rejection;
- local/file visibility restrictions;
- REST list/stats, supported mutation, unsupported capability,
managed-versioning, and asynchronous transaction polling;
- end-to-end IVF_PQ, BTREE, and BITMAP query use through the Doris BE reader.
Regression tests use isolated writable datasets and generate expected result
files through the normal regression-test scripts.
## Decisions requested from reviewers
### Q1: First production executor
The pinned lance-c v0.1.2 exposes synchronous create/drop/list/count
operations but does not expose the Java SDK's distributed segment
build/merge/commit or incremental optimize APIs.
Which Phase-1 execution target is acceptable?
1. A durable asynchronous job executed on one BE using the current direct
lance-c build API, with distributed build and incremental BUILD requiring
lance-c extensions before they are enabled.
2. Extend lance-c first with uncommitted segment build, merge/commit,
optimize, progress, and cancellation APIs, then implement distributed BE
execution as part of this issue.
3. Use an external index-build service where available, with Directory
Catalog mutation deferred until a production executor exists.
My preference is option 2 for the complete production lifecycle. If option 1
is acceptable as an intermediate milestone, it must not claim hard cancellation
or incremental maintenance until the required native APIs exist.
### Q2: Job-control surface
Is extending the existing SHOW BUILD INDEX and CANCEL BUILD INDEX commands
preferable to introducing Lance-specific SHOW/CANCEL job commands? Reuse keeps
a common Doris lifecycle vocabulary, but the result schema may need additive
fields for catalog, operation kind, external transaction ID, and reconciliation
state.
### Q3: REST Phase-1 scope
Should Phase 1:
- support REST SHOW plus create/drop whenever the Namespace service
advertises the corresponding capabilities; or
- limit REST to SHOW/list/stats until asynchronous Namespace transaction
behavior is exercised against a real compatible service?
In both cases Doris will not bypass the Namespace service with direct
dataset mutation.
### Q4: Physical metadata surface
Is a pair of filterable surfaces — one row per physical segment and one row
per segment/fragment mapping — acceptable, or should Phase 1 expose only
segment summaries and defer fragment membership?
References:
- Doris existing CREATE/BUILD/SHOW/CANCEL index command paths
- #65730 and #66340
- Lance distributed indexing: https://lance.org/guide/distributed_indexing/
- Lance Spark CREATE INDEX and SHOW INDEXES
- Lance Namespace index operations
- Pinned Lance Java SDK v9.1.0-beta.3 and lance-c v0.1.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
