u70b3 commented on issue #66497:
URL: https://github.com/apache/doris/issues/66497#issuecomment-5199612350
here is my design proposal for this issue. It builds on #65730 (merged), the
Lance Java SDK / lance-c APIs already
pinned in the repo, and two precedent studies: how Doris implements
mutations for other external catalogs (Iceberg/Hive/Paimon), and lance-spark's
index
DDL (the only landed cross-engine precedent). Open questions are numbered
at the end. /cc @Gabriel39 @zhangstar333
## 1. SQL surface
Reuse the existing index DDL grammar (`createIndex` / `dropIndex` /
`showIndex` in `DorisParser.g4`), extended to accept Lance external tables:
```sql
-- Vector index
CREATE INDEX [IF NOT EXISTS] idx ON lance_ctl.db.tbl (vec_col)
USING IVF_PQ PROPERTIES ("nlist" = "256", "metric" = "l2");
-- Scalar indexes
CREATE INDEX idx ON lance_ctl.db.tbl (col) USING BTREE; -- or BITMAP /
INVERTED
-- Replace / rebuild with the same name (atomic swap via Lance
replace=true)
CREATE OR REPLACE INDEX idx ON lance_ctl.db.tbl (vec_col) USING IVF_PQ;
-- Inspect
SHOW INDEX FROM lance_ctl.db.tbl;
DROP INDEX [IF EXISTS] idx ON lance_ctl.db.tbl;
```
Notes:
- `USING`: extend the enum with Lance concrete types. Vector: `IVF_FLAT |
IVF_SQ | IVF_PQ | IVF_HNSW_FLAT | IVF_HNSW_SQ | IVF_HNSW_PQ`; scalar: `BTREE |
BITMAP | INVERTED` (matching lance-c's `LanceVectorIndexType` /
`LanceScalarIndexType` and the #66340 matrix). PROPERTIES map to Lance
`IndexParams`
(`nlist`, `metric`, ...).
- Same-name semantics follow the Lance SDK's safe default: `CREATE INDEX`
fails if the name already exists; `IF NOT EXISTS` no-ops; `CREATE OR REPLACE
INDEX` maps to Lance `replace=true`, an atomic swap in a single commit.
This keeps lance-spark's replace capability in Doris's explicit style. (Q3)
- `SHOW INDEX` output columns for Lance tables: `Name | Column | Type |
UUID | DatasetVersion | IndexedFragments | UnindexedFragments` — covering the
issue's required "names, UUIDs, columns, types, dataset versions" plus
lance-spark-style coverage stats (backed by `listIndexes` /
`lance_dataset_index_list_json`).
- The number of indexes on a table is the row count of `SHOW INDEX`
(backed by `listIndexes().size()` / `lance_dataset_index_count`); no separate
statement is proposed.
- Index names are case-insensitive and stored lowercase, following Lance /
lance-spark convention.
- Single-column indexes only in the initial implementation.
## 2. Where index operations execute (Q1)
Precedent study: every mutable external catalog in Doris performs
metadata-level mutations inside the FE via an embedded SDK —
`IcebergMetadataOps`,
`HiveMetadataOps`, `PaimonMetadataOps`, `MaxComputeMetadataOps` all
implement the shared `ExternalMetadataOps` interface; a null `metadataOps` means
read-only (JDBC is the read-only example). BE never touches the external
system's metadata. `LanceExternalCatalog` currently has no metadataOps (=
read-only), so the conventional extension point is FE-side index methods
calling the Lance Java SDK.
**Proposal for the initial implementation — FE via Lance Java SDK (JNI).**
`createIndex` / `dropIndex` / `listIndexes` all exist in the pinned lance-core
9.1.0-beta.3. No thrift / RPC / BE changes; follows the
`ExternalMetadataOps` convention.
One honest caveat: unlike a pure metadata mutation, Lance index creation
bundles data-plane compute (column scan + IVF training) into a single monolithic
Rust call, so with this approach that compute runs in native threads
inside the FE process. Lance does expose the primitives for a distributed build
—
per-fragment segment building without commit, then a coordinator commit
(`create_index_uncommitted` / `commitExistingIndexSegments`; lance-spark's
`num_segments` mode uses this). A follow-up could offload building to BE
(the pinned lance-c v0.1.2 already ships the matching Phase-2 C APIs) while FE
keeps initiation and commit, mirroring the Iceberg INSERT pattern. The
initial implementation stays FE-only behind a `LanceIndexOperator` interface so
the later swap does not change SQL semantics.
## 3. Synchronous execution (Q2)
Proposal: synchronous DDL for the initial implementation. Precedents line
up: Doris's internal light `CREATE INDEX` is a synchronous metadata change; the
async `BUILD INDEX` framework (`IndexChangeJob` per partition) is bound to
Doris's tablet / replica / BE-agent model and does not transfer to Lance;
lance-spark's `CREATE INDEX` is synchronous as well.
Semantics: killing the statement before commit leaves nothing persisted
(Lance commits atomically at the end), so interruption is safe. Documentation
will note that build time scales with dataset size. Follow-up candidates:
an async job with progress reporting (the Lance SDK exposes
`IndexBuildProgress`), and/or the REST namespace's native async model.
## 4. Authorization
- `CREATE / CREATE OR REPLACE / DROP INDEX` → `ALTER_PRIV` on the target
table. Matches both precedents: internal CREATE/DROP/BUILD INDEX share a single
ALTER check (`AlterTableCommand.validate`), and external-table ALTER
operations (Iceberg branch/tag, add/drop column) check `ALTER_PRIV`.
- `SHOW INDEX` → `SHOW_PRIV` on the table, matching internal SHOW INDEX.
Stated as design rather than an open question since both precedents agree
— reviewers please flag if you disagree.
## 5. Lifecycle semantics (the "define semantics" scope item)
These follow Lance's transaction semantics directly (see
`docs/src/format/table/transaction.md` in lance-format/lance); Doris does not
invent its own:
- **Versioning**: every CREATE / OR REPLACE / DROP is one Lance
transaction producing a new dataset version with an atomic metadata switch. A
failed or
interrupted build commits nothing — no partial index is ever visible.
- **Concurrency**: in-flight queries are unaffected (#65730 pins the
dataset version at planning time). Two concurrent creations with the same name:
last
committer wins. Index builds are compatible with concurrent appends (new
fragments are simply uncovered); they conflict with overwrite/restore, surfaced
as a retryable error.
- **Failure recovery**: FE crash after commit but before responding → the
index exists; the user verifies via `SHOW INDEX`. `IF NOT EXISTS` / `OR
REPLACE` make retries idempotent. `DROP` on a missing name errors unless
`IF EXISTS`.
- **DROP semantics**: DROP only removes the index from the manifest;
physical index files are reclaimed later by Lance cleanup / VACUUM, so time
travel
is preserved.
- **Metadata refresh**: after a successful mutation, FE invalidates the
table's cached metadata through the existing `RefreshManager` /
`ExtMetaCacheMgr`
mechanism (same as Iceberg DDL). `SHOW INDEX` always reads live from the
dataset at its latest version, never from the catalog cache.
## 6. Scope of the initial implementation
- Vector: `IVF_PQ` and `IVF_FLAT` verified end-to-end (what the #65730
query path consumes); other types accepted syntactically but gated to what tests
cover.
- Scalar: `BTREE`, `BITMAP`, `INVERTED`.
- Catalogs: filesystem (local / file / s3). **REST Namespace: reject with
a clear error.** The REST spec does define v1 index endpoints, but they are
asynchronous (transactionId + status polling), optional for servers to
implement, and have no replace field — a poor fit for the synchronous,
atomic-replace semantics proposed here. (Q4)
- Tests: FE UTs (parse / analyze / privilege / catalog API), regression
tests (create → show → or-replace → drop → failure paths, vector + scalar), docs
PR in apache/doris-website.
## 7. Implementation approach (orientation; details in the PR)
- Reuse the existing command chain rather than adding Lance-specific
statements: CREATE/DROP INDEX already parse into `AlterTableCommand` + ops; SHOW
INDEX into `ShowIndexCommand`. The change is to admit `LanceExternalTable`
at the existing gates (external-table op whitelist, `Alter` dispatch switch,
`ShowIndexCommand` table-type branch) — the same pattern Iceberg
branch/tag support used.
- Catalog side: add index methods (`createIndex` / `dropIndex` /
`listIndexes`) to `LanceExternalCatalog`, calling the Lance Java SDK at the
dataset's
latest version.
- Estimated size: ~2.5k lines including FE UTs and regression tests.
## 8. Coverage of the issue checklist
- **Scope**: create vector / scalar indexes → §1; show the number of
indexes → row count of `SHOW INDEX` (§1); show names / UUIDs / columns / types /
dataset versions → `SHOW INDEX` columns (§1); replace or rebuild with the
same name → `CREATE OR REPLACE INDEX` (§1, Q3); drop by name → `DROP INDEX`
(§1); authorization / concurrency / versioning / failure recovery /
metadata refresh semantics → §4–5.
- **Completion criteria**: FE statements and metadata APIs → §7;
service-side index operations integrated with Lance → §2; unit and regression
tests →
§6; user documentation → §6; verified for vector and scalar index types →
§6.
## Questions for reviewers
- **Q1**: FE-only (Java SDK) for the initial implementation, with the
BE-distributed fragment build as an explicit follow-up path — OK?
- **Q2**: Synchronous DDL initially (no SHOW/CANCEL BUILD INDEX
integration for Lance) — OK?
- **Q3**: Is extending the grammar with `CREATE OR REPLACE INDEX`
acceptable? (Alternative: `PROPERTIES("replace"="true")` — avoids a grammar
change but
hides the semantics.)
- **Q4**: REST Namespace index ops rejected with a clear error initially —
OK?
## References
- lance-spark index DDL:
`docs/src/operations/ddl/{create-index,drop-index,show-indexes}.md` in
lance-format/lance-spark
- Lance transaction & CreateIndex compatibility semantics:
`docs/src/format/table/transaction.md` in lance-format/lance
- lance-namespace REST index endpoints: `IndexApi.md` in
lance-format/lance-namespace
- Doris external-catalog mutation precedent: `IcebergMetadataOps` /
`ExternalMetadataOps`; internal index DDL chain: `AlterTableCommand` /
`ShowIndexCommand` / `BuildIndexOp`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
