XiaoHongbo-Hope opened a new pull request, #8930: URL: https://github.com/apache/paimon/pull/8930
### Purpose Building a generic global index (incl. Lumina vector) on a table with `deletion-vectors.enabled = true` currently fails fast in `GenericGlobalIndexBuilder.scan()`: > Generic global index does not support tables with deletion vectors enabled ... which may cause deleted rows to be indexed. That guard predates the read-side deletion-vector (DV) handling and is now overly conservative. Vector search already removes DV-deleted rows at query time via the live-row pre-filter (`GlobalIndexLiveRowFilter`, reading the *current* snapshot's DVs) and intersecting it into the index hits (`AbstractDataEvolutionVectorRead.preFilters`). So the index can safely be a **superset** of physical rows: deleted rows are filtered on read, and the index never needs rebuilding when DVs change. Normal data-evolution compaction preserves row-ids, and the one case that renumbers them (materialized deletions) already drops the affected indexes via `DataEvolutionCompactGlobalIndexDropper`. ### Changes - Add an opt-in capability `GlobalIndexerFactory.supportsDeletionVectors()` (default `false`), mirroring the existing `supportsFullTextSearch()` pattern. - Enable it only for the **Lumina** vector index (`LuminaVectorGlobalIndexerFactory`; legacy `lumina-vector-ann` inherits it), whose read path is DV-safe. - Gate the build-time DV guard on this capability: `!deletionVectorsEnabled() || supportsDeletionVectors()`. All other generic index types (e.g. eslib) remain rejected. - The build still reads physical rows **without** applying DVs, so global row-ids stay dense (no holes) — no change to the build read path or the `DataSplit` (which intentionally carries only `dataFiles`, not `deletionFiles`). ### Scope Deliberately Lumina-only. The `supportsDeletionVectors()` hook lets other vector / full-text index types opt in later, each with their own regression coverage. ### Tests Follow-up: E2E cases (pre-existing DV before build; new DV after build without rebuild; consistency after DV materialize/compaction) will be added. Feedback welcome on placement given the Lumina IT is gated on the native library. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
