leaves12138 opened a new pull request, #8818:
URL: https://github.com/apache/paimon/pull/8818
## Purpose
Data-evolution append tables can partially update an indexed vector column,
for example by filling an embedding from `NULL` to a value. Incremental
global-index builds previously considered only uncovered row ranges, so an
existing vector index could remain readable but permanently miss those updated
vectors unless users performed a full rebuild.
This change refreshes only affected logical index ranges while keeping the
old index active until its replacement is ready.
## Changes
- Store the data snapshot used to build a vector index in
`GlobalIndexMeta.sourceMeta`, encoded with a `DEIX` magic number and version.
- Select an existing index range for refresh when an active data file:
- intersects the index row range,
- physically contains an indexed field, resolved by stable field ID, and
- has `maxSequenceNumber` newer than the recorded snapshot.
- Refresh legacy vector indexes without this metadata once.
- Merge refreshed ranges with newly uncovered ranges in the Flink and Spark
builders.
- Commit old-index deletion and replacement-index addition atomically.
- Detect concurrent replacements by requiring the expected old index file to
remain active and by preventing an older build generation from replacing a
newer one.
- Preserve the existing behavior for non-data-evolution tables and
non-vector indexes.
## User impact
Users can continuously populate embedding columns and run incremental index
builds. Search continues using the previous complete index during construction,
then switches atomically to the more complete replacement without an
index-unavailable window.
## Tests
- Paimon core targeted tests: 53 passed.
-
`LuminaVectorGlobalIndexITCase#testDataEvolutionUpdateRefreshesIndexAtomically`
on Linux: passed.
- `LuminaVectorIndexTest` on Linux: 10 passed.
- Spotless and `git diff --check`: passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
