JingsongLi opened a new pull request, #8549: URL: https://github.com/apache/paimon/pull/8549
## Summary Introduce the configuration and immutable payload foundation for bucket-local vector indexes on primary-key tables. This PR is intentionally limited to the foundation layer. Manifest maintenance, compaction/write integration, and query routing will follow in stacked PRs. The feature remains disabled by default. ## Changes - Add `pk-vector.*` lifecycle, algorithm, distance, compaction-bound, and refinement options. - Validate that primary-key vector indexes use a `VECTOR<FLOAT>` column, fixed buckets, deletion vectors, and a supported merge engine. - Resolve and canonically hash algorithm-specific index options. - Add row-position-addressable RAW vector sidecars with validation, random access, and exact top-k search for L2, cosine, and inner product. - Add immutable single-source and multi-source ANN payload build/search support through the existing Global Index SPI. - Persist source-file coverage, ordinal layout, vector definition, metric, algorithm, and options fingerprint in segment metadata. - Extend the test vector indexer to support `VECTOR<FLOAT>` fields. ## Testing - [x] `PrimaryKeyVectorIndexOptionsTest` - [x] `PrimaryKeyVectorIndexValidationTest` - [x] `TestVectorGlobalIndexerTest` - [x] `PkVectorSegmentMetaTest` - [x] `RawVectorSidecarTest` - [x] `PkVectorAnnSegmentFileTest` - [x] Clean intermediate commits compiled independently with JDK 8. ## Notes - The feature defaults to `pk-vector.index.state = disabled`. - This is the first PR in a planned stack: foundation, bucket maintainer, write lifecycle, and query path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
