JingsongLi opened a new pull request, #11306:
URL: https://github.com/apache/paimon/pull/11306
### Purpose
Add optional shard-local Bitmap indexes to Spark centroid vector indexes on
Data Evolution tables. Pass a JSON mapping such as `extra_fields =>
'{"tenant_id":"bitmap","status":"bitmap"}'` to `sys.create_global_index`.
- Write one scalar index file per populated centroid and store each field's
type, byte range, and original Bitmap metadata in `GlobalIndexMeta.indexMeta`.
- Apply supported scalar predicates before ANN ranking in single, batch, and
distributed vector search. Skip ANN shard reads when no visible row matches.
- Keep the router, vector shards, and scalar files together for coverage,
rebuilds, drops, and concurrent update checks.
### Tests
- 298 Java tests across Bitmap, centroid metadata, commit conflict
detection, vector indexing, and vector search.
- 35 Spark tests, including manifest layout, distributed Top-K filtering,
incremental range builds, and index group replacement and deletion.
- 74 PyPaimon tests and 53 subtests covering index build, drop, and update
handling.
- Flink common compilation, repository formatting checks, and `git diff
--check`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]