JingsongLi opened a new pull request, #9282: URL: https://github.com/apache/paimon/pull/9282
### Purpose Add a bitmap-backed Multivalue global index for accelerating element-membership predicates on `ARRAY` columns without treating the complete array as a scalar key. ### Changes - Add the Core `ARRAY_CONTAINS` predicate, visitor propagation, Java/JSON serialization, and safe unsupported-predicate fallback. - Add the `multivalue` global indexer, bitmap writer/reader, configuration, and service registration. - Preserve null-array semantics with dedicated null/non-null row bitmaps; ignore null elements inside non-null arrays. - Integrate index construction and pushdown with Data Evolution tables. - Integrate Primary Key table configuration, maintenance, scan localization, and read pushdown. - Let unordered indexers bypass PK external sorting and stream records in deterministic source ordinal space. - Add Flink and Spark `create_global_index` construction support and documentation. ### User impact Paimon Core callers can build an `ARRAY_CONTAINS` predicate and use the Multivalue index to prune rows. Flink and Spark can build the index through `create_global_index`; SQL expression translation to the Core predicate remains connector-specific and is not part of this change. ### Validation - Common predicate serialization, visitor, evaluator, and Multivalue bitmap reader/writer tests. - Core Data Evolution Multivalue table tests. - Primary Key builder, payload, maintenance, scan, and read tests, including null arrays, empty arrays, duplicate elements, and multi-file source ordinals. - Flink Multivalue build/refresh integration and topology tests. - Spark `CreateGlobalIndexProcedureTest` suite. - Non-fast Maven compile with Spotless and Checkstyle for the affected Core, Flink 1, and Spark 3 modules. - `git diff --check`. ### Known follow-ups - Each writer currently buffers distinct-element posting lists for its source group; the feature is intended initially for low-to-moderate element cardinality. A spillable or bounded PK posting builder should follow. - Data Evolution index metadata currently does not persist the build-time element type. Element-type schema evolution needs explicit index invalidation/rebuild or compatible-type metadata before old Multivalue indexes can be reused safely. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
