YannByron opened a new pull request, #11265: URL: https://github.com/apache/paimon/pull/11265
### Purpose Paimon's Spark connector currently advertises only partition columns for runtime filtering. A filtered broadcast join therefore cannot use a non-partition join key's existing scalar index to narrow the probe scan. Expose eligible single-column BTree and Bitmap index keys through `SupportsRuntimeV2Filtering`. Runtime `IN` predicates are combined with existing filters and replan splits at the original snapshot, retaining the reader builder used by Spark's original batch. Index coverage and freshness discovery is deferred until Spark considers runtime filtering, so ordinary scans do not pay that metadata cost. The initial implementation covers latest-snapshot append-only Data Evolution scans with complete, fresh indexes and supported boolean, integer, character, or date keys. Incomplete, stale, or legacy indexes and unsafe scan shapes keep their ordinary behavior. Runtime index work is bounded to 10,000 keys and 1 MiB of literal text per filter; larger filters are ignored. This bound applies after Spark collects the keys. Spark's existing dynamic pruning conditions still apply, including a selective build-side filter and a reusable broadcast exchange under the default configuration. The final join remains responsible for multiplicity and join semantics. This does not change Spark core or the configured scalar-index search mode. ### Tests - `RuntimeScalarIndexFilterTest`: 9 passed. Includes actual broadcast joins for BTree and Bitmap with index evaluation during planning and in readers; asserts reduced index candidate rows as well as duplicate/NULL result equivalence. - Covers fixed snapshots and the original reader factory, static predicates, empty keys, incomplete/stale indexes, writer-only `IGNORE` updates, unsupported scans, string payload limits, and deferred index metadata access using a real FileIO spy. - `RepartitionLargeBlobScanTest` and `PostponeBucketTableTest`: 50 passed. - `SparkPartitionMetadataAuthTest`: 23 passed. - Reactor Spotless and Checkstyle, repository RAT license check, and `git diff --check`: passed. Validated locally with Spark 3.5.9 and Scala 2.12 after packaging the reactor to provide the embedded codegen resources. No performance benchmark is claimed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
