YannByron opened a new pull request, #11265:
URL: https://github.com/apache/paimon/pull/11265

   ### Purpose
   
   Paimon's Spark connector currently advertises only partition columns for 
runtime filtering. A filtered broadcast join therefore cannot use a 
non-partition join key's existing scalar index to narrow the probe scan.
   
   Expose eligible single-column BTree and Bitmap index keys through 
`SupportsRuntimeV2Filtering`. Runtime `IN` predicates are combined with 
existing filters and replan splits at the original snapshot, retaining the 
reader builder used by Spark's original batch. Index coverage and freshness 
discovery is deferred until Spark considers runtime filtering, so ordinary 
scans do not pay that metadata cost.
   
   The initial implementation covers latest-snapshot append-only Data Evolution 
scans with complete, fresh indexes and supported boolean, integer, character, 
or date keys. Incomplete, stale, or legacy indexes and unsafe scan shapes keep 
their ordinary behavior. Runtime index work is bounded to 10,000 keys and 1 MiB 
of literal text per filter; larger filters are ignored. This bound applies 
after Spark collects the keys.
   
   Spark's existing dynamic pruning conditions still apply, including a 
selective build-side filter and a reusable broadcast exchange under the default 
configuration. The final join remains responsible for multiplicity and join 
semantics. This does not change Spark core or the configured scalar-index 
search mode.
   
   ### Tests
   
   - `RuntimeScalarIndexFilterTest`: 9 passed. Includes actual broadcast joins 
for BTree and Bitmap with index evaluation during planning and in readers; 
asserts reduced index candidate rows as well as duplicate/NULL result 
equivalence.
   - Covers fixed snapshots and the original reader factory, static predicates, 
empty keys, incomplete/stale indexes, writer-only `IGNORE` updates, unsupported 
scans, string payload limits, and deferred index metadata access using a real 
FileIO spy.
   - `RepartitionLargeBlobScanTest` and `PostponeBucketTableTest`: 50 passed.
   - `SparkPartitionMetadataAuthTest`: 23 passed.
   - Reactor Spotless and Checkstyle, repository RAT license check, and `git 
diff --check`: passed.
   
   Validated locally with Spark 3.5.9 and Scala 2.12 after packaging the 
reactor to provide the embedded codegen resources. No performance benchmark is 
claimed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to