cshuo opened a new pull request, #19013: URL: https://github.com/apache/hudi/pull/19013
### Describe the issue this Pull Request addresses Spark data skipping based on record level index previously only supported global RLI lookup semantics. When the record level index is partitioned, Spark could not use the partitioned RLI metadata layout to narrow candidate files from record-key predicates. ### Summary and Changelog - Add Spark data-skipping support for partitioned record level index. - Split global and partitioned RLI pruning into dedicated `RecordLevelIndexSupport` implementations. - Route RLI pruning based on the metadata record-index definition, using `HoodieRecordIndex.isPartitioned`. - Add partitioned RLI lookup by grouping `(partition, recordKey)` pairs against the corresponding metadata file groups. - Extract `PartitionedRecordIndexFileGroupLookupFunction` so it can be reused by Spark metadata index lookup and Spark datasource pruning. - Add functional SQL coverage for partitioned RLI pruning, including the max-partitions threshold behavior. ### Impact This improves Spark query pruning when partitioned RLI is enabled and a query contains record-key filters, optionally combined with partition filters. For partitioned RLI, pruning is skipped when too many candidate partitions remain, avoiding expensive metadata fan-out. Global RLI behavior remains covered by the existing test path. ### Risk Level Medium. The change touches Spark datasource file-index pruning and metadata-table lookup paths, but the implementation is scoped to record-level-index pruning and includes dedicated functional coverage. The targeted `TestRecordLevelIndexWithSQL#testPartitionedRliPartitionsThreshold` test passed locally. ### Documentation Update No documentation update is required. This change enables Spark pruning behavior for an existing metadata index mode and does not add a new public user-facing configuration. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change has been documented - [ ] Change has been tested - [ ] Change has been verified -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
