rangareddy commented on issue #17108:
URL: https://github.com/apache/hudi/issues/17108#issuecomment-5391602442
This issue was reviewed as part of the JIRA-migrated backlog triage
(HUDI-9631).
**Findings: this is implemented on `master`. The change that delivered it
carried no JIRA key, which is why this ticket stayed open.**
Delivered by PR #19013, `feat(spark): Support data skipping based on
partitioned RLI` (`ac179e48a0c1`, merged 2026-07-15).
The wiring is complete, not partial. `HoodieFileIndex.scala:123` makes
`RecordLevelIndexSupport.create(...)` the first entry of `indicesSupport`, and
that factory dispatches on whether the RLI is partitioned:
```scala
// RecordLevelIndexSupport.scala:167
def create(spark: SparkSession, metadataConfig: HoodieMetadataConfig,
metaClient: HoodieTableMetaClient): RecordLevelIndexSupport = {
val isPartitioned =
metaClient.getIndexForMetadataPartition(PARTITION_NAME_RECORD_INDEX)
.map[Boolean](indexDef => HoodieRecordIndex.isPartitioned(indexDef))
.orElse(false)
if (isPartitioned) {
new PartitionedRecordLevelIndexSupport(spark, metadataConfig, metaClient)
} else {
new GlobalRecordLevelIndexSupport(spark, metadataConfig, metaClient)
}
}
```
And `PartitionedRecordLevelIndexSupport.lookupCandidateFilesForRecordKeys`
is a real implementation rather than a placeholder - it performs
partition-scoped metadata-table lookups, applies a `MAX_PARTITIONS` threshold,
and falls back with `Option.empty` in the cases where pruning cannot be
determined, so it under-prunes rather than over-prunes.
Test coverage landed with it: `TestRecordLevelIndexWithSQL.scala` was added,
alongside changes to `RecordLevelIndexTestBase.scala` and
`TestGlobalRecordLevelIndexWithSQL.scala`, covering partitioned-RLI creation,
data skipping, partition filters and the threshold fallback.
**One clarification, since it is easy to reach the wrong conclusion here.**
PR #13603 is merged, but it is not the fix for this ticket - it is HUDI-9630,
the writer side, and its own description is what created this one:
> HUDI-9631 ticket created to support pruning on the reader. Right now this
only is implemented for the writer to use this as an index.
Separately, #17116 (HUDI-9648) remains open and is *not* a blocker for this:
it asks for `readPartitionedRecordIndexLocation` APIs over new `RawKey` types
that still do not exist. That is distinct work.
Closing as fixed. Worth resolving JIRA HUDI-9631 to match.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
