rangareddy commented on issue #17393:
URL: https://github.com/apache/hudi/issues/17393#issuecomment-5351244132
This issue was reviewed as part of the JIRA-migrated backlog triage
(HUDI-9130).
**Findings: not delivered on `master`. A related change standardised the
metadata-table reader path, but the capability this ticket specifies does not
exist yet.**
The design goal for this family, as stated on the sibling HUDI-9128
(#17391), is a reader that *"should be able to read an entire file slice in its
entirety"*. There is no such API. Every metadata-table read entry point on
`hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java`
is key-driven:
- `getRecordsByKeyPrefixes` (`:230`)
- `readRecordIndexLocations` / `readRecordIndexLocationsWithKeys` (`:342`,
`:364`, `:436`)
- `readSecondaryIndexLocations` / `readSecondaryIndexLocationsWithKeys`
(`:459`, `:481`)
- `readIndexRecordsWithKeys` (`:496`, `:502`)
and the slice reader states the opposite of a full-slice read explicitly:
```java
// If no keys to lookup, we must return early, otherwise, the hfile lookup
will return all records.
if (sortedKeys.isEmpty()) {
return new EmptyIterator<>();
}
```
The predicate comes from `buildPredicate` (`:736`) as `Predicates.in` over
full keys, or a prefix match for secondary index, and the reuse path asserts
`predicate instanceof Predicates.In` (`:603`).
**What has landed is adjacent but different.** PR #13544, `[HUDI-9592] Read
metadata table through file group reader for spark datasource` (`21e9a0edc2f3`,
merged 2025-07-21), routed the *existing key-lookup* metadata-table reader
through `HoodieFileGroupReader` / `HoodieLsmFileGroupReader`
(`HoodieBackedTableMetadata:617`, `:629`) and removed
`HoodieMetadataLogRecordReader.java`. That is real progress - there is now one
reader path for MDT key lookups - but it does not add full-slice reads.
**One trap worth flagging for anyone triaging this later.** Commit
`30beae426c72` is titled *"[HUDI-9120] Remove HUDI-9130 code changes relating
to FGR (#12935)"*, which reads as though this ticket's work was implemented and
then reverted. It was not. The actual diff is seven files:
```
hudi-common/.../config/HoodieReaderConfig.java | 19 -------
hudi-common/.../table/TestHoodieReaderConfig.java | 66 ----------
hudi-hadoop-common/.../HoodieInputFormatUtils.java | 39 +++------
.../table/action/compact/HoodieCompactor.java | 2 +-
.../MultipleSparkJobExecutionStrategy.java | 2 +-
.../main/scala/org/apache/hudi/DefaultSource.scala | 5 +-
.../org/apache/hudi/TestHoodieSparkSqlWriter.scala | 12 ++--
```
That is a reader **config** revert made while enabling the file group reader
by default for table version 6 - not a metadata-table reader implementation.
The commit subject is misleading.
**Remaining work:** the full-file-slice read capability plus its tests. That
is also the prerequisite for the two dependent siblings, HUDI-9129 (#17392,
expression filter support) and HUDI-9131 (#17394, benchmark), both of which are
still open.
Keeping this open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
