rangareddy commented on issue #17391:
URL: https://github.com/apache/hudi/issues/17391#issuecomment-5351243650
This issue was reviewed as part of the JIRA-migrated backlog triage
(HUDI-9128).
**Findings: the metadata table now reads through the file group reader, but
only for key lookups - the capability this design calls for is not there yet.**
The metadata table is read through the file group reader on `master` today.
`hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java`
constructs either `HoodieLsmFileGroupReader` (`:617`) or
`HoodieFileGroupReader` (`:629`) over the file slice, with a
`HoodieAvroReaderContext` carrying the key predicate:
```java
return HoodieFileGroupReader.<IndexedRecord>builder()
.withReaderContext(readerContext)
.withHoodieTableMetaClient(metadataMetaClient)
.withLatestCommitTime(latestMetadataInstantTime)
.withBaseFileOption(fileSlice.getBaseFile())
.withLogFiles(fileSlice.getLogFiles())
...
```
That arrived via **PR #13544**, `[HUDI-9592] Read metadata table through
file group reader for spark datasource` (`21e9a0edc2f3`, merged 2025-07-21) -
not through this sub-task. The stated goal here ("should be able to read an
entire file slice in its entirety") is NOT satisfied: every MDT read on master
is key-driven and readSliceAndFilterByKeysIntoList returns an EmptyIterator
when no keys are supplied, so no full-slice read path exists.
Two things to be aware of:
- No design artifact was ever produced - there is no RFC or doc for the MDT
file group reader in `rfc/`. The design exists only as shipped code.
- Commit `30beae426c72` ("[HUDI-9120] Remove HUDI-9130 code changes relating
to FGR") looks like a revert of the sibling's work but is actually a
reader-config revert, so it should not be read as history for this family. See
#17393 for the detail.
Suggest the assignee write the design down, since the full-slice capability
it specifies still does not exist and #17392 / #17394 both depend on it.
Keeping this open pending that call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
