rangareddy opened a new pull request, #19721: URL: https://github.com/apache/hudi/pull/19721
### Describe the issue this Pull Request addresses Closes #17030 (HUDI-9454), which asks for the details missing from `hudi-io/hfile_format.md` without naming them. I compared the document against `hudi-io`'s reader and writer and found four gaps, one of which is a factual error. ### Summary and Changelog | Gap | Fix | | --- | --- | | **Block magics incomplete.** The list named 4 of the 12 types in `HFileBlockType`, and `TRABLK"$` was used in the Trailer section without appearing in the list. | Split into what Hudi writes and what its reader additionally accepts (so it can read HBase-written files): the leaf/intermediate index magics, HBase's three bloom block types, `DATABLKE` and the v1 index block. | | **Checksum Type values missing.** Described only as "type of checksum used". | `ChecksumType` defines `NULL` = 0, `CRC32` = 1, `CRC32C` = 2. | | **Multi-level data block index undocumented.** `num_data_index_levels` is listed among the trailer fields, but nothing said what more than one level means, though `HFileLeafIndexBlock` / `HFileIntermediateIndexBlock` exist and `HFileReaderImpl` walks the levels. | New section. Also records that Hudi's writer always emits a single root level and sets `num_data_index_levels` to `1`, so the multi-level path exists only for reading HBase files. | | **File Info keys wrong and incomplete.** Two entries listed, one incorrect. | The doc gave `hfile.MAX_MEMSTORE_TS_KEY`, but `HFileInfo` declares `MAX_MEMSTORE_TS_KEY` and `KEY_VALUE_VERSION` **without** the reserved `hfile.` prefix that `LASTKEY`, `CREATE_TIME_TS`, `AVG_KEY_LEN` and `AVG_VALUE_LEN` carry. Corrected, added the four missing keys, and called the inconsistency out. | Also names the meta block key Hudi stores its bloom filter under, `bloomFilter`. The document mentioned bloom filters in the section overview but never said how to locate one, and it is easy to confuse with HBase's own bloom blocks, which Hudi never writes. ### Verification Documentation only, no code change. Every claim was read out of `hudi-io` on master rather than from memory: - `HFileBlockType` for the 12 magics - `ChecksumType` for the three checksum values - `HFileInfo` for the key names and which ones carry the `hfile.` prefix - `HFileReaderImpl` for the level walk, and `HFileWriterImpl` for `setNumDataIndexLevels(1)` - `HoodieHFile*` reader for the `bloomFilter` meta block key Markdown checked: all 5 internal links resolve to real headings, code fences balanced. ### Impact None on behaviour. The corrected File Info key names matter to anyone reading the file info programmatically: looking up `hfile.MAX_MEMSTORE_TS_KEY` as previously documented finds nothing. ### Risk Level none ### Documentation Update This PR is the documentation update. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable - [x] CI passes on my PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
