rangareddy commented on issue #17428:
URL: https://github.com/apache/hudi/issues/17428#issuecomment-5568707428
Closing this as already done. The ask in the originating comment was
conditional:
> For all the places using varint, let's update the HFile format spec
`hudi-io/hfile_format.md` **if such information is not already in the spec**.
It is already in the spec, for every varint site on both the write and the
read side, and the spec also states the thing that actually catches people out:
there are *two* different varint encodings in an HFile and they are not
interchangeable.
### Hadoop `WritableUtils` VInt, the block-index key length
Written by `HFileIndexBlock.getVarIntBytes`, via `HFileRootIndexBlock:110`
and `HFileMetaIndexBlock:44`; read back at `HFileRootIndexBlock:92-93` through
`IOUtils.decodeVarLongSizeOnDisk` and `IOUtils.readVarLong`.
Documented under **Index Block**, in the Block Index Entry field list:
> **Key Length**: the length of the "Key" part, encoded as a Hadoop
`WritableUtils` variable-length integer (VInt). A value in `[-112, 127]` is a
single byte; otherwise the first byte encodes the number of following
big-endian value bytes and the sign. [...] It is **not** the Protobuf varint
used for the trailer and file info length prefixes (see those sections): the
two are byte-identical for `0..127` but diverge at `>= 128`, so they are not
interchangeable.
The Meta Index shares that entry format, which the same section states ("For
Meta Index, the 'Key Bytes' part is the byte array of the key of the Meta
Block"), so `HFileMetaIndexBlock` is covered by the same description rather
than needing its own.
### Protobuf varint, the length prefixes
Written at `HFileFileInfoBlock:107` and `HFileWriterImpl:203`; read back at
`HFileFileInfoBlock:73` and `HFileTrailer:111`, both via `parseDelimitedFrom`.
Documented in **File Info Block**:
> **File Info**: a small key-value map of metadata serialized in Protobuf,
length-delimited (a Protobuf varint length prefix followed by the `InfoProto`
message, i.e. `writeDelimitedTo` / `parseDelimitedFrom`). This Protobuf varint
is a different encoding from the Hadoop `WritableUtils` VInt used for the
block-index Key Length above.
and again in **Trailer**, with the same cross-reference.
### So
All four write sites and all four read sites are described, each in the
section that owns the bytes, and the VInt-versus-Protobuf distinction is stated
in both directions. I could not find a varint use in
`hudi-io/src/main/java/org/apache/hudi/io/hfile/` that the spec leaves out.
If a specific gap was meant that I have missed, please reopen and point at
it and I will document it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
