deepakpanda93 opened a new pull request, #19730: URL: https://github.com/apache/hudi/pull/19730
### Describe the issue this Pull Request addresses Addresses #16138 (JIRA [HUDI-6616](https://issues.apache.org/jira/browse/HUDI-6616)). HUDI-5760 replaced Kryo with Avro as the serde for delete log blocks, but the tech spec was never updated. The **Delete Block (Id: 1)** section still told readers: > Tombstone of the record to encode a delete. **The following 3 fields are serialized using the KryoSerializer.** That has not been true since HUDI-5760 landed. The issue also asks (@vinothchandar's comment) for positional headers, block uuid headers and "other changes". Those have since been documented independently: `RECORD_POSITIONS` (5), `BLOCK_IDENTIFIER` (6), `IS_PARTIAL` (7), `COMPACTED_BLOCK_TIMES` (4) and `BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS` (8) are all present in the **Headers** table today. I checked each rather than assuming, so this PR is scoped to what actually remains: the block content tables. ### Summary and Changelog One file, `website/learn/tech-specs.md` (+32/−5). `learn/` is an unversioned Docusaurus plugin, so there is a single copy and no versioned duplicates. **1. Delete Block: the payload encoding is versioned, not simply Kryo.** `HoodieDeleteBlock` dispatches on the log block version, and older blocks stay readable, so the section now documents all three encodings rather than only the current one: | Block version | Encoding | Tombstone contents | |---|---|---| | 1 | Kryo-serialized `HoodieKey[]` | Record key and partition path only. **No ordering value.** | | 2 | Kryo-serialized `DeleteRecord[]` | Record key, partition path, ordering value | | 3 | Avro, binary-encoded `HoodieDeleteRecordList` | Record key, partition path, **typed** ordering value | Version 3 is what current writers emit, since `HoodieLogBlock.version` is `3`. The v1 row matters in practice: a reader encountering a v1 block gets no ordering value at all, which the old text gave no way to anticipate. **2. The version 3 record fields**, from `HoodieDeleteRecordList.avsc`: `recordKey` and `partitionPath` as nullable strings, and `orderingVal` as a union of typed wrappers (`BooleanWrapper`, `IntWrapper`, `LongWrapper`, `FloatWrapper`, `DoubleWrapper`, `BytesWrapper`, `StringWrapper`, `DateWrapper`, `DecimalWrapper`, `TimeMicrosWrapper`, `TimestampMicrosWrapper`, `ArrayWrapper`). That typed ordering value is the substantive gain over Kryo, alongside the block no longer needing a JVM with matching Kryo registrations to read. The schema's own doc string confirms the pairing: *"A list of delete records stored in the delete block in log block version 3"*. <!-- SNAPSHOT 1 HERE: rendered Delete Block (Id: 1) section --> **3. Byte widths corrected in two tables.** Both the Delete Block `length` field and the Avro Block `record length` field were documented as **8** bytes. `HoodieDeleteBlock#getContentBytes` and `HoodieAvroDataBlock#serializeRecords` both write them with `output.writeInt`, so both are **4**: ```java // HoodieDeleteBlock#getContentBytes output.writeInt(version); byte[] bytesToWrite = (version <= 2) ? serializeV2() : serializeV3(); output.writeInt(bytesToWrite.length); output.write(bytesToWrite); ``` Worth being explicit that this is not a confusion with the outer log block, which genuinely *does* have an 8-byte `content length` field. These tables describe the bytes **inside** the block content. I also checked that `HoodieAvroDataBlock` has two serialization paths and that the 8-byte reading does not come from the other one: the `@Deprecated getBytes(Schema)` method is a legacy writer that compresses and writes the schema, and is not what current writers use. **4. `format version` renamed to `block version`** in both tables. The value written is `HoodieLogBlock.version`, currently `3`, which is the log **block** version. The log **file** format version is a separate concept that the Versioning section of this same page documents as currently `1`, so labelling this row "version of the log file format" conflated the two. After this change no block table row says "format version"; the remaining occurrences on the page are unrelated prose about `hoodie.table.version` and the Versioning section. <!-- SNAPSHOT 2 HERE: rendered Avro Block (Id: 3) section --> ### Scope note The Avro Block correction is slightly wider than issue #16138, which is about the delete block. I included it for two reasons: it is the same class of error verified the same way, and renaming the Delete Block's `format version` row would otherwise have left the two adjacent tables describing the identical field under different names. Happy to split it out if reviewers would rather keep this strictly to the delete block. ### Verification Every claim was read out of `release-1.2.0` source rather than inferred: `HoodieDeleteBlock`, `HoodieDeleteRecordList.avsc`, `HoodieLogBlock` (for `version = 3`), `SerializationUtils` (confirming v1/v2 are Kryo-backed via `KryoSerializerInstance`), `DeleteRecord`, and `HoodieAvroDataBlock` (both the current `serializeRecords` and the deprecated `getBytes`). `npm run build` passes with the warning block **byte-identical** to a baseline built from the same base commit (`5971a1ac3ba3`), 13,265 lines each. The branch is rebased on that head, so the workflow's changed-file check sees only the one `website/` file. Rendering confirmed under `npm run serve`: both tables render with the corrected widths, the Kryo sentence is gone, all twelve wrapper types appear, and the new `RECORD_POSITIONS` cross-reference resolves to `#headers`. One limitation worth stating: this is verified by reading the format code, not by decoding a delete block off disk at each version. A committer who can confirm the v1/v2/v3 dispatch against real log files would strengthen it, particularly the claim that v1 blocks carry no ordering value. ### Impact Documentation only. No code, config, or behaviour change. It does correct statements that are currently wrong about both the delete block encoding and two field widths. ### Risk Level none ### Documentation Update This PR is the documentation update — the tech spec, https://hudi.apache.org/learn/tech-specs. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable cc @vinothchandar (who raised the original list on the issue), @yihua -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
