dragonknightk opened a new pull request, #1025:
URL: https://github.com/apache/hudi-rs/pull/1025
## Description
Part of #908
`decode_record_index_entry` called `from_avro_datum` for every record, and
that path re-resolves the full `HoodieMetadataRecord` schema, including every
index payload type, on each call. Record-index entries have a fixed layout, so
this change uses direct decode when the container writer schema passes the
layout check, and keeps generic decode as the fallback. The fallback to generic
decode has two triggers: the layout check fails, or direct decode fails. The
layout check is computed once per HFile. On a release build, decoding one
record-index entry drops from about 8.8 us to about 0.45 us (four sample shapes
rotated for 100000 iterations).
The layout check compares field names, order and types to the vendored
`HoodieMetadata.avsc`. A null union branch is one byte regardless of the other
branch. Record-index records leave the other payloads null, so a difference in
the column-stats wrapper union still passes. An extra or reordered top-level
field, or a change to `HoodieRecordIndexInfo` itself, fails the layout check,
because it can change the bytes.
Swapping `fileIdHighBits` and `fileIdLowBits` keeps the same field types, so
direct decode would still succeed but return a file ID with its two halves
swapped. The layout check rejects that schema.
Column-stats encode and decode are left alone, as issue 908 step 1 asked.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
- Direct decode and generic decode agree field by field, including null and
default `HoodieRecordIndexInfo` fields and a Spark-compacted record-index HFile
fixture.
- Empty bytes and a null `recordIndexMetadata` stay deletes.
- The layout check accepts a column-stats-only change and rejects mismatches.
- `decode_record_index_entry` matches generic decode for each of those
schemas, and when direct decode fails after a passing layout check.
- Truncated prefixes and fixed corrupt inputs do not panic, and
`decode_record_index_entry` matches generic decode.
- `apply_hfile_records` matches generic decode on a schema that fails the
layout check.
## AI Disclosure
Human-led contribution. Grok Bot, working for dragonknightk, assisted with
this change, including the code, tests, and description. All submitted content
was reviewed by a human before submission.
(๑•̀ㅂ•́)و✧
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]