linliu-code opened a new pull request, #663:
URL: https://github.com/apache/hudi-rs/pull/663
**Stacked on #639–#662** — review only the last commit.
## The bug
A delete block whose ordering value is anything but a small integer fails to
read:
```
Union index 1490 out of bounds: 2
Cannot convert i64 to usize: -26
```
"out of bounds: **2**" is the tell — the only 2-branch unions are
`recordKey` and `partitionPath`. So the decoder is reading a *record key's*
union index and getting 1490. The byte stream is misaligned; the branch wasn't
chosen wrongly.
**This is a pre-existing hudi-rs defect, not a port regression.** It fails
on the current reader too, through `Table::read`.
## Cause
`orderingVal` is declared here as a union of primitives. Hudi writes a union
of per-type **wrapper records**, and inserted `BooleanWrapper` at position 1 —
so every position from `int` onward names a different type:
| Position | This crate | Hudi (writer) |
|---|---|---|
| 1 | int | BooleanWrapper |
| 2 | long | IntWrapper |
| **3** | **float** | **LongWrapper** |
| 4 | double | FloatWrapper |
| … | … | … |
## Evidence
The delete block in `table_delete_ord_long`, 26 bytes, is exactly
self-consistent under Hudi's schema:
```
04 | 02 02 34 | 02 00 | 06 c0 3e | 02 02 33 | 02 00 | 06 f0 2e | 00
```
- `04` → 2 records
- keys `"4"` and `"3"`, empty partitions, ordering position **3**, varints
**4000** and **3000**
Decoded here, position 3 is a `float` — 4 fixed bytes — so `c0 3e 02 02` is
swallowed and `33` is read as the next record's key union index: zigzag(0x33) =
**−26**, the reported error, byte for byte.
## What changed
1. **The schema** — Hudi's wrapper union, for decoding.
2. **The narrowing template** — position *i* now holds the primitive that
wrapper *i* carries, so the existing "keep `[null, i]`" logic still yields a
primitive Arrow column and the output type is unchanged.
3. **Unwrapping** — `Union(i, Record[("value", v)])` → `Union(1, v)`,
applied **after** narrowing, since narrowing reads the position Hudi wrote and
unwrapping rewrites it.
4. **A cast** — the wrapper is chosen for the *value*, not the column, so a
long ordering column can carry an `IntWrapper` for a small value. The delete
batch's ordering column is cast to the type the data schema declares. The old
schema hid this by calling position 2 a long regardless.
5. **`ArrayWrapper` is rejected** in both places that read the position,
rather than mapped to something that disagrees with its value.
## A correction to an earlier claim
I previously reported that both shapes exist in the wild and could not be
distinguished, after an attempt to swap the schema broke `v6_trips_8i3d` and
`v8_trips_8i3u1d`. That was wrong. Both use position **2**, which is `long`
here and `IntWrapper{int}` there — and a wrapper's field is a bare primitive,
so **both decode the same bytes to the same number**. Those fixtures never
distinguished the schemas. They broke on the Arrow *type* of the result, which
is what item 4 addresses.
All 16 fixtures with delete blocks are consistent with the wrapper schema;
none requires the primitive form.
## Older tables
Not supported, deliberately. No fixture uses the primitive layout, and the
two cannot be told apart from the bytes: where they differ, decoding usually
fails (this bug), and at position 2 both succeed with the same value. **If you
know of tables written with the older layout, please say so** — the alternative
is a decode-and-retry fallback, which I did not take.
## Tests
11 previously-ignored cases un-ignored and passing: ordering by long,
decimal, timestamp, plus multi-log, watermark and instant-range cases that were
blocked behind the same decode.
The schema's own unit tests asserted the old index space; they now assert
the wire order, which is what a delete block's index actually refers to. Two
new tests cover unwrapping and the `ArrayWrapper` rejection.
Full workspace green: 1177 lib + 79 table-read + 39 datafusion + 21 + 12.
Ignored drops 19 → 8.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
