voonhous opened a new pull request, #20161:
URL: https://github.com/apache/hudi/pull/20161
### Describe the issue this Pull Request addresses
Fixes #20157. Spark 4.0 reads a parquet variant group by position as
`[value, metadata]`, while Hudi writes `[metadata, value]` (#18334).
`Spark40HoodieParquetReadSupport` works around that by reordering the group in
the requested schema. Since #18674 the reorder, when handed the catalyst
schema, covers top-level variant columns only.
A variant below a struct, in an array or in a map value therefore keeps the
file order on the row-based path. Once an update sends the read through the
file group reader (MOR) or the Spark-record merge handle (COW), the read fails
with `MALFORMED_VARIANT`, and the COW rewrite persists the swapped bytes for
the rows it carries over.
### Summary and Changelog
- `reorderVariantFields` walks the catalyst schema through structs, arrays
and maps and reorders every group it types as `VariantType`. A plain struct
with the same parquet shape is still left alone, at any depth. Without a
catalyst schema the shape-based reorder is unchanged.
- `TestSpark40HoodieParquetReadSupport`: nested variants are reordered in
every layout; plain structs are not.
- `TestVariantDataType`: variants in a struct, an array and a map read back
after an update, COW and MOR, both record types. Runs on Spark 4.0+.
<details>
<summary>Local runs, Spark 4.0.2 (CI builds no Spark 4.0 profile)</summary>
| State | Unit test | End-to-end test |
|:--|:--|:--|
| without the change | nested reorder test fails | fails,
`MALFORMED_VARIANT` |
| with the change | 8 of 8 pass | passes |
With the change, reads after an update are correct for `struct<inner:
variant>`, `struct<mid: struct<inner: variant>>`, `array<variant>`,
`map<string, variant>` and `array<struct<inner: variant>>`, on COW and MOR with
both record types, and the COW rewrite writes `metadata` and `value` into the
right columns.
`TestVariantDataType` on Spark 4.0.2: 12 pass, 13 skipped by their Spark
version assumptions, 1 fails ("Test Spark 4.0 rejects reads rewritten by
pushVariantIntoScan"), which is the failure #20158 fixes.
</details>
### Impact
Spark 4.0 only. Base files that a COW rewrite already wrote with swapped
bytes are not repaired by this change.
### Risk Level
low
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
