voonhous opened a new pull request, #20351:
URL: https://github.com/apache/hudi/pull/20351
### Describe the issue this Pull Request addresses
Closes #20349
A COW upsert on the default AVRO record type fails with `AvroTypeException:
Invalid default for field v: null not a {record ...}` when a base file stores a
shredded variant column as REQUIRED and the table schema has it nullable. Spark
4.1 produces such files with default configs: a DataFrame insert of a non-null
`parse_json` column, or a METADATA_ONLY bootstrap of a vanilla Spark parquet
file. Once it happens, no later upsert into that file group succeeds.
### Summary and Changelog
- `VariantSchemaUtils.swapShreddedVariantsAt`: at a shredded position the
swapped-in schema stays nullable when the base side is nullable.
`toShreddedReadSchema` therefore keeps the requested nullability, and reading a
REQUIRED column as nullable is a safe widening. A nullable side is never
narrowed.
- Tests: unit (`TestVariantSchemaUtils`), parquet read through
`HoodieAvroParquetReader` (`TestHoodieVariantReconstructionRoundTrip`), and
both user routes end to end (`TestVariantShreddingMixedLayouts`). Each fails
without the fix.
<details>
<summary>Details</summary>
**Cause.** `toShreddedReadSchema` swaps each requested variant for its
on-disk shredded form. At a shredded position it took the file's schema
wholesale, nullability included, and `swapShreddedVariantFields` rebuilt the
requested field around it with `withSchema`, keeping the requested `null`
default on a non-null record.
**Both directions.** The rule is keyed on the base side, so it also covers
`alignShreddedVariants` (base = file, used by `HoodieMergeHelper`). There, a
nullable file column (null default) under a non-null requested variant hit the
same invalid default; it now stays nullable. The opposite mismatch in
`toShreddedReadSchema` (nullable file, non-null request) is unchanged: the
file's nullable group is kept, since a REQUIRED projection of an OPTIONAL
column would not read.
**Depth.** The rule applies at every position the swap walks: top-level
columns, record members, array elements and map values.
**Tests.**
- `TestVariantSchemaUtils`: requested nullability kept at top level, record
member, array element and map value; opposite mismatch kept nullable; Avro
round trip; `alignShreddedVariants` with a nullable file column.
- `TestHoodieVariantReconstructionRoundTrip`: a REQUIRED shredded `v`
written by `AvroParquetWriter`, read at a nullable unshredded request; typed,
partly residual, scalar and variant-null rows rebuild.
- `TestVariantShreddingMixedLayouts`: the DataFrame route on both record
types, and a COW upsert added to the METADATA_ONLY bootstrap test.
</details>
### Impact
Fixes the failing upsert. No change for files whose variant nullability
already matches the table schema.
### Risk Level
low: only shredded positions whose nullability differs between the file and
the request are affected. At record fields those failed outright; a REQUIRED
array element or map value is now read at the requested nullable type instead
of the file's.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
