voonhous opened a new pull request, #20036:
URL: https://github.com/apache/hudi/pull/20036
### Describe the issue this Pull Request addresses
Closes #20035
PushVariantIntoScan rewrites struct paths only, so a variant that is an
array element or a map value reaches the reader as native VariantType and the
reader context leaves it alone (#19783). That proves the rule skipped those
columns, not that the value coming back through Hudi's readers is right.
### Summary and Changelog
Test only. One new test in `TestVariantShreddingMixedLayouts` over `v
variant`, `arr array<variant>`, `m map<string, variant>` and `items
array<struct<inner: variant>>`:
- Sweeps COW/MOR x `spark.sql.variant.pushVariantIntoScan` on/off x
AVRO/SPARK record types (8 legs), plus MOR avro data blocks on table version 9
(2 legs).
- Inserts five rows, then updates ids 3 and 4 through the log; id 4 gets a
null map value, a null struct member and, on the SPARK legs, a null array
element.
- Checks `variant_get`, casts, filters served off log and base rows,
`posexplode` and `explode`, and whole-collection reads compared as VariantVal
JSON.
- Asserts in the plan that the array element, the map value and the struct
member are native VariantType on both arms, while the top-level `v` is a
projection struct only with the conf on.
The null array element runs on the SPARK legs only: the Avro write path
forces parquet-avro's old list structure, which cannot write a null array
element of any type ("Array contains a null element at 1"). Not
variant-specific.
### Impact
None. Test only, gated to Spark 4.1+.
### Risk Level
none. Run locally on master `7430134280ae` with `-Dspark4.1` (Spark 4.1.1):
10 of 10 legs pass, scalastyle clean. CI exercises it on spark4.2.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
