voonhous opened a new issue, #20035: URL: https://github.com/apache/hudi/issues/20035
### Task Description **What needs to be done:** Confirm that a variant reached through an array element or a map value is read **correctly**, not merely left unrewritten. Struct paths are covered by #19783; for arrays and maps that commit only says they "stay native" ([SparkAdapter](https://github.com/apache/hudi/blob/7430134280ae5684daee64ece7660a77577bf65f/hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/hudi/SparkAdapter.scala#L519), [BaseSpark4Adapter](https://github.com/apache/hudi/blob/7430134280ae5684daee64ece7660a77577bf65f/hudi-spark-datasource/hudi-spark4-common/src/main/scala/org/apache/spark/sql/adapter/BaseSpark4Adapter.scala#L333)), which proves PushVariantIntoScan skipped them, not that the value coming back through Hudi's readers is right, least of all on a MOR merge beside a top-level column that was rewritten in the same scan. Existing coverage on master: `arr[0]` on MOR with the SPARK record type and parquet logs ([array-element test](https://github.com/apache/hudi/blob/7430134280ae5684daee64ece7660a77577bf65f/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/schema/TestVariantShreddingMixedLayouts.scala#L1229)) and `items[0].inner` on COW. No map-valued variant anywhere, no avro-block leg, no null element or null value. Deliverable: one test in TestVariantShreddingMixedLayouts over `v variant`, `arr array<variant>`, `m map<string, variant>` and `items array<struct<inner: variant>>`, sweeping COW/MOR x pushVariantIntoScan on/off x AVRO/SPARK record types plus MOR avro data blocks on table version 9, with an update through the log that nulls a map value, a struct member and (SPARK legs) an array element. It checks `variant_get`, casts, filters, `explode`/`posexplode`, whole-collection reads, and asserts in the plan that the collections are native on both arms while the top-level `v` is a projection struct only with the conf on. Result so far, master `7430134280ae` on Spark 4.1.1: all 10 legs pass. PR follows. **Why this task is needed:** Closes the review question above with evidence instead of a code read, and pins the collection paths so a later reader change cannot regress them silently. <details> <summary>Out of scope and side finding</summary> - release-1.2.1 is unverified for these shapes; its ported suite covers top-level and struct-nested only, and its MOR conf-on legs already fail on #19783's gap (see the do-not-merge evidence PR against that branch). - The Avro write path forces parquet-avro's old list structure, so writing an array with a null element of ANY type fails on COW and compaction with "Array contains a null element at 1". Not variant-specific; the test keeps the null element on the SPARK legs. </details> ### Task Type Test enhancement ### Related Issues **Parent feature issue:** n/a **Related issues:** #19783, #19775, #20032 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
