voonhous opened a new pull request, #20034: URL: https://github.com/apache/hudi/pull/20034
### Describe the issue this Pull Request addresses **Do not merge.** Test-only draft that makes CI show the current Spark 4.1 behaviour of release-1.2.1 before #19783 is backported. Expected to go red in `test-spark-java17-scala-dml-tests` (spark4.1), step "Scala UT - Common & Spark". release-1.2.1 carries #18674 but not #19783 (master `c59987a024cd`), so its reader context overlays and projects the PushVariantIntoScan struct at the top level only, and its projector has no null guard. Spark 4.1 ships `spark.sql.variant.pushVariantIntoScan=true`. ### Summary and Changelog Adds `TestVariantPushVariantIntoScan`, the pushVariantIntoScan legs of master's `TestVariantShreddingMixedLayouts` (#19688, #19783) ported over unshredded data (this branch has no shredding writer). It sweeps table type x conf arm x record type on a top-level `v variant` and a nested `s struct<inner: variant>`, with an insert and an update that nulls one row's variant. Both arms expect the same rows; every leg runs to the end and prints `LEG PASS` / `LEG FAIL`, with the conf-off arm first so all verdicts precede a crash. Local run on `ddd37c05521a`, `-Dspark4.1`, JDK 17: | Leg | Result | |---|---| | COW, top-level and nested, both arms, both record types | pass | | MOR, conf off, top-level and nested, both record types | pass | | MOR, conf on, top-level, AVRO and SPARK | `where v is null` returns no row for the nulled variant | | MOR, conf on, nested, AVRO and SPARK | JVM SIGBUS (`BUS_ADRALN` in `StubRoutines::forward_copy_longs`) | What to look for in the job log: two `LEG FAIL toplevel:mor:true:*` lines, then `# SIGBUS` on the first `nested:mor:true` leg. ### Impact None. Test only; the suite is gated to Spark 4.1+ and will ship with the backport. ### Risk Level none ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
