voonhous opened a new pull request, #20155: URL: https://github.com/apache/hudi/pull/20155
### Describe the issue this Pull Request addresses Validation of Spark 4.1+ `PushVariantIntoScan` on master (#18285). Every sweep of `spark.sql.variant.pushVariantIntoScan` expects the same rows on both arms, so only the plan can prove that the `true` arm really ran with the rewrite and did not pass as a copy of the `false` arm. Most sweeps in `TestVariantShreddingMixedLayouts` already assert that through `variantProjectionPushedIntoScan`. The first one, "variant_get filters and projections resolve per file across mixed layouts", had no plan assertion on its COW and MOR legs, and nothing pinned the ReadSchema of the scan itself. ### Summary and Changelog Test only. - `VariantShreddingTestSupport.scanReadSchema(sql)`: the ReadSchema of the file scan as its explain line prints it (the catalog string of the scan's required schema). - The mixed-layout `variant_get` test pins it per arm on COW and MOR: `struct<id:int,v:struct<0:bigint>>` with the rule on, `struct<id:int,v:variant>` with it off. - The partial-update read in the MOR leg pins the eight-member projection struct, which is what its "seven of eight pushed fields are null" expectation rests on. - `variantProjectionPushedIntoScan` keeps its behaviour and shares the scan lookup with the new helper. <details> <summary>Where the rewrite is already pinned</summary> `variantProjectionPushedIntoScan` is asserted per arm in: time travel and incremental V1/V2, bootstrap, nested `variant_get` (COW, MOR parquet logs, MOR avro blocks), top-level sibling widening, nested sibling widening (COW and MOR), and in #20127 and #20136. The collections test asserts the scan's required schema directly. The array-element test sweeps the conf with nothing to pin: the rule rewrites struct paths only, which the collections test pins. </details> Green on master as-is. No production change. ### Impact None. ### Risk Level none ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
