voonhous opened a new pull request, #20033: URL: https://github.com/apache/hudi/pull/20033
### Describe the issue this Pull Request addresses Closes #20032 Spark 4.0 ships `spark.sql.variant.pushVariantIntoScan` off (it is on from 4.1, SPARK-54454). Once a user sets it, Spark 4.0 rewrites a VARIANT column into the same projection struct 4.1 produces, but 4.0's readers cannot evaluate it and Hudi does not align log records to it. Nothing on 4.0 recognised the struct, so the read fell through into schema-change handling instead of failing. ### Summary and Changelog Spark 4.0 with the conf on is unsupported. The read now fails on the driver, before any task runs, naming the column, the conf and the fix. - `SparkAdapter`: new no-op hook `validateVariantProjectionReadable`; `isVariantProjectionStruct` doc corrected (the rewrite exists on every 4.x, 4.1 only changed the default). - `BaseSpark4Adapter`: `isVariantProjectionStruct` moves here from `Spark4_1Adapter`, so 4.0 recognises the shape too. - `Spark4_0Adapter`: overrides the hook to throw `HoodieNotSupportedException`. - `HoodieFileGroupReaderBasedFileFormat`: calls the hook first thing in `buildReaderWithPartitionValues`. - Tests: adapter unit test (throws on 4.0, passes on 4.1+, covers root and struct-nested projections); functional test on COW and MOR gated to Spark 4.0, with the flag-on query that does not touch the variant column as the positive control. ### Impact Spark 4.0 only, and only when the internal conf is set by hand. Default reads on every version are unchanged. Spark 4.1+ behaviour is unchanged: the moved override returns the same value it did before. ### Risk Level low. CI builds and tests only the spark3.5 and spark4.2 profiles, so it compiles the shared pieces and runs the adapter unit test on 4.2 but never exercises `Spark4_0Adapter` or the 4.0-gated tests; those need a local `-Dspark4.0` run (see the last commit's hand-off). ### Documentation Update none. The conf is Spark's, internal, and the error message carries the guidance. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
