voonhous opened a new pull request, #20158: URL: https://github.com/apache/hudi/pull/20158
### Describe the issue this Pull Request addresses Spark 4.0 with `spark.sql.variant.pushVariantIntoScan=true` is unsupported (#20032), and #20033 added a guard that fails such a read with an error naming the conf. CI builds no Spark 4.0 profile, so neither the guard nor its test had run. On Spark 4.0.2 the test fails: a query that reads no variant column gets past the guard and dies with `HoodieSchemaException: Illegal initial character: 0`. The guard checks the required schema of the scan. `PushVariantIntoScan` rewrites every variant column of the relation, into a placeholder `struct<0:boolean>` when no expression asks for it, so the rewrite sits in the data schema of every scan of the table. Converting that data schema to a HoodieSchema then trips on the field name `0`. ### Summary and Changelog - `HoodieFileGroupReaderBasedFileFormat.buildReaderWithPartitionValues` validates the data schema instead of the required schema. - `SparkAdapter.validateVariantProjectionReadable`: parameter renamed to `schema`, doc updated. - `TestVariantDataType` "Test Spark 4.0 rejects reads rewritten by pushVariantIntoScan": the query that reads no variant now expects the guard. It expected rows, which it never returned. <details> <summary>Spark 4.0.2, conf on, COW and MOR, AVRO and SPARK record types, top-level and nested variant</summary> | Query | Before | After | |:--|:--|:--| | `variant_get(v, ...)`, `cast(v as string)`, `where v is null` | guard message | guard message | | `select id, ts` | `Illegal initial character: 0` | guard message | | `select count(*)` | not run | guard message | Guard message: `Column 'v' was rewritten by spark.sql.variant.pushVariantIntoScan into a variant projection struct, which Hudi does not support on Spark 4.0.2. Set spark.sql.variant.pushVariantIntoScan=false (the Spark 4.0 default) or read the table with Spark 4.1+.` The test fails before the change and passes with it. `TestBaseSpark4AdapterVariantMethods` passes on 4.0.2 either way. </details> ### Impact Spark 4.0 only, and only with the internal conf turned on: reads that already failed now fail with the message that names the conf. The hook stays a no-op on Spark 3.x, 4.1 and 4.2. ### Risk Level low ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
