voonhous opened a new pull request, #20142: URL: https://github.com/apache/hudi/pull/20142
### Describe the issue this Pull Request addresses Validation of Spark 4.1+ `PushVariantIntoScan` on master, schema-evolution item 5 of the checklist on #18285 (https://github.com/apache/hudi/issues/18285#issuecomment-5869224297): rename the variant column under schema-on-read, then read it with pushdown on. Finding: with pushdown on the read fails through the schema-on-read guard (`validateNoShreddedVariants`), which names the renamed column, on COW from the base-file reader and on MOR through the file group reader. The checklist expected a Spark codegen failure on this arm; that failure is on the other arm: with pushdown off the read dies in the generated `UnsafeProjection` with a `ClassCastException` from `SpecificInternalRow` to `VariantVal`. Both arms are loud, shredding plays no part, and there are no silent nulls under schema-on-read. Every failure on the way, on schema-on-read and schema-on-write, and what unblocks it, is tracked in #20141; real support is #18285. The DDL itself is refused under default configs (#19766), so the test sets `hoodie.datasource.write.schema.allow.auto.evolution.column.drop=true` around the DDL only. ### Summary and Changelog Test only. `TestVariantShreddingMixedLayouts` gets "Rename of the variant column under schema-on-read fails fast on both pushVariantIntoScan arms": COW shredded, COW unshredded and MOR shredded with a log block. Pins the DDL as its own instant with no data file rewritten and the old name still in the footer, the catalog column `w variant`, the guard message on `select id, w` and `variant_get(w, ...)`, the cast failure with pushdown off, and that a projection without the variant still reads on both arms. <details> <summary>Observed while probing, filed as #20139 and #20140</summary> - `count(*)` on a schema-on-read variant table fails in the vectorized reader once an internal schema is committed, add-column included: the empty-projection carve-out skips the guard but hands the reader the unpruned merged schema with the variant as a struct, and Spark's `ParquetToSparkSchemaConverter` refuses it (#20139). It surfaces as an NPE from `HoodieVectorizedParquetRecordReader.close`, which runs before `initBatch` created the map it iterates and masks the real exception. - `HoodieFileGroupReaderBasedFileFormat.buildReaderWithPartitionValues` writes the per-query batch decision into the session conf `spark.sql.parquet.enableVectorizedReader`, so one variant query turns vectorization off for the rest of the session (#20140). The existing `count(*)` pin in "Schema-on-read reads of shredded variant files fail fast" passes only because of that. This test does not pin `count(*)`. </details> Result: green on master as-is. No production change. ### Impact None. ### Risk Level none ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
