voonhous commented on code in PR #20033:
URL: https://github.com/apache/hudi/pull/20033#discussion_r4089701004
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -291,6 +291,9 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath:
String,
filters: Seq[Filter],
options: Map[String, String],
hadoopConf: Configuration):
PartitionedFile => Iterator[InternalRow] = {
+ // Driver side, once per scan: Spark 4.0 cannot read a PushVariantIntoScan
projection struct and
+ // has to fail here rather than in the schema-change path (#20032).
+ sparkAdapter.validateVariantProjectionReadable(requiredSchema)
Review Comment:
Confirmed, no 4.0 path reaches those consumers ahead of this hook. The
projection struct only enters Hudi through the `requiredSchema` that Spark's
rule hands a `HadoopFsRelation` scan, and every data-table `HadoopFsRelation`
is built by `HoodieHadoopFsRelationFactory` with
`HoodieFileGroupReaderBasedFileFormat`. The only other one
(`BaseFileOnlyRelation.toHadoopFsRelation`, legacy format) is taken only for
the metadata table, which has no VARIANT columns. Of the consumers,
`overlayVariantProjections` and `SparkSchemaTransformUtils` run under the
reader built here, and `HoodieSparkSchemaConverters` is gated on
`gteqSpark4_1`. `supportBatch` runs earlier on the driver, but it only picks
batch vs row before this throws. Hudi-internal reads (compaction, clustering)
build their schema from the table schema, so they never carry the struct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
