wombatu-kun commented on code in PR #19834:
URL: https://github.com/apache/hudi/pull/19834#discussion_r3955951818
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -318,6 +318,18 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath:
String,
partitionSchema.fields.foreach(f => exclusionFields.add(f.name))
val requestedStructType = StructType(readRequiredSchema.fields ++
partitionSchema.fields.filter(f => mandatoryFields.contains(f.name) &&
!isNestedPartitionField(f.name)))
val requestedSchema = HoodieSchemaUtils.pruneDataSchema(schema,
HoodieSchemaConversionUtils.convertStructTypeToHoodieSchema(requestedStructType,
sanitizedTableName), exclusionFields)
+ // pruneDataSchema keeps a union (a member0..memberN struct on the Spark
side), a BLOB and a VARIANT
+ // whole, so where Spark's nested schema pruning asked for only some of
their inner fields the reader
+ // emits a wider struct than requestedStructType declares. Bind the output
projection to the emitted
+ // shape for those columns so it can drop the extra inner fields by name;
everywhere else the two
+ // agree and the projection stays the pass-through it is today.
+ val readerStructType =
HoodieSchemaConversionUtils.convertHoodieSchemaToStructType(requestedSchema)
+ val projectionInputSchema = StructType(requestedStructType.fields.map { f
=>
Review Comment:
`canBeUnion` has no depth guard, so a projection whose columns are all
nullable and named `memberN` makes the whole top-level requested schema a
union, `pruneDataSchemaInternal` hands the reader the entire table schema, and
`SELECT member0 FROM <mor table>` comes back with `_hoodie_commit_time`.
`projectionInputSchema` only widens the types of columns it already lists, so
keying it off `readerStructType`'s fields instead of `requestedStructType`'s
would drop the extra top-level ones the way it already drops extra nested ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
