yihua commented on code in PR #19123:
URL: https://github.com/apache/hudi/pull/19123#discussion_r3825830025
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -265,9 +271,9 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath:
String,
val exclusionFields = new java.util.HashSet[String]()
exclusionFields.add("op")
partitionSchema.fields.foreach(f => exclusionFields.add(f.name))
- val requestedStructType = StructType(requiredSchema.fields ++
partitionSchema.fields.filter(f => mandatoryFields.contains(f.name)))
+ val requestedStructType = StructType(requiredSchema.fields ++
partitionSchema.fields.filter(f => mandatoryFields.contains(f.name) &&
!isNestedPartitionField(f.name)))
Review Comment:
It depends on how the partition paths are generated. For simple partition
fields, i.e., the partition path is the same as the partition column value,
parsing from the path is OK. For a timestamp-based partition path that
requires a transformation of the partition column, the partition path and value
differ. For partition pruning or attaching partition path meta column, it's
better to get them from the path, and we can handle the timestamp key generator
separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
