dbtsai commented on a change in pull request #23474: [SPARK-26551][SQL] Fix
schema pruning error when selecting one complex field and having is not null
predicate on another one
URL: https://github.com/apache/spark/pull/23474#discussion_r245781612
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala
##########
@@ -119,7 +119,12 @@ private[sql] object ParquetSchemaPruning extends
Rule[LogicalPlan] {
.distinct.partition(_.contentAccessed)
optRootFields.filter { opt =>
- !rootFields.exists(_.field.name == opt.field.name)
+ !rootFields.exists { root =>
+ val rootFieldType = StructType(Array(root.field))
+ val optFieldType = StructType(Array(opt.field))
+ val merged = optFieldType.merge(rootFieldType)
+ root.field.name == opt.field.name && merged.sameType(optFieldType)
+ }
Review comment:
When we read `name.first` and check if `name` is not `null`, we are marking
`name` as `contentAccessed = false` to avoid reading the entire `name` column.
Now, the issue is when we read `name.first` and check if `name.middle` is
not `null`, we are still marking `name.middle` as `contentAccessed = false`
resulting `java.lang.IllegalArgumentException: middle does not exist.`.
To fix the root cause, and avoid the misunderstanding of the meaning of
`contentAccessed`, we might mark `contentAccessed = true` in the second case;
thus, this change here will not be required.
What do you think?
Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
