dbtsai commented on issue #23474: [SPARK-26551][SQL] Fix schema pruning error when selecting one complex field and having is not null predicate on another one URL: https://github.com/apache/spark/pull/23474#issuecomment-452064572 When we read `name.first` and check if `name` is not `null`, we are marking `name` as `contentAccessed = false` to avoid reading the entire `name` column. Now, the issue is when we read `name.first` and check if `name.middle` is not `null`, we are still marking `name.middle` as `contentAccessed = false` resulting `java.lang.IllegalArgumentException: middle does not exist.`. To fix the root cause, and avoid the misunderstanding of the meaning of `contentAccessed`, we might mark `contentAccessed = true` in the second case; thus, this change here will not be required. What do you think? Thanks.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
