dbtsai commented on issue #23474: [SPARK-26551][SQL] Fix schema pruning error when selecting one complex field and having is not null predicate on another one URL: https://github.com/apache/spark/pull/23474#issuecomment-452427310 If it's really difficult to mark `name.middle` as `contentAccessed = true` in this case (which I feel is a less hacky solution), can we reformat the code with the following with documentation? ```scala !rootFields.exists { root => root.field.name == opt.field.name && { // If the merged field type of root and opt field is different from opt field type, // we will keep it. // For example, when root field type is `struct<name:struct<last:string>>`, // and opt field type is `struct<name:struct<middle:string>>`, the merged field type will be // `struct<name:struct<last:string,middle:string>>`. Since the merged one contains more // nested fields than opt field type, we have to keep it. val rootFieldType = StructType(Array(root.field)) val optFieldType = StructType(Array(opt.field)) val merged = optFieldType.merge(rootFieldType) merged.sameType(optFieldType) } ``` Add @hvanhovell @gatorsmile for more input. Thanks!
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
