cloud-fan commented on a change in pull request #35147:
URL: https://github.com/apache/spark/pull/35147#discussion_r783945149
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SchemaPruning.scala
##########
@@ -158,18 +180,28 @@ object SchemaPruning extends Rule[LogicalPlan] {
/**
* Builds a pruned logical relation from the output of the output relation
and the schema of the
- * pruned base relation.
+ * pruned base relation
*/
private def buildPrunedRelation(
outputRelation: LogicalRelation,
- prunedBaseRelation: HadoopFsRelation) = {
+ prunedBaseRelation: HadoopFsRelation,
+ prunedMetadataSchema: StructType) = {
val prunedOutput = getPrunedOutput(outputRelation.output,
prunedBaseRelation.schema)
Review comment:
how about
```
val prunedOutput = getPrunedOutput(outputRelation.output,
prunedBaseRelation.schema)
val metadataCols = outputRelation.output.collect {
case MetadataAttribute(attr) => attr.withDataType(prunedMetadataSchema)
}
assert(metadataCols.length == 1)
outputRelation.copy(relation = prunedBaseRelation, output = prunedOutput ++
metadataCols)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
