cloud-fan commented on a change in pull request #35147:
URL: https://github.com/apache/spark/pull/35147#discussion_r783950279
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SchemaPruning.scala
##########
@@ -158,18 +180,28 @@ object SchemaPruning extends Rule[LogicalPlan] {
/**
* Builds a pruned logical relation from the output of the output relation
and the schema of the
- * pruned base relation.
+ * pruned base relation
*/
private def buildPrunedRelation(
outputRelation: LogicalRelation,
- prunedBaseRelation: HadoopFsRelation) = {
+ prunedBaseRelation: HadoopFsRelation,
+ prunedMetadataSchema: StructType) = {
val prunedOutput = getPrunedOutput(outputRelation.output,
prunedBaseRelation.schema)
Review comment:
actually, this can be even simpler. We only care about the final schema
here, as `getPrunedOutput` is basically generating new attributes from the
final schema and storing expr IDs from `relation.output`
```
val prunedOutput = getPrunedOutput(outputRelation.output, finalSchema)
outputRelation.copy(relation = prunedBaseRelation, output = prunedOutput)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
