cloud-fan commented on a change in pull request #35660:
URL: https://github.com/apache/spark/pull/35660#discussion_r815817246
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
##########
@@ -327,16 +323,7 @@ abstract class QueryPlan[PlanType <: QueryPlan[PlanType]]
val existingAttrMappingSet = transferAttrMapping.map(_._2).toSet
newValidAttrMapping.filterNot { case (_, a) =>
existingAttrMappingSet.contains(a) }
}
- val resultAttrMapping = if (canGetOutput(plan)) {
- // We propagate the attributes mapping to the parent plan node to
update attributes, so
- // the `newAttr` must be part of this plan's output.
Review comment:
I was not asking why table is fine but view hits the bug. I was asking
why we produce an invalid Filter operator. Even with the table case, you can
also see `!Filter (...`, and the `!` indicates invalid plan. This seems like a
feature of DataFrame API and IIUC the rule `ResolveMissingReferences` can fix
the invalid `Filter` operator.
So my suggestion is: `DeduplicateRelations` should only kick in if the
plan's children are all resolved and valid. e.g.
```
val newPlan = renewDuplicatedRelations(mutable.HashSet.empty, plan)._1
if (newPlan.find(p => p.resolved && p.missingInput.nonEmpty).isDefined) {
// Wait for `ResolveMissingReferences` to resolve missing attributes first
newPlan
} else {
newPlan.resolveOperatorsUpWithPruning... {
case p: LogicalPlan if !p.childrenResolved => p
// To resolve duplicate expression IDs for Join.
case j @ Join(left, right, _, _, _) if !j.duplicateResolved =>
j.copy(right = dedupRight(left, right))
// Resolve duplicate output for LateralJoin.
...
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
