Takeshi Yamamuro created SPARK-33035:
----------------------------------------
Summary: Updates the obsoleted entries of attribute mapping in
QueryPlan#transformUpWithNewOutput
Key: SPARK-33035
URL: https://issues.apache.org/jira/browse/SPARK-33035
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.0.1, 3.1.0
Reporter: Takeshi Yamamuro
This ticket aims at fixing corner-case bugs in the
`QueryPlan#transformUpWithNewOutput` that is used to propagate updated
`ExprId`s in a bottom-up way. Let's say we have a rule to simply assign new
`ExprId`s in a projection list like this;
{code}
case class TestRule extends Rule[LogicalPlan] {
override def apply(plan: LogicalPlan): LogicalPlan =
plan.transformUpWithNewOutput {
case p @ Project(projList, _) =>
val newPlan = p.copy(projectList = projList.map { _.transform {
// Assigns a new `ExprId` for references
case a: AttributeReference => Alias(a, a.name)()
}}.asInstanceOf[Seq[NamedExpression]])
val attrMapping = p.output.zip(newPlan.output)
newPlan -> attrMapping
}
}
{code}
Then, this rule is applied into a plan below;
{code}
(3) Project [a#5, b#6]
+- (2) Project [a#5, b#6]
+- (1) Project [a#5, b#6]
+- LocalRelation <empty>, [a#5, b#6]
{code}
In the first transformation, the rule assigns new `ExprId`s in `(1) Project`
(e.g., a#5 AS a#7, b#6 AS b#8). In the second transformation, the rule corrects
the input references of `(2) Project` first by using attribute mapping given
from `(1) Project` (a#5->a#7 and b#6->b#8) and then assigns new `ExprId`s
(e.g., a#7 AS a#9, b#8 AS b#10). But, in the third transformation, the rule
fails because it tries to correct the references of `(3) Project` by using
incorrect attribute mapping (a#7->a#9 and b#8->b#10) even though the correct
one is a#5->a#9 and b#6->b#10.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
