[
https://issues.apache.org/jira/browse/SPARK-61232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-61232:
----------------------------------
Fix Version/s: 4.3.1
> UnionExec whole-stage codegen fails when a child output repeats an ExprId
> -------------------------------------------------------------------------
>
> Key: SPARK-61232
> URL: https://issues.apache.org/jira/browse/SPARK-61232
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0, 4.3.0, 5.0.0, 4.4.0
> Reporter: XiDuo You
> Assignee: XiDuo You
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0, 4.3.1
>
>
> UnionExec's whole-stage codegen path projects each child output column to the
> union's output, but it binds that projection by ExprId through
> BindReferences. When a child output repeats an ExprId (e.g. SELECT p, p, w),
> every reference to that ExprId resolves to the first occurrence, so:
> - the second union output column reads the first column's value (wrong
> results), and
> - the generated code emits that input variable's declaration twice, which
> Janino rejects with "CompileException: Redefinition of local variable
> inputadapter_value_3".
> The repeated-ExprId shape is normally removed by the analyzer's union-child
> deduplication. It reaches a physical UnionExec when
> spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved and
> spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled are disabled,
> or through a path that skips that deduplication.
> The codegen path must read each union output position from its own child
> position, which is what doExecute does when it concatenates child rows. The
> fix binds the per-child projection to BoundReference(i) instead of the
> child's Attribute and drops the BindReferences call in UnionExec.doConsume.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
