Kent Yao created SPARK-57039:
--------------------------------
Summary: Fold InnerJoin with single-row
LocalRelation/OneRowRelation into Project
Key: SPARK-57039
URL: https://issues.apache.org/jira/browse/SPARK-57039
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Kent Yao
h2. Background
Currently {{ConvertToLocalRelation}} (Optimizer.scala) folds {{Project}} /
{{Filter}} / {{Limit}} over {{LocalRelation}}, but does *not* fold
{{InnerJoin(LocalRelation(single-row), other, Inner)}} or
{{InnerJoin(OneRowRelation(), other, Inner)}} — even when the single-row side
has no streaming / Unevaluable hazards and the user gave no join hint.
This leaves a redundant join node that:
* Forces broadcast/shuffle planning for what is logically a constant attach.
* Blocks downstream constant-folding and column pruning of the single-row
side's projected attributes.
* Is structurally identical to what {{DecorrelateInnerQuery.scala:435}} flags
as a TODO: {{// TODO add a more general rule to optimize join with
OneRowRelation}}.
h2. Proposal
Add two complementary rules in {{Optimizer.scala}}:
# *{{ConvertToLocalRelation}} case 5/6* (left/right symmetric arms) — fold
{{Inner Join}} where one side is {{LocalRelation(out, [row], false, _)}}
(single-row, ≥1 column) into {{Project(literals ++ other.output, other)}},
optionally wrapped in {{Filter(cond, ...)}}. Literal values preserve the
original {{ExprId}} via {{Alias(Literal.create(...))(exprId)}}.
# *{{FoldInnerJoinWithOneRowRelation}}* (new independent rule, tree-pattern
{{INNER_LIKE_JOIN}}) — fold {{Inner Join}} where one side is
{{OneRowRelation()}} (single-row, 0 columns) into {{other}} or {{Filter(cond,
other)}}. {{OneRowRelation}} does *not* publish {{LOCAL_RELATION}}
tree-pattern, so a single combined rule using
{{transformWithPruning(_.containsPattern(LOCAL_RELATION))}} would silently miss
it — hence two rules.
Four narrowings (all strictly conservative):
|| Dimension || Accepted || Rejected || Reason ||
| JoinType | {{Inner}} only | LeftOuter / RightOuter / FullOuter / LeftSemi /
LeftAnti / Cross | Outer needs null-pad; Semi/Anti needs existence semantics;
Cross has no condition (already handled). |
| JoinHint | {{JoinHint.NONE}} only | BROADCAST / SHUFFLE_HASH / SHUFFLE_MERGE
/ SHUFFLE_REPLICATE_NL | User hints take priority over plan rewrite. |
| Row count | {{data.length == 1}} | 0 (covered by {{PropagateEmptyRelation}})
/ >1 | Inlining >1 rows as Literals would LoC-explode and require Cartesian
materialization. |
| Condition | {{!cond.exists(hasUnevaluableExpr)}} && {{!other.isStreaming}} |
ScalarSubquery / DynamicPruning / streaming | Unevaluable expressions can't
safely move into Filter; streaming has state-store ordering constraints. |
h2. Test coverage
* New {{FoldInnerJoinWithOneRowRelationSuite}}: 9 tests (OneRow × table
left/right, no-cond, LeftOuter negative, ArrayType / MapType / nested
StructType columns preserved, nested struct-of-array, Unevaluable-condition
negative).
* Existing {{ConvertToLocalRelationSuite}}: +4 tests (T7-T10) including a
strong-assertion ExprId preservation test ({{collectFirst { case _: Join
}.isEmpty && output.length == 4}}).
* All 19 new unit tests verified RED-sensitive (fail under deliberately-broken
rule, pass under correct impl).
* Plan-stability regression: {{TPCDSV1_4 + TPCDSV1_4Stats + TPCDSV2_7 +
TPCDSV2_7Stats + TPCDSModified + TPCDSModifiedStats + TPCH}} = *322 / 322 PASS,
0 plan-diff*.
h2. Backward compatibility
No user-facing change. Query results are bit-identical; only the optimized
logical plan shape changes (one less Join, one more Project/Filter). No new
SQLConf knob — the four narrowings make this strictly safe.
h2. Related work
Trino: {{RemoveRedundantCrossJoin}} + {{EvaluateZeroLimit}}. DuckDB:
{{ConstantPropagation}} over cartesian. Doris: handled in FE constant-folding
pass.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
