This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 0ea808880e22 [SPARK-48027][SQL][FOLLOWUP] Add comments for the other
code branch
0ea808880e22 is described below
commit 0ea808880e22e2b6cc97a3e946123bec035ade93
Author: beliefer <belie...@163.com>
AuthorDate: Tue May 14 13:26:17 2024 +0800
[SPARK-48027][SQL][FOLLOWUP] Add comments for the other code branch
### What changes were proposed in this pull request?
This PR propose to add comments for the other code branch.
### Why are the changes needed?
https://github.com/apache/spark/pull/46263 missing the comments for the
other code branch.
### Does this PR introduce _any_ user-facing change?
'No'.
### How was this patch tested?
N/A
### Was this patch authored or co-authored using generative AI tooling?
'No'.
Closes #46536 from beliefer/SPARK-48027_followup.
Authored-by: beliefer <belie...@163.com>
Signed-off-by: Wenchen Fan <wenc...@databricks.com>
---
.../catalyst/optimizer/InjectRuntimeFilter.scala | 21 ++++++++++++---------
1 file changed, 12 insertions(+), 9 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
index 3bb7c4d1ceca..176e927b2d21 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
@@ -123,21 +123,20 @@ object InjectRuntimeFilter extends Rule[LogicalPlan] with
PredicateHelper with J
case ExtractEquiJoinKeys(joinType, lkeys, rkeys, _, _, left, right, _) =>
// Runtime filters use one side of the [[Join]] to build a set of join
key values and prune
// the other side of the [[Join]]. It's also OK to use a superset of
the join key values
- // (ignore null values) to do the pruning.
+ // (ignore null values) to do the pruning. We can also extract from
the other side if the
+ // join keys are transitive, and the other side always produces a
superset output of join
+ // key values. Any join side always produce a superset output of its
corresponding
+ // join keys, but for transitive join keys we need to check the join
type.
// We assume other rules have already pushed predicates through join
if possible.
// So the predicate references won't pass on anymore.
if (left.output.exists(_.semanticEquals(targetKey))) {
extract(left, AttributeSet.empty, hasHitFilter = false,
hasHitSelectiveFilter = false,
currentPlan = left, targetKey = targetKey).orElse {
- // We can also extract from the right side if the join keys are
transitive, and
- // the right side always produces a superset output of join left
keys.
- // Let's look at an example
+ // An example that extract from the right side if the join keys
are transitive.
// left table: 1, 2, 3
// right table, 3, 4
- // left outer join output: (1, null), (2, null), (3, 3)
- // left key output: 1, 2, 3
- // Any join side always produce a superset output of its
corresponding
- // join keys, but for transitive join keys we need to check the
join type.
+ // right outer join output: (3, 3), (null, 4)
+ // right key output: 3, 4
if (canPruneLeft(joinType)) {
lkeys.zip(rkeys).find(_._1.semanticEquals(targetKey)).map(_._2)
.flatMap { newTargetKey =>
@@ -152,7 +151,11 @@ object InjectRuntimeFilter extends Rule[LogicalPlan] with
PredicateHelper with J
} else if (right.output.exists(_.semanticEquals(targetKey))) {
extract(right, AttributeSet.empty, hasHitFilter = false,
hasHitSelectiveFilter = false,
currentPlan = right, targetKey = targetKey).orElse {
- // We can also extract from the left side if the join keys are
transitive.
+ // An example that extract from the left side if the join keys are
transitive.
+ // left table: 1, 2, 3
+ // right table, 3, 4
+ // left outer join output: (1, null), (2, null), (3, 3)
+ // left key output: 1, 2, 3
if (canPruneRight(joinType)) {
rkeys.zip(lkeys).find(_._1.semanticEquals(targetKey)).map(_._2)
.flatMap { newTargetKey =>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
