cloud-fan commented on code in PR #46263:
URL: https://github.com/apache/spark/pull/46263#discussion_r1582495367
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala:
##########
@@ -120,34 +132,49 @@ object InjectRuntimeFilter extends Rule[LogicalPlan] with
PredicateHelper with J
hasHitSelectiveFilter = hasHitSelectiveFilter ||
isLikelySelective(condition),
currentPlan,
targetKey)
- case ExtractEquiJoinKeys(_, lkeys, rkeys, _, _, left, right, _) =>
+ case ExtractEquiJoinKeys(joinType, lkeys, rkeys, _, _, left, right, _) =>
// Runtime filters use one side of the [[Join]] to build a set of join
key values and prune
// the other side of the [[Join]]. It's also OK to use a superset of
the join key values
// (ignore null values) to do the pruning.
// We assume other rules have already pushed predicates through join
if possible.
// So the predicate references won't pass on anymore.
if (left.output.exists(_.semanticEquals(targetKey))) {
- extract(left, AttributeSet.empty, hasHitFilter = false,
hasHitSelectiveFilter = false,
Review Comment:
Let's clarify the expected strategy a bit more: For the exact join key
match, like the left table here, it's always OK to generate the runtime filter
using this left table, no matter what the join type is. This is because left
table always produce a superset of output of the join output regarding the left
keys.
For transitive join key match, it's different. The right table here does not
always generate a superset output regarding left keys. Let's look at an example
```
left table: 1, 2, 3
right table, 3, 4
left outer join output: 1, 2, 3
```
So we can't use right table to generate runtime filter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
