cloud-fan commented on code in PR #36361:
URL: https://github.com/apache/spark/pull/36361#discussion_r858716045
##########
sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala:
##########
@@ -267,14 +267,20 @@ class InjectRuntimeFilterSuite extends QueryTest with
SQLTestUtils with SharedSp
// `MergeScalarSubqueries` can duplicate subqueries in the optimized plan,
but the subqueries will
// be reused in the physical plan.
def getNumBloomFilters(plan: LogicalPlan, scalarSubqueryCTEMultiplicator:
Int = 1): Integer = {
- val numBloomFilterAggs = plan.collectWithSubqueries {
- case Aggregate(_, aggregateExpressions, _) =>
- aggregateExpressions.collect {
- case Alias(AggregateExpression(bfAgg: BloomFilterAggregate, _, _, _,
_), _) =>
- assert(bfAgg.estimatedNumItemsExpression.isInstanceOf[Literal])
- assert(bfAgg.numBitsExpression.isInstanceOf[Literal])
- 1
+ val numBloomFilterAggs = plan.collect {
+ case Filter(condition, _) => condition.collect {
+ case subquery: org.apache.spark.sql.catalyst.expressions.ScalarSubquery
Review Comment:
I like the previous code more as it precisely matches `Filter` and
`ScalarSubquery`. +1 to this change.
My major concern is `scalarSubqueryCTEMultiplicator`. This makes the test
really hard to write as we need to manually think about if scalar subqueries
merging can be applied or not to the testing query.
Can we turn off this optimization completely in this test suite? We can add
one more test to verify the case that scalar subqueries merging is beneficial
to bloom filter join, by explicitly enabling the optimizer rule.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
