GithubZhitao commented on a change in pull request #33650:
URL: https://github.com/apache/spark/pull/33650#discussion_r732443239
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala
##########
@@ -40,37 +40,43 @@ object PushDownUtils extends PredicateHelper {
def pushFilters(
scanBuilder: ScanBuilder,
filters: Seq[Expression]): (Seq[sources.Filter], Seq[Expression]) = {
+ // A map from translated data source leaf node filters to original
catalyst filter
+ // expressions. For a `And`/`Or` predicate, it is possible that the
predicate is partially
+ // pushed down. This map can be used to construct a catalyst filter
expression from the
+ // input filter, or a superset(partial push down filter) of the input
filter.
Review comment:
Thanks for your reply. But how can I get the all filters or how to
define "un-pushable" ? I run some cases and found it weird some conditions
were classified to post-scan Filters, wherever I extends
SupportsPushDownFilters or SupportsPushDownV2Filters. Only some isNotNull(#v)
in the **pushFilters** function . Here is our case:
**SELECT * FROM iteblog_tab1 t1 LEFT SEMI JOIN iteblog_tab2 t2 ON t1.k= t2.k
AND t2._c0 < 2**
In the case we partition iteblog_tab1 and iteblog_tab2 by k. Following is
our calling exlain result . Why the t2._c0 < 2 are not happened in the Scan
but in the post-Scan. Code version is updated to newest master branch.
== Parsed Logical Plan ==
'Project [*]
+- 'Join LeftSemi, (('t1.k = 't2.k) AND ('t2._c0 < 2))
:- 'SubqueryAlias t1
: +- 'UnresolvedRelation [iteblog_tab1], [], false
+- 'SubqueryAlias t2
+- 'UnresolvedRelation [iteblog_tab2], [], false
== Analyzed Logical Plan ==
_c0: string, k: int
Project [_c0#38, k#39]
+- Join LeftSemi, ((k#39 = k#59) AND (cast(_c0#58 as int) < 2))
:- SubqueryAlias t1
: +- SubqueryAlias iteblog_tab1
: +- View (`iteblog_tab1`, [_c0#38,k#39])
: +- RelationV2[_c0#38, k#39]
/home/i-bozhitao/spark/spark-warehouse/iteblog_tab1
+- SubqueryAlias t2
+- SubqueryAlias iteblog_tab2
+- View (`iteblog_tab2`, [_c0#58,k#59])
+- RelationV2[_c0#58, k#59]
/home/user/spark/spark-warehouse/iteblog_tab2
== Optimized Logical Plan ==
Join LeftSemi, (k#39 = k#59)
:- Filter dynamicpruning#72 [k#39]
: : +- Project [k#59]
: : +- Filter (cast(_c0#58 as int) < 2)
: : +- RelationV2[_c0#58, k#59]
/home/user/spark/spark-warehouse/iteblog_tab2
: +- RelationV2[_c0#38, k#39] /home/user/spark/spark-warehouse/iteblog_tab1
+- Project [k#59]
+- Filter (cast(_c0#58 as int) < 2)
+- RelationV2[_c0#58, k#59]
/home/user/spark/spark-warehouse/iteblog_tab2
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- BroadcastHashJoin [k#39], [k#59], LeftSemi, BuildRight, false
:- Project [_c0#38, k#39]
: +- BatchScan[_c0#38, k#39] CSVScan DataFilters: [], Format: csv,
Location: InMemoryFileIndex(1
paths)[file:/home/user/spark/spark-warehouse/iteblog_tab1], PartitionFilters:
[], ReadSchema: struct<_c0:string>, PushedFilters---------: [] RuntimeFilters:
[dynamicpruningexpression(k#39 IN dynamicpruning#72)]
: +- SubqueryAdaptiveBroadcast dynamicpruning#72, 0, false,
Project [k#59], [k#59]
: +- AdaptiveSparkPlan isFinalPlan=false
: +- Project [k#59]
: +- Filter (cast(_c0#58 as int) < 2)
: +- BatchScan[_c0#58, k#59] CSVScan DataFilters: [],
Format: csv, Location: InMemoryFileIndex(1
paths)[file:/home/user/spark/spark-warehouse/iteblog_tab2], PartitionFilters:
[], ReadSchema: struct<_c0:string>, PushedFilters---------: [] RuntimeFilters:
[]
+- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int,
true] as bigint)),false), [id=#105]
+- Project [k#59]
+- Filter (cast(_c0#58 as int) < 2)
+- BatchScan[_c0#58, k#59] CSVScan DataFilters: [], Format: csv,
Location: InMemoryFileIndex(1
paths)[file:/home/i-bozhitao/spark/spark-warehouse/iteblog_tab2],
PartitionFilters: [], ReadSchema: struct<_c0:string>, PushedFilters---------:
[] RuntimeFilters: []
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
