jovany-wang commented on pull request #29065:
URL: https://github.com/apache/spark/pull/29065#issuecomment-662255897
Hi @wangyum , This is a nice PR to me. But some issues in my mind should be
thrown here.
I didn't do more perf between MinMax and Bloom, but in my personal sense,
these may effect the perf of different cases.
So how about making these things more general? like:
```
DynamicFilter
|
Is the filtering key partitioned?
/ \
Y N
/ \
DPP filter Choose a best filter for it. (from MinMax,
Bloom or other filters such as index filter, etc)
Note: Not all of the filters can be
pushed to scan.
```
That is just a rough idea, but the key point is to make DynamicFilter(or
name it RuntimeFilter) more general(that means both of MinMaxFilter,
BloomFilter and DPPFilter are DynamicFilter), so that it will be easy to get
extended.
I have seen another proposal about RuntimeFilter(MinMax) before, so making
things easy to be extended should be important as well as the perf result.
Feel free to point my incorrect understanding out, thx.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
