On Mon, Mar 27, 2017 at 11:42 AM, Sailesh Mukil <[email protected]> wrote: > I will be working on a patch to add min/max filter support in Impala, and > as a first step, specifically target the KuduScanNode, since the Kudu > client is already able to accept a Min and a Max that it would internally > use to filter during its scans. Below is a brief design proposal. > > *Goal:* > > To leverage runtime min/max filter support in Kudu for the potential speed > up of queries over Kudu tables. Kudu does this by taking a min and a max > that Impala will provide and only return values in the range Impala is > interested in. > > *[min <= range we're interested in >= max]* > > *Proposal:* > > > - As a first step, plumb the runtime filter code from > *exec/hdfs-scan-node-base.cc/h > <http://hdfs-scan-node-base.cc/h>* to *exec/scan-node.cc/h > <http://scan-node.cc/h>*, so that it can be applied to *KuduScanNode* > cleanly as well, since *KuduScanNode* and *HdfsScanNodeBase* both > inherit from *ScanNode.*
Quick comment: please make sure your solution also applies to KuduScanNodeMt. > - Reuse the *ColumnStats* class (exec/parquet-column-stats.h) or > implement a lighter weight version of it to process and store the Min and > the Max on the build side of the join. > - Once the Min and Max values are added to the existing runtime filter > structures, as a first step, we will ignore the Min and Max values for > non-Kudu tables. Using them for non-Kudu tables can come in as a following > patch(es). > - Similarly, the bloom filter will be ignored for Kudu tables, and only > the Min and Max values will be used, since Kudu does not accept bloom > filters yet. (https://issues.apache.org/jira/browse/IMPALA-3741) > - Applying the bloom filter on the Impala side of the Kudu scan (i.e. in > KuduScanNode) is not in the scope of this patch. > > > *Complications:* > > - We have to make sure that finding the Min and Max values on the build > side doesn't regress certain workloads, since the difference between > generating a bloom filter and generating a Min and a Max, is that a bloom > filter can be type agnostic (we just take a raw hash over the data) whereas > a Min and a Max have to be type specific.
