Thanks for sending this around, Sailesh. parquet-column-stats.h is somewhat tied to parquet statistics and will soon need more parquet-specific fields (null_count and distinct_count). It will also need extension to copy strings and handle there memory when tracking min/max values for variable length data across row batches. I'm not sure how these changes will affect its suitability for your purposes, but I thought I'd give a quick heads up.
On Mon, Mar 27, 2017 at 11:35 PM, Todd Lipcon <[email protected]> wrote: > Sounds reasonable to me as well. > > The only thing I'd add is that, if it's easy to design the code to be > extended to pushing small 'IN (...)' predicate for low-cardinality filters, > that would be great. eg if the filter can start as an IN(...) and then if > it exceeds 32K (or whatever arbitrary threshold), "collapse" it to the > min/max range predicate? > > This should have a big advantage for partition pruning in low-cardinality > joins against hash-partitioned tables. > > -Todd > > > > On Mon, Mar 27, 2017 at 2:29 PM, Matthew Jacobs <[email protected]> wrote: > > > Thanks for writing this up, Sailesh. It sounds reasonable. > > > > On Mon, Mar 27, 2017 at 2:24 PM, Sailesh Mukil <[email protected]> > > wrote: > > > On Mon, Mar 27, 2017 at 11:49 AM, Marcel Kornacker < > [email protected]> > > > wrote: > > > > > >> On Mon, Mar 27, 2017 at 11:42 AM, Sailesh Mukil <[email protected] > > > > >> wrote: > > >> > I will be working on a patch to add min/max filter support in > Impala, > > and > > >> > as a first step, specifically target the KuduScanNode, since the > Kudu > > >> > client is already able to accept a Min and a Max that it would > > internally > > >> > use to filter during its scans. Below is a brief design proposal. > > >> > > > >> > *Goal:* > > >> > > > >> > To leverage runtime min/max filter support in Kudu for the potential > > >> speed > > >> > up of queries over Kudu tables. Kudu does this by taking a min and a > > max > > >> > that Impala will provide and only return values in the range Impala > is > > >> > interested in. > > >> > > > >> > *[min <= range we're interested in >= max]* > > >> > > > >> > *Proposal:* > > >> > > > >> > > > >> > - As a first step, plumb the runtime filter code from > > >> > *exec/hdfs-scan-node-base.cc/h > > >> > <http://hdfs-scan-node-base.cc/h>* to *exec/scan-node.cc/h > > >> > <http://scan-node.cc/h>*, so that it can be applied to > > *KuduScanNode* > > >> > cleanly as well, since *KuduScanNode* and *HdfsScanNodeBase* both > > >> > inherit from *ScanNode.* > > >> > > >> Quick comment: please make sure your solution also applies to > > >> KuduScanNodeMt. > > >> > > > > > > Thanks for the input, I'll make sure to do that. > > > > > > > > >> > > >> > - Reuse the *ColumnStats* class (exec/parquet-column-stats.h) or > > >> > implement a lighter weight version of it to process and store the > > Min > > >> and > > >> > the Max on the build side of the join. > > >> > - Once the Min and Max values are added to the existing runtime > > filter > > >> > structures, as a first step, we will ignore the Min and Max > values > > for > > >> > non-Kudu tables. Using them for non-Kudu tables can come in as a > > >> following > > >> > patch(es). > > >> > - Similarly, the bloom filter will be ignored for Kudu tables, > and > > >> only > > >> > the Min and Max values will be used, since Kudu does not accept > > bloom > > >> > filters yet. (https://issues.apache.org/jira/browse/IMPALA-3741) > > >> > - Applying the bloom filter on the Impala side of the Kudu scan > > (i.e. > > >> in > > >> > KuduScanNode) is not in the scope of this patch. > > >> > > > >> > > > >> > *Complications:* > > >> > > > >> > - We have to make sure that finding the Min and Max values on the > > >> build > > >> > side doesn't regress certain workloads, since the difference > > between > > >> > generating a bloom filter and generating a Min and a Max, is > that a > > >> bloom > > >> > filter can be type agnostic (we just take a raw hash over the > data) > > >> whereas > > >> > a Min and a Max have to be type specific. > > >> > > > > > > -- > Todd Lipcon > Software Engineer, Cloudera >
