Aman Sinha created DRILL-3759:
---------------------------------
Summary: Make partition pruning multi-phased to reduce the working
set kept in memory
Key: DRILL-3759
URL: https://issues.apache.org/jira/browse/DRILL-3759
Project: Apache Drill
Issue Type: Improvement
Components: Query Planning & Optimization
Affects Versions: 1.1.0
Reporter: Aman Sinha
Assignee: Jinfeng Ni
Currently, partition pruning gets all file names in the table and applies the
pruning. Suppose the files are spread out over several directories and there
is a filter on dirN, this is not efficient - both in terms of elapsed time
and memory usage. This has been seen in a few use cases recently.
Wherever possible, we should ideally perform the pruning in N steps (where N is
the number of directory levels referenced in the filter conditions):
1. Get the directory and filenames at level i
2. Materialize into the in-memory table
3. Apply interpreter-based evaluation of filter condition
4. Determine qualifying directories, increment i and repeat from step 1
This multi phase approach may not be possible for certain types of filters -
e,g for disjunctions. This analysis needs to be done.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
