[
https://issues.apache.org/jira/browse/DRILL-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14933531#comment-14933531
]
Aman Sinha commented on DRILL-3759:
-----------------------------------
In order to support the multi-phase approach outlined above in description, we
will need a placeholder in the FileSelection class to keep track of
directories. Currently, FileSelection holds the expanded FileStatuses. We
could have a List<List<FileStatus>> as a container for directories only. The
index in the outer list will be the depth of the directory in the hierarchy.
My initial thought is that the directory would be expanded only when requested
- for instance during partition pruning, but I haven't considered the
implications for other metadata operations.
Note, there's a separate discussion about doing the partition pruning type of
metadata operations during execution phase instead of planning phase - see
DRILL-3838. My understanding is that would require design and be a longer term
solution. This particular JIRA needs to be addressed in the near term to
handle scaling issues. [~jnadeau] let me know your thoughts if any.
> Make partition pruning multi-phased to reduce the working set kept in memory
> ----------------------------------------------------------------------------
>
> Key: DRILL-3759
> URL: https://issues.apache.org/jira/browse/DRILL-3759
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
> Affects Versions: 1.1.0
> Reporter: Aman Sinha
> Assignee: Aman Sinha
>
> Currently, partition pruning gets all file names in the table and applies the
> pruning. Suppose the files are spread out over several directories and there
> is a filter on dirN, this is not efficient - both in terms of elapsed time
> and memory usage. This has been seen in a few use cases recently.
> Wherever possible, we should ideally perform the pruning in N steps (where N
> is the number of directory levels referenced in the filter conditions):
> 1. Get the directory and filenames at level i
> 2. Materialize into the in-memory table
> 3. Apply interpreter-based evaluation of filter condition
> 4. Determine qualifying directories, increment i and repeat from step 1
>
> This multi phase approach may not be possible for certain types of filters -
> e,g for disjunctions. This analysis needs to be done.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
