[
https://issues.apache.org/jira/browse/DRILL-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955213#comment-14955213
]
Michael England commented on DRILL-3759:
----------------------------------------
I can add a use case which is impacted by this bug from a performance point of
view.
Copied from the user list:
Hi,
I am doing a performance test on a Drill v1.1 query over directories. I have
the following two queries which are effectively doing the same thing:
SELECT count(*) FROM `stg`.`ws`.`./2015/10/13` where columns[7] like '%blah%'
and columns[10] = ‘xx’
SELECT count(*) FROM `stg`.`ws`.`./2015` where dir0='10' and dir1='13' and
columns[7] like '%blah%' and columns[10] = ‘xx’
I have noticed that the second query that includes ‘where dir0=’10’ and
dir1=’13’ ‘ in the where clause takes ~29 seconds to run, however the first
query takes ~8 seconds to run. The Fragment profiles show the same explain plan
and very similar operator profiles. The second queries fragment profile starts
from 22 seconds whereas the first profile starts from 1 second.
Can this performance issue be caused by the Calcite parser taking longer to
understand the where clause?
Thanks,
Mike
> Make partition pruning multi-phased to reduce the working set kept in memory
> ----------------------------------------------------------------------------
>
> Key: DRILL-3759
> URL: https://issues.apache.org/jira/browse/DRILL-3759
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
> Affects Versions: 1.1.0
> Reporter: Aman Sinha
> Fix For: Future
>
>
> Currently, partition pruning gets all file names in the table and applies the
> pruning. Suppose the files are spread out over several directories and there
> is a filter on dirN, this is not efficient - both in terms of elapsed time
> and memory usage. This has been seen in a few use cases recently.
> Wherever possible, we should ideally perform the pruning in N steps (where N
> is the number of directory levels referenced in the filter conditions):
> 1. Get the directory and filenames at level i
> 2. Materialize into the in-memory table
> 3. Apply interpreter-based evaluation of filter condition
> 4. Determine qualifying directories, increment i and repeat from step 1
>
> This multi phase approach may not be possible for certain types of filters -
> e,g for disjunctions. This analysis needs to be done.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
