[
https://issues.apache.org/jira/browse/DRILL-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15004845#comment-15004845
]
ASF GitHub Bot commented on DRILL-3765:
---------------------------------------
GitHub user jinfengni opened a pull request:
https://github.com/apache/drill/pull/255
DRILL-3765: Move partitioning pruning to HepPlanner to avoid the perf…
…ormance overhead for redundant rule execution.
Add fall back option in planner.
Run the regression/unit with this feature turned on /off.
Please see DRILL-3765 for some performance number.
Regarding the planner options:
1. planner.enable_join_hep_opt is replaced with planner.enable_hep_opt.
This controls whether HepPlanner would be used in drill logical planning. If
this option is off, it would fall back to Volcano planner.
2. Add new planning option : planner.enable_hep_partition_pruning, which
will turn on/off this feature. If this option is off, it will fall back to the
code prior to this patch.
@amansinha100 , could you please review this patch?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jinfengni/incubator-drill DRILL-3765
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/255.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #255
----
commit 739ac9482605cd2593b569ce2ec22fd176f6afbd
Author: Jinfeng Ni <[email protected]>
Date: 2015-11-03T23:51:10Z
DRILL-3765: Move partitioning pruning to HepPlanner to avoid the
performance overhead for redundant rule execution.
Add fall back option in planner.
----
> Partition prune rule is unnecessary fired multiple times.
> ----------------------------------------------------------
>
> Key: DRILL-3765
> URL: https://issues.apache.org/jira/browse/DRILL-3765
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
> Reporter: Jinfeng Ni
> Assignee: Jinfeng Ni
>
> It seems that the partition prune rule may be fired multiple times, even
> after the first rule execution has pushed the filter into the scan operator.
> Since partition prune has to build the vectors to contain the partition /file
> / directory information, to invoke the partition prune rule unnecessary may
> lead to big memory overhead.
> Drill planner should avoid the un-necessary partition prune rule, in order to
> reduce the chance of hitting OOM exception, while the partition prune rule is
> executed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
