[
https://issues.apache.org/jira/browse/DRILL-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15007223#comment-15007223
]
ASF GitHub Bot commented on DRILL-3765:
---------------------------------------
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/255#discussion_r44974839
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java
---
@@ -116,20 +117,32 @@ public DrillSqlWorker(QueryContext context) {
private RuleSet[] getRules(QueryContext context) {
StoragePluginRegistry storagePluginRegistry = context.getStorage();
- RuleSet drillLogicalRules = DrillRuleSets.mergedRuleSets(
+
+ // Ruleset for the case where VolcanoPlanner is used for everything :
join, filter/project pushdown, partition pruning.
+ RuleSet drillLogicalVolOnlyRules = DrillRuleSets.mergedRuleSets(
DrillRuleSets.getDrillBasicRules(context),
+ DrillRuleSets.getPruneScanRules(context),
DrillRuleSets.getJoinPermRules(context),
DrillRuleSets.getDrillUserConfigurableLogicalRules(context));
+
+ // Ruleset for the case where join planning is done in Hep-LOPT,
filter/project pushdown and parttion pruning are done in VolcanoPlanner
+ RuleSet drillLogicalHepJoinRules = DrillRuleSets.mergedRuleSets(
+ DrillRuleSets.getDrillBasicRules(context),
+ DrillRuleSets.getPruneScanRules(context),
+ DrillRuleSets.getDrillUserConfigurableLogicalRules(context));
+
+ // Ruleset for the case where join planning and partition pruning is
done in Hep, filter/project pushdown are done in VolcanoPlanner
--- End diff --
Exactly. The ruleset defined here is used in Frameworks.VolcanoPlanner. In
case that join / partition pruning is done in Hep, we do not put the
join/partition pruning rules in the ruleset here. In stead, the join/partition
pruning rules are specified when HepPlanner is built.
> Partition prune rule is unnecessary fired multiple times.
> ----------------------------------------------------------
>
> Key: DRILL-3765
> URL: https://issues.apache.org/jira/browse/DRILL-3765
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
> Reporter: Jinfeng Ni
> Assignee: Aman Sinha
>
> It seems that the partition prune rule may be fired multiple times, even
> after the first rule execution has pushed the filter into the scan operator.
> Since partition prune has to build the vectors to contain the partition /file
> / directory information, to invoke the partition prune rule unnecessary may
> lead to big memory overhead.
> Drill planner should avoid the un-necessary partition prune rule, in order to
> reduce the chance of hitting OOM exception, while the partition prune rule is
> executed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
