[
https://issues.apache.org/jira/browse/CALCITE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZiLin Chen updated CALCITE-4677:
--------------------------------
Description:
When we enable top-down optimization, the VolcanoPlanner couldn't be completely
compatible with the old one.
Especially when we using rules matching RelNodes which convention is not NONE,
like rules that want to convert different conventions between physical node or
some rules that want to do some optimization for serval physical nodes. It
could be found that rules can't fire, because in top-down optimization physical
nodes can only be created by implementation rules(logical node -> physical
node) but not rules matching physical node(physical node -> physical node). Try
to use enforcer instead of rules to convert node between different conventions.
Example:
Whenever new physical nodes are on produce, we need to fire rules in case of
different physical conventions conversion.
DeriveTrait Task deal with this case, while some physical node maybe missing.
There 3 rules, but in TopDownRuleDriver rule 3 can not be applied.
1. LogicalSortToTopNRule
2. LogicalSortToLimitMemSortRule
3. TopNToSomeRelNodeRule (Need Match Physical Convention Node like TopN)
LogicalAggreate -> 1. RelSet1
- LogicalSort -> 2. RelSet2 fires LogicalSortToTopNRule and
LogicalSortToLimitMemSortRule
- OtherNode -> 3. RelSet3
ApplyRuleTask LogicalSortToLimitMemSortRule is already scheduled.
ApplyRuleTask LogicalSortToTopNRule still on the stack.
when RelSet3 on Merge (Merge with other RelSets), then RelSet2
clearProcessed(): subset.resetTaskState() and then RelSet1 clearProcessed():
subset.resetTaskState()
because RelSet1 is root so OptimizeGroup will be scheduled again
Now the task stack
Task Stack:
Task about RelSet1
Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule
Task about RelSet3
Task about RelSet1 root schedule again
Task about RelSet2
Task about RelSet3 (current task)
next
Task Stack:
Task about RelSet1
Task about RelSet2
Task about RelSet3
Task about RelSet1
Task about RelSet2 RelSubSet setOptimized so taskState -> Completed. rels on
RelSet (LogicalSort and LimitMemSort)
next...
Task Stack:
Task about RelSet1
Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule.
Here we found when TopN is on produce, every RelSubSet can not fire
OptimizeInputTask. so TopNToSomeRelNodeRule can not fire
was:
When we enable top-down optimization, the VolcanoPlanner couldn't be completely
compatible with the old one.
Especially when we using rules matching RelNodes which convention is not NONE,
like rules that want to convert different conventions between physical node or
some rules that want to do some optimization for serval physical nodes. It
could be found that rules can't fire, because in top-down optimization physical
nodes can only be created by implementation rules(logical node -> physical
node) but not rules matching physical node(physical node -> physical node).
Example:
Whenever new physical nodes are on produce, we need to fire rules in case of
different physical conventions conversion.
DeriveTrait Task deal with this case, while some physical node maybe missing.
There 3 rules, but in TopDownRuleDriver rule 3 can not be applied.
1. LogicalSortToTopNRule
2. LogicalSortToLimitMemSortRule
3. TopNToSomeRelNodeRule (Need Match Physical Convention Node like TopN)
LogicalAggreate -> 1. RelSet1
- LogicalSort -> 2. RelSet2 fires LogicalSortToTopNRule and
LogicalSortToLimitMemSortRule
- OtherNode -> 3. RelSet3
ApplyRuleTask LogicalSortToLimitMemSortRule is already scheduled.
ApplyRuleTask LogicalSortToTopNRule still on the stack.
when RelSet3 on Merge (Merge with other RelSets), then RelSet2
clearProcessed(): subset.resetTaskState() and then RelSet1 clearProcessed():
subset.resetTaskState()
because RelSet1 is root so OptimizeGroup will be scheduled again
Now the task stack
Task Stack:
Task about RelSet1
Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule
Task about RelSet3
Task about RelSet1 root schedule again
Task about RelSet2
Task about RelSet3 (current task)
next
Task Stack:
Task about RelSet1
Task about RelSet2
Task about RelSet3
Task about RelSet1
Task about RelSet2 RelSubSet setOptimized so taskState -> Completed. rels on
RelSet (LogicalSort and LimitMemSort)
next...
Task Stack:
Task about RelSet1
Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule.
Here we found when TopN is on produce, every RelSubSet can not fire
OptimizeInputTask. so TopNToSomeRelNodeRule can not fire
> Different physical conventions conversion on new physical node producing
> -------------------------------------------------------------------------
>
> Key: CALCITE-4677
> URL: https://issues.apache.org/jira/browse/CALCITE-4677
> Project: Calcite
> Issue Type: Bug
> Components: core
> Affects Versions: 1.28.0
> Reporter: ZiLin Chen
> Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When we enable top-down optimization, the VolcanoPlanner couldn't be
> completely compatible with the old one.
> Especially when we using rules matching RelNodes which convention is not
> NONE, like rules that want to convert different conventions between physical
> node or some rules that want to do some optimization for serval physical
> nodes. It could be found that rules can't fire, because in top-down
> optimization physical nodes can only be created by implementation
> rules(logical node -> physical node) but not rules matching physical
> node(physical node -> physical node). Try to use enforcer instead of rules to
> convert node between different conventions.
>
> Example:
> Whenever new physical nodes are on produce, we need to fire rules in case of
> different physical conventions conversion.
> DeriveTrait Task deal with this case, while some physical node maybe missing.
>
> There 3 rules, but in TopDownRuleDriver rule 3 can not be applied.
> 1. LogicalSortToTopNRule
> 2. LogicalSortToLimitMemSortRule
> 3. TopNToSomeRelNodeRule (Need Match Physical Convention Node like TopN)
>
> LogicalAggreate -> 1. RelSet1
> - LogicalSort -> 2. RelSet2 fires LogicalSortToTopNRule and
> LogicalSortToLimitMemSortRule
> - OtherNode -> 3. RelSet3
>
> ApplyRuleTask LogicalSortToLimitMemSortRule is already scheduled.
> ApplyRuleTask LogicalSortToTopNRule still on the stack.
>
> when RelSet3 on Merge (Merge with other RelSets), then RelSet2
> clearProcessed(): subset.resetTaskState() and then RelSet1 clearProcessed():
> subset.resetTaskState()
> because RelSet1 is root so OptimizeGroup will be scheduled again
>
> Now the task stack
> Task Stack:
> Task about RelSet1
> Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule
> Task about RelSet3
> Task about RelSet1 root schedule again
> Task about RelSet2
> Task about RelSet3 (current task)
>
> next
> Task Stack:
> Task about RelSet1
> Task about RelSet2
> Task about RelSet3
> Task about RelSet1
> Task about RelSet2 RelSubSet setOptimized so taskState -> Completed. rels on
> RelSet (LogicalSort and LimitMemSort)
>
> next...
> Task Stack:
> Task about RelSet1
> Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule.
> Here we found when TopN is on produce, every RelSubSet can not fire
> OptimizeInputTask. so TopNToSomeRelNodeRule can not fire
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
