[
https://issues.apache.org/jira/browse/CALCITE-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961619#comment-16961619
]
Xiening Dai commented on CALCITE-2970:
--------------------------------------
I spent some time debugging this issue. I think I know what is going on.
When useAbstractConvertersForConversion is set to true, planner inserts an
AbstractConverter node to convert input into the collation trait that's
required by SortMergedJoin. And during the ExpandConversionRule, the planner
calls into RelCollationTraitDef.convert() and adds an LogicalSort to the input.
The new LogicalSort, combined with other existing logical nodes, significantly
expand the search space as they trigger a large number of rules, such as
ProjectMergeRule, AggregateProjectMergeRule, etc. These rules produce too many
alternatives which then would trigger more rules. So the optimization latency
spikes (from 3s to 52s).
I did a quick experiment to create EnumerableSort, instead of LogicalSort,
within RelCollationDef.convert(). The planning time is down from 52s to 9s. And
join 6 way can be completed with 12s. See -
https://github.com/xndai/calcite/tree/calcite-2970
I understand we cannot just create EnumerableSort as there maybe other
conventions. But I think it's totally reasonable to just create a corresponding
physical sort when convention is known. When we know exactly the required
convention and collation trait, there's no reason to create yet another logical
node.
In order to achieve that, the framework would need to provide a way to create a
physical node with given convention not through user rules. I don't have a
clear solution in my mind, and I would like to hear your comments [~hyuan]
[~zabetak] [~julianhyde]. I think this will be an important enhancement to the
calcite framework.
> Performance issue when enabling abstract converter for EnumerableConvention
> ---------------------------------------------------------------------------
>
> Key: CALCITE-2970
> URL: https://issues.apache.org/jira/browse/CALCITE-2970
> Project: Calcite
> Issue Type: Bug
> Components: core
> Reporter: Haisheng Yuan
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> If we enable the use of abstract converter for {{EnumerableConvention}}, by
> making {{useAbstractConvertersForConversion}} return true,
> {{JDBCTest.testJoinManyWay}} will not complete.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
