[
https://issues.apache.org/jira/browse/CALCITE-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
TANG Wen-hui updated CALCITE-2223:
----------------------------------
Comment: was deleted
(was: [~vladimirsitnikov] Thanks for your
PR(https://github.com/apache/calcite/pull/855), That's really helpful for us.
But when we run a sql like this:
sql = "select t1.cint,max(t4.cdouble) as mac " +
"from mysql.temp t1 " +
"inner join pg.temp2 t2 " +
"on t1.cint = t2.cint " +
"inner join mysql2.temp t3 " +
"on t2.cint = t3.cint " +
"inner join spark.temp2 t4 " +
"on t3.cint = t4.cint " +
"where t1.cfloat < 100 and t2.cstring <> 'hello' " +
"and t4.cdate <> '2012-01-01' and t3.ctimestamp <> '2012-01-01 18:03:19' " +
"group by t1.cint, t3.cstring, t4.cdouble " +
"order by t3.cstring desc, t1.cint, t4.cdouble asc ";
It cost almost 101879 ms to find the best plan , and we found that most of the
time spend on ProjectMergeRule especially when both topProject and
bottomProject are permutations (Sometimes it will cause RelSet merge). So we
set the importance of topProject and bottomProject to 0.0 when both projects
are permutations to reduce the time and it works. but we not sure if it is a
right way . It seems that the new project is better.)
> ProjectMergeRule is infinitely matched when is applied after
> ProjectReduceExpressionsRule
> -----------------------------------------------------------------------------------------
>
> Key: CALCITE-2223
> URL: https://issues.apache.org/jira/browse/CALCITE-2223
> Project: Calcite
> Issue Type: Bug
> Reporter: Volodymyr Vysotskyi
> Priority: Critical
> Labels: pull-request-available
> Attachments:
> TestLimitWithExchanges_testPushLimitPastUnionExchange.png, heap_overview.png,
> provenance_contents.png
>
>
> For queries like this:
> {code:sql}
> select t1.f from (select cast(f as int) f, f from (select cast(f as int) f
> from (values('1')) t(f))) as t1
> {code}
> OOM is thrown when {{ProjectMergeRule}} is applied before applying
> {{ProjectReduceExpressionsRule}} in VolcanoPlanner.
> A simple test to reproduce this issue (in {{RelOptRulesTest}}):
> {code:java}
> @Test public void testOomProjectMergeRule() {
> RelBuilder relBuilder =
> RelBuilder.create(RelBuilderTest.config().build());
> RelNode relNode = relBuilder
> .values(new String[]{"f"}, "1")
> .project(
> relBuilder.alias(
> relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER),
> "f"))
> .project(
> relBuilder.alias(
> relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER),
> "f0"),
> relBuilder.alias(relBuilder.field(0), "f"))
> .project(
> relBuilder.alias(relBuilder.field(0), "f"))
> .build();
> RelOptPlanner planner = relNode.getCluster().getPlanner();
> RuleSet ruleSet =
> RuleSets.ofList(
> ReduceExpressionsRule.PROJECT_INSTANCE,
> new ProjectMergeRuleWithLongerName(),
> EnumerableRules.ENUMERABLE_PROJECT_RULE,
> EnumerableRules.ENUMERABLE_VALUES_RULE);
> Program program = Programs.of(ruleSet);
> RelTraitSet toTraits =
> relNode.getCluster().traitSet()
> .replace(0, EnumerableConvention.INSTANCE);
> RelNode output = program.run(planner, relNode, toTraits,
> ImmutableList.<RelOptMaterialization>of(),
> ImmutableList.<RelOptLattice>of());
> // check for output
> }
> /**
> * ProjectMergeRule inheritor which has
> * class name greater than ProjectReduceExpressionsRule class name
> (String.compareTo()).
> *
> * It is needed for RuleQueue.popMatch() method
> * to apply this rule before ProjectReduceExpressionsRule.
> */
> private static class ProjectMergeRuleWithLongerName extends
> ProjectMergeRule {
> public ProjectMergeRuleWithLongerName() {
> super(true, RelFactories.LOGICAL_BUILDER);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
