[
https://issues.apache.org/jira/browse/HIVE-19101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16473118#comment-16473118
]
Jesus Camacho Rodriguez commented on HIVE-19101:
------------------------------------------------
[~vitalii], HIVE-19433 may fix the issue. I agree there is no reason to not use
the Calcite original methods, we need to check what are the differences between
the Hive and Calcite version: if there are contributions that have been made to
Hive but not ported to Calcite, we should push them to Calcite and remove the
methods from the Hive side when we upgrade to new version. If there are no
differences, it does not make sense to have the same code in two different
places.
> Apply rule [HiveJoinPushTransitivePredicatesRule] is getting stuck when there
> are huge number of predicates
> ------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-19101
> URL: https://issues.apache.org/jira/browse/HIVE-19101
> Project: Hive
> Issue Type: Bug
> Components: CBO
> Affects Versions: 2.1.1, 2.2.0, 2.3.0, 2.3.1, 2.3.2
> Reporter: Ganesha Shreedhara
> Priority: Major
> Attachments: queries
>
>
> Hive query is getting stuck during the optimisation phase while applying
> HiveJoinPushTransitivePredicatesRule when there are huge number of predicates.
>
> *DEBUG Log:*
> {code:java}
> 2018-04-04T11:22:47,991 [user: ganeshas] -1 DEBUG
> [6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main] plan.RelOptPlanner: call#10963:
> Apply rule [ReduceExpressionsRule(Join)] to
> [rel#881:HiveJoin.HIVE.[](left=HepRelVertex#879,right=HepRelVertex#887,condition==($1,
> $73),joinType=inner,algorithm=none,cost=not available)]
> 2018-04-04T11:22:48,359 [user: ganeshas] -1 DEBUG
> [6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main] plan.RelOptPlanner: call#10964:
> Apply rule [HiveJoinAddNotNullRule] to
> [rel#881:HiveJoin.HIVE.[](left=HepRelVertex#879,right=HepRelVertex#887,condition==($1,
> $73),joinType=inner,algorithm=none,cost=not available)]
> 2018-04-04T11:22:48,360 [user: ganeshas] -1 DEBUG
> [6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main] plan.RelOptPlanner: call#10965:
> Apply rule [HiveJoinPushTransitivePredicatesRule] to
> [rel#881:HiveJoin.HIVE.[](left=HepRelVertex#879,right=HepRelVertex#887,condition==($1,
> $73),joinType=inner,algorithm=none,cost=not available)]{code}
>
> *Thread Status:*
>
> {code:java}
> "6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main" prio=5 tid=0x00007ff18e006800
> nid=0x1c03 runnable [0x0000700008176000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOfRange(Arrays.java:2694)
> at java.lang.String.<init>(String.java:203)
> at java.lang.StringBuilder.toString(StringBuilder.java:405)
> at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:95)
> at org.apache.calcite.rex.RexCall.toString(RexCall.java:100)
> at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:84)
> at org.apache.calcite.rex.RexCall.toString(RexCall.java:100)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference.infer(HiveRelMdPredicates.java:516)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference.inferPredicates(HiveRelMdPredicates.java:426)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates.getPredicates(HiveRelMdPredicates.java:186)
> at GeneratedMetadataHandler_Predicates.getPredicates_$(Unknown Source)
> at GeneratedMetadataHandler_Predicates.getPredicates(Unknown Source)
> at
> org.apache.calcite.rel.metadata.RelMetadataQuery.getPulledUpPredicates(RelMetadataQuery.java:721)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveJoinPushTransitivePredicatesRule.onMatch(HiveJoinPushTransitivePredicatesRule.java:83)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:314)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:502)
> at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:381)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:275)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:72)
> at org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:206)
> at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:193)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.hepPlan(CalcitePlanner.java:1575)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPreJoinOrderingTransforms(CalcitePlanner.java:1448)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1174)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1096)
> at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:113)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:997)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:149)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:106)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:905)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:920)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:330)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11206)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:251)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:265)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:527)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1358)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1499)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1278)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1268)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:213)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:447)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:924)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:848)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:735)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Locked ownable synchronizers:
> - <0x00000007d5cfcab0> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> {code}
>
>
> This issue seems to be because of optimisation done on CBO (Specifically on
> calcite transitive predicate inference rule) in hive 2.1.1 and later
> versions.
> This issue is not there in hive1.2.1 version.
>
> The queries to reproduce this issue is in this attachment : [^queries].
> There is a view (test_view) created by combining the results from 16 tables
> using UNION ALL.
> The query is getting stuck when I run a select query which performs join
> between view (test_view) and a table (test1).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
