[ 
https://issues.apache.org/jira/browse/HIVE-19101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430613#comment-16430613
 ] 

Ganesha Shreedhara commented on HIVE-19101:
-------------------------------------------

[~jcamachorodriguez] Could you please help with this? Please let me know if you 
need more details on this.

> Apply rule [HiveJoinPushTransitivePredicatesRule] is getting stuck when there 
> are huge number of predicates 
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-19101
>                 URL: https://issues.apache.org/jira/browse/HIVE-19101
>             Project: Hive
>          Issue Type: Bug
>          Components: CBO
>    Affects Versions: 2.1.1, 2.2.0, 2.3.0, 2.3.1, 2.3.2
>            Reporter: Ganesha Shreedhara
>            Priority: Major
>         Attachments: queries
>
>
> Hive query is getting stuck during the optimisation phase while applying 
> HiveJoinPushTransitivePredicatesRule when there are huge number of predicates.
>  
> *DEBUG Log:*
> {code:java}
> 2018-04-04T11:22:47,991 [user: ganeshas] -1 DEBUG 
> [6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main] plan.RelOptPlanner: call#10963: 
> Apply rule [ReduceExpressionsRule(Join)] to 
> [rel#881:HiveJoin.HIVE.[](left=HepRelVertex#879,right=HepRelVertex#887,condition==($1,
>  $73),joinType=inner,algorithm=none,cost=not available)]
> 2018-04-04T11:22:48,359 [user: ganeshas] -1 DEBUG 
> [6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main] plan.RelOptPlanner: call#10964: 
> Apply rule [HiveJoinAddNotNullRule] to 
> [rel#881:HiveJoin.HIVE.[](left=HepRelVertex#879,right=HepRelVertex#887,condition==($1,
>  $73),joinType=inner,algorithm=none,cost=not available)]
> 2018-04-04T11:22:48,360 [user: ganeshas] -1 DEBUG 
> [6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main] plan.RelOptPlanner: call#10965: 
> Apply rule [HiveJoinPushTransitivePredicatesRule] to 
> [rel#881:HiveJoin.HIVE.[](left=HepRelVertex#879,right=HepRelVertex#887,condition==($1,
>  $73),joinType=inner,algorithm=none,cost=not available)]{code}
>  
> *Thread Status:*
>  
> {code:java}
> "6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main" prio=5 tid=0x00007ff18e006800 
> nid=0x1c03 runnable [0x0000700008176000]
>  java.lang.Thread.State: RUNNABLE
>  at java.util.Arrays.copyOfRange(Arrays.java:2694)
>  at java.lang.String.<init>(String.java:203)
>  at java.lang.StringBuilder.toString(StringBuilder.java:405)
>  at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:95)
>  at org.apache.calcite.rex.RexCall.toString(RexCall.java:100)
>  at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:84)
>  at org.apache.calcite.rex.RexCall.toString(RexCall.java:100)
>  at 
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference.infer(HiveRelMdPredicates.java:516)
>  at 
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference.inferPredicates(HiveRelMdPredicates.java:426)
>  at 
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates.getPredicates(HiveRelMdPredicates.java:186)
>  at GeneratedMetadataHandler_Predicates.getPredicates_$(Unknown Source)
>  at GeneratedMetadataHandler_Predicates.getPredicates(Unknown Source)
>  at 
> org.apache.calcite.rel.metadata.RelMetadataQuery.getPulledUpPredicates(RelMetadataQuery.java:721)
>  at 
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveJoinPushTransitivePredicatesRule.onMatch(HiveJoinPushTransitivePredicatesRule.java:83)
>  at 
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:314)
>  at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:502)
>  at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:381)
>  at 
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:275)
>  at 
> org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:72)
>  at org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:206)
>  at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:193)
>  at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.hepPlan(CalcitePlanner.java:1575)
>  at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPreJoinOrderingTransforms(CalcitePlanner.java:1448)
>  at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1174)
>  at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1096)
>  at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:113)
>  at 
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:997)
>  at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:149)
>  at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:106)
>  at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:905)
>  at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:920)
>  at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:330)
>  at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11206)
>  at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:251)
>  at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:265)
>  at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:527)
>  at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1358)
>  at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1499)
>  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1278)
>  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1268)
>  at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
>  at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:213)
>  at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:447)
>  at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:924)
>  at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:848)
>  at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:735)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:606)
>  at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
>  at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Locked ownable synchronizers:
>  - <0x00000007d5cfcab0> (a 
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> {code}
>  
>  
> This issue seems to be  because of optimisation done on CBO (Specifically on 
> calcite transitive predicate inference rule) in hive 2.1.1 and later 
> versions. 
> This issue is not there in hive1.2.1 version.
>  
> The queries to reproduce this issue is in this attachment : [^queries]. 
> There is a view (test_view) created by combining the results from 16 tables 
> using UNION ALL. 
> The query is getting stuck when I run a select query which performs join 
> between view (test_view) and a table (test1). 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to