[
https://issues.apache.org/jira/browse/HIVE-30080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-30080 started by Krisztian Kasa.
---------------------------------------------
> Queries can get stuck compiling during SharedWorkOptimizer
> ----------------------------------------------------------
>
> Key: HIVE-30080
> URL: https://issues.apache.org/jira/browse/HIVE-30080
> Project: Hive
> Issue Type: Bug
> Affects Versions: 4.2.1
> Reporter: Urmas Tamassy
> Assignee: Krisztian Kasa
> Priority: Major
> Labels: compilation, pull-request-available
> Attachments: swo_hang.q
>
>
> As noted in the title, certain types of queries can get stuck during Shared
> work optimizer step during compilation as indicated in HiveServer2 logs and
> in the thread dumps taken while the query is stuck:
> {code:java}
> INFO org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer:
> [9240622d-5efe-464e-9c38-c8008504a54e9240622d-5efe-464e-9c38-c8008504a54e
> HiveServer2-Handler-Pool: Thread-3389141]: SharedWorkOptimizer start {code}
> {code:java}
> at java.util.HashMap$HashIterator.<init>([email protected]/HashMap.java:1593)
> at java.util.HashMap$KeyIterator.<init>([email protected]/HashMap.java:1626)
> at java.util.HashMap$KeySet.iterator([email protected]/HashMap.java:991) at
> java.util.HashSet.iterator([email protected]/HashSet.java:182) at
> org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.findDescendantWorkOperators(SharedWorkOptimizer.java:2081)
> at
> org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.validPreConditions(SharedWorkOptimizer.java:1944)
> at
> org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.sharedWorkOptimization(SharedWorkOptimizer.java:471)
> at
> org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.runSharedWorkOptimization(SharedWorkOptimizer.java:287)
> at
> org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.transform(SharedWorkOptimizer.java:184)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:252)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:186)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.compilePlan(SemanticAnalyzer.java:13247)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13472)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:467)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:360)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:188)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:360)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:225) at
> org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:110) at
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:500) at
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:452) at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:416) at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:410) at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:127)
> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:235)
> {code}
> Attaching [^swo_hang.q] test query to reproduce the issue. As noted in the
> comments in the file the test requires hive.auto.convert.join=false.
> At this time it is uncertain what the exact conditions are for the issue to
> occur as the query needs to be slightly complex (3-way join with additional
> conditions, group by etc.) and may depend on the input table metadata.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
