Urmas Tamassy created HIVE-30080:
------------------------------------
Summary: Queries can get stuck compiling during SharedWorkOptimizer
Key: HIVE-30080
URL: https://issues.apache.org/jira/browse/HIVE-30080
Project: Hive
Issue Type: Bug
Affects Versions: 4.2.1
Reporter: Urmas Tamassy
Attachments: swo_hang.q
As noted in the title, certain types of queries can get stuck during Shared
work optimizer step as indicated in HiveServer2 logs and in the thread dumps
taken while the query is stuck:
{code:java}
INFO org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer:
[9240622d-5efe-464e-9c38-c8008504a54e9240622d-5efe-464e-9c38-c8008504a54e
HiveServer2-Handler-Pool: Thread-3389141]: SharedWorkOptimizer start {code}
{code:java}
at java.util.HashMap$HashIterator.<init>([email protected]/HashMap.java:1593) at
java.util.HashMap$KeyIterator.<init>([email protected]/HashMap.java:1626) at
java.util.HashMap$KeySet.iterator([email protected]/HashMap.java:991) at
java.util.HashSet.iterator([email protected]/HashSet.java:182) at
org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.findDescendantWorkOperators(SharedWorkOptimizer.java:2081)
at
org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.validPreConditions(SharedWorkOptimizer.java:1944)
at
org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.sharedWorkOptimization(SharedWorkOptimizer.java:471)
at
org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.runSharedWorkOptimization(SharedWorkOptimizer.java:287)
at
org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.transform(SharedWorkOptimizer.java:184)
at
org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:252)
at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:186)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.compilePlan(SemanticAnalyzer.java:13247)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13472)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:467)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:360)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:188)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:360)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:225) at
org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:110) at
org.apache.hadoop.hive.ql.Driver.compile(Driver.java:500) at
org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:452) at
org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:416) at
org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:410) at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:127)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:235)
{code}
Attaching [^swo_hang.q] test query to reproduce the issue. As noted in the
comments in the file the test requires hive.auto.convert.join=false.
At this time it is uncertain what the exact conditions are for the issue to
occur as the query needs to be slightly complex (3-way join with additional
conditions, group by etc.) and may depend on the input table metadata.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
