[
https://issues.apache.org/jira/browse/HIVE-30080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Urmas Tamassy updated HIVE-30080:
---------------------------------
Description:
As noted in the title, certain types of queries can get stuck during Shared
work optimizer step during compilation as indicated in HiveServer2 logs and in
the thread dumps taken while the query is stuck:
{code:java}
INFO org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer:
[9240622d-5efe-464e-9c38-c8008504a54e9240622d-5efe-464e-9c38-c8008504a54e
HiveServer2-Handler-Pool: Thread-3389141]: SharedWorkOptimizer start {code}
{code:java}
at java.util.HashMap$HashIterator.<init>([email protected]/HashMap.java:1593) at
java.util.HashMap$KeyIterator.<init>([email protected]/HashMap.java:1626) at
java.util.HashMap$KeySet.iterator([email protected]/HashMap.java:991) at
java.util.HashSet.iterator([email protected]/HashSet.java:182) at
org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.findDescendantWorkOperators(SharedWorkOptimizer.java:2081)
at
org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.validPreConditions(SharedWorkOptimizer.java:1944)
at
org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.sharedWorkOptimization(SharedWorkOptimizer.java:471)
at
org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.runSharedWorkOptimization(SharedWorkOptimizer.java:287)
at
org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.transform(SharedWorkOptimizer.java:184)
at
org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:252)
at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:186)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.compilePlan(SemanticAnalyzer.java:13247)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13472)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:467)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:360)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:188)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:360)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:225) at
org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:110) at
org.apache.hadoop.hive.ql.Driver.compile(Driver.java:500) at
org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:452) at
org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:416) at
org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:410) at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:127)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:235)
{code}
Attaching [^swo_hang.q] test query to reproduce the issue. As noted in the
comments in the file the test requires hive.auto.convert.join=false.
At this time it is uncertain what the exact conditions are for the issue to
occur as the query needs to be slightly complex (3-way join with additional
conditions, group by etc.) and may depend on the input table metadata.
was:
As noted in the title, certain types of queries can get stuck during Shared
work optimizer step as indicated in HiveServer2 logs and in the thread dumps
taken while the query is stuck:
{code:java}
INFO org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer:
[9240622d-5efe-464e-9c38-c8008504a54e9240622d-5efe-464e-9c38-c8008504a54e
HiveServer2-Handler-Pool: Thread-3389141]: SharedWorkOptimizer start {code}
{code:java}
at java.util.HashMap$HashIterator.<init>([email protected]/HashMap.java:1593) at
java.util.HashMap$KeyIterator.<init>([email protected]/HashMap.java:1626) at
java.util.HashMap$KeySet.iterator([email protected]/HashMap.java:991) at
java.util.HashSet.iterator([email protected]/HashSet.java:182) at
org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.findDescendantWorkOperators(SharedWorkOptimizer.java:2081)
at
org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.validPreConditions(SharedWorkOptimizer.java:1944)
at
org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.sharedWorkOptimization(SharedWorkOptimizer.java:471)
at
org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.runSharedWorkOptimization(SharedWorkOptimizer.java:287)
at
org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.transform(SharedWorkOptimizer.java:184)
at
org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:252)
at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:186)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.compilePlan(SemanticAnalyzer.java:13247)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13472)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:467)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:360)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:188)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:360)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:225) at
org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:110) at
org.apache.hadoop.hive.ql.Driver.compile(Driver.java:500) at
org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:452) at
org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:416) at
org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:410) at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:127)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:235)
{code}
Attaching [^swo_hang.q] test query to reproduce the issue. As noted in the
comments in the file the test requires hive.auto.convert.join=false.
At this time it is uncertain what the exact conditions are for the issue to
occur as the query needs to be slightly complex (3-way join with additional
conditions, group by etc.) and may depend on the input table metadata.
> Queries can get stuck compiling during SharedWorkOptimizer
> ----------------------------------------------------------
>
> Key: HIVE-30080
> URL: https://issues.apache.org/jira/browse/HIVE-30080
> Project: Hive
> Issue Type: Bug
> Affects Versions: 4.2.1
> Reporter: Urmas Tamassy
> Priority: Major
> Labels: compilation
> Attachments: swo_hang.q
>
>
> As noted in the title, certain types of queries can get stuck during Shared
> work optimizer step during compilation as indicated in HiveServer2 logs and
> in the thread dumps taken while the query is stuck:
> {code:java}
> INFO org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer:
> [9240622d-5efe-464e-9c38-c8008504a54e9240622d-5efe-464e-9c38-c8008504a54e
> HiveServer2-Handler-Pool: Thread-3389141]: SharedWorkOptimizer start {code}
> {code:java}
> at java.util.HashMap$HashIterator.<init>([email protected]/HashMap.java:1593)
> at java.util.HashMap$KeyIterator.<init>([email protected]/HashMap.java:1626)
> at java.util.HashMap$KeySet.iterator([email protected]/HashMap.java:991) at
> java.util.HashSet.iterator([email protected]/HashSet.java:182) at
> org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.findDescendantWorkOperators(SharedWorkOptimizer.java:2081)
> at
> org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.validPreConditions(SharedWorkOptimizer.java:1944)
> at
> org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.sharedWorkOptimization(SharedWorkOptimizer.java:471)
> at
> org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.runSharedWorkOptimization(SharedWorkOptimizer.java:287)
> at
> org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.transform(SharedWorkOptimizer.java:184)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:252)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:186)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.compilePlan(SemanticAnalyzer.java:13247)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13472)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:467)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:360)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:188)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:360)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:225) at
> org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:110) at
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:500) at
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:452) at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:416) at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:410) at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:127)
> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:235)
> {code}
> Attaching [^swo_hang.q] test query to reproduce the issue. As noted in the
> comments in the file the test requires hive.auto.convert.join=false.
> At this time it is uncertain what the exact conditions are for the issue to
> occur as the query needs to be slightly complex (3-way join with additional
> conditions, group by etc.) and may depend on the input table metadata.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
