[
https://issues.apache.org/jira/browse/HIVE-24467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
guojh updated HIVE-24467:
-------------------------
Description:
When hive execute jobs in parallel(control by “hive.exec.parallel” parameter),
ConditionalTasks remove the tasks that not selected in parallel, because there
are thread safety issues, some task may not remove from the dependent task
tree. This is a very serious bug, which causes some stage task not trigger
execution.
In our production cluster, the query run three conditional task in parallel,
{code:java}
// code placeholder
2020-12-01T22:18:13,764 INFO [HiveServer2-Background-Pool: Thread-233156]
ql.Driver: Starting task [Stage-38:MAPRED] in parallel
2020-12-01T22:19:01,766 INFO [HiveServer2-Background-Pool: Thread-233156]
ql.Driver: Starting task [Stage-25:CONDITIONAL] in parallel
2020-12-01T22:19:01,766 INFO [HiveServer2-Background-Pool: Thread-233156]
ql.Driver: Starting task [Stage-33:CONDITIONAL] in parallel
2020-12-01T22:19:01,766 INFO [HiveServer2-Background-Pool: Thread-233156]
ql.Driver: Starting task [Stage-37:CONDITIONAL] in parallel
2020-12-01T22:20:17,774 INFO [HiveServer2-Background-Pool: Thread-233156]
ql.Driver: Task:Stage-3:MAPRED false false falseParent:Stage-2:MAPRED
status:trueParent:Stage-12:MAPRED status:trueParent:Stage-16:MAPRED
status:trueParent:Stage-20:MAPRED status:trueParent:Stage-31:MAPRED status:false
2020-12-01T22:20:17,774 ERROR [HiveServer2-Background-Pool: Thread-233156]
ql.Driver: Miss stage: Stage-3for queryid
hive_20201201221740_805852c0-60a7-4141-96e9-196f83b2705e
2020-12-01T22:20:17,774 ERROR [HiveServer2-Background-Pool: Thread-233156]
ql.Driver: Miss stage for queryid
hive_20201201221740_805852c0-60a7-4141-96e9-196f83b2705e : FAILED: Some Execute
Stage miss error
{code}
was:When hive execute jobs in parallel(control by “hive.exec.parallel”
parameter), ConditionalTasks remove the tasks that not selected in parallel,
because there are thread safety issues, some task may not remove from the
dependent task tree. This is a very serious bug, which causes some stage task
not trigger execution.
> ConditionalTask remove tasks that not selected exists thread safety problem
> ---------------------------------------------------------------------------
>
> Key: HIVE-24467
> URL: https://issues.apache.org/jira/browse/HIVE-24467
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Affects Versions: 2.3.4
> Reporter: guojh
> Priority: Major
>
> When hive execute jobs in parallel(control by “hive.exec.parallel”
> parameter), ConditionalTasks remove the tasks that not selected in parallel,
> because there are thread safety issues, some task may not remove from the
> dependent task tree. This is a very serious bug, which causes some stage task
> not trigger execution.
> In our production cluster, the query run three conditional task in parallel,
> {code:java}
> // code placeholder
> 2020-12-01T22:18:13,764 INFO [HiveServer2-Background-Pool: Thread-233156]
> ql.Driver: Starting task [Stage-38:MAPRED] in parallel
> 2020-12-01T22:19:01,766 INFO [HiveServer2-Background-Pool: Thread-233156]
> ql.Driver: Starting task [Stage-25:CONDITIONAL] in parallel
> 2020-12-01T22:19:01,766 INFO [HiveServer2-Background-Pool: Thread-233156]
> ql.Driver: Starting task [Stage-33:CONDITIONAL] in parallel
> 2020-12-01T22:19:01,766 INFO [HiveServer2-Background-Pool: Thread-233156]
> ql.Driver: Starting task [Stage-37:CONDITIONAL] in parallel
> 2020-12-01T22:20:17,774 INFO [HiveServer2-Background-Pool: Thread-233156]
> ql.Driver: Task:Stage-3:MAPRED false false falseParent:Stage-2:MAPRED
> status:trueParent:Stage-12:MAPRED status:trueParent:Stage-16:MAPRED
> status:trueParent:Stage-20:MAPRED status:trueParent:Stage-31:MAPRED
> status:false
> 2020-12-01T22:20:17,774 ERROR [HiveServer2-Background-Pool: Thread-233156]
> ql.Driver: Miss stage: Stage-3for queryid
> hive_20201201221740_805852c0-60a7-4141-96e9-196f83b2705e
> 2020-12-01T22:20:17,774 ERROR [HiveServer2-Background-Pool: Thread-233156]
> ql.Driver: Miss stage for queryid
> hive_20201201221740_805852c0-60a7-4141-96e9-196f83b2705e : FAILED: Some
> Execute Stage miss error
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
