[
https://issues.apache.org/jira/browse/HIVE-15662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated HIVE-15662:
-----------------------------
Description:
Check startTime in SparkTask to make sure startTime is not less than
submitTime. We saw a corner case when the sparkTask is finished in less than 1
second, the startTime may not be set because RemoteSparkJobMonitor will sleep
for 1 second then check the state, in this case, right after sleep for one
second, the spark job is already completed. one example query with 3 spark
tasks, the second one finished quickly for around 1 second:
{code}
SparkTask1:
"finishTime":1484638391978
"submitTime":1484638385933
"startTime": 1484638386973
SparkTask2:
"finishTime":1484638393019
"submitTime":1484638391979
"startTime": 1484638386973
SparkTask3:
"finishTime":1484638432123
"submitTime":1484638393020
"startTime": 1484638394057
{code}
was:
Check startTime in SparkTask to make sure startTime is not less than
submitTime. We saw a corner case when the sparkTask is finished in less than 1
second, the startTime may not be set because RemoteSparkJobMonitor will sleep
for 1 second then check the state, in this case, right after sleep for one
second, the spark job is already completed.
> check startTime in SparkTask to make sure startTime is not less than
> submitTime
> -------------------------------------------------------------------------------
>
> Key: HIVE-15662
> URL: https://issues.apache.org/jira/browse/HIVE-15662
> Project: Hive
> Issue Type: Bug
> Reporter: zhihai xu
> Assignee: zhihai xu
> Priority: Minor
> Attachments: HIVE-15662.000.patch
>
>
> Check startTime in SparkTask to make sure startTime is not less than
> submitTime. We saw a corner case when the sparkTask is finished in less than
> 1 second, the startTime may not be set because RemoteSparkJobMonitor will
> sleep for 1 second then check the state, in this case, right after sleep for
> one second, the spark job is already completed. one example query with 3
> spark tasks, the second one finished quickly for around 1 second:
> {code}
> SparkTask1:
> "finishTime":1484638391978
> "submitTime":1484638385933
> "startTime": 1484638386973
> SparkTask2:
> "finishTime":1484638393019
> "submitTime":1484638391979
> "startTime": 1484638386973
> SparkTask3:
> "finishTime":1484638432123
> "submitTime":1484638393020
> "startTime": 1484638394057
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
