[
https://issues.apache.org/jira/browse/HIVE-15386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15741126#comment-15741126
]
zhihai xu commented on HIVE-15386:
----------------------------------
thanks for the review [~lirui]! Submit time is when the spark job is submitted
to spark. Start time is when the job is running.
there may be some delay between submit time and start time. for example, wait
to get resource to run the job.
PerfLogEnd for SPARK_SUBMIT_TO_RUNNING is called when the spark job enters
running state
{code}
perfLogger.PerfLogEnd(CLASS_NAME,
PerfLogger.SPARK_SUBMIT_TO_RUNNING);
{code}
the end time for SPARK_SUBMIT_TO_RUNNING is the same as the start time.
Looks like the org.apache.hadoop.hive.ql.hooks.ATSHook uses Utilities to get MR
tasks and Tez task currently.
{code}
int numMrJobs = Utilities.getMRTasks(plan.getRootTasks()).size();
int numTezJobs = Utilities.getTezTasks(plan.getRootTasks()).size();
{code}
> Expose Spark task counts and stage Ids information in SparkTask from
> SparkJobMonitor
> ------------------------------------------------------------------------------------
>
> Key: HIVE-15386
> URL: https://issues.apache.org/jira/browse/HIVE-15386
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Affects Versions: 2.2.0
> Reporter: zhihai xu
> Assignee: zhihai xu
> Attachments: HIVE-15386.000.patch, HIVE-15386.001.patch
>
>
> Expose Spark task counts and stage Ids information in SparkTask from
> SparkJobMonitor. So these information can be used by hive hook to monitor
> spark jobs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
