[
https://issues.apache.org/jira/browse/SPARK-16019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Vanzin resolved SPARK-16019.
------------------------------------
Resolution: Fixed
Assignee: Marcelo Vanzin
Fix Version/s: 2.3.0
> Eliminate unexpected delay during spark on yarn job launch
> ----------------------------------------------------------
>
> Key: SPARK-16019
> URL: https://issues.apache.org/jira/browse/SPARK-16019
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Reporter: Olasoji
> Assignee: Marcelo Vanzin
> Priority: Minor
> Fix For: 2.3.0
>
>
> Currently when launching a job in yarn mode, there will be an added delay of
> about "spark.yarn.report.interval" seconds before launch. By default this
> parameter is set to 1 second, however if a user increases this for whatever
> reason, an unexpected startup delay is introduced.
> The "waitForApplication" function called during job submission and launch
> will eventually call monitorApplication and so sleep for the configured
> interval time before checking the state of the job and proceeding with job
> startup.
> One solution would be to add an "interval" argument to the monitorApplication
> function. This added flexibility would allow callers who don't need to wait
> to set a lower wait interval or none.
> Patch to follow soon
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
