Olasoji created SPARK-16019:
-------------------------------
Summary: Eliminate unexpected delay during spark on yarn job launch
Key: SPARK-16019
URL: https://issues.apache.org/jira/browse/SPARK-16019
Project: Spark
Issue Type: Improvement
Components: YARN
Reporter: Olasoji
Currently when launching a job in yarn mode, there will be an added delay of
about "spark.yarn.report.interval" seconds before launch. By default this
parameter is set to 1 second, however if a user increases this for whatever
reason, an unexpected startup delay is introduced.
The "waitForApplication" function called during job submission and launch will
eventually call monitorApplication and so sleep for the configured interval
time before checking the state of the job and proceeding with job startup.
One solution would be to add an "interval" argument to the monitorApplication
function. This added flexibility would allow callers who don't need to wait to
set a lower wait interval or none.
Patch to follow soon
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
