GitHub user vanzin opened a pull request:
https://github.com/apache/spark/pull/18380
[SPARK-16019][yarn] Use separate RM poll interval when starting client AM.
Currently the code monitoring the launch of the client AM uses the value of
spark.yarn.report.interval as the interval for polling the RM; if someone
has that value to a really large interval, it would take that long to detect
that the client AM has started, which is not expected.
Instead, have a separate config for the interval to use when the client AM
is
starting. The other config is still used in cluster mode, and to detect the
status of the client AM after it is already running.
Tested by running client and cluster mode apps with a modified value of
spark.yarn.report.interval, verifying client AM launch is detected before
that interval elapses.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/spark SPARK-16019
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18380.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18380
----
commit a117dd37362d01b724b49974fa77fba85362ee12
Author: Marcelo Vanzin <[email protected]>
Date: 2017-06-21T20:05:58Z
[SPARK-16019][yarn] Use separate RM poll interval when starting client AM.
Currently the code monitoring the launch of the client AM uses the value of
spark.yarn.report.interval as the interval for polling the RM; if someone
has that value to a really large interval, it would take that long to detect
that the client AM has started, which is not expected.
Instead, have a separate config for the interval to use when the client AM
is
starting. The other config is still used in cluster mode, and to detect the
status of the client AM after it is already running.
Tested by running client and cluster mode apps with a modified value of
spark.yarn.report.interval, verifying client AM launch is detected before
that interval elapses.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
