GitHub user vanzin opened a pull request:
https://github.com/apache/spark/pull/18370
[SPARK-9825][yarn] Do not overwrite final Hadoop config entries.
When localizing the gateway config files in a YARN application, avoid
overwriting final configs by distributing the gateway files to a separate
directory, and explicitly loading them into the Hadoop config, instead
of placing those files before the cluster's files in the classpath.
The implementation is a little hacky, but mostly because there's no API
that tells you which are the files that are loaded by a YarnConfiguration
object; so the list of files was obtained empirically and is hardcoded
in the code.
Tested with existing unit tests, and by verifying the behavior in a YARN
cluster (final values are not overridden, non-final values are).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/spark SPARK-9825
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18370.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18370
----
commit 4c35cce59b6d52bba93f9805a0d5a7dd0be3f6ad
Author: Marcelo Vanzin <[email protected]>
Date: 2017-06-20T21:33:43Z
[SPARK-9825][yarn] Do not overwrite final Hadoop config entries.
When localizing the gateway config files in a YARN application, avoid
overwriting final configs by distributing the gateway files to a separate
directory, and explicitly loading them into the Hadoop config, instead
of placing those files before the cluster's files in the classpath.
The implementation is a little hacky, but mostly because there's no API
that tells you which are the files that are loaded by a YarnConfiguration
object; so the list of files was obtained empirically and is hardcoded
in the code.
Tested with existing unit tests, and by verifying the behavior in a YARN
cluster (final values are not overridden, non-final values are).
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
