Github user redbaron commented on the pull request:
https://github.com/apache/spark/pull/4142#issuecomment-70942000
IMHO Spark should mimic what Hadoop MR is doing, which is building a
Configuration object and then serializing into xml file and then shipping this
xml to workers. It allows to make executors config to be entirely client-driven.
Copying files verbatim from the directory leads to not being able to
override hadoop config params via "-D<param>=<value>" switch (don't know if it
is supported by spark, but if it is not, it should be) or can cause surprises
if config is influenced by other means (for instance different defaults for
client side version of libs).
Both original patch and this one suffer from more serious problem (not
saying that it should prevent this patch from being merged or anything like
that, just sharing my experience): different parts of spark may request
different configs to be shipped to AM, like Spark SQL requires hive-site.xml in
a similar way.
I am not familiar with spark internals, but if SparkContext instance is
available to submitting client, then it can "query" it to get list of configs
to push to AM, then users of SparkContext (like SQLContext) can register
additional configs to be transferred at job deploy time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
