Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/2739#issuecomment-60275209
Hi @jacek-lewandowski,
Now that I finally noticed you built this on top of branch-1.1, some of the
choices you made make a lot more sense. (I always assume people are working on
master, since it's generally preferable to add new features to master first.)
One huge difference in master, which lead to a lot of my comments, is
SPARK-2098. That fix added the ability of all daemons - including Master and
Worker - to read the spark-defaults.conf file. So, if you build on top of that,
you need zero code dealing with loading config data, and can rely on SparkConf
for everything. Then, you could have something like:
class SSLOptions(conf: SparkConf, module: String)
That would load options like this:
sslEnabled = conf.getOption(s"spark.$module.ssl.enabled")
.orElse(conf.getOption("spark.ssl.enabled"))
.getOrElse(false)
Then you have module-specific configuration and a global fallback. What do
you think?
On the subject of distributing the configuration, I think it's sort of ok
to rely on that, for the time being, for standalone mode. Long term, it would
be better to allow each job to be able to distribute its own configuration, so
that it's easy for admins and users to use different certificates for the
daemons and for the jobs, for example.
On Yarn, I still believe we should not have this requirement - since when
using Spark-on-Yarn, Spark is kind of a client-side thing and shouldn't require
any changes in the cluster . The needed files should be distributed
automatically by Spark and made available to executors. That should be doable
by disabling certificate validation (so that the hostnames don't matter) or
using wildcard certificates (assuming everything is in the same sub-domain). If
that's not enough to cover all user cases, we can leave other enhancements for
later.
I'm not familiar enough with mesos to be able to suggest anything.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
