Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/3571#issuecomment-67722678
Hi @jacek-lewandowski ,
I think that's an improvement (assuming that you'd remove all the code
dealing with the separate ssl.conf, which is still there), but there's still
something I'm no seeing. How are the SSL properties propagated from the
launcher to the executors?
If I set different options in my launcher that do not match those in the
worker's configuration, it seems that things will fail with your new code.
The reason why I pointed out the Yarn code is because that's exactly what
it does. The properties you set *in your launch process* are used by the
executors. You don't need to configure anything on the edge nodes. In
standalone mode, of course you need to configure the worker daemons themselves,
but you don't need to manage *client* configuration in every worker node.
Anyway, it seems you're not planning to handle Yarn in this change at all
(I didn't notice any changes there?), so we can always treat that as a future
enhancement. I'll let other reviewers chime in on whether they think this
limitation is acceptable in standalone mode. If I misunderstood something about
your code, apologies, and please correct me!
Looking at just the last commit, the only comment I had was the somehow
confouding use of `SparkModule` and `namespace`. It doesn't seem like the
`SparkModule` type is adding much functionality here... also, I'd rather keep
the config for master/worker and the config for the history server in separate
namespaces, although that's not a big deal (since you can use different config
files to achieve the same thing).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
