akirillov edited a comment on issue #22071: [SPARK-25088][CORE][MESOS][DOCS] Update Rest Server docs & defaults. URL: https://github.com/apache/spark/pull/22071#issuecomment-465826399 Hi @squito, if I may, I have a few comments to share regarding this change and probably you can help me to understand it in better details. From [Spark Security Doc page](https://spark.apache.org/docs/latest/security.html#standalone-mode-only) it looks like `spark.master.rest.enabled` property belongs to Spark Standalone Master only. IMO, It makes sense, because it's the server side who is responsible for enabling and disabling REST endpoint according to the [source code](https://github.com/apache/spark/blob/v2.4.0/core/src/main/scala/org/apache/spark/deploy/master/Master.scala#L154-L157). Now, from the [comment in the code](https://github.com/apache/spark/blob/v2.4.0/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L176-L180) of `SparkSubmit.scala` it looks like REST-based gateway introduced in Spark 1.3 is the default way of communication with Spark Standalone Master with a fallback to legacy RPC. And here's the problem: now `spark.master.rest.enabled` became a client-side property which defaults to `false` for both Mesos and Spark Standalone and breaks **all** spark-submit calls to Mesos Dispatcher if this property is not provided explicitly. Now users have to provide it every time for Mesos cluster mode while REST is the only mode supported by the dispatcher. Looks like a breaking change. Another note is regarding `spark.authenticate.secret` assertion in [MesosClusterDispatcher](https://github.com/apache/spark/blob/v2.4.0/resource-managers/mesos/src/main/scala/org/apache/spark/deploy/mesos/MesosClusterDispatcher.scala#L54-L60). Dispatcher will be unable to start if `spark.authenticate.secret` is set. It looks like not so great behaviour if it doesn't allow to run a program at all if some flag is set. Which also looks like a breaking change. Suggestions: - it looks natural for `spark.master.rest.enabled` to be handled in SparkSubmit _only_ when used with Spark Standalone Master. It can be completely ignored with Mesos Dispatcher because REST interface is the only endpoint it exposes - defaulting Spark Standalone Master's `spark.master.rest.enabled` to `false` switches the communication to 'legacy' RPC protocol which is marked as a fallback-only since Spark 1.3. It's not clear how this protocol is going to be supported in the future. It looks safer to document _how to disable_ REST endpoint for security purposes instead while preserving REST communications by default - instead of throwing an assertion error - log a `WARN` message in Mesos Dispatcher if `spark.authenticate.secret` is provided in configuration Please let me know what you think. Thanks.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
