akirillov commented on issue #22071: [SPARK-25088][CORE][MESOS][DOCS] Update Rest Server docs & defaults. URL: https://github.com/apache/spark/pull/22071#issuecomment-465826399 Hi @squito, if I may, I have a few comments to share regarding this change and probably you can help me to understand it in better details. From [Spark Security Doc page](https://spark.apache.org/docs/latest/security.html#standalone-mode-only) it looks like `spark.master.rest.enabled` property belongs to Spark Standalone Master only. IMO, It makes sense, because it's the server side who is responsible for enabling and disabling REST endpoint according to the [source code](https://github.com/apache/spark/blob/v2.4.0/core/src/main/scala/org/apache/spark/deploy/master/Master.scala#L154-L157). Now, from the [comment in the code] (https://github.com/apache/spark/blob/v2.4.0/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L176-L180) of `SparkSubmit.scala` it looks like REST-based gateway introduced in Spark 1.3 is the default way of communication with Spark Standalone Master with a fallback to legacy RPC. And here's the problem: now `spark.master.rest.enabled` became a client-side property, not Spark Standalone Master property. So in order to communicate with Master, a user have to provide `spark.master.rest.enabled` with each job she submits because without it, the submission will fail. This is a pretty breaking change. What is the reason to fail the start of [MesosClusterDispatcher](https://github.com/apache/spark/blob/v2.4.0/resource-managers/mesos/src/main/scala/org/apache/spark/deploy/mesos/MesosClusterDispatcher.scala#L54-L60) if `spark.authenticate.secret` is set? Mesos uses different mechanisms for authentication and there's no need to fail the Dispatcher - this property can be just ignored or logged at most. TL;DR It looks like `spark.master.rest.enabled` should reside only in Spark Standalone Master code and shouldn't leak into the client side of `SparkSubmit`. Also, it can be completely ignored with Mesos Dispatcher, because REST interface is the only endpoint it exposes. Please let me know what you think. Thanks.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
