Hi, We have a spark cluster and we wanted to add some security for it. I was looking at the documentation (in http://spark.apache.org/docs/latest/security.html) and had some questions.
1. Do all executors listen by the same blockManager port? For example, in yarn there are multiple executors per node, do they all listen to the same port? 2. Are ports defined in earlier version (e.g. http://spark.apache.org/docs/1.6.1/security.html) and removed in the latest (such as spark.executor.port and spark.fileserver.port) gone and can be blocked? 3. If I define multiple workers per node in spark standalone mode, how do I set the different ports for each worker (there is only one spark.worker.ui.port / SPARK_WORKER_WEBUI_PORT definition. Do I have to start each worker separately to configure a port?) The same is true for the worker port (SPARK_WORKER_PORT) 4. Is it possible to encrypt the logs instead of just limiting with permissions the log directory? 5. Is the communication between the servers encrypted (e.g. using ssh?) 6. Are there any additional best practices beyond what is written in the documentation? Thanks, Assaf.