First a bit of context: We use Spark on a platform where each user start workers as needed. This has the advantage that all permission management is handled by the OS, so the users can only read files they have permission to.
To do this, we have some utility that does the following: - start a master - start worker managers on a number of servers - "submit" the Spark driver program - the driver then talks to the master, tell it how many executors it needs - the master tell the worker nodes to start executors and talk to the driver - the executors are started >From here on, the master doesn't do much, neither do the process manager on the worker nodes. What I would like to do is simplify this to: - Start the driver program - Start executors on a number of servers, telling them where to find the driver - The executors connect directly to the driver Is there a way I could do this without the master and worker managers? Thanks! -- Mathieu Longtin 1-514-803-8977
