Hi, I'm doing some work to get Apache Spark running in Aurora and it seems to work reasonably well without many changes to Spark, the only issue I'm running into is launching it over many instances.
Spark runs in a driver executor model, where the driver coordinates works on the executors. The problem I have is that I want to launch the executors and drivers independently, ie I want to have 10 executors and 1 driver. I can accomplish this by having 2 jobs, a driver and an executor job, but launching this seems a bit complicated to orchestrate. Another option would be to declare the job with 2 tasks, have the driver run on shard 0 and executors on the rest. Has anyone had any experience with running similar systems in Aurora? I imagine Heron must have to do something similar, launching the topology master and workers. Chris
