Hi Chris,
we are spawning an internal batch processing framework on Aurora, consisting of a single master and multiple workers. We opted for the 2 jobs idea. Main advantage I see with this approach that you actually keep separate things separate without having to teach all external systems (service discovery, load balancer, your monitoring solution, etc) that the first instance is different. Best Regards, Stephan ? ________________________________ From: Chris Bannister <[email protected]> Sent: Tuesday, December 22, 2015 2:47 PM To: [email protected] Subject: Launching master/slave jobs in Auora Hi, I'm doing some work to get Apache Spark running in Aurora and it seems to work reasonably well without many changes to Spark, the only issue I'm running into is launching it over many instances. Spark runs in a driver executor model, where the driver coordinates works on the executors. The problem I have is that I want to launch the executors and drivers independently, ie I want to have 10 executors and 1 driver. I can accomplish this by having 2 jobs, a driver and an executor job, but launching this seems a bit complicated to orchestrate. Another option would be to declare the job with 2 tasks, have the driver run on shard 0 and executors on the rest. Has anyone had any experience with running similar systems in Aurora? I imagine Heron must have to do something similar, launching the topology master and workers. Chris
