Yes. Fair schedulwr only helps concurrency within an application. With multiple shells you'd either need something like Yarn/Mesos or careful math on resources as you said
Sent on the new Sprint Network from my Samsung Galaxy S®4. <div>-------- Original message --------</div><div>From: Arun Patel <arunp.bigd...@gmail.com> </div><div>Date:04/22/2015 6:28 AM (GMT-05:00) </div><div>To: user <user@spark.apache.org> </div><div>Subject: Scheduling across applications - Need suggestion </div><div> </div>I believe we can use the properties like --executor-memory --total-executor-cores to configure the resources allocated for each application. But, in a multi user environment, shells and applications are being submitted by multiple users at the same time. All users are requesting resources with different properties. At times, some users are not getting resources of the cluster. How to control resource usage in this case? Please share any best practices followed. As per my understanding, Fair scheduler can used for scheduling tasks within an application but not across multiple applications. Is this correct? Regards, Arun
