If I'm understanding you correctly, then you are correct that the fair
scheduler doesn't currently do everything that you want to achieve. Fair
scheduler pools currently can be configured with a minimum number of cores
that they will need before accepting Tasks, but there isn't a way to
restrict
Mark,
I'm trying to configure spark cluster to share resources between two pools.
I can do that by assigning minimal shares (it works fine), but that means
specific amount of cores is going to be wasted by just being ready to run
anything. While that's better, than nothing, I'd like to specify
It's 2 -- and it's pretty hard to point to a line of code, a method, or
even a class since the scheduling of Tasks involves a pretty complex
interaction of several Spark components -- mostly the DAGScheduler,
TaskScheduler/TaskSchedulerImpl, TaskSetManager, Schedulable and Pool, as
well as the
Hi,
I'm trying to understand how this thing works underneath. Let's say I have
two types of jobs - high important, that might use small amount of cores
and has to be run pretty fast. And less important, but greedy - uses as
many cores as available. So, the idea is to use two corresponding pools.
