Yes, if you are asking about developing a new priority queue job scheduling feature and not just about how job scheduling currently works in Spark, the that's a dev list issue. The current job scheduling priority is at the granularity of pools containing jobs, not the jobs themselves; so if you require strictly job-level priority queuing, that would require a new development effort -- and one that I expect will involve a lot of tricky corner cases.
Sorry for misreading the nature of your initial inquiry. On Sun, Jan 11, 2015 at 7:36 AM, Alessandro Baretta <alexbare...@gmail.com> wrote: > Cody, > > While I might be able to improve the scheduling of my jobs by using a few > different pools with weights equal to, say, 1, 1e3 and 1e6, effectively > getting a small handful of priority classes. Still, this is really not > quite what I am describing. This is why my original post was on the dev > list. Let me then ask if there is any interest in having priority queue job > scheduling in Spark. This is something I might be able to pull off. > > Alex > > On Sun, Jan 11, 2015 at 6:21 AM, Cody Koeninger <c...@koeninger.org> > wrote: > >> If you set up a number of pools equal to the number of different priority >> levels you want, make the relative weights of those pools very different, >> and submit a job to the pool representing its priority, I think youll get >> behavior equivalent to a priority queue. Try it and see. >> >> If I'm misunderstandng what youre trying to do, then I don't know. >> >> >> On Sunday, January 11, 2015, Alessandro Baretta <alexbare...@gmail.com> >> wrote: >> >>> Cody, >>> >>> Maybe I'm not getting this, but it doesn't look like this page is >>> describing a priority queue scheduling policy. What this section discusses >>> is how resources are shared between queues. A weight-1000 pool will get >>> 1000 times more resources allocated to it than a priority 1 queue. Great, >>> but not what I want. I want to be able to define an Ordering on make my >>> tasks representing their priority, and have Spark allocate all resources to >>> the job that has the highest priority. >>> >>> Alex >>> >>> On Sat, Jan 10, 2015 at 10:11 PM, Cody Koeninger <c...@koeninger.org> >>> wrote: >>> >>>> >>>> http://spark.apache.org/docs/latest/job-scheduling.html#configuring-pool-properties >>>> >>>> "Setting a high weight such as 1000 also makes it possible to >>>> implement *priority* between pools—in essence, the weight-1000 pool >>>> will always get to launch tasks first whenever it has jobs active." >>>> >>>> On Sat, Jan 10, 2015 at 11:57 PM, Alessandro Baretta < >>>> alexbare...@gmail.com> wrote: >>>> >>>>> Mark, >>>>> >>>>> Thanks, but I don't see how this documentation solves my problem. You >>>>> are referring me to documentation of fair scheduling; whereas, I am asking >>>>> about as unfair a scheduling policy as can be: a priority queue. >>>>> >>>>> Alex >>>>> >>>>> On Sat, Jan 10, 2015 at 5:00 PM, Mark Hamstra <m...@clearstorydata.com >>>>> > wrote: >>>>> >>>>>> -dev, +user >>>>>> >>>>>> http://spark.apache.org/docs/latest/job-scheduling.html >>>>>> >>>>>> >>>>>> On Sat, Jan 10, 2015 at 4:40 PM, Alessandro Baretta < >>>>>> alexbare...@gmail.com> wrote: >>>>>> >>>>>>> Is it possible to specify a priority level for a job, such that the >>>>>>> active >>>>>>> jobs might be scheduled in order of priority? >>>>>>> >>>>>>> Alex >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >
