Yup, this page describes the two levels of scheduling (within and across applications): http://spark.incubator.apache.org/docs/latest/job-scheduling.html
On Jan 17, 2014, at 12:21 PM, Reynold Xin <[email protected]> wrote: > It does. > > There are two scheduling levels here. > > The first level is what the cluster manager does. The standalone cluster > manager for Spark only supports FIFO at the moment at the level of > applications. > > Regarding Spark itself. Within a single Spark application, both FIFO and > fair scheduling are supported, regardless of what your cluster manager is > > > > On Fri, Jan 17, 2014 at 12:17 PM, Evan Chan <[email protected]> wrote: > >> What is the reason that standalone mode doesn't support the fair scheduler? >> Does that mean that Mesos coarse mode also doesn't support the fair >> scheduler? >> >> >> On Tue, Jan 14, 2014 at 8:10 PM, Matei Zaharia <[email protected] >>> wrote: >> >>> This is true for now, we didn’t want to replicate those systems. But it >>> may change if we see demand for fair scheduling in our standalone cluster >>> manager. >>> >>> Matei >>> >>> On Jan 14, 2014, at 6:32 PM, Xia, Junluan <[email protected]> wrote: >>> >>>> Yes, Spark depends on Yarn or Mesos for application level scheduling. >>>> >>>> -----Original Message----- >>>> From: Nan Zhu [mailto:[email protected]] >>>> Sent: Tuesday, January 14, 2014 9:43 PM >>>> To: [email protected] >>>> Subject: Re: Is there any plan to develop an application level fair >>> scheduler? >>>> >>>> Hi, Junluan, >>>> >>>> Thank you for the reply >>>> >>>> but for the long-term plan, Spark will depend on Yarn and Mesos for >>> application level scheduling in the coming versions? >>>> >>>> Best, >>>> >>>> -- >>>> Nan Zhu >>>> >>>> >>>> On Tuesday, January 14, 2014 at 12:56 AM, Xia, Junluan wrote: >>>> >>>>> Are you sure that you must deploy spark in standalone mode?(it >>> currently only support FIFO) >>>>> >>>>> If you could setup Spark on Yarn or Mesos, then it has supported Fair >>> scheduler in application level. >>>>> >>>>> -----Original Message----- >>>>> From: Nan Zhu [mailto:[email protected]] >>>>> Sent: Tuesday, January 14, 2014 10:13 AM >>>>> To: [email protected] (mailto: >>> [email protected]) >>>>> Subject: Is there any plan to develop an application level fair >>> scheduler? >>>>> >>>>> Hi, All >>>>> >>>>> Is there any plan to develop an application level fair scheduler? >>>>> >>>>> I think it will have more value than a fair scheduler within the >>> application (actually I didn’t understand why we want to fairly share the >>> resource among jobs within the application, in usual, users submit >>> different applications, not jobs)… >>>>> >>>>> Best, >>>>> >>>>> -- >>>>> Nan Zhu >>>>> >>>>> >>>> >>>> >>> >>> >> >> >> -- >> -- >> Evan Chan >> Staff Engineer >> [email protected] | >> >> <http://www.ooyala.com/> >> <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala>< >> http://www.twitter.com/ooyala> >>
