Mark & Ayan, thanks for the inputs. *Is there anyway can we setup the scheduler mode in Spark Cluster level besides application (SC level).*
Currently in YARN is in FAIR mode and manually we ensure that Spark Application also in FAIR mode however noticed that Applications are not releasing the resources as soon as the tasks are done when we mention Dynamic allocation = true and did not specify any explicit Executor allocation. At this moment, we are specifying the Min and Max Executor allocation at Spark Application level in order to ensure that all of our ETL Spark Applications can run parallel without any resource issues. It would be great if you could throw more insight on the how to set the preemption within yarn and Spark. Thanks & Regards, Gokula Krishnan* (Gokul)* On Thu, Jul 20, 2017 at 6:46 PM, ayan guha <guha.a...@gmail.com> wrote: > Hi > > As Mark said, scheduler mode works within application ie within a Spark > Session and Spark context. This is also clear if you think where you set > the configuration - in a Spark Config which used to build a context. > > If you are using Yarn as resource manager, however, you can set YARN with > fair scheduler. If you do so, both of your applications will get "Fair" > treatment from Yarn, ie get resources in round robin manner. If you want > your App A to give up resources while using them, you need to set > preemption within Yarn and priority of applications so that preemption can > kick in. > > HTH... > > Best, Ayan > > On Fri, Jul 21, 2017 at 7:11 AM, Mark Hamstra <m...@clearstorydata.com> > wrote: > >> The fair scheduler doesn't have anything to do with reallocating resource >> across Applications. >> >> https://spark.apache.org/docs/latest/job-scheduling.html#sch >> eduling-across-applications >> https://spark.apache.org/docs/latest/job-scheduling.html#sch >> eduling-within-an-application >> >> On Thu, Jul 20, 2017 at 2:02 PM, Gokula Krishnan D <email2...@gmail.com> >> wrote: >> >>> Mark, Thanks for the response. >>> >>> Let me rephrase my statements. >>> >>> "I am submitting a Spark application(*Application*#A) with >>> scheduler.mode as FAIR and dynamicallocation=true and it got all the >>> available executors. >>> >>> In the meantime, submitting another Spark Application (*Application* >>> # B) with the scheduler.mode as FAIR and dynamicallocation=true but it got >>> only one executor. " >>> >>> Thanks & Regards, >>> Gokula Krishnan* (Gokul)* >>> >>> On Thu, Jul 20, 2017 at 4:56 PM, Mark Hamstra <m...@clearstorydata.com> >>> wrote: >>> >>>> First, Executors are not allocated to Jobs, but rather to Applications. >>>> If you run multiple Jobs within a single Application, then each of the >>>> Tasks associated with Stages of those Jobs has the potential to run on any >>>> of the Application's Executors. Second, once a Task starts running on an >>>> Executor, it has to complete before another Task can be scheduled using the >>>> prior Task's resources -- the fair scheduler is not preemptive of running >>>> Tasks. >>>> >>>> On Thu, Jul 20, 2017 at 1:45 PM, Gokula Krishnan D <email2...@gmail.com >>>> > wrote: >>>> >>>>> Hello All, >>>>> >>>>> We are having cluster with 50 Executors each with 4 Cores so can avail >>>>> max. 200 Executors. >>>>> >>>>> I am submitting a Spark application(JOB A) with scheduler.mode as FAIR >>>>> and dynamicallocation=true and it got all the available executors. >>>>> >>>>> In the meantime, submitting another Spark Application (JOB B) with the >>>>> scheduler.mode as FAIR and dynamicallocation=true but it got only one >>>>> executor. >>>>> >>>>> Normally this situation occurs when any of the JOB runs with the >>>>> Scheduler.mode= FIFO. >>>>> >>>>> 1) Have your ever faced this issue if so how to overcome this?. >>>>> >>>>> I was in the impression that as soon as I submit the JOB B the Spark >>>>> Scheduler should distribute/release few resources from the JOB A and share >>>>> it with the JOB A in the Round Robin fashion?. >>>>> >>>>> Appreciate your response !!!. >>>>> >>>>> >>>>> Thanks & Regards, >>>>> Gokula Krishnan* (Gokul)* >>>>> >>>> >>>> >>> >> > > > -- > Best Regards, > Ayan Guha >
