Hi Karthik, The resourceOffer() method is invoked from a class implementing the SchedulerBackend interface; in the case of a standalone cluster, it's invoked from a CoarseGrainedSchedulerBackend (in the makeOffers() method). If you look in TaskSchedulerImpl.submitTasks(), it calls backend.reviveOffers() at the end -- which signals to the SchedulerBackend that it should offer some resources. This somewhat confusing interface exists for historical reasons: Spark originally used only Mesos for scheduling, which relies on resource offers for scheduling, and so standalone mode was done in a similar way to maximize code re-use.
The scheduler/local folder contains code specific to running in local mode. With local mode, all code runs in a single JVM (as opposed to having one JVM for the driver, one for the cluster master, and one for each worker). This mode is typically used for testing. -Kay On Wed, Jul 16, 2014 at 8:52 AM, rapelly kartheek <kartheek.m...@gmail.com> wrote: > Hi, > > I am trying to understand how the resource allocation happens in spark. I > understand the resourceOffer method in taskScheduler. This method takes > care of locality factor while allocating the resources. This resourceOffer > method gets invoked by the corresponding cluster manager. > > I am working on stand-alone spark cluster. But I am not able to locate > starting point of the resource allocation. I want to understand from end to > end how exactly resource allocations happens, given a new Application to > the Spark. Also, apart from mesos, there is a folder called Local in the > Scheduler. Can someone tell me which all files should I look into to > understand the resource allocation in stand-alone spark cluster and what is > this "Local" for? > > Thanks in advance!! > Karthik. >
