--- On Wed, 10/3/10, Allen Wittenauer <[email protected]> wrote:
> From: Allen Wittenauer <[email protected]> > Subject: Re: Fair scheduler fairness question > To: [email protected] > Date: Wednesday, 10 March, 2010, 16:06 > On 3/10/10 7:38 AM, "Neo Anderson" > <[email protected]> > wrote: > > > I am learning how fair scheduler manage the jobs to > allow each job share > > resource over time; but don't know if my understanding > is correct or not. > > > > My scenario is that I have 3 data nodes and the > cluster is configured using > > fair scheduler with three pools launched (e.g. A, B, > C). Each pool is > > configured with > '<maxRunningJobs>1</maxRunningJobs>.' Now the > clients try to > > submit 4 jobs (e.g. submitjob()) to 3 differt pools. > For instance, > > > > the first job is submitted to pool A > > the second job is submitted to pool B > > the third job is submitted to pool B > > the fourth job is submitted to pool C > > > > So I expect that the first 3 jobs will occupy the free > slots (the slots should > > be fool now.) Then the fourth job is submitted. But > since the slots are full, > > and the fourth job should also have a slot executing > its job; therefore, the > > third job will be terminated (or kill) so that the > fourth job can be launched. > > > > Is my scenario correct? > > And if I am right, is there any key word searchable in > the log to observe such > > activites (the job that is being killed e.g. the third > job)? > > > A lot of it depends upon timing. If there is a long > enough pause between > job 1 and job 2, job 1 will take every slot available to > it. As job 1's > slots finish, job 2 and 4 would get those > slots. As job 2 finishes, job 3 > will get its slots. > > Slots are only freed by force if the scheduler you are > using has > pre-emption. I think some versions of fair share may > have it. Entire jobs > are never killed. > > > > At the moment I use hadoop 0.20.2 and I can not find code that relates to 'preempt' function; however, I read the jira MAPREDUCE-551 saying preempt function is already been fixed at version 0.20.0. Also, I can find some functons that relates to 'preemption' e.g. 'protected void preemptTasksIfNecessary()' in the patch. I am confused now - which function in version 0.20.2 (or 0.20.1) is used to preempt unnecessary tasks (so that slots can be freed for other tasks/ jobs to run)? Thanks you for your help.
