Saptarshi Guha wrote:
Hello,
I recall asking this question but this is in addition to what I'ev askd.
Firstly, to recap my question and Arun's specific response:

-- On May 20, 2008, at 9:03 AM, Saptarshi Guha wrote: > Hello, > -- Does the "Data-local map tasks" counter mean the number of tasks that the had the input data already present on the machine on they are running on? -- i.e the wasn't a need to ship the data to them.
Response from Arun
-- Yes. Your understanding is correct. More specifically it means that the map-task got scheduled on a machine on which one of the -- replicas of it's input-split-block was present and was served by the datanode running on that machine. *smile* Arun


Now, Is Hadoop designed to schedule a map task on a machine which has one of the replicas of it's input split block?
Yes.
Failing that, does then assign a map task on machine close to one that contains a replica of it's input split block?
The scheduling is tasktracker based rather than split based. By that what I mean is that the tasktracker asks for a task and the JT schedules a task to that tracker. If there is any split that is data local to the tasktracker and not yet scheduled, it will be assigned to the tracker. If no such split can be found the JT will assign a high priority split to it. The priority amongst the splits is based on their ordering given by the jobclient. By default its sorted on split size (decreasing order). Either the split is data-local (on the same machine), rack local (within the same rack) or is not-local. There is no other measure of closeness. The scheduling problem is 'given a tasktracker find out the best split' rather than 'given a split find out the best/closest tracker'.
Are there any performance metrics for this?

Many thanks
Saptarshi


*/Saptarshi Guha | [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> | http://www.stat.purdue.edu/~sguha <http://www.stat.purdue.edu/%7Esguha>/*



Reply via email to