[
https://issues.apache.org/jira/browse/MAPREDUCE-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13059852#comment-13059852
]
Steve Loughran commented on MAPREDUCE-2636:
-------------------------------------------
right now the JT is ignorant of where blocks live on a server, only that they
are server-local. Indeed, the Namenode doesn't know either. It would be quite a
large change to add this information, and if it took up more namenode memory,
large-filesystem sites would be reluctant to adopt the improvement.
You'd also have to take into account not just the source disk, but the output
disks, and maybe the location of the any intermediate/overspill storage; the JT
would need to know not just how many slots there were free, but which disks
each active task is reading and writing, either by having this data pushed to
it, or by checking prior to scheduling.
Like I said, a lot of work. Rather than rushing to do this, I'd recommend you
come up with a way of measuring the conflict that is occurring. That way
different clusters (with different #s of disks/server) could get data on how
much of an issue this is -and whether adding more HDDs to a server improves
things, or, as more tasks get executed on multicore CPUs, whether it gets worse.
> Scheduling over disks horizontally
> ----------------------------------
>
> Key: MAPREDUCE-2636
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2636
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: job submission
> Reporter: Evert Lammerts
> Priority: Minor
>
> Based on this message:
> http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201106.mbox/browser
> The JT schedules tasks on nodes based on metadata it gets from the NN. The
> namenode does not know on which disk a block resides. It might happen that on
> a node running 4 tasks, all read from the same disk. This can affect
> performance.
> An optimization might be to schedule horizontally over disks instead of
> nodes. Any ideas?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
