[
https://issues.apache.org/jira/browse/MAPREDUCE-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jaehoon ko updated MAPREDUCE-5946:
----------------------------------
Component/s: (was: security)
task
Description:
In map task, merge starts only after the last spill is completely written to
disk. This is not necessary nor efficient because the last spill should to be
reloaded soon for merge, probably immediately because spills are merged in the
order of their sizes and the last spill is likely smallest. OS page cache is
not the answer due to its opportunistic nature.
I'm starting to work on this. Please give me your thoughts.
was:
In map task, merge starts only after the last spill is completely written to
disk. This is not necessary nor efficient because the last spill should to be
reloaded soon for merge, probably immediately because spills are merged in the
order of their sizes and the last spill is likely smallest. OS page cache is
not the answer due to its opportunistic nature.
I'm starting to work on it. Please give me your thoughts.
> Last spill of map task is not necessary for final merge
> -------------------------------------------------------
>
> Key: MAPREDUCE-5946
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5946
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance, task
> Affects Versions: 2.4.0
> Reporter: jaehoon ko
> Assignee: jaehoon ko
> Labels: newbie, performance
>
> In map task, merge starts only after the last spill is completely written to
> disk. This is not necessary nor efficient because the last spill should to be
> reloaded soon for merge, probably immediately because spills are merged in
> the order of their sizes and the last spill is likely smallest. OS page cache
> is not the answer due to its opportunistic nature.
> I'm starting to work on this. Please give me your thoughts.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
