[ http://issues.apache.org/jira/browse/HADOOP-331?page=all ]
Devaraj Das updated HADOOP-331:
-------------------------------
Attachment: 331.patch
Thanks Owen for the detailed review. Attached is the new patch.
> map outputs should be written to a single output file with an index
> -------------------------------------------------------------------
>
> Key: HADOOP-331
> URL: http://issues.apache.org/jira/browse/HADOOP-331
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.3.2
> Reporter: eric baldeschwieler
> Assigned To: Devaraj Das
> Attachments: 331-design.txt, 331-initial3.patch, 331.patch, 331.txt
>
>
> The current strategy of writing a file per target map is consuming a lot of
> unused buffer space (causing out of memory crashes) and puts a lot of burden
> on the FS (many opens, inodes used, etc).
> I propose that we write a single file containing all output and also write an
> index file IDing which byte range in the file goes to each reduce. This will
> remove the issue of buffer waste, address scaling issues with number of open
> files and generally set us up better for scaling. It will also have
> advantages with very small inputs, since the buffer cache will reduce the
> number of seeks needed and the data serving node can open a single file and
> just keep it open rather than needing to do directory and open ops on every
> request.
> The only issue I see is that in cases where the task output is substantiallyu
> larger than its input, we may need to spill multiple times. In this case, we
> can do a merge after all spills are complete (or during the final spill).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
