[
https://issues.apache.org/jira/browse/HADOOP-5299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12675788#action_12675788
]
eric baldeschwieler commented on HADOOP-5299:
---------------------------------------------
owen,
A couple of tricks on the map side could really cut down your numbers. The
interesting hack of passing "open file info" around from maps to reduces and
skipping the name node for example. Storing indexes at the end of maps would
halve file opens of course. Requires more thought before discarding.
On coupling MR to HDFS... I agree that one should be able to run MR on systems
without an HDFS, absent a compelling reason not to. That doesn't seem to
preclude taking advantage of an existing HDFS if a performance gain could be
found.
Milind,
I don't see reduce side to be compelling enough to check in code, since it
complicates things without fixing the basic flaw of storing lots of temp data
outside of the distributed FS, where it can be more easily managed. I'd be
interested in seeing experimental data and patches to play with. But I'd hate
to see us check anything like this in, without good supporting data for why it
makes real systems better.
> Reducer inputs should be spilled to HDFS rather than local disk.
> ----------------------------------------------------------------
>
> Key: HADOOP-5299
> URL: https://issues.apache.org/jira/browse/HADOOP-5299
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.19.0
> Environment: All
> Reporter: Milind Bhandarkar
>
> Currently, both map outputs and reduce inputs are stored on local disks of
> tasktrackers. (Un) Availability of local disk space for intermediate data is
> seen as a major factor in job failures.
> The suggested solution is to store these intermediate data on HDFS (maybe
> with replication factor of 1). However, the main blocker issue with that
> solution is that lots of temporary names (proportional to total number of
> maps), can overwhelm the namenode, especially since the map outputs are
> typically small (most produce one block output).
> Also, as we see in many applications, the map outputs can be estimated more
> accurately, and thus users can plan accordingly, based on available local
> disk space.
> However, the reduce input sizes can vary a lot, especially for skewed data
> (or because of bad partitioning.)
> So, I suggest that it makes more sense to keep map outputs on local disks,
> but the reduce inputs (when spilled from reducer memory) should go to HDFS.
> Adding a configuration variable to indicate the filesystem to be used for
> reduce-side spills would let us experiment and compare the efficiency of this
> new scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
