[
https://issues.apache.org/jira/browse/HADOOP-3288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591541#action_12591541
]
Raghu Angadi commented on HADOOP-3288:
--------------------------------------
> Read-ahead achieves striping because writing stripes files.
In that sense, striping is really independent of read-ahead.. no?
If writing stripes across disks, wouldn't most of negatives of RAID0 that Doug
mentioned apply to this as well? A more detailed proposal might make it more
clear.. but I am not able to see fundamental difference between simple DataNode
striping across disks and RAID0. DataNode striping may not what is proposed
here.
> Serial streaming performance should be Math.min(ideal client performance,
> ideal serial hdfs performance)
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3288
> URL: https://issues.apache.org/jira/browse/HADOOP-3288
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.16.3, 0.18.0
> Environment: Mac OS X 10.5.2, Java 6
> Reporter: Sam Pullara
> Fix For: 0.18.0
>
>
> I looked at all the code long and hard and this was my analysis (could be
> wrong, I'm not an expert on this codebase):
> Current Serial HDFS performance = Average Datanode Performance
> Average Datanode Performance = Average Disk Performance (even if you have
> more than one)
> We should have:
> Ideal Serial HDFS Performance = Sum of Ideal Datanode Performance
> Ideal Datanode Performance = Sum of disk performance
> When you read a single file serially from HDFS there are a number of
> limitations that come into play:
> 1) Blocks on multiple datanodes will be load balanced between them -
> averaging the performance of the datanodes
> 2) Blocks on multiple disks in a single datanode are load balanced between
> them - averaging the performance of the disks
> I think that all this could be fixed if we actually prefetched fully read
> blocks on the client until the client can no longer keep up with the data or
> there is another bottleneck like network bandwidth.
> This seems like a reasonably common use case though not the typical MapReduce
> case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
