[
https://issues.apache.org/jira/browse/HDFS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13623343#comment-13623343
]
Suresh Srinivas commented on HDFS-347:
--------------------------------------
I have refrained from commenting. What I have seen is lack of understanding,
not being receptive to suggestions, and complete disregard for the comments
posted by a long term committer.
bq. The only purpose of the patch with 'Jenkins' in the name is to get a
Jenkins run. It is not for review. It was generated with 'git diff master' run
from the HDFS-347 branch. You could do a similar thing with subverison by
checking out two copies and diffing them. I recommend looking at the commits in
subversion or git.
What is the basis of this statement? I had several times mentioned in the
voting thread that, I choose to review merge patch with several reasons why it
works for me. Still you make the statement that "The only purpose of the patch
with 'Jenkins' in the name is to get a Jenkins run".
Repeated requests for generating clean patch has been ignored.
Yes, you need to generate the right patch. Not just a patch that is close to
the right patch. If the patch is not the right one, Jenkins +1 has no meaning.
Many comments have tried to indicate this, several times
[here|https://issues.apache.org/jira/browse/HDFS-347?focusedCommentId=13572860&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13572860],
[here|https://issues.apache.org/jira/browse/HDFS-347?focusedCommentId=13620632&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13620632]
and
[here|https://issues.apache.org/jira/browse/HDFS-347?focusedCommentId=13621601&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13621601].
If this is not clear, you could ask others to guide you on how to do it
(though the helpful tips on how to do it are ignored).
bq. can you please add these style comments to the style cleanup JIRA
HDFS-4661? Loading this JIRA makes my web browser slow to a crawl.
I can load this on my mobile device over wireless connection. No problem. If
the comments are related to the clean merge patch, it should be made in this
jira, isn't it?
bq. The difference between my patch and yours is this...
After describing how to do it, you chose to ignore. After it is done and works,
you seem to be saying it does not count because the different is not
significant? It does not matter whether it is a few lines of difference or one
line. Right merge patch needs to be generated.
In the interest of making progress on this issue, [~tlipcon] can you please
help merge the patch with cleanly and correctly generated merge patch? If you
are busy, I will work with [~szetszwo] on getting patch merged to trunk.
[~szetszwo] can you please indicate if a +1 from Jenkins on a clean merge patch
is sufficient to merge this change to trunk or you would like to see any more
changes?
> DFS read performance suboptimal when client co-located on nodes with data
> -------------------------------------------------------------------------
>
> Key: HDFS-347
> URL: https://issues.apache.org/jira/browse/HDFS-347
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, hdfs-client, performance
> Reporter: George Porter
> Assignee: Colin Patrick McCabe
> Attachments: 2013.01.28.design.pdf, 2013.01.31.consolidated2.patch,
> 2013.01.31.consolidated.patch, 2013.02.15.consolidated4.patch,
> 2013-04-01-jenkins.patch, all.tsv, a.patch, BlockReaderLocal1.txt,
> full.patch, HADOOP-4801.1.patch, HADOOP-4801.2.patch, HADOOP-4801.3.patch,
> HDFS-347-016_cleaned.patch, HDFS-347.016.patch, HDFS-347.017.clean.patch,
> HDFS-347.017.patch, HDFS-347.018.clean.patch, HDFS-347.018.patch2,
> HDFS-347.019.patch, HDFS-347.020.patch, HDFS-347.021.patch,
> HDFS-347.022.patch, HDFS-347.024.patch, HDFS-347.025.patch,
> HDFS-347.026.patch, HDFS-347.027.patch, HDFS-347.029.patch,
> HDFS-347.030.patch, HDFS-347.033.patch, HDFS-347.035.patch,
> HDFS-347-branch-20-append.txt, hdfs-347-merge.txt, hdfs-347-merge.txt,
> hdfs-347-merge.txt, hdfs-347.png, hdfs-347.txt, local-reads-doc
>
>
> One of the major strategies Hadoop uses to get scalable data processing is to
> move the code to the data. However, putting the DFS client on the same
> physical node as the data blocks it acts on doesn't improve read performance
> as much as expected.
> After looking at Hadoop and O/S traces (via HADOOP-4049), I think the problem
> is due to the HDFS streaming protocol causing many more read I/O operations
> (iops) than necessary. Consider the case of a DFSClient fetching a 64 MB
> disk block from the DataNode process (running in a separate JVM) running on
> the same machine. The DataNode will satisfy the single disk block request by
> sending data back to the HDFS client in 64-KB chunks. In BlockSender.java,
> this is done in the sendChunk() method, relying on Java's transferTo()
> method. Depending on the host O/S and JVM implementation, transferTo() is
> implemented as either a sendfilev() syscall or a pair of mmap() and write().
> In either case, each chunk is read from the disk by issuing a separate I/O
> operation for each chunk. The result is that the single request for a 64-MB
> block ends up hitting the disk as over a thousand smaller requests for 64-KB
> each.
> Since the DFSClient runs in a different JVM and process than the DataNode,
> shuttling data from the disk to the DFSClient also results in context
> switches each time network packets get sent (in this case, the 64-kb chunk
> turns into a large number of 1500 byte packet send operations). Thus we see
> a large number of context switches for each block send operation.
> I'd like to get some feedback on the best way to address this, but I think
> providing a mechanism for a DFSClient to directly open data blocks that
> happen to be on the same machine. It could do this by examining the set of
> LocatedBlocks returned by the NameNode, marking those that should be resident
> on the local host. Since the DataNode and DFSClient (probably) share the
> same hadoop configuration, the DFSClient should be able to find the files
> holding the block data, and it could directly open them and send data back to
> the client. This would avoid the context switches imposed by the network
> layer, and would allow for much larger read buffers than 64KB, which should
> reduce the number of iops imposed by each read block operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
