[
https://issues.apache.org/jira/browse/HDFS-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138383#comment-14138383
]
Colin Patrick McCabe commented on HDFS-6581:
--------------------------------------------
bq. I don't recall any performance numbers were published before merging to
trunk. Looking through the Jira even now I don't see any numbers. Maybe it's
there - feel free to give a pointer.
There are a bunch of performance numbers on HDFS-4953 (a subtask of HDFS-4949).
Enabling zero-copy reads was an important goal of HDFS-4949. On HDFS-4953,
Todd and I discussed how to get close to the optimal 9GB/s per core. More
generally, we knew the HDFS read path could push a few gigs a second even
without zero-copy because we'd done the testing beforehand (some of the results
are on HDFS-347, some of that discussion is scattered other places). So we
were confident that HDFS-4949 was a performance improvement at the time of the
merge.
bq. The current direction is based on broad discussion with the community here
and on HDFS-5851, and the overall consensus was to keep DataTransferProtocol in
the picture for now.
I'm fine with keeping {{DataTransferProtocol}} in the picture for now. But you
should still be able to show a performance improvement when reading (not
writing) the data which you have previously written to the single replica in
memory. If there are flaws in this (for example, if short-circuit doesn't yet
work for these blocks, causing slowness) then we should address this before
merging.
We should also quantify exactly what the write performance is, even if it's not
as good as it could be. This will tell us what we need to work on in the
future and what the priorities should be.
> Write to single replica in memory
> ---------------------------------
>
> Key: HDFS-6581
> URL: https://issues.apache.org/jira/browse/HDFS-6581
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Reporter: Arpit Agarwal
> Assignee: Arpit Agarwal
> Attachments: HDFS-6581.merge.01.patch, HDFS-6581.merge.02.patch,
> HDFS-6581.merge.03.patch, HDFSWriteableReplicasInMemory.pdf,
> Test-Plan-for-HDFS-6581-Memory-Storage.pdf
>
>
> Per discussion with the community on HDFS-5851, we will implement writing to
> a single replica in DN memory via DataTransferProtocol.
> This avoids some of the issues with short-circuit writes, which we can
> revisit at a later time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
