[
https://issues.apache.org/jira/browse/HDFS-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138165#comment-14138165
]
Colin Patrick McCabe commented on HDFS-6581:
--------------------------------------------
It's exciting to see this making progress.
One thing that I would really like to see is an eviction policy other than
least recently used (LRU). LRU has been well-documented to be a very poor
policy for scan workloads, which most of HDFS' workloads are. It would be nice
to see something like 2Q (See
http://www.tedunangst.com/flak/post/2Q-buffer-cache-algorithm), or at least a
pluggable interface that would allow this to be implemented. I think that very
few users, if any, will want LRU.
Do we have performance benchmarks for this? It would be nice to see that would
could get a substantial fraction of memory bandwidth when writing to a single
replica in-memory. That usually means somewhere between 5 gigs/s and 10
gigs/s. Perhaps it's too early to ask for these, since we haven't implemented
HDFS-6933. But it would be good to start thinking about this. Also, some of
the recent work to use native checksums has substantially reduced the checksum
overhead in the write path so maybe we can get an estimate without HDFS-6933.
> Write to single replica in memory
> ---------------------------------
>
> Key: HDFS-6581
> URL: https://issues.apache.org/jira/browse/HDFS-6581
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Reporter: Arpit Agarwal
> Assignee: Arpit Agarwal
> Attachments: HDFS-6581.merge.01.patch, HDFS-6581.merge.02.patch,
> HDFS-6581.merge.03.patch, HDFSWriteableReplicasInMemory.pdf,
> Test-Plan-for-HDFS-6581-Memory-Storage.pdf
>
>
> Per discussion with the community on HDFS-5851, we will implement writing to
> a single replica in DN memory via DataTransferProtocol.
> This avoids some of the issues with short-circuit writes, which we can
> revisit at a later time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
