[
https://issues.apache.org/jira/browse/HDFS-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14140914#comment-14140914
]
Arpit Agarwal commented on HDFS-6581:
-------------------------------------
bq. There are a bunch of performance numbers on HDFS-4953 (a subtask of
HDFS-4949). Enabling zero-copy reads was an important goal of HDFS-4949. On
HDFS-4953, Todd and I discussed how to get close to the optimal 9GB/s per core.
HDFS-4953 is not a subtask of HDFS-4949, and you yourself have said that it is
an independent change. Also no numbers were cited or provided as evidence when
calling the merge vote for HDFS-4949 so I don't think the community could
reasonably be expected to take that into consideration.
I acknowledge your concern that the eviction scheme needs tuning and what we
have in here is mostly a placeholder. IMO we cannot come up with an ideal
scheme with synthetic benchmarks. For that we'd require HDFS applications to
run real world workloads using the feature, and it is hard to convince anyone
to spend time doing that unless at least the base changes are in a mainline
branch. The feature is entirely optional, so we do not risk breaking any
existing users. The cost of indefinitely maintaining the feature in a branch is
rather high given the regularity of changes happening in trunk.
That said, I see the primary concern being that short-circuit reads must be
able to work from memory and that read performance is not regressed. I think
that is an entirely reasonable request and will provide evidence to support
that before we completing the merge.
> Write to single replica in memory
> ---------------------------------
>
> Key: HDFS-6581
> URL: https://issues.apache.org/jira/browse/HDFS-6581
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Reporter: Arpit Agarwal
> Assignee: Arpit Agarwal
> Attachments: HDFS-6581.merge.01.patch, HDFS-6581.merge.02.patch,
> HDFS-6581.merge.03.patch, HDFS-6581.merge.04.patch, HDFS-6581.merge.05.patch,
> HDFS-6581.merge.06.patch, HDFSWriteableReplicasInMemory.pdf,
> Test-Plan-for-HDFS-6581-Memory-Storage.pdf
>
>
> Per discussion with the community on HDFS-5851, we will implement writing to
> a single replica in DN memory via DataTransferProtocol.
> This avoids some of the issues with short-circuit writes, which we can
> revisit at a later time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
