[
https://issues.apache.org/jira/browse/HDFS-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14141558#comment-14141558
]
Arpit Agarwal commented on HDFS-6581:
-------------------------------------
Rebase is no silver bullet. Conflicts still need to be resolved manually.
Colin, explaining how to use git is a little condescending.
bq. I don't think creating a usable eviction strategy is that hard. I can help
with it, but probably not for a few days.
It would be very little code to convert the current FIFO approach to something
like LFU but writing code is easy and demonstrating it actually helps HDFS
clients is harder. For your caching feature the measurement was fairly
straightforward. The logic for deciding which replicas need to be in memory was
outside HDFS. For this feature we'd first need to define "better scheme" and
we'd need help from other stack components for evaluation. Think of this
feature as providing the overall framework including API, protocol changes and
DN support. If there is no argument with the framework design then there should
be no objection to doing the eviction fine tuning (which is a very small
proportion of the patch, perhaps less than 5% content wise) post-merge. And to
restate, we cannot get clients to start evaluating it until the changes are in
mainline.
bq. Can you give a summary of which tasks you are planning on completing prior
to the merge and which after?
Please see the test plan attached by Xiaoyu for functional/regression testing.
Pre-merge we'll make the eviction scheme pluggable via an interface and provide
evidence that short circuit read performance is as good or better than what it
is currently to address your concerns. I've outlined the post-merge plan above.
This broadly covers write pipeline optimization, crc evaluations and tuning
eviction.
> Write to single replica in memory
> ---------------------------------
>
> Key: HDFS-6581
> URL: https://issues.apache.org/jira/browse/HDFS-6581
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Reporter: Arpit Agarwal
> Assignee: Arpit Agarwal
> Attachments: HDFS-6581.merge.01.patch, HDFS-6581.merge.02.patch,
> HDFS-6581.merge.03.patch, HDFS-6581.merge.04.patch, HDFS-6581.merge.05.patch,
> HDFS-6581.merge.06.patch, HDFS-6581.merge.07.patch,
> HDFSWriteableReplicasInMemory.pdf, Test-Plan-for-HDFS-6581-Memory-Storage.pdf
>
>
> Per discussion with the community on HDFS-5851, we will implement writing to
> a single replica in DN memory via DataTransferProtocol.
> This avoids some of the issues with short-circuit writes, which we can
> revisit at a later time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
