[
https://issues.apache.org/jira/browse/HDFS-5851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14004031#comment-14004031
]
Arpit Agarwal commented on HDFS-5851:
-------------------------------------
Minutes from Google Hangout:
Wrt to the mechanism to support memory caching there was high level agreement
on the implementation phases roughly as:
* 1st phase - streaming socket write, but mlock on DN side so that it keeps it
for readers.
** Make this work for a single replica
** Separately (in another Jira) investigate write-pipeline improvements because
the write-pipeline has not been optimized. This should give us some initial
performance numbers and one can start using this mechanism. [~tlipcon] (?) has
a prototype.
* 2nd phase - Explore short-circuit write, but datanode still mlocks. We had a
quick discussion on short-circuit write being tricky
** Recovery issues (RBW)
** Client can do things that can get the DN confused (e.g. truncate/append the
file after close)
* Future phases
** Add lazy replication to other replicas (note earlier phases allowed only 1
replica)
** Direct writes to memory by memory-mapping the file
Discussion on discardability:
* Shouldn't this be property of file (such a replica count of 1) rather than a
a property of /.reserved/ddm?
** This needs further discussion on the jira.
* Why the two layer approach?
** We don't want to necessarily put load on NN for intermediate files and hence
the 2nd layer.
> Support memory as a storage medium
> ----------------------------------
>
> Key: HDFS-5851
> URL: https://issues.apache.org/jira/browse/HDFS-5851
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
> Affects Versions: 3.0.0
> Reporter: Arpit Agarwal
> Assignee: Arpit Agarwal
> Attachments:
> SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf,
> SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf
>
>
> Memory can be used as a storage medium for smaller/transient files for fast
> write throughput.
> More information/design will be added later.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
