[
https://issues.apache.org/jira/browse/HDFS-5851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14005285#comment-14005285
]
Arpit Agarwal commented on HDFS-5851:
-------------------------------------
Hi [~eric14], I apologize for the long delayed response.
bq. The case of a local short circuit read having access to the open file is
interesting... does this pin the memory until the possibly misbehaved client
process closes the socket / FD?
Yes this is correct. This should be the existing behavior with short-circuit
reads.
bq. Single replicas? Why would one want to triple replicate discardable memory?
One should at least have the option to only keep a single local copy in HDFS.
We will have a single replica to begin with. There are use cases for triple
replicas to share memory across sessions/applications. Multiple replicas would
have to be optional as you suggest.
bq. If we can not prevent random access writes to DDM (we could presumably
limit this in client API), then I don't think we can checksum or replicate
until a file is closed. My gut is delaying such until close is the right call...
Direct writes to DN memory are appealing for performance reasons but there are
some open questions including the ones you raised. Hence phase 1 will avoid
short-circuit writes.
I will let Sanjay respond to the remaining questions since he is more familiar
with those aspects.
> Support memory as a storage medium
> ----------------------------------
>
> Key: HDFS-5851
> URL: https://issues.apache.org/jira/browse/HDFS-5851
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
> Affects Versions: 3.0.0
> Reporter: Arpit Agarwal
> Assignee: Arpit Agarwal
> Attachments:
> SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf,
> SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf
>
>
> Memory can be used as a storage medium for smaller/transient files for fast
> write throughput.
> More information/design will be added later.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
