[
https://issues.apache.org/jira/browse/HDFS-5182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13867073#comment-13867073
]
Todd Lipcon commented on HDFS-5182:
-----------------------------------
bq. it will check to see if the segment is anchorable, and increment the anchor
count before performing the mmap. The anchor count will stay incremented until
the mmap is closed.
That seems much longer than necessary -- don't we want clients to be able to
keep mmaps around in their cache for very long periods of time? And then, when
the user requests the read, we can "anchor" the mmap only for the duration of
time for which the user holds onto the zero-copy buffer? Once the user returns
the zero-copy buffer, we can decrement the count and allow the DN to evict the
block from the cache.
bq. One exception is if the client passes the ReadOption.SKIP_CHECKSUMS flag.
In that case, we do not need to consult the anchor flag because we are willing
to tolerate bad data being returned or SIGBUS.
I disagree on this. Just because you want to skip checksumming doesn't mean you
can tolerate SIGBUS. For example, many file formats have their own checksums,
so we can safely skip HDFS checksumming, but we still want to ensure that we're
only reading locked (i.e safe) memory via mmap.
bq. The only function of this socket is to cause a close notification to be
sent to the datanode when the client closes (or vice versa). (When a UNIX
domain socket closes, the remote end gets a close notification).
Maybe this can be put into a separate JIRA, and first implement just a very
simple timeout-based mechanism? The DN could change the anchor flag to a magic
value which invalidates the segment and then close it after some amount of
time. Then if the client looks at it again it will know to invalidate.
> BlockReaderLocal must allow zero-copy reads only when the DN believes it's
> valid
> ---------------------------------------------------------------------------------
>
> Key: HDFS-5182
> URL: https://issues.apache.org/jira/browse/HDFS-5182
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
> Affects Versions: 3.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
>
> BlockReaderLocal must allow zero-copy reads only when the DN believes it's
> valid. This implies adding a new field to the response to
> REQUEST_SHORT_CIRCUIT_FDS. We also need some kind of heartbeat from the
> client to the DN, so that the DN can inform the client when the mapped region
> is no longer locked into memory.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
