[
https://issues.apache.org/jira/browse/HDFS-5182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13867175#comment-13867175
]
Todd Lipcon commented on HDFS-5182:
-----------------------------------
bq. What I was referring to here is where a client has specifically requested
an mmap region using the zero-copy API and the SKIP_CHECKSUMS option. In that
case, the user is clearly going to be reading without any guarantees from us.
If the user just uses the normal (non-zero-copy, non-mmap) read path, SIGBUS
will not be an issue.
Why not two separate flags? One flag saying "SKIP_CHECKSUMS" (ie "I will do my
own checksumming") and another flag for "NO_REQUIRE_MLOCK" or "UNSAFE_IO" or
something, which means you're OK with SIGBUS.
ie there are really three levels of guarantee we can provide:
1) Normal HDFS semantics: a read will only return correct data, and if it
fails, a nice error code will return.
2) Skip-checksums semantics: a read will return data which might be corrupt. If
it fails, a nice error code will return.
3) Unsafe semantics: a read will return data which might be corrupt. If it
fails, either a nice error code or a SIGBUS.
There are a lot of applications that are OK with #2 but not #3. #3 is really
hard to deal with since a bad disk in the cluster would SIGBUS everything
running on the machine pretty fast, and we don't currently have any way of
handling it.
> BlockReaderLocal must allow zero-copy reads only when the DN believes it's
> valid
> ---------------------------------------------------------------------------------
>
> Key: HDFS-5182
> URL: https://issues.apache.org/jira/browse/HDFS-5182
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
> Affects Versions: 3.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
>
> BlockReaderLocal must allow zero-copy reads only when the DN believes it's
> valid. This implies adding a new field to the response to
> REQUEST_SHORT_CIRCUIT_FDS. We also need some kind of heartbeat from the
> client to the DN, so that the DN can inform the client when the mapped region
> is no longer locked into memory.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
