HDFS-12251. Add document for StreamCapabilities. (Lei (Eddy) Xu)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/fe334178 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/fe334178 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/fe334178 Branch: refs/heads/HADOOP-13345 Commit: fe3341786a0d61f404127bf21d1afc85b2f21d38 Parents: a6fdeb8 Author: Lei Xu <l...@apache.org> Authored: Fri Aug 4 11:21:58 2017 -0700 Committer: Lei Xu <l...@apache.org> Committed: Fri Aug 4 11:21:58 2017 -0700 ---------------------------------------------------------------------- .../src/site/markdown/filesystem/filesystem.md | 24 ++++++++++++++++++++ .../src/site/markdown/HDFSErasureCoding.md | 19 ++++++++++++++++ 2 files changed, 43 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/fe334178/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md index b56666c..d7e57ce 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md @@ -1210,3 +1210,27 @@ try { It is notable that this is *not* done in the Hadoop codebase. This does not imply that robust loops are not recommended âmore that the concurrency problems were not considered during the implementation of these loops. + + +## <a name="StreamCapability"></a> interface `StreamCapabilities` + +The `StreamCapabilities` provides a way to programmatically query the +capabilities that an `OutputStream` supports. + +```java +public interface StreamCapabilities { + boolean hasCapability(String capability); +} +``` + +### `boolean hasCapability(capability)` + +Return true if the `OutputStream` has the desired capability. + +The caller can query the capabilities of a stream using a string value. +It currently supports to query: + + * `StreamCapabilties.HFLUSH` ("*hflush*"): the capability to flush out the data + in client's buffer. + * `StreamCapabilities.HSYNC` ("*hsync*"): capability to flush out the data in + client's buffer and the disk device. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/hadoop/blob/fe334178/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md index 1c0a2de..88293ba 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md @@ -199,3 +199,22 @@ Below are the details about each command. * `[-disablePolicy -policy <policyName>]` Disable an erasure coding policy. + +Limitations +----------- + +Certain HDFS file write operations, i.e., `hflush`, `hsync` and `append`, +are not supported on erasure coded files due to substantial technical +challenges. + +* `append()` on an erasure coded file will throw `IOException`. +* `hflush()` and `hsync()` on `DFSStripedOutputStream` are no-op. Thus calling +`hflush()` or `hsync()` on an erasure coded file can not guarantee data +being persistent. + +A client can use [`StreamCapabilities`](../hadoop-common/filesystem/filesystem.html#interface_StreamCapabilities) +API to query whether a `OutputStream` supports `hflush()` and `hsync()`. +If the client desires data persistence via `hflush()` and `hsync()`, the current +remedy is creating such files as regular 3x replication files in a +non-erasure-coded directory, or using `FSDataOutputStreamBuilder#replicate()` +API to create 3x replication files in an erasure-coded directory. --------------------------------------------------------------------- To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
