Sean Busbey updated HBASE-18784:
       Resolution: Fixed
    Fix Version/s:     (was: 1.5.0)
     Release Note: 
<!-- markdown -->

If HBase is run on top of Apache Hadoop libraries that support the needed APIs, 
it will verify that underlying Filesystem implementations provide the needed 
durability mechanisms to safely operate. The needed APIs *should* be present in 
Hadoop 3 release and Hadoop 2 releases starting in the Hadoop 2.9 series. If 
the APIs are not available, HBase behaves as it has in previous releases (that 
is, it moves forward assuming such a check would pass).

Where this check fails, it is unsafe to rely on HBase in a production setting. 
In the event of process or node failure, the HBase RegionServer process may 
fail to have access to all the data it previously wrote to its write ahead log, 
resulting in data loss. In the event of process or node failure, the HBase 
master process may lose all or part of the write ahead log that it relies on 
for cluster management operations, leaving the cluster in an inconsistent state 
that we aren't sure it could recover from.

Notably, the LocalFileSystem implementation provided by Hadoop reports 
(accurately) via these new APIs that it can not provide the durability HBase 
needs to operate. As such, the current instructions for single-node HBase 
operation have been updated both with a) how to bypass this safety check and b) 
a strong warning about the dire consequences of doing so outside of a dev/test 
           Status: Resolved  (was: Patch Available)

> Use of filesystem that requires hflush / hsync / append / etc should query 
> outputstream capabilities
> ----------------------------------------------------------------------------------------------------
>                 Key: HBASE-18784
>                 URL: https://issues.apache.org/jira/browse/HBASE-18784
>             Project: HBase
>          Issue Type: Improvement
>          Components: Filesystem Integration
>    Affects Versions: 1.4.0, 2.0.0-alpha-2
>            Reporter: Sean Busbey
>            Assignee: Sean Busbey
>            Priority: Major
>             Fix For: 2.0.0
>         Attachments: HBASE-18784-branch-1.v2.patch, HBASE-18784.0.patch, 
> HBASE-18784.1.patch, HBASE-18784.2.patch
> In places where we rely on the underlying filesystem holding up the promises 
> of hflush/hsync (most importantly the WAL), we should use the new interfaces 
> provided by HDFS-11644 to fail loudly when they are not present (e.g. on S3, 
> on EC mounts, etc).

This message was sent by Atlassian JIRA

Reply via email to