[
https://issues.apache.org/jira/browse/HBASE-20429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591970#comment-16591970
]
Zach York commented on HBASE-20429:
-----------------------------------
[~apurtell] Makes sense. We'll tackle FS redo in a separate JIRA.
Have you tested using a consistency layer such as S3Guard? That should be able
to handle most of the consistency guarantees so you don't have RS aborts when
you encounter consistencies after writing storefiles. Alternatively, you could
add a retry in the failure logic instead of aborting the RS immediately when a
file doesn't exist.
FWIW, I have seen a large number of heavy write workloads working well with S3
given you use a consistency layer (EmrFS Consistent View in my case) so that
might be sufficient for your case until you can upgrade to a version that
(hopefully) contains the FSredo work.
> Support for mixed or write-heavy workloads on non-HDFS filesystems
> ------------------------------------------------------------------
>
> Key: HBASE-20429
> URL: https://issues.apache.org/jira/browse/HBASE-20429
> Project: HBase
> Issue Type: Umbrella
> Reporter: Andrew Purtell
> Priority: Major
>
> We can support reasonably well use cases on non-HDFS filesystems, like S3,
> where an external writer has loaded (and continues to load) HFiles via the
> bulk load mechanism, and then we serve out a read only workload at the HBase
> API.
> Mixed workloads or write-heavy workloads won't fare as well. In fact, data
> loss seems certain. It will depend in the specific filesystem, but all of the
> S3 backed Hadoop filesystems suffer from a couple of obvious problems,
> notably a lack of atomic rename.
> This umbrella will serve to collect some related ideas for consideration.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
