[
https://issues.apache.org/jira/browse/HADOOP-11708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14358797#comment-14358797
]
Sean Busbey commented on HADOOP-11708:
--------------------------------------
I *think* the short term solution for fixing HBase on HDFS Encryption is to
document FSDataOutputStream to say write + sync are thread safe and have those
two methods sync on the wrapped stream since that will be the same monitor
we're already paying for on DFSOS.
Doing so would mean we could remove ~10 synchronization blocks in DFSOS (some
of them are unneeded and just about all of them are questionable, and I can't
find a rationalization for them).
As a follow-on, we add a FSDataOutputStream that isn't threadsafe and says as
much. We can do this compatibly by either making it an option (in
FSDataOutputStream construction or in configs), by making it a new API, or
making it a documented breaking change.
> CryptoOutputStream synchronization differences from DFSOutputStream break
> HBase
> -------------------------------------------------------------------------------
>
> Key: HADOOP-11708
> URL: https://issues.apache.org/jira/browse/HADOOP-11708
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.6.0
> Reporter: Sean Busbey
> Assignee: Sean Busbey
> Priority: Critical
>
> For the write-ahead-log, HBase writes to DFS from a single thread and sends
> sync/flush/hflush from a configurable number of other threads (default 5).
> FSDataOutputStream does not document anything about being thread safe, and it
> is not thread safe for concurrent writes.
> However, DFSOutputStream is thread safe for concurrent writes + syncs. When
> it is the stream FSDataOutputStream wraps, the combination is threadsafe for
> 1 writer and multiple syncs (the exact behavior HBase relies on).
> When HDFS Transparent Encryption is turned on, CryptoOutputStream is inserted
> between FSDataOutputStream and DFSOutputStream. It is proactively labeled as
> not thread safe, and this composition is not thread safe for any operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
