[
https://issues.apache.org/jira/browse/HADOOP-11708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14359060#comment-14359060
]
Sean Busbey commented on HADOOP-11708:
--------------------------------------
{quote}
Short term? I'd say note the concurrency expectations of HBase and an output
stream & come up with a change for the CryptoOutputStream which implements
consistent concurrency semantics. The runup to a release is not the time to
change something so foundational.
{quote}
{quote}
+1 for fixing CryptoOutputStream to implement the same expectations of HDFS.
{quote}
This is a bad idea, IMO. It's chasing one undocumented and likely broken
implementation with another one. If we're not going to update
FSDataOutputStream, we should just document and rely on downstream to fix their
reliance on undocumented behavior.
> CryptoOutputStream synchronization differences from DFSOutputStream break
> HBase
> -------------------------------------------------------------------------------
>
> Key: HADOOP-11708
> URL: https://issues.apache.org/jira/browse/HADOOP-11708
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.6.0
> Reporter: Sean Busbey
> Assignee: Sean Busbey
> Priority: Critical
>
> For the write-ahead-log, HBase writes to DFS from a single thread and sends
> sync/flush/hflush from a configurable number of other threads (default 5).
> FSDataOutputStream does not document anything about being thread safe, and it
> is not thread safe for concurrent writes.
> However, DFSOutputStream is thread safe for concurrent writes + syncs. When
> it is the stream FSDataOutputStream wraps, the combination is threadsafe for
> 1 writer and multiple syncs (the exact behavior HBase relies on).
> When HDFS Transparent Encryption is turned on, CryptoOutputStream is inserted
> between FSDataOutputStream and DFSOutputStream. It is proactively labeled as
> not thread safe, and this composition is not thread safe for any operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
