Sean Busbey created HADOOP-11708:
------------------------------------
Summary: CryptoOutputStream synchronization differences from
DFSOutputStream break HBase
Key: HADOOP-11708
URL: https://issues.apache.org/jira/browse/HADOOP-11708
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.6.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Priority: Critical
For the write-ahead-log, HBase writes to DFS from a single thread and sends
sync/flush/hflush from a configurable number of other threads (default 5).
FSDataOutputStream does not document anything about being thread safe, and it
is not thread safe for concurrent writes.
However, DFSOutputStream is thread safe for concurrent writes + syncs. When it
is the stream FSDataOutputStream wraps, the combination is threadsafe for 1
writer and multiple syncs (the exact behavior HBase relies on).
When HDFS Transparent Encryption is turned on, CryptoOutputStream is inserted
between FSDataOutputStream and DFSOutputStream. It is proactively labeled as
not thread safe, and this composition is not thread safe for any operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
