[
https://issues.apache.org/jira/browse/HADOOP-10150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14120262#comment-14120262
]
Terence Spies commented on HADOOP-10150:
----------------------------------------
After looking at the encryption proposal, I’m very concerned about the security
of the chosen mechanism. As I understand it, the idea is to use AES in counter
mode to encrypt the data (which offers no integrity protection) and rely on the
existing CRC32 checksums to detect data tampering. The problem here is that
the CRC32 checksums are unkeyed, and are quite easy to defeat by an active
attacker. The net result is that the attacker can, even through they cannot
know the value of individual bits, can trivially flip the value of any bit they
desire in the file. This may be detected by the CRC32 checksum, but it’s not
difficult to defeat this checksum mechanism by making trial bit flips to
compensate.
At a bare minimum, the checksum mechanism should be replaced with a HMAC/CMAC
based keyed checksum mechanism. (The one thing I could not find is if the
checksum file is encrypted — if it isn’t, then the plaintext checksums leak
information about the underlying plaintext. HMAC/CMACs would prevent that from
happening.)
In general, not using an authenticated encryption mode is pretty dangerous
here. Separated MACs enable detect of tampering, but the mechanism needs to be
carefully implemented to prevent attackers from using error or timing
information to insert tampered data into files.
I understand the desire to keep the file seekable, and also not change the size
of the underlying file. My suggestion would be stay with a mode that encrypts
and decrypts as a block cipher, but keep the block size small enough that you
can seek with a smallish buffer. In terms of file size, most of the block
modes will support ciphertext stealing, which enables the block size to be
changed to whatever byte size is required.
One suggestion would be to look at a mode like OCB, which gives an
authenticated mode with very little overhead, and also supports associated
data. The associated data feature would enable the block position and IV to be
incorporated, giving seekability. As an example, if the file was encrypted
with a 128 byte block size, the associated data (similarly to the CTR mode
index) would be the position of the block within the file and the IV for the
file.
This would also have the upside of producing an authentication tag for each
block, which could at some point be added to some metadata to give
cryptographic integrity. Note that we would also want to turn off the CRC32
checksums, as they would leak data about the underlying plaintext.
> Hadoop cryptographic file system
> --------------------------------
>
> Key: HADOOP-10150
> URL: https://issues.apache.org/jira/browse/HADOOP-10150
> Project: Hadoop Common
> Issue Type: New Feature
> Components: security
> Affects Versions: 3.0.0
> Reporter: Yi Liu
> Assignee: Yi Liu
> Labels: rhino
> Fix For: 2.6.0
>
> Attachments: CryptographicFileSystem.patch, HADOOP cryptographic file
> system-V2.docx, HADOOP cryptographic file system.pdf,
> HDFSDataAtRestEncryptionAlternatives.pdf,
> HDFSDataatRestEncryptionAttackVectors.pdf,
> HDFSDataatRestEncryptionProposal.pdf, cfs.patch, extended information based
> on INode feature.patch
>
>
> There is an increasing need for securing data when Hadoop customers use
> various upper layer applications, such as Map-Reduce, Hive, Pig, HBase and so
> on.
> HADOOP CFS (HADOOP Cryptographic File System) is used to secure data, based
> on HADOOP “FilterFileSystem” decorating DFS or other file systems, and
> transparent to upper layer applications. It’s configurable, scalable and fast.
> High level requirements:
> 1. Transparent to and no modification required for upper layer
> applications.
> 2. “Seek”, “PositionedReadable” are supported for input stream of CFS if
> the wrapped file system supports them.
> 3. Very high performance for encryption and decryption, they will not
> become bottleneck.
> 4. Can decorate HDFS and all other file systems in Hadoop, and will not
> modify existing structure of file system, such as namenode and datanode
> structure if the wrapped file system is HDFS.
> 5. Admin can configure encryption policies, such as which directory will
> be encrypted.
> 6. A robust key management framework.
> 7. Support Pread and append operations if the wrapped file system supports
> them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
