[
https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16192301#comment-16192301
]
John Zhuge commented on HADOOP-14872:
-------------------------------------
TestKDiag and TestDataNodeVolumeFailure failures are known issues.
But these failures are puzzling. Memory corruptions in native code:
{noformat}
Running org.apache.hadoop.crypto.TestCryptoStreamsNormal
Running org.apache.hadoop.crypto.TestCryptoCodec
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.599 sec - in
org.apache.hadoop.crypto.TestCryptoOutputStreamClosing
Running org.apache.hadoop.crypto.TestCryptoStreams
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.329 sec - in
org.apache.hadoop.crypto.key.TestValueQueue
Running org.apache.hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec
*** Error in `/usr/lib/jvm/java-8-oracle/jre/bin/java': free(): corrupted
unsorted chunks: 0x00007f51c8a86fa0 ***
Aborted (core dumped)
Running org.apache.hadoop.crypto.TestCryptoStreamsWithJceAesCtrCryptoCodec
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.887 sec - in
org.apache.hadoop.crypto.TestCryptoCodec
Running org.apache.hadoop.crypto.TestCryptoStreamsForLocalFS
*** Error in `/usr/lib/jvm/java-8-oracle/jre/bin/java': free(): corrupted
unsorted chunks: 0x00007ffae813ab40 ***
Aborted (core dumped)
...
Running org.apache.hadoop.metrics2.lib.TestMutableRollingAverages
*** Error in `/usr/lib/jvm/java-8-oracle/jre/bin/java': free(): corrupted
unsorted chunks: 0x00007fc194b43c00 ***
Aborted (core dumped)
{noformat}
> CryptoInputStream should implement unbuffer
> -------------------------------------------
>
> Key: HADOOP-14872
> URL: https://issues.apache.org/jira/browse/HADOOP-14872
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 2.6.4
> Reporter: John Zhuge
> Assignee: John Zhuge
> Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch,
> HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch,
> HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch
>
>
> Discovered in IMPALA-5909.
> Opening an encrypted HDFS file returns a chain of wrapped input streams:
> {noformat}
> HdfsDataInputStream
> CryptoInputStream
> DFSInputStream
> {noformat}
> If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer,
> FSDataInputStream#unbuffer will be called:
> {code:java}
> try {
> ((CanUnbuffer)in).unbuffer();
> } catch (ClassCastException e) {
> throw new UnsupportedOperationException("this stream does not " +
> "support unbuffering.");
> }
> {code}
> If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If
> the application is not careful, tons of UOEs will show up in logs.
> In comparison, opening an non-encrypted HDFS file returns this chain:
> {noformat}
> HdfsDataInputStream
> DFSInputStream
> {noformat}
> DFSInputStream implements CanUnbuffer.
> It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:
> * Release buffer, cache, or any other resource when instructed
> * Able to call its wrapped DFSInputStream unbuffer
> * Avoid the UOE described above. Applications may not handle the UOE very
> well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
