viirya commented on a change in pull request #2350:
URL: https://github.com/apache/hadoop/pull/2350#discussion_r504441080
##########
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
##########
@@ -302,11 +303,20 @@ public synchronized long getBytesWritten() {
public synchronized void end() {
}
- private native static void initIDs();
-
- private native int compressBytesDirect();
-
- private native int compressBytesDirectHC();
-
- public native static String getLibraryName();
+ private int compressDirectBuf() {
+ if (uncompressedDirectBufLen == 0) {
+ return 0;
+ } else {
+ // Set the position and limit of `uncompressedDirectBuf` for reading
+ uncompressedDirectBuf.limit(uncompressedDirectBufLen).position(0);
+ compressedDirectBuf.clear();
Review comment:
We can remove the `clear` and `limit` calls at the call site.
```java
// Re-initialize the lz4's output direct-buffer
compressedDirectBuf.clear();
compressedDirectBuf.limit(0);
```
But we cannot remove `clear` in `compressDirectBuf` method.
First, lz4-java cannot accept the `compressedDirectBuf` if it is called with
`limit(0)`.
```
[ERROR] Failures:
[ERROR] TestCompressorDecompressor.testCompressorDecompressor:69 Expected
to find 'testCompressorDecompressor error !!!' but got unexpected exception:
net.jpountz.lz4.LZ4Exception: maxDestLen is too small
at
net.jpountz.lz4.LZ4JNICompressor.compress(LZ4JNICompressor.java:69)
at net.jpountz.lz4.LZ4Compressor.compress(LZ4Compressor.java:158)
at
org.apache.hadoop.io.compress.lz4.Lz4Compressor.compressDirectBuf(Lz4Compressor.java:310)
at
org.apache.hadoop.io.compress.lz4.Lz4Compressor.compress(Lz4Compressor.java:237)
at
org.apache.hadoop.io.compress.CompressDecompressTester$CompressionTestStrategy$2.assertCompression(CompressDecompressTester.java:286)
at
org.apache.hadoop.io.compress.CompressDecompressTester.test(CompressDecompressTester.java:114)
at
org.apache.hadoop.io.compress.TestCompressorDecompressor.testCompressorDecompressor(TestCompressorDecompressor.java:66)
```
Second, even we remove `limit` in the call site, one test still failed:
```
[ERROR]
testCompressorDecompressor(org.apache.hadoop.io.compress.TestCompressorDecompressor)
Time elapsed: 0.539 s <<< FAILURE!
java.lang.AssertionError:
org.apache.hadoop.io.compress.lz4.Lz4Compressor_org.apache.hadoop.io.compress.lz4.Lz4Decompressor-
empty stream compressed output size != 4 expected:<4> but was:<65796>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at
org.apache.hadoop.io.compress.CompressDecompressTester$CompressionTestStrategy$3.assertCompression(CompressDecompressTester.java:334)
at
org.apache.hadoop.io.compress.CompressDecompressTester.test(CompressDecompressTester.java:114)
at
org.apache.hadoop.io.compress.TestCompressorDecompressor.testCompressorDecompressor(TestCompressorDecompressor.java:66)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
