steveloughran commented on a change in pull request #2297:
URL: https://github.com/apache/hadoop/pull/2297#discussion_r489422661
##########
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.java
##########
@@ -276,13 +258,29 @@ public void end() {
// do nothing
}
- private native static void initIDs();
+ private int decompressBytesDirect() throws IOException {
+ if (compressedDirectBufLen == 0) {
+ return 0;
+ } else {
+ // Set the position and limit of `compressedDirectBuf` for reading
+ compressedDirectBuf.limit(compressedDirectBufLen).position(0);
+ // There is compressed input, decompress it now.
+ int size = Snappy.uncompressedLength((ByteBuffer) compressedDirectBuf);
+ if (size > uncompressedDirectBuf.remaining()) {
+ throw new IOException("Could not decompress data. " +
+ "uncompressedDirectBuf length is too small.");
+ }
+ size = Snappy.uncompress((ByteBuffer) compressedDirectBuf,
+ (ByteBuffer) uncompressedDirectBuf);
+ compressedDirectBufLen = 0;
+ compressedDirectBuf.limit(compressedDirectBuf.capacity()).position(0);
+ return size;
+ }
+ }
- private native int decompressBytesDirect();
-
int decompressDirect(ByteBuffer src, ByteBuffer dst) throws IOException {
assert (this instanceof SnappyDirectDecompressor);
-
+
Review comment:
please stop the IDE removing trailing whitespace on lines which haven't
been edited; complicates life
##########
File path: hadoop-common-project/hadoop-common/pom.xml
##########
@@ -363,6 +363,10 @@
<artifactId>wildfly-openssl-java</artifactId>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>org.xerial.snappy</groupId>
Review comment:
<scope>provided</scope>
##########
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.java
##########
@@ -276,13 +258,29 @@ public void end() {
// do nothing
}
- private native static void initIDs();
+ private int decompressBytesDirect() throws IOException {
+ if (compressedDirectBufLen == 0) {
+ return 0;
+ } else {
+ // Set the position and limit of `compressedDirectBuf` for reading
+ compressedDirectBuf.limit(compressedDirectBufLen).position(0);
+ // There is compressed input, decompress it now.
+ int size = Snappy.uncompressedLength((ByteBuffer) compressedDirectBuf);
+ if (size > uncompressedDirectBuf.remaining()) {
+ throw new IOException("Could not decompress data. " +
+ "uncompressedDirectBuf length is too small.");
Review comment:
use name of config option which users can tun
##########
File path: hadoop-project-dist/pom.xml
##########
@@ -341,7 +340,6 @@
<argument>--openssllib=${openssl.lib}</argument>
<argument>--opensslbinbundle=${bundle.openssl.in.bin}</argument>
<argument>--openssllibbundle=${bundle.openssl}</argument>
-
<argument>--snappybinbundle=${bundle.snappy.in.bin}</argument>
<argument>--snappylib=${snappy.lib}</argument>
<argument>--snappylibbundle=${bundle.snappy}</argument>
Review comment:
what about the others snappylibs?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
