Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/8150 )
Change subject: IMPALA-5990: Part 1: JNI-based LZ4 de/compression ...................................................................... Patch Set 2: (9 comments) http://gerrit.cloudera.org:8080/#/c/8150/2/be/src/service/fe-support.cc File be/src/service/fe-support.cc: http://gerrit.cloudera.org:8080/#/c/8150/2/be/src/service/fe-support.cc@450 PS2, Line 450: // whether to compress or decompress the 'src' into 'dst'. > May be worth commenting about GetPrimitiveArrayCritical() and lifecycle of Done http://gerrit.cloudera.org:8080/#/c/8150/2/be/src/service/fe-support.cc@451 PS2, Line 451: extern "C" > Is extern required for this method? Also, no return type? Done http://gerrit.cloudera.org:8080/#/c/8150/2/be/src/service/fe-support.cc@452 PS2, Line 452: NativeLz4Internal(JNIEnv* env, jbyteArray src, jint src_off, jint src_len, > Sounds good to me. Thanks. Done http://gerrit.cloudera.org:8080/#/c/8150/2/be/src/service/fe-support.cc@453 PS2, Line 453: jbyteArray dst, jint dst_off, jint dst_len, jboolean compress) { > DCHECK_GE(dst_len, LZ4_compressBound(src_size))? Why DCHECK? It's not a precondition of this function. The compression will fail gracefully. http://gerrit.cloudera.org:8080/#/c/8150/2/be/src/service/fe-support.cc@454 PS2, Line 454: 0 > nit: Not sure if this was intentional, but you are passing a nullptr argume My understanding is that the copied bytes would also be freed via ReleasePrimitiveArrayCritical(), so I don't see the flow changing even if is_copy is true. I added code to deal with is_copy and added comments to explain. http://gerrit.cloudera.org:8080/#/c/8150/2/be/src/service/fe-support.cc@455 PS2, Line 455: if (src_data == nullptr) return -1; > Do we still need to call ReleasePrimitiveArrayCritical() in this case? Also See documentation and example in: http://docs.oracle.com/javase/7/docs/technotes/guides/jni/spec/functions.html I added comments here. http://gerrit.cloudera.org:8080/#/c/8150/2/fe/src/main/java/org/apache/impala/service/FeSupport.java File fe/src/main/java/org/apache/impala/service/FeSupport.java: http://gerrit.cloudera.org:8080/#/c/8150/2/fe/src/main/java/org/apache/impala/service/FeSupport.java@285 PS2, Line 285: */ > Probably worth commenting about the format of the output buffer. Done http://gerrit.cloudera.org:8080/#/c/8150/2/fe/src/main/java/org/apache/impala/service/FeSupport.java@289 PS2, Line 289: if (dstLen <= 0) { : throw new InternalException( : "Payload is too big for LZ4 compression: " + payload.length); : } > Move this below L294? I don't think that's right because NativeLz4CompressBound() returns 0 if the payload is too big. If we do the check after adding 8 in L294, then the check won't work anymore. http://gerrit.cloudera.org:8080/#/c/8150/2/fe/src/main/java/org/apache/impala/service/FeSupport.java@306 PS2, Line 306: ByteBuffer.wrap(dst).putInt(0, payload.length); : ByteBuffer.wrap(dst).putInt(4, compressedSize); > Can't we return the resized buffer directly here? Basically merge the logic Not sure what you mean. A Java byte[] cannot be shrunk without copying it into a new one. The ByteBuffer trick works without copying. The purpose of the ByteBuffer version is to hand it to a thrift binary which will only copy as many bytes as the ByteBuffer has left (which might be different than the size of the byte[] backing the ByteBuffer). We could switch to using a ByteBuffer everywhere but that seems kinda complicated. -- To view, visit http://gerrit.cloudera.org:8080/8150 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I237802944875b07080db0159ff8ec548150fd95e Gerrit-Change-Number: 8150 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Behm <[email protected]> Gerrit-Reviewer: Alex Behm <[email protected]> Gerrit-Reviewer: Bharath Vissapragada <[email protected]> Gerrit-Reviewer: Philip Zeyliger <[email protected]> Gerrit-Comment-Date: Fri, 29 Sep 2017 03:59:17 +0000 Gerrit-HasComments: Yes
