kkewwei commented on code in PR #16836:
URL: https://github.com/apache/lucene/pull/16836#discussion_r4281998421
##########
lucene/core/src/java/org/apache/lucene/util/compress/LZ4.java:
##########
@@ -147,6 +147,95 @@ public static int decompress(DataInput compressed, int
decompressedLen, byte[] d
return dOff;
}
+ /**
+ * Same as {@link #decompress(DataInput, int, byte[], int)}, but reads the
compressed bytes from
+ * {@code compressed[cOff:cOff+cLen]}. Reading from a heap array avoids the
per-byte overhead of
+ * {@link DataInput#readByte()}, which dominates decompression when the
input is an {@code
+ * IndexInput}.
+ */
+ public static int decompress(
Review Comment:
Could we wrap the `compressed` in `ByteArrayDataInput` and reuse the
existing`DataInput`, which can reduce code duplication?
##########
lucene/core/src/java/org/apache/lucene/codecs/lucene90/LZ4WithPresetDictCompressionMode.java:
##########
@@ -144,6 +154,29 @@ public void decompress(DataInput in, int originalLength,
int offset, int length,
assert bytes.isValid();
}
+ /**
+ * Decompresses {@code decompressedLength} bytes of a block that holds
{@code blockLength}
+ * bytes. Compressible blocks are read into a byte[] first, since decoding
from a heap array is
+ * much faster than reading tokens one by one from the {@link DataInput}.
Incompressible blocks
+ * are mostly literals that are already copied in bulk, so they are
decoded from the {@link
+ * DataInput} to avoid copying them twice.
+ */
+ private int decompress(
+ DataInput in,
+ int compressedLength,
+ int blockLength,
+ int decompressedLength,
+ byte[] dest,
+ int dOff)
+ throws IOException {
+ if (compressedLength > blockLength - (blockLength >>> 3)) {
+ return LZ4.decompress(in, decompressedLength, dest, dOff);
+ }
+ compressed = ArrayUtil.growNoCopy(compressed, compressedLength);
Review Comment:
For cluster with many segments, we have observed that the`buffer` can
sometimes retain a significant amount of memory. I’m concerned that adding
another`compressed` may further increase memory usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
