dongjoon-hyun commented on code in PR #59998:
URL: https://github.com/apache/spark/pull/59998#discussion_r4436991104
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala:
##########
@@ -105,6 +113,28 @@ private[sql] class
BoundedZstdCompressionCodec(compressionLevel: Int)
compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH +
bytesWritten)
compressedBuffer
}
+
+ override def compress(allocator: BufferAllocator, uncompressedBuffer:
ArrowBuf): ArrowBuf = {
+ val compressed = super.compress(allocator, uncompressedBuffer)
+ val length = compressed.writerIndex()
+ // Requiring at least half of the buffer back also limits the extra memory
that the copy needs
+ // while both buffers are alive. Empty and raw outputs are already in a
buffer of their size.
+ if (allocator.getRoundingPolicy.getRoundedSize(length) >
compressed.capacity() / 2) {
Review Comment:
Thank you, @peter-toth. You are right, and small batches turned out to be
worse than my table showed: at 10 rows, `getRecordBatch` and `close` cost up to
31.8% more with 8 threads. I added a floor of 4 KiB on the saving, as you
suggested. All the measured batches are now within 1% of the codec without the
shrink, and the floor gives up less than 4 KiB per Arrow buffer, about 32 KiB
for the 10,000-row `lineitem` batch. A 64 KiB floor would have held nearly
twice as much memory at that size.
I also changed the input of the halving test to 20 MiB, because below 16 MiB
the floor already keeps an output that does not halve its buffer. The PR
description has the new numbers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
