comphead commented on code in PR #6985:
URL: https://github.com/apache/datafusion-comet/pull/6985#discussion_r4488344789
##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/CachedBatchIpc.scala:
##########
@@ -234,7 +238,9 @@ private[comet] object CachedBatchIpc {
// own buffers by now, so this does not touch it.
root.clear()
- val out = new ChunkedByteBufferOutputStream(chunkSize,
ByteBuffer.allocate)
+ val out = new ChunkedByteBufferOutputStream(
+ math.min(chunkSize.toLong, messageSize(recordBatch)).toInt,
+ ByteBuffer.allocate)
Review Comment:
The numbers in the description come from a probe that is not in the PR.
Could you run `CometInMemoryCacheBenchmark.runBuildBenchmark` before and after,
or add the probe there as a write-only case, so the gain can be reproduced? I
have not run it, so I do not know whether build time moves or only the
allocation rate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
