lhotari commented on PR #23525: URL: https://github.com/apache/pulsar/pull/23525#issuecomment-2447010383
@liangyepianzhou Regarding performance optimizations for compression in Pulsar, there's also work that should be done. For example for gzip compression/decompression this is very inefficient: https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-common/src/main/java/org/apache/pulsar/common/compression/CompressionCodecZLib.java#L60-L85 . Another detail is that the current implementation isn't using "zero copy" approaches that are available. For example in Snappy: https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-common/src/test/java/org/apache/pulsar/common/compression/CompressionCodecSnappyJNI.java#L34-L64 In BookKeeper, I added zero-copy for calculating checksums in https://github.com/apache/bookkeeper/pull/4196. The ByteBufVisitor approach could be used to avoid copying source buffers to an extra nio buffer. Calling Netty's io.netty.buffer.CompositeByteBuf#nioBuffer will allocate a new nio ByteBuffer in the heap and copy the content there. That's not very great from performance perspective, especially when we want to reduce allocations and garbage. With the ByteBufVisitor approach it's possible to read the source direct byte buffers without extra copies. **Have you considered in addressing this performance issue in the Pulsar message compression solution?** -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
