liangyepianzhou commented on PR #23525: URL: https://github.com/apache/pulsar/pull/23525#issuecomment-2467552702
> @liangyepianzhou Regarding performance optimizations for compression in Pulsar, there's also work that should be done. For example for gzip compression/decompression this is very inefficient: > > https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-common/src/main/java/org/apache/pulsar/common/compression/CompressionCodecZLib.java#L60-L85 > > . > Another detail is that the current implementation isn't using "zero copy" approaches that are available. > For example in Snappy: > https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-common/src/test/java/org/apache/pulsar/common/compression/CompressionCodecSnappyJNI.java#L34-L64 > > > In BookKeeper, I added zero-copy for calculating checksums in [apache/bookkeeper#4196](https://github.com/apache/bookkeeper/pull/4196). The ByteBufVisitor approach could be used to avoid copying source buffers to an extra nio buffer. > Calling Netty's io.netty.buffer.CompositeByteBuf#nioBuffer will allocate a new nio ByteBuffer in the heap and copy the content there. That's not very great from performance perspective, especially when we want to reduce allocations and garbage. With the ByteBufVisitor approach it's possible to read the source direct byte buffers without extra copies. > **Have you considered in addressing this performance issue in the Pulsar message compression solution?** @lhotari I try to optimize it yesterday. but I found that Pulsar did not use `CompositeByteBuf` when sending messages. Single sending uses `io.netty.buffer.UnpooledHeapByteBufmemory`, while batch sending uses `io.netty.buffer.PooledUnsafeDirectByteBufmemory`. They all have memory addresses or array, so the current implementation already uses zero copy. It seems that there is no need to optimize? And the example you gave, `CompressionCodecSnappyJNI.java`, is a compression class used in testing. It seems that there is no need to compress it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
