lhotari opened a new issue, #23586: URL: https://github.com/apache/pulsar/issues/23586
### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Motivation Here's an example of CompressionCodeZLib which has several opportunities for optimizations: https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-common/src/main/java/org/apache/pulsar/common/compression/CompressionCodecZLib.java#L60-L85 ### Solution The `java.util.zip.Deflater` class has contained methods for using `ByteBuffer` [input](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/zip/Deflater.html#setInput(java.nio.ByteBuffer)) and [output](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/zip/Deflater.html#deflate(java.nio.ByteBuffer,int)) since Java 11. In the case of Java 11+, the code could be optimized. Since the Pulsar Java client is Java 8+, using the `ByteBuffer` methods would require the use of reflection (unless a multi-release jar file is used with separate classes for Java 8 and Java 11). There's a reflection example in different situation in [BookKeeper's Java9IntHash class](https://github.com/apache/bookkeeper/blob/master/circe-checksum/src/main/java/com/scurrilous/circe/checksum/Java9IntHash.java). Regarding performance on Java 11+, the first problem is that it's using a heap buffer for the compressed buffer. A direct buffer would be more optimal when using the ByteBuffer methods with Deflater. For Netty ByteBuf input, it's possible to achieve zero copy in most cases by using Netty ByteBuf's `nioBuffer` method. It's notable that using `nioBuffer` method will cause copies when the Netty ByteBuf input is a CompositeByteBuf. Netty doesn't have a good way for zero copy of CompositeByteBuf input. In BookKeeper, there's a solution for checksum calculation in the https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/util/ByteBufVisitor.java class, which can visit all buffer parts to avoid extra copies. A similar solution would be applicable to compression. ### Alternatives _No response_ ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
