GGraziadei opened a new issue, #8652: URL: https://github.com/apache/storm/issues/8652
This proposal introduces zstd as an optional compression codec to address the performance bottlenecks often associated with the current GZIP implementation. Located in the serialization layer of storm-client, GZIP frequently introduces unnecessary latency in real-time topologies; however, industry benchmarks from Cloudflare and Kafka (KIP-390) consistently demonstrate that zstd delivers significantly faster compression and decompression speeds without compromising the compression ratio. To maximize flexibility, I propose making the compression level user-configurable, with a recommended default of 3 to provide an optimal balance between CPU efficiency and data density. Implementing zstd will allow Storm users to reduce serialisation/deserialization processing latency and lower CPU overhead, bringing the project in line with modern data-streaming performance standards. Resources: - https://blog.cloudflare.com/new-standards/ - https://medium.com/trendyol-tech/optimizing-kafka-performance-through-data-compression-330fb31a0827 - https://themoah.medium.com/is-zstd-really-fast-7814abb4afcd - https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Support+Compression+Level -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
