jackluo923 opened a new pull request, #13782: URL: https://github.com/apache/pinot/pull/13782
This pull request enhances tar archive compression and decompression by adding support for ZStandard and LZ4, alongside GZip. The code is refactored for easy extensibility, allowing new compressors to be added with minimal effort if supported by Apache Commons Compress. Compression typically infers the compressor from the file extension, but it can also be specified by name. Decompression uses magic numbers to automatically identify the compressor, utilizing Apache Commons Compress functionality. This update is a step towards enabling ZStandard compression for segment data in Deepstore. As noted in the flamegraph profiling in issue #13319, GZip compression is currently a CPU bottleneck, consuming 5-10% of overall CPU usage in petabyte-scale production environments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
