Re: [PR] KAFKA-7632: Support Compression Level [kafka]
dongjinleekr commented on PR #10826: URL: https://github.com/apache/kafka/pull/10826#issuecomment-2228295007 @junrao @mimaison @clolov @ozturkberkay I greatly appreciate you finalizing this PR! After changing jobs, I could not get involved with the Kafka community since I had to focus on our in-house fork, aka Navercorp (@naver) Kafka. Now, I just restarted KIP-780 and am testing it with our in-house tiered storage plugin. As soon as I complete the work, I will re-open the PR. Thank you again, and stay tuned! :bowing_man: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-7632: Support Compression Level [kafka]
mimaison commented on PR #10826: URL: https://github.com/apache/kafka/pull/10826#issuecomment-2154623171 Closing this PR since https://github.com/apache/kafka/pull/15516 has been merged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-7632: Support Compression Level [kafka]
mimaison closed pull request #10826: KAFKA-7632: Support Compression Level URL: https://github.com/apache/kafka/pull/10826 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-7632: Support Compression Level [kafka]
mimaison commented on PR #10826: URL: https://github.com/apache/kafka/pull/10826#issuecomment-2048127235 I opened https://github.com/apache/kafka/pull/15516 to implement this KIP. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-7632: Support Compression Level [kafka]
ijuma commented on code in PR #10826: URL: https://github.com/apache/kafka/pull/10826#discussion_r850506417 ## clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java: ## @@ -188,6 +190,12 @@ public class ProducerConfig extends AbstractConfig { + " values are none, gzip, snappy, lz4, or zstd. " + "Compression is of full batches of data, so the efficacy of batching will also impact the compression ratio (more batching means better compression)."; +/** compression.level */ +public static final String COMPRESSION_LEVEL_CONFIG = "compression.level"; +private static final String COMPRESSION_LEVEL_DOC = "The compression level for all data generated by the producer. The default level and valid value is up to " ++ "compression.type. (none, snappy: not available. gzip: 1~9. lz4: 1~17. " Review Comment: I think the reason why @dongjinleekr didn't include those is that they may change at the compression library level. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-7632: Support Compression Level [kafka]
mimaison commented on PR #10826: URL: https://github.com/apache/kafka/pull/10826#issuecomment-1931723005 I updated the [KIP](https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Support+Compression+Level) based on [Jun's suggestion](https://github.com/apache/kafka/pull/10826#issuecomment-1795952612) and sent an update to the VOTE thread. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-7632: Support Compression Level [kafka]
mimaison commented on PR #10826: URL: https://github.com/apache/kafka/pull/10826#issuecomment-1801941525 Yes it would be good to get this over the line. @dongjinleekr Do you think you'll have time to update it? or would you need some help? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-7632: Support Compression Level [kafka]
junrao commented on PR #10826: URL: https://github.com/apache/kafka/pull/10826#issuecomment-1795952612 One discussion of the PR is around choosing the default value for `compression.level` (https://github.com/apache/kafka/pull/10826#discussion_r917108950). This is tricky since it's specific to the compression type. I am wondering if it's better to adjust the KIP to support `compression.{gzip|lz4|zstd}.level` instead. This will make it more aligned with the future KIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-780%3A+Support+fine-grained+compression+options#KIP780:Supportfinegrainedcompressionoptions-Zstd.1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-7632: Support Compression Level [kafka]
ozturkberkay commented on PR #10826: URL: https://github.com/apache/kafka/pull/10826#issuecomment-1786847820 Hi, I was wondering if this PR would be prioritized in the foreseeable future given there has been a lot of work done already -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org