[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281803#comment-15281803 ] Jiangjie Qin commented on KAFKA-3565: - Got it. Yes, if the messages are keyed messages and the

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281777#comment-15281777 ] Gwen Shapira commented on KAFKA-3565: - Why do I need to override the handler? The hash function is

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281735#comment-15281735 ] Jiangjie Qin commented on KAFKA-3565: - Thanks for the clarification, Gwen. Just curious, by "each

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281213#comment-15281213 ] Gwen Shapira commented on KAFKA-3565: - This is consistent with my calculations, except that 90% of the

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281192#comment-15281192 ] Jiangjie Qin commented on KAFKA-3565: - [~gwenshap] I am not sure if 32K is minor. Typically a mirror

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15280933#comment-15280933 ] Gwen Shapira commented on KAFKA-3565: - 32K per partition sounds pretty minor to me. You'd need 30,000

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15280863#comment-15280863 ] Guozhang Wang commented on KAFKA-3565: -- Yes, that is a valid concern. Maybe we can add a note in the

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15280721#comment-15280721 ] Jiangjie Qin commented on KAFKA-3565: - [~junrao] [~guozhang] I think it makes sense to just use 32K as

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15280213#comment-15280213 ] Jun Rao commented on KAFKA-3565: Since that's an existing issue, perhaps file a new jira? > Producer's

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15279541#comment-15279541 ] Guozhang Wang commented on KAFKA-3565: -- I can do that, under this JIRA or file a separate one? >

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15279536#comment-15279536 ] Jun Rao commented on KAFKA-3565: [~guozhang], do you want to patch the producer to use the default buffer

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277142#comment-15277142 ] Guozhang Wang commented on KAFKA-3565: -- You are referring to {{new Compressor(buffer,

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277072#comment-15277072 ] Jiangjie Qin commented on KAFKA-3565: - [~guozhang] It seems that we are using the default buffer size

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277023#comment-15277023 ] Guozhang Wang commented on KAFKA-3565: -- Actually, in the producer's GZIP and SNAPPY compression code,

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275974#comment-15275974 ] Jiangjie Qin commented on KAFKA-3565: - Actually, never mind. The buffer size is not the batch size,

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275971#comment-15275971 ] Jiangjie Qin commented on KAFKA-3565: - The batch size was 80K. It is not quite clear to me how this

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275970#comment-15275970 ] Jiangjie Qin commented on KAFKA-3565: - [~junrao] Yes, the value bound was supposed to be included in

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275959#comment-15275959 ] Jun Rao commented on KAFKA-3565: [~becket_qin], also, it seems that you patched ProducerPerformance with

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275957#comment-15275957 ] Jun Rao commented on KAFKA-3565: [~becket_qin], thanks for confirming this. I guess defaulting the buffer

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275948#comment-15275948 ] Jiangjie Qin commented on KAFKA-3565: - BTW, I just created KAFKA-3677 to provide a tool to help user

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275934#comment-15275934 ] Jiangjie Qin commented on KAFKA-3565: - [~ijuma] We should absolutely mention the 8 bytes overhead. And

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275930#comment-15275930 ] Guozhang Wang commented on KAFKA-3565: -- Just curious what is the {{batch.size}} in your test code

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275924#comment-15275924 ] Jiangjie Qin commented on KAFKA-3565: - [~junrao] Yes, you are right. I changed the producer side

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275847#comment-15275847 ] Ismael Juma commented on KAFKA-3565: [~becket_qin], would you be able to submit a PR that adds a note

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275793#comment-15275793 ] Jun Rao commented on KAFKA-3565: [~becket_qin], thanks for the latest analysis. The different data size on

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275431#comment-15275431 ] Jiangjie Qin commented on KAFKA-3565: - [~junrao] I think I figured out the reason why 0.9 consumer has

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15274969#comment-15274969 ] Jiangjie Qin commented on KAFKA-3565: - [~junrao] I didn't have time to run the test yet. I will

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15274917#comment-15274917 ] Jun Rao commented on KAFKA-3565: [~becket_qin], any new findings on the consumer performance? Thanks. >

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15272750#comment-15272750 ] Jun Rao commented on KAFKA-3565: [~becket_qin], thanks for the latest consumer results. Yes, the snappy

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15272619#comment-15272619 ] Jiangjie Qin commented on KAFKA-3565: - [~junrao] I ran the tests again with more data and it looks the

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271795#comment-15271795 ] Jiangjie Qin commented on KAFKA-3565: - [~junrao] I noticed that the consumer tests finished very

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271185#comment-15271185 ] Jun Rao commented on KAFKA-3565: [~becket_qin], thanks for the results. As I was looking at the results

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269854#comment-15269854 ] Jiangjie Qin commented on KAFKA-3565: - Woops...Touched the the yubi key accidentally... > Producer's

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269491#comment-15269491 ] Jiangjie Qin commented on KAFKA-3565: - [~junrao] Thanks for help looking into this. I updated run 11

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269113#comment-15269113 ] Jun Rao commented on KAFKA-3565: [~becket_qin], thanks for the explanation. The results now make sense to

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268998#comment-15268998 ] Jiangjie Qin commented on KAFKA-3565: - [~junrao] The tests run above have two different linger.ms

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268985#comment-15268985 ] Jun Rao commented on KAFKA-3565: [~becket_qin], thanks for the new results. Are the stats in test_run9 and

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268242#comment-15268242 ] Jiangjie Qin commented on KAFKA-3565: - [~junrao] I updated the results in the Google sheet. The

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15267803#comment-15267803 ] Jun Rao commented on KAFKA-3565: Jiangjie, thanks for the results. It seems that the overall producer

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265529#comment-15265529 ] Jiangjie Qin commented on KAFKA-3565: - [~gwenshap] I ran a few tests and it seems the performance

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264905#comment-15264905 ] Gwen Shapira commented on KAFKA-3565: - This is marked as critical. [~becket_qin] and [~ijuma] - are

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256788#comment-15256788 ] Ismael Juma commented on KAFKA-3565: Thanks for your work on this [~becket_qin]. I finally have a bit

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254550#comment-15254550 ] Jiangjie Qin commented on KAFKA-3565: - [~jkreps] [~ijuma] I ran the tests a few more times and

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252352#comment-15252352 ] Jiangjie Qin commented on KAFKA-3565: - [~jkreps] Yes, I agree that "free" calls can make a difference

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252152#comment-15252152 ] Jay Kreps commented on KAFKA-3565: -- Oh yeah, I think the critical question is what is the bottleneck? In

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252139#comment-15252139 ] Jay Kreps commented on KAFKA-3565: -- [~becket_qin] Yeah I think you are saying something like "previously

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252123#comment-15252123 ] Jiangjie Qin commented on KAFKA-3565: - [~jkreps] The google doc should be accessible now. I think

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252069#comment-15252069 ] Jay Kreps commented on KAFKA-3565: -- [~ijuma] Yeah your data eliminates my guess. If the problem were the

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251422#comment-15251422 ] Jiangjie Qin commented on KAFKA-3565: - [~ijuma] [~junrao] [~jkreps] I just finished the parameterized

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249601#comment-15249601 ] Ismael Juma commented on KAFKA-3565: I ran the tests a couple of times with various settings to check

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249346#comment-15249346 ] Ismael Juma commented on KAFKA-3565: [~becket_qin], the results were reproducible when I ran them last

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15248198#comment-15248198 ] Jay Kreps commented on KAFKA-3565: -- I think [~becket_qin] noted a good possible explanation. The dominate

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15247198#comment-15247198 ] Jiangjie Qin commented on KAFKA-3565: - [~junrao] [~ijuma] I tried a few more things including 1)

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15246765#comment-15246765 ] Ismael Juma commented on KAFKA-3565: Interesting about the tuning that may be required due to the

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15246565#comment-15246565 ] Jiangjie Qin commented on KAFKA-3565: - [~ijuma] I see. So we do know that the throughput of a

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15246460#comment-15246460 ] Jun Rao commented on KAFKA-3565: [~becket_qin], in the benchmark result that Ismael posted, each message

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15245007#comment-15245007 ] Ismael Juma commented on KAFKA-3565: Becket, 1. Yes, the non-compressed throughput change is

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15244946#comment-15244946 ] Jiangjie Qin commented on KAFKA-3565: - [~ijuma] A few comments: 1. The performance improvement

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243862#comment-15243862 ] Ismael Juma commented on KAFKA-3565: Interesting. Good to know that there are cases where the new code

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243850#comment-15243850 ] Jiangjie Qin commented on KAFKA-3565: - The producer performance I ran was with similar tweaks you made

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243840#comment-15243840 ] Jiangjie Qin commented on KAFKA-3565: - [~ijuma] What is your other settings? e.g.

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243699#comment-15243699 ] Ismael Juma commented on KAFKA-3565: By the way, the results with compression include the following

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243695#comment-15243695 ] Ismael Juma commented on KAFKA-3565: [~becket_qin], do you think you can look into this? We need to