Re: Kafka Producer performance - 400GB of transfer on single instance taking > 72 hours?

oing load testing recently can with 3 brokers I can write 100MB/s (using Java clients). > > -Dave > > -Original Message- > From: Dominik Safaric [mailto:dominiksafa...@gmail.com] > Sent: Thursday, August 25, 2016 11:51 AM > To: users@kafka.apache.org > Subject: Re: K

RE: Kafka Producer performance - 400GB of transfer on single instance taking > 72 hours?

[mailto:dominiksafa...@gmail.com] Sent: Thursday, August 25, 2016 11:51 AM To: users@kafka.apache.org Subject: Re: Kafka Producer performance - 400GB of transfer on single instance taking > 72 hours? Dear Dana, > I would recommend > other tools for bulk transfers. What tools/languages

Re: Kafka Producer performance - 400GB of transfer on single instance taking > 72 hours?

Dear Dana, > I would recommend > other tools for bulk transfers. What tools/languages would you rather recommend then using Python? I could for sure accomplish the same by using the native Java Kafka Producer API, but should this really affect the performance under the assumption that the

Re: Synchronous producer performance is very low

The bottom line is synchronous will never be as fast as asynchronous. When using asynchronous mode you get the benefits of batching, compression, reduced network roundtrips, etc. You do loose the ability to achieve absolute ordering, but do you really need that? Can you design you system not to?

Multithreading to improve producer performance.

Hi All, I'm new to Kafka and in the process of writing a producer. Just to give you the context, my producer reads a binary file, decodes it according to a predefined structure (message length followed by the message) and publishes the decoded messages based on its type to the topic. For instance

Re: Synchronous producer performance is very low

Hi Grant, Thanks for the reply. Yes I meant data is received on the other end in 1sec while using asynchronous producer. We are using the new producer and absolute ordering is our prime concern too. Thanks, Santhakumari On Mon, Oct 19, 2015 at 9:12 PM, Grant Henke wrote: >

Warning in producer performance test

kafka-producer-perf-test.sh --broker-list 192.168.1.107:9092 --message-size 600 --messages 17000 --topic jun8 start.time, end.time, compression, message.size, batch.size, total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec [2015-06-19 14:07:46,196] WARN Failed to send producer

Re: Experiences testing new producer performance across multiple threads/producer counts

-Original Message- From: Guozhang Wang [mailto:wangg...@gmail.com] Sent: 14 May 2015 18:57 To: users@kafka.apache.org Subject: Re: Experiences testing new producer performance across multiple threads/producer counts Regarding the issue that adding more partitions kill the performance: I would

RE: Experiences testing new producer performance across multiple threads/producer counts

re-visit this and do deeper profiling over the next week and will revert here with findings. Regards Garry -Original Message- From: Guozhang Wang [mailto:wangg...@gmail.com] Sent: 14 May 2015 18:57 To: users@kafka.apache.org Subject: Re: Experiences testing new producer performance

Re: Experiences testing new producer performance across multiple threads/producer counts

Regarding the issue that adding more partitions kill the performance: I would suspect it maybe due to not-sufficient batching. Note that in the new producer batching is done per-partition, and if linger.ms setting low, partition data may not be batched enough before they got sent to the brokers.

Experiences testing new producer performance across multiple threads/producer counts

Hi, I talked with Gwen at Strata last week and promised to share some of my experiences benchmarking an app reliant on the new producer. I'm using relatively meaty boxes running my producer code (24 core/64GB RAM) but I wasn't pushing them until I got them on the same 10GB fabric as the Kafka

Re: Experiences testing new producer performance across multiple threads/producer counts

Thanks for sharing this, Garry. I actually did similar tests before but unfortunately lost the test data because my laptop rebooted and I forgot to save the dataŠ Anyway, several things to verify: 1. Remember KafkaProducer holds lock per partition. So if you have only one partition in the target

Re: Experiences testing new producer performance across multiple threads/producer counts

Hey Garry, Super interesting. We honestly never did a ton of performance tuning on the producer. I checked the profiles early on in development and we fixed a few issues that popped up in deployment, but I don't think anyone has done a really scientific look. If you (or anyone else) want to dive

Re: kafka producer performance test

and it should be there. -Jay On Wed, Oct 1, 2014 at 7:55 PM, Sa Li sal...@gmail.com wrote: Hi, All I built a 3-node kafka cluster, I want to make performance test, I found someone post following thread, that is exactly the problem I have: - While testing kafka producer

Re: kafka producer performance test

wrote: Hi, All I built a 3-node kafka cluster, I want to make performance test, I found someone post following thread, that is exactly the problem I have: - While testing kafka producer performance, I found 2 testing scripts. 1) performance testing script in kafka

Re: kafka producer performance test

post following thread, that is exactly the problem I have: - While testing kafka producer performance, I found 2 testing scripts. 1) performance testing script in kafka distribution bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages

Re: kafka producer performance test

On Wed, Oct 1, 2014 at 7:55 PM, Sa Li sal...@gmail.com wrote: Hi, All I built a 3-node kafka cluster, I want to make performance test, I found someone post following thread, that is exactly the problem I have: - While testing kafka producer performance, I

Re: kafka producer performance test

the problem I have: - While testing kafka producer performance, I found 2 testing scripts. 1) performance testing script in kafka distribution bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages 1000 --topic test --threads 10

kafka producer performance test

Hi, All I built a 3-node kafka cluster, I want to make performance test, I found someone post following thread, that is exactly the problem I have: - While testing kafka producer performance, I found 2 testing scripts. 1) performance testing script in kafka distribution bin/kafka

Re: kafka producer performance test

following thread, that is exactly the problem I have: - While testing kafka producer performance, I found 2 testing scripts. 1) performance testing script in kafka distribution bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages 1000 --topic test --threads

Re: kafka producer performance test

PM, Sa Li sal...@gmail.com wrote: Hi, All I built a 3-node kafka cluster, I want to make performance test, I found someone post following thread, that is exactly the problem I have: - While testing kafka producer performance, I found 2 testing scripts. 1) performance testing

Re: kafka producer performance test

post following thread, that is exactly the problem I have: - While testing kafka producer performance, I found 2 testing scripts. 1) performance testing script in kafka distribution bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages 1000 --topic test

Re: producer performance

, Chen Song chen.song...@gmail.com wrote: While testing kafka producer performance, I found 2 testing scripts. 1) performance testing script in kafka distribution bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages 1000 --topic test --threads 10 --message-size 100

Re: producer performance

...@gmail.com wrote: While testing kafka producer performance, I found 2 testing scripts. 1) performance testing script in kafka distribution bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages 1000 --topic test --threads 10 --message-size 100 --batch-size 1

Re: Kafka producer performance test sending 0x0 byte messages

had planned to try and edit the producer performance tester source and do the following 1. Add an option to read sample data from provided text file. (thought would be to add a file with 1-5000 rows, whatever I thought my batch size might be) 2. Load sample file

Kafka producer performance test sending 0x0 byte messages

Hi folks I was doing some performance testing using the built in Kafka performance tester and it seems like it sends messages of size n bytes but with all bytes having the value 0x0. Is that correct? Reading the source seemed to indicate that too but I'm not a Scala developer so I could be

Re: Kafka producer performance test sending 0x0 byte messages

Daniel, We have the same question. We noticed that the compression tests we ran using the built in performance tester was not realistic. I think on disk compression was 200:1. (yes that is two hundred to one) I had planned to try and edit the producer performance tester source and do

Re: Kafka producer performance test sending 0x0 byte messages

Yes, this is a problem and will indeed affect the producer performance when compression is turned on. Perhaps we should fill in the values with some randomized bytes. Could you file a jira for this? Thanks, Jun On Sun, Jun 29, 2014 at 11:24 PM, Daniel Compton d...@danielcompton.net wrote: Hi

Re: Kafka producer performance test sending 0x0 byte messages

that is two hundred to one) I had planned to try and edit the producer performance tester source and do the following 1. Add an option to read sample data from provided text file. (thought would be to add a file with 1-5000 rows, whatever I thought my batch size might be) 2. Load

Re: Analysis of producer performance -- and Producer-Kafka reliability

: Tuesday, April 9, 2013 1:23 PM Subject: Re: Analysis of producer performance Jun, Thank you for your comments. I'll reply point by point for clarity. 1. We were aware of the migration tool but since we haven't used Kafka for production yet we just started using the 0.8 version directly. 2. I

Re: Analysis of producer performance -- and Producer-Kafka reliability

From: Piotr Kozikowski pi...@liveramp.com To: users@kafka.apache.org Sent: Tuesday, April 9, 2013 1:23 PM Subject: Re: Analysis of producer performance Jun, Thank you for your comments. I'll reply point by point for clarity. 1. We were

Re: Analysis of producer performance

compression or using a cheaper compression codec like snappy. Thanks, Jun On Fri, Apr 12, 2013 at 4:08 PM, Piotr Kozikowski pi...@liveramp.comwrote: Hi all, I posted an update on the post ( https://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/) to test the effect of disabling ack

Re: Analysis of producer performance -- and Producer-Kafka reliability

/ HBase - http://sematext.com/spm From: Piotr Kozikowski pi...@liveramp.com To: users@kafka.apache.org Sent: Tuesday, April 9, 2013 1:23 PM Subject: Re: Analysis of producer performance Jun, Thank you

Re: Analysis of producer performance -- and Producer-Kafka reliability

: Tuesday, April 9, 2013 1:23 PM Subject: Re: Analysis of producer performance Jun, Thank you for your comments. I'll reply point by point for clarity. 1. We were aware of the migration tool but since we haven't used Kafka for production yet we just started

Re: Analysis of producer performance -- and Producer-Kafka reliability

But it shouldn't almost never happen. Obviously I mean it should almost never happen. Not shouldn't. Philip

Re: Analysis of producer performance

Hi all, I posted an update on the post ( https://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/) to test the effect of disabling ack messages from brokers. It appears this only makes a big difference (~2x improvement ) when using synthetic log messages, but only a modest 12

Re: Analysis of producer performance

of producer performance Jun, Thank you for your comments. I'll reply point by point for clarity. 1. We were aware of the migration tool but since we haven't used Kafka for production yet we just started using the 0.8 version directly. 2. I hadn't seen those particular slides, very

Re: Analysis of producer performance

, 2013 1:23 PM Subject: Re: Analysis of producer performance Jun, Thank you for your comments. I'll reply point by point for clarity. 1. We were aware of the migration tool but since we haven't used Kafka for production yet we just started using the 0.8 version directly. 2. I hadn't seen those

Re: Analysis of producer performance

- http://sematext.com/spm From: Piotr Kozikowski pi...@liveramp.com To: users@kafka.apache.org Sent: Tuesday, April 9, 2013 1:23 PM Subject: Re: Analysis of producer performance Jun, Thank you for your comments. I'll reply point by point for clarity. 1

Re: Analysis of producer performance

: Analysis of producer performance Jun, Thank you for your comments. I'll reply point by point for clarity. 1. We were aware of the migration tool but since we haven't used Kafka for production yet we just started using the 0.8 version directly. 2. I hadn't seen those particular slides

Re: Analysis of producer performance

From: Piotr Kozikowski pi...@liveramp.com To: users@kafka.apache.org Sent: Tuesday, April 9, 2013 1:23 PM Subject: Re: Analysis of producer performance Jun, Thank you for your comments. I'll reply point by point for clarity. 1. We were aware of the migration tool but since we

Re: Analysis of producer performance

socket on the client side. Jun On Mon, Apr 8, 2013 at 4:42 PM, Piotr Kozikowski pi...@liveramp.com wrote: Hi, At LiveRamp we are considering replacing Scribe with Kafka, and as a first step we run some tests to evaluate producer performance. You can find our preliminary results here: https

Re: Analysis of producer performance

on the built-in producer performance tool, I know that we started getting very different numbers once we switched to use real (actual production log) messages. Compression may also be a factor in case it wasn't configured the same way in those tests. 3. In the latency section, there are two tests, one

Analysis of producer performance

Hi, At LiveRamp we are considering replacing Scribe with Kafka, and as a first step we run some tests to evaluate producer performance. You can find our preliminary results here: https://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/. We hope this will be useful for some folks