Just to narrow it down .. can u get rid of the bolt and see what numbers you get ? Which Kafka spout are you using (new/old) ? I assume you mean msgs when you say ‘transactions’. Did you mean you are getting 48k aggregate Msgs/sec or is that 48k per worker ?
Sent from Yahoo Mail for iPhone On Sunday, January 6, 2019, 11:29 PM, Bernard Quizon <[email protected]> wrote: Hi, Before anything else, I'm using Apache Storm 1.2.1 and Apache Kafka 1.1.1. On my initial tests, I have: - 6 supervisor nodes - 1 worker per node - 7 KafkaSpout executors per worker - 1 bolt (that does nothing) per worker - 0 Ackers - 2 Kafka brokers with 42 partitions on the topic With that configuration, no matter how I change some other configs (Please see list below), my topology is capping out at around 48k TPS.Please note that CPU usage, for the Supervisor nodes, is only around 20% and Network usage is only around 20Mbps for both the Kafka Brokers and Supervisor nodes, well below the network capacities. Now, I have increased the supervisor nodes from 6 to 12 and used a new topic with 82 partitions, thinking that scaling out could help the performance.So is this the new configuration: - 12 supervisor nodes - 1 worker per node - 7 KafkaSpout executors per worker - 1 bolt (that does nothing) per worker - 0 Ackers - 2 Kafka brokers with 84 partitions on the topic And I'm still getting around 48k TPS. Some other configs I played around with: - max.poll.records - max.spout.pending - processing.guarantee - offset.commit.period.ms - max.uncommitted.offsets - poll.timeout.ms - fetch.min.bytes - fetch.max.bytes - max.partition.fetch.bytes - receive.buffer.bytes - fetch.max.wait.ms - topology.executor.receive.buffer.size - topology.executor.send.buffer.size - topology.receiver.buffer.size - topology.transfer.buffer.size Am I missing something here? Why is the throughput not improving? Just to add, I have also done some performance isolation tests on both Kafka and Storm.On a distributed consumer using Spark and Kafka, I was able to get around 700K TPS (So we know that Kafka isn't the issue here).And also I could get around 400k TPS on a custom Storm topology with 1 spout that generates random transactions and 1 bolt that does nothing. I feel like the numbers don't add up and the topology shouldn't be capping out at around 48K TPS.Your suggestions would be very much appreciated. Thanks, Bernard
