I suspect having only one bolt per worker may be holding things a back a bit. Also check if enabling/disabling load aware messaging makes any diff. BTW..what is your avg event size ?
Sent from Yahoo Mail for iPhone On Tuesday, January 8, 2019, 7:06 PM, Bernard Quizon <[email protected]> wrote: Hi, I've tried using the patched `storm-kafka-client` and my throughput was up from 48K TPS to 585K TPS using my 6 supervisor node setup.Thank you Stig and Roshan. -Bernard On Mon, Jan 7, 2019 at 5:47 PM Bernard Quizon <[email protected]> wrote: Wow, thanks Stig! That's very helpful. On Mon, Jan 7, 2019 at 4:50 PM Stig Rohde Døssing <[email protected]> wrote: I'm not sure if it's the only issue, but you might be affected by https://issues.apache.org/jira/browse/STORM-3102. Den man. 7. jan. 2019 kl. 09.37 skrev Bernard Quizon <[email protected]>: Hi Roshan, Just to narrow it down .. can u get rid of the bolt and see what numbers you get ? > Thanks, I will try that. Which Kafka spout are you using (new/old) ? > The new one, from kafka-clients. I assume you mean msgs when you say ‘transactions’. > Yes that would be messages Did you mean you are getting 48k aggregate Msgs/sec or is that 48k per worker ? > That would be the throughput of the whole topology, not per worker. Thanks, Bernard On Mon, Jan 7, 2019 at 4:24 PM Roshan Naik <[email protected]> wrote: Just to narrow it down .. can u get rid of the bolt and see what numbers you get ? Which Kafka spout are you using (new/old) ? I assume you mean msgs when you say ‘transactions’. Did you mean you are getting 48k aggregate Msgs/sec or is that 48k per worker ? Sent from Yahoo Mail for iPhone On Sunday, January 6, 2019, 11:29 PM, Bernard Quizon <[email protected]> wrote: Hi, Before anything else, I'm using Apache Storm 1.2.1 and Apache Kafka 1.1.1. On my initial tests, I have: - 6 supervisor nodes - 1 worker per node - 7 KafkaSpout executors per worker - 1 bolt (that does nothing) per worker - 0 Ackers - 2 Kafka brokers with 42 partitions on the topic With that configuration, no matter how I change some other configs (Please see list below), my topology is capping out at around 48k TPS.Please note that CPU usage, for the Supervisor nodes, is only around 20% and Network usage is only around 20Mbps for both the Kafka Brokers and Supervisor nodes, well below the network capacities. Now, I have increased the supervisor nodes from 6 to 12 and used a new topic with 82 partitions, thinking that scaling out could help the performance.So is this the new configuration: - 12 supervisor nodes - 1 worker per node - 7 KafkaSpout executors per worker - 1 bolt (that does nothing) per worker - 0 Ackers - 2 Kafka brokers with 84 partitions on the topic And I'm still getting around 48k TPS. Some other configs I played around with: - max.poll.records - max.spout.pending - processing.guarantee - offset.commit.period.ms - max.uncommitted.offsets - poll.timeout.ms - fetch.min.bytes - fetch.max.bytes - max.partition.fetch.bytes - receive.buffer.bytes - fetch.max.wait.ms - topology.executor.receive.buffer.size - topology.executor.send.buffer.size - topology.receiver.buffer.size - topology.transfer.buffer.size Am I missing something here? Why is the throughput not improving? Just to add, I have also done some performance isolation tests on both Kafka and Storm.On a distributed consumer using Spark and Kafka, I was able to get around 700K TPS (So we know that Kafka isn't the issue here).And also I could get around 400k TPS on a custom Storm topology with 1 spout that generates random transactions and 1 bolt that does nothing. I feel like the numbers don't add up and the topology shouldn't be capping out at around 48K TPS.Your suggestions would be very much appreciated. Thanks, Bernard
