Hi, I've tried using the patched `storm-kafka-client` and my throughput was up from 48K TPS to 585K TPS using my 6 supervisor node setup. Thank you Stig and Roshan.
-Bernard On Mon, Jan 7, 2019 at 5:47 PM Bernard Quizon < [email protected]> wrote: > Wow, thanks Stig! That's very helpful. > > On Mon, Jan 7, 2019 at 4:50 PM Stig Rohde Døssing <[email protected]> > wrote: > >> I'm not sure if it's the only issue, but you might be affected by >> https://issues.apache.org/jira/browse/STORM-3102. >> >> Den man. 7. jan. 2019 kl. 09.37 skrev Bernard Quizon < >> [email protected]>: >> >>> Hi Roshan, >>> >>> Just to narrow it down .. can u get rid of the bolt and see what >>> numbers you get ? >>> > Thanks, I will try that. >>> >>> Which Kafka spout are you using (new/old) ? >>> > The new one, from kafka-clients. >>> >>> I assume you mean msgs when you say ‘transactions’. >>> > Yes that would be messages >>> >>> Did you mean you are getting 48k aggregate Msgs/sec or is that 48k per >>> worker ? >>> > That would be the throughput of the whole topology, not per worker. >>> >>> Thanks, >>> Bernard >>> >>> On Mon, Jan 7, 2019 at 4:24 PM Roshan Naik <[email protected]> >>> wrote: >>> >>>> Just to narrow it down .. can u get rid of the bolt and see what >>>> numbers you get ? >>>> >>>> Which Kafka spout are you using (new/old) ? >>>> >>>> I assume you mean msgs when you say ‘transactions’. >>>> Did you mean you are getting 48k aggregate Msgs/sec or is that 48k per >>>> worker ? >>>> >>>> >>>> >>>> Sent from Yahoo Mail for iPhone >>>> <https://overview.mail.yahoo.com/?.src=iOS> >>>> >>>> On Sunday, January 6, 2019, 11:29 PM, Bernard Quizon < >>>> [email protected]> wrote: >>>> >>>> Hi, >>>> >>>> Before anything else, I'm using Apache Storm 1.2.1 and Apache Kafka >>>> 1.1.1. >>>> On my initial tests, I have: >>>> >>>> - *6 supervisor nodes * >>>> - 1 worker per node >>>> - 7 KafkaSpout executors per worker >>>> - 1 bolt (that does nothing) per worker >>>> - 0 Ackers >>>> - 2 Kafka brokers with *42 partitions *on the topic >>>> >>>> With that configuration, no matter how I change some other configs >>>> (Please see list below), my topology is capping out at around 48k TPS. >>>> Please note that CPU usage, for the Supervisor nodes, is only around >>>> 20% and Network usage is only around 20Mbps for both the Kafka Brokers and >>>> Supervisor nodes, well below the network capacities. >>>> >>>> Now, I have increased the supervisor nodes from 6 to 12 and used a new >>>> topic with 82 partitions, thinking that scaling out could help the >>>> performance. >>>> So is this the new configuration: >>>> >>>> - *12 supervisor nodes * >>>> - 1 worker per node >>>> - 7 KafkaSpout executors per worker >>>> - 1 bolt (that does nothing) per worker >>>> - 0 Ackers >>>> - 2 Kafka brokers with *84 partitions* on the topic >>>> >>>> And I'm still getting around 48k TPS. >>>> >>>> Some other configs I played around with: >>>> >>>> - max.poll.records >>>> - max.spout.pending >>>> - processing.guarantee >>>> - offset.commit.period.ms >>>> - max.uncommitted.offsets >>>> - poll.timeout.ms >>>> - fetch.min.bytes >>>> - fetch.max.bytes >>>> - max.partition.fetch.bytes >>>> - receive.buffer.bytes >>>> - fetch.max.wait.ms >>>> - topology.executor.receive.buffer.size >>>> - topology.executor.send.buffer.size >>>> - topology.receiver.buffer.size >>>> - topology.transfer.buffer.size >>>> >>>> >>>> Am I missing something here? Why is the throughput not improving? >>>> >>>> Just to add, I have also done some performance isolation tests on both >>>> Kafka and Storm. >>>> On a distributed consumer using Spark and Kafka, I was able to get >>>> around 700K TPS (So we know that Kafka isn't the issue here). >>>> And also I could get around 400k TPS on a custom Storm topology with 1 >>>> spout that generates random transactions and 1 bolt that does nothing. >>>> I feel like the numbers don't add up and the topology shouldn't be >>>> capping out at around 48K TPS. >>>> Your suggestions would be very much appreciated. >>>> >>>> >>>> Thanks, >>>> Bernard >>>> >>>>
