Wow, thanks Stig! That's very helpful. On Mon, Jan 7, 2019 at 4:50 PM Stig Rohde Døssing <[email protected]> wrote:
> I'm not sure if it's the only issue, but you might be affected by > https://issues.apache.org/jira/browse/STORM-3102. > > Den man. 7. jan. 2019 kl. 09.37 skrev Bernard Quizon < > [email protected]>: > >> Hi Roshan, >> >> Just to narrow it down .. can u get rid of the bolt and see what >> numbers you get ? >> > Thanks, I will try that. >> >> Which Kafka spout are you using (new/old) ? >> > The new one, from kafka-clients. >> >> I assume you mean msgs when you say ‘transactions’. >> > Yes that would be messages >> >> Did you mean you are getting 48k aggregate Msgs/sec or is that 48k per >> worker ? >> > That would be the throughput of the whole topology, not per worker. >> >> Thanks, >> Bernard >> >> On Mon, Jan 7, 2019 at 4:24 PM Roshan Naik <[email protected]> wrote: >> >>> Just to narrow it down .. can u get rid of the bolt and see what >>> numbers you get ? >>> >>> Which Kafka spout are you using (new/old) ? >>> >>> I assume you mean msgs when you say ‘transactions’. >>> Did you mean you are getting 48k aggregate Msgs/sec or is that 48k per >>> worker ? >>> >>> >>> >>> Sent from Yahoo Mail for iPhone >>> <https://overview.mail.yahoo.com/?.src=iOS> >>> >>> On Sunday, January 6, 2019, 11:29 PM, Bernard Quizon < >>> [email protected]> wrote: >>> >>> Hi, >>> >>> Before anything else, I'm using Apache Storm 1.2.1 and Apache Kafka >>> 1.1.1. >>> On my initial tests, I have: >>> >>> - *6 supervisor nodes * >>> - 1 worker per node >>> - 7 KafkaSpout executors per worker >>> - 1 bolt (that does nothing) per worker >>> - 0 Ackers >>> - 2 Kafka brokers with *42 partitions *on the topic >>> >>> With that configuration, no matter how I change some other configs >>> (Please see list below), my topology is capping out at around 48k TPS. >>> Please note that CPU usage, for the Supervisor nodes, is only around 20% >>> and Network usage is only around 20Mbps for both the Kafka Brokers and >>> Supervisor nodes, well below the network capacities. >>> >>> Now, I have increased the supervisor nodes from 6 to 12 and used a new >>> topic with 82 partitions, thinking that scaling out could help the >>> performance. >>> So is this the new configuration: >>> >>> - *12 supervisor nodes * >>> - 1 worker per node >>> - 7 KafkaSpout executors per worker >>> - 1 bolt (that does nothing) per worker >>> - 0 Ackers >>> - 2 Kafka brokers with *84 partitions* on the topic >>> >>> And I'm still getting around 48k TPS. >>> >>> Some other configs I played around with: >>> >>> - max.poll.records >>> - max.spout.pending >>> - processing.guarantee >>> - offset.commit.period.ms >>> - max.uncommitted.offsets >>> - poll.timeout.ms >>> - fetch.min.bytes >>> - fetch.max.bytes >>> - max.partition.fetch.bytes >>> - receive.buffer.bytes >>> - fetch.max.wait.ms >>> - topology.executor.receive.buffer.size >>> - topology.executor.send.buffer.size >>> - topology.receiver.buffer.size >>> - topology.transfer.buffer.size >>> >>> >>> Am I missing something here? Why is the throughput not improving? >>> >>> Just to add, I have also done some performance isolation tests on both >>> Kafka and Storm. >>> On a distributed consumer using Spark and Kafka, I was able to get >>> around 700K TPS (So we know that Kafka isn't the issue here). >>> And also I could get around 400k TPS on a custom Storm topology with 1 >>> spout that generates random transactions and 1 bolt that does nothing. >>> I feel like the numbers don't add up and the topology shouldn't be >>> capping out at around 48K TPS. >>> Your suggestions would be very much appreciated. >>> >>> >>> Thanks, >>> Bernard >>> >>>
