I'm not sure if it's the only issue, but you might be affected by https://issues.apache.org/jira/browse/STORM-3102.
Den man. 7. jan. 2019 kl. 09.37 skrev Bernard Quizon < [email protected]>: > Hi Roshan, > > Just to narrow it down .. can u get rid of the bolt and see what numbers > you get ? > > Thanks, I will try that. > > Which Kafka spout are you using (new/old) ? > > The new one, from kafka-clients. > > I assume you mean msgs when you say ‘transactions’. > > Yes that would be messages > > Did you mean you are getting 48k aggregate Msgs/sec or is that 48k per > worker ? > > That would be the throughput of the whole topology, not per worker. > > Thanks, > Bernard > > On Mon, Jan 7, 2019 at 4:24 PM Roshan Naik <[email protected]> wrote: > >> Just to narrow it down .. can u get rid of the bolt and see what >> numbers you get ? >> >> Which Kafka spout are you using (new/old) ? >> >> I assume you mean msgs when you say ‘transactions’. >> Did you mean you are getting 48k aggregate Msgs/sec or is that 48k per >> worker ? >> >> >> >> Sent from Yahoo Mail for iPhone >> <https://overview.mail.yahoo.com/?.src=iOS> >> >> On Sunday, January 6, 2019, 11:29 PM, Bernard Quizon < >> [email protected]> wrote: >> >> Hi, >> >> Before anything else, I'm using Apache Storm 1.2.1 and Apache Kafka 1.1.1. >> On my initial tests, I have: >> >> - *6 supervisor nodes * >> - 1 worker per node >> - 7 KafkaSpout executors per worker >> - 1 bolt (that does nothing) per worker >> - 0 Ackers >> - 2 Kafka brokers with *42 partitions *on the topic >> >> With that configuration, no matter how I change some other configs >> (Please see list below), my topology is capping out at around 48k TPS. >> Please note that CPU usage, for the Supervisor nodes, is only around 20% >> and Network usage is only around 20Mbps for both the Kafka Brokers and >> Supervisor nodes, well below the network capacities. >> >> Now, I have increased the supervisor nodes from 6 to 12 and used a new >> topic with 82 partitions, thinking that scaling out could help the >> performance. >> So is this the new configuration: >> >> - *12 supervisor nodes * >> - 1 worker per node >> - 7 KafkaSpout executors per worker >> - 1 bolt (that does nothing) per worker >> - 0 Ackers >> - 2 Kafka brokers with *84 partitions* on the topic >> >> And I'm still getting around 48k TPS. >> >> Some other configs I played around with: >> >> - max.poll.records >> - max.spout.pending >> - processing.guarantee >> - offset.commit.period.ms >> - max.uncommitted.offsets >> - poll.timeout.ms >> - fetch.min.bytes >> - fetch.max.bytes >> - max.partition.fetch.bytes >> - receive.buffer.bytes >> - fetch.max.wait.ms >> - topology.executor.receive.buffer.size >> - topology.executor.send.buffer.size >> - topology.receiver.buffer.size >> - topology.transfer.buffer.size >> >> >> Am I missing something here? Why is the throughput not improving? >> >> Just to add, I have also done some performance isolation tests on both >> Kafka and Storm. >> On a distributed consumer using Spark and Kafka, I was able to get around >> 700K TPS (So we know that Kafka isn't the issue here). >> And also I could get around 400k TPS on a custom Storm topology with 1 >> spout that generates random transactions and 1 bolt that does nothing. >> I feel like the numbers don't add up and the topology shouldn't be >> capping out at around 48K TPS. >> Your suggestions would be very much appreciated. >> >> >> Thanks, >> Bernard >> >>
