Perhaps you can try some things depending on how feasible they it is for your case: 1- Try to design and deploy your topology to maximize communication within a worker process instead fo across worker processes. 2- Alternatively you can spin up multiple single worker instances of you toplo3- Do you need ACKing enabled ? if not disable acking as it has a perf hit.4- Play with the batch size a bit. ()5- Try the upcoming 2.0 stuff from the master branch. That should have much better perf out of the box. Take a look at suggestions in Performance.md for tuning it. Pay attention to the new names for some of the tunables like batch size.6- Just in case u have not already done ... make sure debugging or tracing is disabled and log level is at warn.
For measuring the raw speed of the messaging between spouts and bolts (within and across process), I tend to use the ConstSpout*java topologies from here https://github.com/apache/storm/tree/master/examples/storm-perf/src/main/java/org/apache/storm/perf You may want to change the message size used in them to suit your needs. On Thursday, August 2, 2018, 11:48:38 AM PDT, Jonathan Kim <jonathankim1...@gmail.com> wrote: Hi, I’m trying to optimize storm to process at least a million messages per second with each message ranging in size from 1kb to 5mb. I’ve been looking at other benchmarks and the highest I’ve seen was 43 mb per second which is awfully slow. Is it possible to achieve this task without spinning up hundreds of nodes to avoid spending too much on hardware to support it? Or is storm
