As far as we can tell, if you run without ackers, the Linux OOM killer is what eventually prevents Storm from flooding the topology anymore. ZMQ will happily accept tuples from upstream and buffer them off-heap until the Java component can drain the tuples (which may never happen). Same applies for any kind of topology WITH ackers but with too much geometric tuple growth.
It's true that disabling ackers will improve performance, but at that point it's kind of just up to you ensuring that downstream components are fast enough to keep up with upstream traffic. We aren't trying to send 300k tuple/s/node, so we find the overhead of ackers to be a more than acceptable cost. Michael Michael Rose (@Xorlev <https://twitter.com/xorlev>) Senior Platform Engineer, FullContact <http://www.fullcontact.com/> [email protected] On Sun, Nov 10, 2013 at 12:15 PM, Philip O'Toole <[email protected]> wrote: > This is a question about Storm 0.82. > > What are the implications of running without any acking? I realise that > reliability is then not guaranteed, but without ackers I understand that > the "topology.max.spout.pending" doesn't apply. But the Storm docs state > that it is highly-desirable to set this value to prevent "queue explosion". > > > https://github.com/nathanmarz/storm/wiki/Running-topologies-on-a-production-cluster > > Yet the docs also say that by disabling acking, one can improve > performance. > > https://github.com/nathanmarz/storm/wiki/Guaranteeing-message-processing > > So, is running without ackers something to be considered for production > systems? And if so, what throttles the Storm system from flooding the > topology with too many tuples (which I presumes eats up memory, heap, > whatever)? > > Thanks, > > Philip >
