Github user roshannaik commented on the issue:
https://github.com/apache/storm/pull/2241
@revans2
About that "better than sliced bread" :
how could i not be offended.. at least briefly ;-) but you could buy me
lunch if this PR turns out better than you were initially afraid of. It was
perhaps a very "low latency" -1 for any PR in the history of Storm. :-)
Yes there are rough edges and some bugs... but I do dream of being able
nail it all the way in one go.
Your observation about the **very high latency for low throughput** topos.
That is clearly a problem with batch not filling up and not getting flushed.
The 5 sec latency corresponds to the 'topology.flush.tuple.freq.millis' setting
(default 5sec). So at each step between Spout->Bolt and Bolt->Bolt if the its
waiting for 5 sec then you are likely to see such ridiculous latency numbers.
Given that, I think the solution must be evident by now ... but will state
it here for the benefit of other readers to whom it may not be:
Tweak one or both of these two settings:
- **topology.producer.batch.size** : for low throughput topos setting this
to 1 is a good idea. In the new system the throughput penalty is not that for a
small batch much compared to larger batch size.
- **topology.flush.tuple.freq.millis**: You could reduce this for more
frequent batch flushing. It causes the timer thread to take up more cpu... but
if throughput fluctuates between very and very low over the day, then is
setting maybe better to meet latency SLA.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
