What is the optimal max spout pending to use in a topology ? I found this thread here: http://mail-archives.apache.org/mod_mbox/storm-user/201402.mbox/%3cca+avhzatfg_s88lkombvommkh-rafwr6szy0i8b8tm3rfab...@mail.gmail.com%3E that didn't seem to have a follow up.
Part of it says to "Start with a max spout pending that is for sure too small -- one for trident, or the number of executors for storm -- and increase it until you stop seeing changes in the flow. You'll probably end up with something near 2*(throughput in recs/sec)*(end-to-end latency) (2x the Little's law capacity)." Does this make sense for a Max Spout Pending value ? I expect my topology to have a throughput of around 80,000/s and I've been seeing a complete latency of around 300ms, so given this formula, I'd want 2*80000*.3 = 48,000 Max Spout Pending. This seems absurdly high to me.. -- Raphael Hsieh
