Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/582#issuecomment-122081360
@chuanlei I like auto-tuning max spout pending, but I am confused what we
are tuning it for. The current code seems to increase the size of
max-spout-pending if the number of fully acked tuples is increasing, and
reduces it if the number appears to be decreasing. Why do we bother with this
at all? If we just set the max spout pending to a large enough number we will
get the same result.
The reason behind having a max-spout-pending is so we don't blow the java
heap, by having too many messages in flight at any point in time, and balance
the throughput with the latency. If we have more messages in flight the
latency will go up, but the throughput increases. If we have fewer in flight
the latency is much lower but the throughput is also lower.
I would really prefer to see a solution where a user can set a latency
target, ideally 95th percentile or 99th percentile target and the spout adjusts
the max-spout-pending to try and hit that latency target, with an upper bound
on max-spout-pending so we don't overflow the heap.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
