Nick Allen created METRON-1897:
----------------------------------
Summary: Warn About Max Spout Pending Exceeded in Profiler Topology
Key: METRON-1897
URL: https://issues.apache.org/jira/browse/METRON-1897
Project: Metron
Issue Type: Improvement
Reporter: Nick Allen
If the user sets the max spout pending value too low for the Profiler topology,
it can cause the topology to stall and stop consuming messages.
This is an annoying issue to debug because I am unable to find anything logged
by Storm indicating that the max spout pending limit is breached. When this
limit is exceeded it just appears that the KafkaSpout stops consuming messages
from Kafka and everything else would indicate the topology is healthy.
The Profiler topology is the only Metron topology that uses Storm's windowing
support. Storm will queue up messages for the window (by default 30 seconds)
and process each of the tuples in the window as one unit. This causes Storm to
need to "hang on" to tuples longer than the other topologies and this requires
a greater max spout pending value.
Ideally, we need to make it more obvious that exceeding the max spout pending
value in the Profiler topology is causing the KafkaSpout to stop consuming
messages.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
