Stig Rohde Døssing created STORM-2648:
-----------------------------------------
Summary: Kafka spout can't show acks/fails and complete latency
when auto commit is enabled
Key: STORM-2648
URL: https://issues.apache.org/jira/browse/STORM-2648
Project: Apache Storm
Issue Type: New Feature
Components: storm-kafka-client
Affects Versions: 2.0.0, 1.1.0, 1.2.0
Reporter: Stig Rohde Døssing
Priority: Minor
Fix For: 1.1.1
The storm-kafka-client spout currently emits tuples with no message ids if auto
commit is enabled. This causes the ack/fail/complete latency counters in Storm
UI to be 0. In some cases this is desirable because the user may not care, and
doesn't want the overhead of Storm tracking tuples. [~avermeerbergen] expressed
a desire to be able to use auto commit without these counters being disabled,
presumably to monitor topology performance(?)
In order to support having acks/fails and complete latency when using auto
commit, we need to make the spout always emit tuples with anchors. Since the
spout doesn't care about acks or fails when auto commit is enabled, we should
make the spout ack/fail methods return immediately if auto commit is enabled.
I think we don't lose much by switching to this approach. Users that want the
statistics in Storm UI can enable auto commit and leave topology ackers at some
non-zero number. Users that don't care and don't want the overhead of having
Storm track the tuples can set topology ackers to 0, which should make the
spout behave a lot like it does now.
The only use case I can think of where this won't work is if someone with
multiple spouts in a topology needs one to be auto commit and one to be
at-least-once, and they can't live with the overhead of tuple tracking for the
auto commit spout. If this is a real use case, it can be worked around with an
extra configuration parameter to switch whether tuples are emitted unanchored,
but I'd rather keep it simple for now.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
