[
https://issues.apache.org/jira/browse/STORM-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stig Rohde Døssing closed STORM-2600.
-------------------------------------
Resolution: Duplicate
> Improve or replace storm-kafka-monitor
> --------------------------------------
>
> Key: STORM-2600
> URL: https://issues.apache.org/jira/browse/STORM-2600
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-kafka-monitor
> Affects Versions: 2.0.0
> Reporter: Stig Rohde Døssing
> Priority: Minor
>
> The storm-kafka-monitor module, which is used by Storm UI to show offset lag
> for topologies with Kafka spouts, has some shortcomings:
> * The Storm UI integration code doesn't seem to be able to support topic
> subscriptions that change after topology submission. The UI code
> (https://github.com/apache/storm/blob/64e29f365c9b5d3e15b33f33ab64e200345333e4/storm-core/src/jvm/org/apache/storm/utils/TopologySpoutLag.java#L91)
> gets the topic list it should request offset lag for via the spout's
> getComponentConfiguration method, as far as I can tell through this call
> https://github.com/apache/storm/blob/9e31509d47c4e91c1009f55c7ccf321d7d7e63aa/storm-client/src/jvm/org/apache/storm/topology/TopologyBuilder.java#L541.
> It seems like the component configuration is intended to be static once the
> topology has started running. This prevents us from showing the right topic
> list for subscriptions that are not known at submission time, which is
> currently the case for Pattern subscriptions. The topic list for that type of
> subscription isn't known until the spout has started the KafkaConsumer in
> {{ISpout.open()}}. I don't see a way to fix this, unless there is some way to
> update the component configuration when the subscription changes.
> * The jar is installed along with the cluster, and depends on the Kafka
> version specified in Storm's root POM. Kafka guarantees backwards compatible
> client-server communication for one release only, so there's a potential
> coupling between Storm cluster version and Kafka version. If users want to
> update the Kafka version in storm-kafka-monitor, they have to rebuild that
> module and replace the jar in their Storm install.
> * The UI integration uses the storm-kafka-monitor Bash script to start the
> monitoring code, in order to avoid a dependency between storm-core and
> storm-kafka-monitor. This prevents the UI integration from working on
> Windows. We could supply a Windows script as well, but then we'd need to keep
> the two in sync.
> I am wondering if these problems could be solved by implementing offset lag
> monitoring via the metrics system instead. The spout could periodically seek
> to the log end offset and submit a metric for how far behind the committed
> offset is, then seek back to where it left off.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
