[
https://issues.apache.org/jira/browse/STORM-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16062505#comment-16062505
]
Priyank Shah commented on STORM-2600:
-------------------------------------
[~Srdo] Regarding first point, because the lag request is a http pull request
from UI, as long as kafkaSpoutConfig.getSubscription().getTopicsString();
returns the correct value it will work since the open method would have been
called eventually. The only change we would need is that when the rebalance
listener is invoked we would need to keep track of the topics currently
subscribed. For e.g. PatternSubscription can have an instance variable called
Set<String> topics that would be correctly updated anytime onPartitionsRevoked
or onPartitionsAssigned is called. We can use that instance variable to return
the value when getTopicsString is called on that object. Does that work?
Regarding point 2, we could move the kafka client version to
storm-kafka-monitor module. Can you elaborate a bit more on your concern? Is it
that if kafka cluster is upgraded, storm-kafka-monitor wont work? In that case
storm-kafka-client module will also have to be updated and topology jar
rebuilt. Correct? I think in general, we have a certain compatibility
restriction in that a given storm release works or has been tested with a
certain version of kafka clients. Correct me if i am wrong.
Regarding 3, the main reason for creating a separate module and calling bash
script from storm UI is so that storm-core does not have a direct kafka
dependency since that made sense. For windows, cygwin can be a workaround.
Plus, i dont know how many windows users we have. We can start a thread to see
if there is really a user base for which we need to support a use case.
I dont know the details about how metrics would work. We could have opinions
from other people. If that is indeed the right way to go about it then I am all
for changing it.
> Improve or replace storm-kafka-monitor
> --------------------------------------
>
> Key: STORM-2600
> URL: https://issues.apache.org/jira/browse/STORM-2600
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-kafka-monitor
> Affects Versions: 2.0.0
> Reporter: Stig Rohde Døssing
> Priority: Minor
>
> The storm-kafka-monitor module, which is used by Storm UI to show offset lag
> for topologies with Kafka spouts, has some shortcomings:
> * The Storm UI integration code doesn't seem to be able to support topic
> subscriptions that change after topology submission. The UI code
> (https://github.com/apache/storm/blob/64e29f365c9b5d3e15b33f33ab64e200345333e4/storm-core/src/jvm/org/apache/storm/utils/TopologySpoutLag.java#L91)
> gets the topic list it should request offset lag for via the spout's
> getComponentConfiguration method, as far as I can tell through this call
> https://github.com/apache/storm/blob/9e31509d47c4e91c1009f55c7ccf321d7d7e63aa/storm-client/src/jvm/org/apache/storm/topology/TopologyBuilder.java#L541.
> It seems like the component configuration is intended to be static once the
> topology has started running. This prevents us from showing the right topic
> list for subscriptions that are not known at submission time, which is
> currently the case for Pattern subscriptions. The topic list for that type of
> subscription isn't known until the spout has started the KafkaConsumer in
> {{ISpout.open()}}. I don't see a way to fix this, unless there is some way to
> update the component configuration when the subscription changes.
> * The jar is installed along with the cluster, and depends on the Kafka
> version specified in Storm's root POM. Kafka guarantees backwards compatible
> client-server communication for one release only, so there's a potential
> coupling between Storm cluster version and Kafka version. If users want to
> update the Kafka version in storm-kafka-monitor, they have to rebuild that
> module and replace the jar in their Storm install.
> * The UI integration uses the storm-kafka-monitor Bash script to start the
> monitoring code, in order to avoid a dependency between storm-core and
> storm-kafka-monitor. This prevents the UI integration from working on
> Windows. We could supply a Windows script as well, but then we'd need to keep
> the two in sync.
> I am wondering if these problems could be solved by implementing offset lag
> monitoring via the metrics system instead. The spout could periodically seek
> to the log end offset and submit a metric for how far behind the committed
> offset is, then seek back to where it left off.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
