[
https://issues.apache.org/jira/browse/STORM-1136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323104#comment-15323104
]
ASF GitHub Bot commented on STORM-1136:
---------------------------------------
Github user priyank5485 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1451#discussion_r66499861
--- Diff: storm-core/src/ui/public/topology.html ---
@@ -384,8 +389,51 @@ <h2 id="topology-resources-header">Topology
resources</h2>
$('#topology-configuration [data-toggle="tooltip"]').tooltip();
$('#topology-actions [data-toggle="tooltip"]').tooltip();
$('#topology-visualization [data-toggle="tooltip"]').tooltip();
+
+ var lagUrl = "/api/v1/topology/"+topologyId+"/lag";
+ $.getJSON(lagUrl,function(lagResponse,status,jqXHR) {
+ if (lagResponse !== null && lagResponse !== undefined &&
lagResponse instanceof Array && lagResponse.length > 0) {
+ var kafkaSpoutsLagTemplate =
$(template).filter("#topology-kafka-spouts-lag-template").html();
+ var spoutsErrorTemplate =
$(template).filter("#topology-spouts-lag-error-template").html();
+
+ var kafkaSpoutLags = lagResponse.filter(function(ele)
{return ele.spoutType === "KAFKA";});
+ var isJson = function (input) {
+ try {
+ JSON.parse(input);
+ } catch (e) {
+ return false;
+ }
+ return true;
+ };
+ var kafkaSpoutsValidResults =
kafkaSpoutLags.filter(function (ele) {return isJson(ele.spoutLagResult);});
+ var kafkaSpoutsErrorResults =
kafkaSpoutLags.filter(function (ele) {return !isJson(ele.spoutLagResult);});
+ var data = {};
+ if (kafkaSpoutsValidResults.length > 0) {
+ data.kafkaSpoutsLagResults = [];
+ kafkaSpoutsValidResults.forEach(function(ele) {
+ var spoutLagResult = JSON.parse(ele.spoutLagResult);
+ spoutLagResult.forEach(function(ele2) {
--- End diff --
@abellina The spoutLagResult property is actually an array. Hence the
flattening.
Let me elaborate the limitations of valid and invalid approach. The
TopologySpoutLag class in storm-core called by ui server does not know anything
about if the spoutLagResult is valid or not. Reason is, the way to get lag
information for kafka and other spouts is handled by making a shell call
handled by an external module since we did not want any direct dependency on
storm core. For now we have kafka, but we can add other types of spouts as
well. Plus the fields in ui or the template for a different type of spout could
be different. The only commonality I found was if something went wrong for
getting lag info for underlying spout(kafka or other) then an error message is
sent in response. This will work for all types of spouts. To do what you are
saying we will have to inspect the response in TopologySpoutLag class. I felt
it did not matter if it were ui doing that or server. I preferred ui. Let me
know if it makes sense or not and if you still think we need to change
something.
> Provide a bin script to check consumer lag from KafkaSpout to Kafka topic
> offsets and integrate with storm ui
> -------------------------------------------------------------------------------------------------------------
>
> Key: STORM-1136
> URL: https://issues.apache.org/jira/browse/STORM-1136
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-kafka
> Reporter: Sriharsha Chintalapani
> Assignee: Priyank Shah
>
> We store kafkaspout offsets in zkroot + id path in zookeeper. Kafka provides
> a utility and a protocol request to fetch latest offsets into topic
> {code}
> example:
> bin/kafka-run-classh.sh kafka.tools.GetOffsetTool
> {code}
> we should provide a way for the user to check how far the kafka spout read
> into topic and whats the lag. If we can expose this via UI even better.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
