[jira] [Commented] (STORM-599) UI is slow due to fetching heartbeats from ZK
[ https://issues.apache.org/jira/browse/STORM-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14277076#comment-14277076 ] Robert Joseph Evans commented on STORM-599: --- [~sriharsha] You are correct I verified it too. Hmm, we are running with this on based on a different code base that is working, it could have something to do with how often the cache is being updated. I will revert the checking, reopen this and then we can investigate further. Thanks for finding this. UI is slow due to fetching heartbeats from ZK - Key: STORM-599 URL: https://issues.apache.org/jira/browse/STORM-599 Project: Apache Storm Issue Type: Improvement Affects Versions: 0.9.3 Reporter: Derek Dagit Assignee: Derek Dagit Priority: Minor Fix For: 0.10.0 The method getTopologyInfo fetches every heartbeat from ZooKeeper in order to gather statistics. The UI calls this method via thrift to render the topology page. When topologies are launched with thousands of executors, this fetching from ZK dramatically slows things down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-599) UI is slow due to fetching heartbeats from ZK
[ https://issues.apache.org/jira/browse/STORM-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14277273#comment-14277273 ] ASF GitHub Bot commented on STORM-599: -- GitHub user d2r opened a pull request: https://github.com/apache/storm/pull/381 [Storm 599] Use use nimbus's cached heartbeats rather than fetching again from ZK, take two This is a retry of the reverted pull request #356. In #356, a merge error (we think?) on the branch used for the pull request caused part of the change on one line to go missing. ```Diff - beats (.executor-beats storm-cluster-state storm-id (:executor-node+port assignment)) + beats (map-val :heartbeat @(:heartbeats-cache nimbus)) ``` The new code should have been: ```Diff + beats (map-val :heartbeat (get @(:heartbeats-cache nimbus) storm-id)) ``` This caused incorrect retrieval of heartbeats and therefore metrics were not populated as noted by @harshach [here](https://issues.apache.org/jira/browse/STORM-599?focusedCommentId=14275512page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14275512). Testing again with this branch in a clean environment shows that metrics are updated as expected once per `nimbus.monitor.freq.secs`, which defaults to 10s. You can merge this pull request into a Git repository by running: $ git pull https://github.com/d2r/storm STORM-599-use-cached-hbs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/381.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #381 commit 1d973c6838118e64024505269b82baffd0f41f8d Author: Derek Dagit der...@yahoo-inc.com Date: 2015-01-14T17:12:30Z use hb cache instead of fetching via zk commit fc952d4c64f52e29cd162ffacee984ef36ab9447 Author: Derek Dagit der...@yahoo-inc.com Date: 2015-01-14T17:13:05Z Remove unnecessary code UI is slow due to fetching heartbeats from ZK - Key: STORM-599 URL: https://issues.apache.org/jira/browse/STORM-599 Project: Apache Storm Issue Type: Improvement Affects Versions: 0.9.3 Reporter: Derek Dagit Assignee: Derek Dagit Priority: Minor The method getTopologyInfo fetches every heartbeat from ZooKeeper in order to gather statistics. The UI calls this method via thrift to render the topology page. When topologies are launched with thousands of executors, this fetching from ZK dramatically slows things down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-599) UI is slow due to fetching heartbeats from ZK
[ https://issues.apache.org/jira/browse/STORM-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14266832#comment-14266832 ] ASF GitHub Bot commented on STORM-599: -- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/356 UI is slow due to fetching heartbeats from ZK - Key: STORM-599 URL: https://issues.apache.org/jira/browse/STORM-599 Project: Apache Storm Issue Type: Improvement Affects Versions: 0.9.3 Reporter: Derek Dagit Assignee: Derek Dagit Priority: Minor The method getTopologyInfo fetches every heartbeat from ZooKeeper in order to gather statistics. The UI calls this method via thrift to render the topology page. When topologies are launched with thousands of executors, this fetching from ZK dramatically slows things down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-599) UI is slow due to fetching heartbeats from ZK
[ https://issues.apache.org/jira/browse/STORM-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257528#comment-14257528 ] ASF GitHub Bot commented on STORM-599: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/356#issuecomment-67997939 The changes look fine to me. My only concern would be around what happens if the cache has not been updated and is empty, but it looks like it will return an empty map, and that looks like what happened before when there were no heartbeats. +1 UI is slow due to fetching heartbeats from ZK - Key: STORM-599 URL: https://issues.apache.org/jira/browse/STORM-599 Project: Apache Storm Issue Type: Improvement Affects Versions: 0.9.3 Reporter: Derek Dagit Assignee: Derek Dagit Priority: Minor The method getTopologyInfo fetches every heartbeat from ZooKeeper in order to gather statistics. The UI calls this method via thrift to render the topology page. When topologies are launched with thousands of executors, this fetching from ZK dramatically slows things down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-599) UI is slow due to fetching heartbeats from ZK
[ https://issues.apache.org/jira/browse/STORM-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256209#comment-14256209 ] ASF GitHub Bot commented on STORM-599: -- GitHub user d2r opened a pull request: https://github.com/apache/storm/pull/356 [STORM-599] Use use nimbus's cached heartbeats rather than fetching again from ZK In my manual testing, this seems to cut the time to load a topology page by around 50%. This also removes some apparently unnecessary code. You can merge this pull request into a Git repository by running: $ git pull https://github.com/d2r/storm apache-use-hb-cache-for-topo-info Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/356.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #356 commit 3907b2d780b796e754bf3566d29567cff6e77679 Author: Derek Dagit der...@yahoo-inc.com Date: 2014-12-13T15:36:31Z use the cache instead of downloading from zk commit 40be5ddb94b674f8fcf2c804c7af96fbfe6a Author: Derek Dagit der...@yahoo-inc.com Date: 2014-12-13T15:36:51Z Remove unnecessary code UI is slow due to fetching heartbeats from ZK - Key: STORM-599 URL: https://issues.apache.org/jira/browse/STORM-599 Project: Apache Storm Issue Type: Improvement Affects Versions: 0.9.3 Reporter: Derek Dagit Assignee: Derek Dagit Priority: Minor The method getTopologyInfo fetches every heartbeat from ZooKeeper in order to gather statistics. The UI calls this method via thrift to render the topology page. When topologies are launched with thousands of executors, this fetching from ZK dramatically slows things down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
