[jira] [Commented] (STORM-3202) Include offset information to spout metrics and remove storm-kafka-monitor
[ https://issues.apache.org/jira/browse/STORM-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807094#comment-16807094 ] Jungtaek Lim commented on STORM-3202: - [~Srdo] Please feel free to take this over. I thought I can make it soon but it wasn't and I couldn't even start. > Include offset information to spout metrics and remove storm-kafka-monitor > -- > > Key: STORM-3202 > URL: https://issues.apache.org/jira/browse/STORM-3202 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Priority: Major > > To provide offset information on Kafka spout (old and new), we have > storm-kafka-monitor module which is being run by UI (shell). This approach > requires UI doing too many things - basically UI process does most of things > via interacting with Nimbus - and also running external shell process in UI > process per opening topology page doesn't look right. > We could just let Spout include offset information into spout metric, and let > UI leverage the information. I have been thinking about this approach but > forgot about addressing it while thinking about generalizing the format. Now > I think we don't have to put too much effort to generalize format, because > Kafka spout is used mainly. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (STORM-3202) Include offset information to spout metrics and remove storm-kafka-monitor
[ https://issues.apache.org/jira/browse/STORM-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned STORM-3202: --- Assignee: (was: Jungtaek Lim) > Include offset information to spout metrics and remove storm-kafka-monitor > -- > > Key: STORM-3202 > URL: https://issues.apache.org/jira/browse/STORM-3202 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Priority: Major > > To provide offset information on Kafka spout (old and new), we have > storm-kafka-monitor module which is being run by UI (shell). This approach > requires UI doing too many things - basically UI process does most of things > via interacting with Nimbus - and also running external shell process in UI > process per opening topology page doesn't look right. > We could just let Spout include offset information into spout metric, and let > UI leverage the information. I have been thinking about this approach but > forgot about addressing it while thinking about generalizing the format. Now > I think we don't have to put too much effort to generalize format, because > Kafka spout is used mainly. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-3366) Workers with low CPU may fail to start in Cgroups or docker containers
[ https://issues.apache.org/jira/browse/STORM-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated STORM-3366: -- Labels: pull-request-available (was: ) > Workers with low CPU may fail to start in Cgroups or docker containers > -- > > Key: STORM-3366 > URL: https://issues.apache.org/jira/browse/STORM-3366 > Project: Apache Storm > Issue Type: Improvement >Reporter: Aaron Gresch >Assignee: Aaron Gresch >Priority: Major > Labels: pull-request-available > > We started implementing storm workers in docker containers and limiting the > max CPU to be as specified by the worker. We found that although a small > amount of CPU may run the worker code successfully, it prevents starting the > worker within the worker startup timeout. > > It would be nice to have a minimum cpu setting that forces a minimum amount > of cpu to be used for the workers to allow them to come up. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (STORM-3202) Include offset information to spout metrics and remove storm-kafka-monitor
[ https://issues.apache.org/jira/browse/STORM-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806996#comment-16806996 ] Stig Rohde Døssing edited comment on STORM-3202 at 4/1/19 5:10 PM: --- [~kabhwan] Are you working on this? You are set as assignee. I was considering looking at this. was (Author: srdo): [~kabhwan] Are you working on this? You are set as assignee. > Include offset information to spout metrics and remove storm-kafka-monitor > -- > > Key: STORM-3202 > URL: https://issues.apache.org/jira/browse/STORM-3202 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Major > > To provide offset information on Kafka spout (old and new), we have > storm-kafka-monitor module which is being run by UI (shell). This approach > requires UI doing too many things - basically UI process does most of things > via interacting with Nimbus - and also running external shell process in UI > process per opening topology page doesn't look right. > We could just let Spout include offset information into spout metric, and let > UI leverage the information. I have been thinking about this approach but > forgot about addressing it while thinking about generalizing the format. Now > I think we don't have to put too much effort to generalize format, because > Kafka spout is used mainly. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (STORM-3328) Allow overriding function name in BasicDRPCTopology
[ https://issues.apache.org/jira/browse/STORM-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Gresch closed STORM-3328. --- Resolution: Fixed > Allow overriding function name in BasicDRPCTopology > --- > > Key: STORM-3328 > URL: https://issues.apache.org/jira/browse/STORM-3328 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Aaron Gresch >Assignee: Aaron Gresch >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (STORM-2156) Add RocksDB instance in Nimbus and write heartbeat based metrics to it
[ https://issues.apache.org/jira/browse/STORM-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing reopened STORM-2156: --- Reopening, the PR has a different issue number: I can't tell if this is resolved, sorry. > Add RocksDB instance in Nimbus and write heartbeat based metrics to it > -- > > Key: STORM-2156 > URL: https://issues.apache.org/jira/browse/STORM-2156 > Project: Apache Storm > Issue Type: Improvement >Reporter: Alessandro Bellina >Assignee: Aaron Gresch >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > There should be a RocksDB instance in Nimbus where we write metrics from the > heartbeats. This should allow us to replace storage for the statistics we see > in the UI and expand the abilities of UIs to allow for time series charting. > Eventually this data will likely come via thrift to Nimbus as the overall > metric system is overhauled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-2156) Add RocksDB instance in Nimbus and write heartbeat based metrics to it
[ https://issues.apache.org/jira/browse/STORM-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing updated STORM-2156: -- Fix Version/s: (was: 2.0.0) > Add RocksDB instance in Nimbus and write heartbeat based metrics to it > -- > > Key: STORM-2156 > URL: https://issues.apache.org/jira/browse/STORM-2156 > Project: Apache Storm > Issue Type: Improvement >Reporter: Alessandro Bellina >Assignee: Aaron Gresch >Priority: Major > Labels: pull-request-available > Time Spent: 3h 40m > Remaining Estimate: 0h > > There should be a RocksDB instance in Nimbus where we write metrics from the > heartbeats. This should allow us to replace storage for the statistics we see > in the UI and expand the abilities of UIs to allow for time series charting. > Eventually this data will likely come via thrift to Nimbus as the overall > metric system is overhauled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-2156) Add RocksDB instance in Nimbus and write heartbeat based metrics to it
[ https://issues.apache.org/jira/browse/STORM-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing resolved STORM-2156. --- Resolution: Fixed Fix Version/s: 2.0.0 The associated PR is merged, marking as resolved. Please reopen if there's something left to do here. > Add RocksDB instance in Nimbus and write heartbeat based metrics to it > -- > > Key: STORM-2156 > URL: https://issues.apache.org/jira/browse/STORM-2156 > Project: Apache Storm > Issue Type: Improvement >Reporter: Alessandro Bellina >Assignee: Aaron Gresch >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > There should be a RocksDB instance in Nimbus where we write metrics from the > heartbeats. This should allow us to replace storage for the statistics we see > in the UI and expand the abilities of UIs to allow for time series charting. > Eventually this data will likely come via thrift to Nimbus as the overall > metric system is overhauled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-433) Give users visibility to the depth of queues at each bolt
[ https://issues.apache.org/jira/browse/STORM-433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806722#comment-16806722 ] Stig Rohde Døssing commented on STORM-433: -- [~dagit] This seems like it should be reasonably easy to implement on master. Metrics v2 was merged, and the queues were simplified since 1.x. I don't think anyone is working on this currently, but feel free to pick it up. > Give users visibility to the depth of queues at each bolt > - > > Key: STORM-433 > URL: https://issues.apache.org/jira/browse/STORM-433 > Project: Apache Storm > Issue Type: Wish > Components: storm-core >Reporter: Dane Hammer >Assignee: Abhishek Agarwal >Priority: Minor > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > I envision being able to browse the Storm UI and see where queues of tuples > are backing up. > Today if I see latencies increasing at a bolt, it may not be due to anything > specific to that bolt, but that it is backed up behind an overwhelmed bolt > (which has too low of parallelism or too high of latency). > I would expect this could use sampling like the metrics reported to the UI > today, and just retrieve data from netty about the state of the queues. I > wouldn't imagine supporting zeromq on the first pass. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
