Inline image seems broken. Could you make it available? Regarding internal storage, below is what I observed from other projects while working with metrics. JStorm is using RocksDB in Nimbus to store metrics internally (only given time frame), and provide plugin to store metrics to external storages. That said, all metrics are in result sent to Nimbus and handled, which is different from Storm metrics feature. And Heron runs separate process, Metrics Manager, which stores topology's metrics via in memory, and expose REST APIs to query. But it seems limited to serve only recent 3 hours. For longer data Heron also exposes metrics sink (like a metrics consumer) to store metrics to external storages.
For external storages I think it should be selectable since it requires users to set up. We just have to think about how to store metrics without installing external things (having limitations are fine), and how to represent them. For UI perspective, linking to Grafana is what Heron is providing. I didn't give it a try but it might be better to take a look at. Btw, IMO, if we can show dashboards without dependencies that would be amazing, and having some dependencies which are easy to install and maintain would be still nice. Thanks, Jungtaek Lim (HeartSaVioR) 2016년 8월 1일 (월) 오전 1:43, Alessandro Bellina <[email protected]>님이 작성: > > HeartSaVIoR > > This is different than STORM-1994. By the way added some screenshots there > to clarify. > > I agree it would be good to reorganize the statistics shown in the UI. > Perhaps having two parts to this, one goes to a dashboard like Grafana, > while the simpler metrics (snapshots, or aggregations) go to the regular > Storm UI. The reason for some of them going to the Storm UI is that I think > that having some stats next to the place where actions can be performed on > a topology is beneficial (as they currently are). > > With regards to Jerry's suggestion, we tried a simple test using MySQL > just as a PoC and c3/d3 as the charting library. The choices were made > purely because we wanted to see the charts quickly, so not suggesting we > use MySQL for time series data. We know probably storing in Druid or > another time series database would be ideal, but harder to set up. Here is > a screenshot showing average user cpu time (ms) per minute for each worker: > > [image: Inline image] > > We could probably then turn around and request data from Druid (or more > generally, the "metrics store") and show in the Storm UI. Additionally, > Grafana looks like it can read from Druid, so folks could use that as a > front end for detailed dashboards. That said, perhaps some version of this > in the Topology page at a coarser resolution would be beneficial for a > quick glimpse, linking to the detailed dashboard. > > Thanks, > > Alessandro > > > > On Sunday, July 31, 2016 1:23 AM, Jungtaek Lim <[email protected]> wrote: > > > Hi Boyang, > > AFAIK, there's no discussion on Storm UI yet. Could you share your sketched > UI for this? > And is it on top of STORM-1994 > <https://issues.apache.org/jira/browse/STORM-1994>? I mean what Alessandro > > submitted. > > A random thought from myself is that we shows too many numbers even > topology info page, which is not intuitive at a glance. It might be better > to show them to graph or at least more abstracted from overview page, and > show details on other pages. (component detail page, or another) > > It might be also better to show graph of values for time-series manner (I > think dashboard like Grafana is the best way to show numbers as intuitive > way), but it requires metrics to be stored and served to time-series > manner, so it's not a near-future goal. > > Thanks, > Jungtaek Lim (HeartSaVIoR) > > 2016년 7월 30일 (토) 오전 12:19, Boyang(Jerry) Peng > <[email protected]>님이 작성: > > > > Hello Everyone, > > I would like to make enhancements on the UI by adding dynamic graphs to > > display stats of a topology. For example, it would be nice to see > > performance and resource usage metrics be displayed in a graph on a per > > worker basis. I know JStorm's UI has graphs but I am not sure to what > > kind. What are our plans to improve storm UI? What portions of JStorm's > > UI might we use? > > Best, > > Jerry > > >
