Sorry about that, attached on this email. The chart is pretty simple/bare
bones.
On Sunday, July 31, 2016 5:25 PM, Jungtaek Lim <[email protected]> wrote:
Inline image seems broken. Could you make it available?
Regarding internal storage, below is what I observed from other projects while
working with metrics.JStorm is using RocksDB in Nimbus to store metrics
internally (only given time frame), and provide plugin to store metrics to
external storages. That said, all metrics are in result sent to Nimbus and
handled, which is different from Storm metrics feature.And Heron runs separate
process, Metrics Manager, which stores topology's metrics via in memory, and
expose REST APIs to query. But it seems limited to serve only recent 3 hours.
For longer data Heron also exposes metrics sink (like a metrics consumer) to
store metrics to external storages.
For external storages I think it should be selectable since it requires users
to set up. We just have to think about how to store metrics without installing
external things (having limitations are fine), and how to represent them.
For UI perspective, linking to Grafana is what Heron is providing. I didn't
give it a try but it might be better to take a look at.
Btw, IMO, if we can show dashboards without dependencies that would be amazing,
and having some dependencies which are easy to install and maintain would be
still nice.
Thanks,Jungtaek Lim (HeartSaVioR)
2016년 8월 1일 (월) 오전 1:43, Alessandro Bellina <[email protected]>님이
작성:
HeartSaVIoR
This is different than STORM-1994. By the way added some screenshots there to
clarify.
I agree it would be good to reorganize the statistics shown in the UI. Perhaps
having two parts to this, one goes to a dashboard like Grafana, while the
simpler metrics (snapshots, or aggregations) go to the regular Storm UI. The
reason for some of them going to the Storm UI is that I think that having some
stats next to the place where actions can be performed on a topology is
beneficial (as they currently are).
With regards to Jerry's suggestion, we tried a simple test using MySQL just as
a PoC and c3/d3 as the charting library. The choices were made purely because
we wanted to see the charts quickly, so not suggesting we use MySQL for time
series data. We know probably storing in Druid or another time series database
would be ideal, but harder to set up. Here is a screenshot showing average user
cpu time (ms) per minute for each worker:
We could probably then turn around and request data from Druid (or more
generally, the "metrics store") and show in the Storm UI. Additionally, Grafana
looks like it can read from Druid, so folks could use that as a front end for
detailed dashboards. That said, perhaps some version of this in the Topology
page at a coarser resolution would be beneficial for a quick glimpse, linking
to the detailed dashboard.
Thanks,
Alessandro
On Sunday, July 31, 2016 1:23 AM, Jungtaek Lim <[email protected]> wrote:
Hi Boyang,
AFAIK, there's no discussion on Storm UI yet. Could you share your sketched
UI for this?
And is it on top of STORM-1994
<https://issues.apache.org/jira/browse/STORM-1994>? I mean what Alessandro
submitted.
A random thought from myself is that we shows too many numbers even
topology info page, which is not intuitive at a glance. It might be better
to show them to graph or at least more abstracted from overview page, and
show details on other pages. (component detail page, or another)
It might be also better to show graph of values for time-series manner (I
think dashboard like Grafana is the best way to show numbers as intuitive
way), but it requires metrics to be stored and served to time-series
manner, so it's not a near-future goal.
Thanks,
Jungtaek Lim (HeartSaVIoR)
2016년 7월 30일 (토) 오전 12:19, Boyang(Jerry) Peng
<[email protected]>님이 작성:
> Hello Everyone,
> I would like to make enhancements on the UI by adding dynamic graphs to
> display stats of a topology. For example, it would be nice to see
> performance and resource usage metrics be displayed in a graph on a per
> worker basis. I know JStorm's UI has graphs but I am not sure to what
> kind. What are our plans to improve storm UI? What portions of JStorm's
> UI might we use?
> Best,
> Jerry
