Hi Łukasz, See answers inline.
Regard, Mikhail. On Wed, Jun 26, 2019 at 7:47 AM Łukasz Gajowy <lgaj...@apache.org> wrote: > Hi Mikhail! > > Together with Kamil we're investigating the possibilities of creating > alerts for anomalies for the metrics collected from various tests (load, IO > tests, other performance tests). This is unfortunately impossible to do in > Perfkit explorer tool that we're using for displaying the metrics right now > [1]. Therefore we're considering a switch to some other solution. > > This is why we'd like to ask you some questions about the Community > Metrics tool. It is set up using Grafana that has the alerting feature out > of the box so it is a natural candidate. Moreover, it let's keep the > infrastructure as code which is also a big plus. Unfortunately alerting > feature does not work with BigQuery as a data source for Grafana [2]. > I'd say it is worth looking into. Keeping most of metrics in one place is much more convenient than have multiple tools in different locations. > > The questions: > > 1. What do you think of adding test related metrics as separate > dashboards in the existing Grafana instance? > > This should be completely fine and I don't see any blockers. > > 1. We were thinking of setting up a cloud SQL Postgres instance for > storing test metrics and reference this source in our Grafana dashboards. > Won't this approach collide in any way with existing setup? > > You can reuse existing PSQL DB that is utilized by Grafana. It's already hosted in GCP and should be available for you to use. Some permissions configuration might need configuring though. Even if you utilize separate DB, Grafana supports multiple data sources, so there should be no issues. > > 1. Have you tried setting up alerts in Grafana for community metrics? > Do you expect any blockers there? > > We do not have email configured in Grafana, however we utilize alerts on freshness dashboard that are later checked by metrics prober job <https://builds.apache.org/view/A-D/view/Beam/view/All/job/beam_Prober_CommunityMetrics/> . > I also CCed the devlist for visibility and comments (if any). > > Thanks! > Łukasz > > [1] https://s.apache.org/io-test-dashboards > [2] https://github.com/doitintl/bigquery-grafana/issues/67 >
