Hi Mikhail! Together with Kamil we're investigating the possibilities of creating alerts for anomalies for the metrics collected from various tests (load, IO tests, other performance tests). This is unfortunately impossible to do in Perfkit explorer tool that we're using for displaying the metrics right now [1]. Therefore we're considering a switch to some other solution.
This is why we'd like to ask you some questions about the Community Metrics tool. It is set up using Grafana that has the alerting feature out of the box so it is a natural candidate. Moreover, it let's keep the infrastructure as code which is also a big plus. Unfortunately alerting feature does not work with BigQuery as a data source for Grafana [2]. The questions: 1. What do you think of adding test related metrics as separate dashboards in the existing Grafana instance? 2. We were thinking of setting up a cloud SQL Postgres instance for storing test metrics and reference this source in our Grafana dashboards. Won't this approach collide in any way with existing setup? 3. Have you tried setting up alerts in Grafana for community metrics? Do you expect any blockers there? I also CCed the devlist for visibility and comments (if any). Thanks! Łukasz [1] https://s.apache.org/io-test-dashboards [2] https://github.com/doitintl/bigquery-grafana/issues/67
