Not yet, I just learned about the migration to a new frontend, including
a new backend (InfluxDB instead of BigQuery).
- Are the metrics available on metrics.beam.apache.org?
Is http://metrics.beam.apache.org online? I was never able to access it.
- What is the feature delta between usinig metrics.beam.apache.org (much
better UI) and using apache-beam-testing.appspot.com?
AFAIK it is an ongoing migration and the delta appears to be high.
- Can we notice regressions faster than release cadence?
Absolutely! A report with the latest numbers including statistics about
the growth of metrics would be useful.
- Can we get automated alerts?
I think we could setup a Jenkins job to do this.
-Max
On 09.07.20 20:26, Kenneth Knowles wrote:
Questions:
- Are the metrics available on metrics.beam.apache.org
<http://metrics.beam.apache.org>?
- What is the feature delta between usinig metrics.beam.apache.org
<http://metrics.beam.apache.org> (much better UI) and using
apache-beam-testing.appspot.com <http://apache-beam-testing.appspot.com>?
- Can we notice regressions faster than release cadence?
- Can we get automated alerts?
Kenn
On Thu, Jul 9, 2020 at 10:21 AM Maximilian Michels <[email protected]
<mailto:[email protected]>> wrote:
Hi,
We recently saw an increase in latency migrating from Beam 2.18.0 to
2.21.0 (Python SDK with Flink Runner). This proofed very hard to debug
and it looks like each version in between the two versions let to
increased latency.
This is not the first time we saw issues when migrating, another
time we
had a decline in checkpointing performance and thus added a
checkpointing test [1] and dashboard [2] (see checkpointing widget).
That makes me wonder if we should monitor performance (throughput /
latency) for basic use cases as part of the release testing. Currently,
our release guide [3] mentions running examples but not evaluating the
performance. I think it would be good practice to check relevant charts
with performance measurements as part of of the release process. The
release guide should reflect that.
WDYT?
-Max
PS: Of course, this requires tests and metrics to be available. This PR
adds latency measurements to the load tests [4].
[1] https://github.com/apache/beam/pull/11558
[2]
https://apache-beam-testing.appspot.com/explore?dashboard=5751884853805056
[3] https://beam.apache.org/contribute/release-guide/
[4] https://github.com/apache/beam/pull/12065
