Very nice Dariusz +1 to putting the daily report on Slack to help prompt early comments and investigation on change.
A couple very minor comments: a) Would it make sense to track number of tests run as well as duration? b) Labels on the Y-axis / a title with units would be a good addition to the charts (sorry, bugbear of mine) I've recently added retry-on-failure code to SolrIO and the tests required doing things like verifying the timeout is observed. This increased the test runtime, and your alerting to all devs would help communicate that and hopefully prompt folk to raise any concerns early. Thanks for sharing, Tim On Sun, Apr 15, 2018 at 1:20 AM, Kenneth Knowles <k...@google.com> wrote: > This is very cool. So is it easy for someone to integrate the proposal to > regularly run Nexmark benchmarks and get those on the dashboard? (or a > separate one to keep IOs in their own page) > > Kenn > > On Fri, Apr 13, 2018 at 9:02 AM Dariusz Aniszewski < > dariusz.aniszew...@polidea.com> wrote: > >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> *Hello Beam devs!As you might already noticed, together with Łukasz >> Gajowy, Kamil Szewczyk and Katarzyna Kucharczyk (all directly cc’d here) >> we’re working on adding some performance tests to the project. We were >> following directions from the Testing I/O Transforms in Apache Beam >> <https://beam.apache.org/documentation/io/testing/> site (which we plan to >> update in near future).We started from testing various FileBasedIOs as part >> of BEAM-3060 <https://issues.apache.org/jira/browse/BEAM-3060>. So far we >> have tests for: - TextIO (with and without compression)- AvroIO- XmlIO- >> TFRecordIOthat may run on following filesystems: - local- GCS- HDFS (except >> for TFRecordIO, see BEAM-3945 >> <https://issues.apache.org/jira/browse/BEAM-3945>)Besides FileBasedIOs we >> also covered: - HadoopInputFormatIO- MongoDBIO- JdbcIO (in this case test >> was there, but was disabled; we fixed it and enabled)- HCatalogIO >> (currently in PR <https://github.com/apache/beam/pull/5097>)While currently >> all the tests are maven-based, we responded to ongoing Gradle migration and >> created PR <https://github.com/apache/beam/pull/5003> that allows running >> them via Gradle.All of those tests are executed on daily basis on Apache >> Jenkins <https://builds.apache.org/> and their results are published to >> individual BigQuery tables. There is also a dashboard on which tests >> results may be viewed and >> compared:https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688 >> <https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688>As >> we have some amount of tests already, we’re currently working on a tool >> that will analyze the results and search for anomalies, so devs are >> notified if degraded performance is observed. You can find proposal >> document >> here:https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA >> <https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA>We >> welcome you to share your thoughts on performance tests in general as well >> as proposed solution for anomaly detection.Best,Dariusz Aniszewski* >> >