That is correct - for now, we're running tests on Dataflow only. There were plans to run them on Spark and Flink (and possibly more runners) but we had some difficulties on the way. We decided to focus on Dataflow, at least for now. Currently the tests are quite flaky, so the priority is to make them more stable. Meanwhile we're providing the all necessary "infrastructure" (hence the anomaly detection proposal).
If anyone is willing to contribute in this area, those seem to be the biggest blockers for Spark and Flink: https://issues.apache.org/jira/browse/BEAM-3370 https://issues.apache.org/jira/browse/BEAM-3371 Best regards, Łukasz 2018-04-16 18:28 GMT+02:00 Pablo Estrada <pabl...@google.com>: > This is very cool! > Are these dashboards for tests running on Dataflow only? Are there plans > for other runners? : ) > -P. > > On Mon, Apr 16, 2018 at 9:23 AM Chamikara Jayalath <chamik...@google.com> > wrote: > >> Thanks Dariusz. This sounds great. Added some comments. >> Also, +Jeff Gardner <gardn...@google.com> who has experience on >> performance regression analysis of integration tests. >> >> Thanks, >> Cham >> >> >> On Mon, Apr 16, 2018 at 2:58 AM Łukasz Gajowy <lukasz.gaj...@gmail.com> >> wrote: >> >>> @Etienne +1 to doing that! :) if we have both results (Nexmark and >>> IOITs) in BQ we could use the same (similar?) tools to detect anomalies >>> captured by Nexmark (if there's need for doing that). >>> >>> 2018-04-16 11:17 GMT+02:00 Etienne Chauchot <echauc...@apache.org>: >>> >>>> Very nice to see the dashboards ! >>>> >>>> Regarding Kenn's comment: Nexmark supports outputing the results to >>>> Bigquery so it could be easily integrated into the dashboards. We're, with >>>> Kenn, scheduling Nexmark runs. We could configure the output to bigquery >>>> dashboard tables ? >>>> WDYT? >>>> >>>> Etienne >>>> Le samedi 14 avril 2018 à 23:20 +0000, Kenneth Knowles a écrit : >>>> >>>> This is very cool. So is it easy for someone to integrate the proposal >>>> to regularly run Nexmark benchmarks and get those on the dashboard? (or a >>>> separate one to keep IOs in their own page) >>>> >>>> Kenn >>>> >>>> On Fri, Apr 13, 2018 at 9:02 AM Dariusz Aniszewski < >>>> dariusz.aniszew...@polidea.com> wrote: >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> *Hello Beam devs!As you might already noticed, together with Łukasz >>>> Gajowy, Kamil Szewczyk and Katarzyna Kucharczyk (all directly cc’d here) >>>> we’re working on adding some performance tests to the project. We were >>>> following directions from the Testing I/O Transforms in Apache Beam >>>> <https://beam.apache.org/documentation/io/testing/> site (which we plan to >>>> update in near future).We started from testing various FileBasedIOs as part >>>> of BEAM-3060 <https://issues.apache.org/jira/browse/BEAM-3060>. So far we >>>> have tests for: - TextIO (with and without compression)- AvroIO- XmlIO- >>>> TFRecordIOthat may run on following filesystems: - local- GCS- HDFS (except >>>> for TFRecordIO, see BEAM-3945 >>>> <https://issues.apache.org/jira/browse/BEAM-3945>)Besides FileBasedIOs we >>>> also covered: - HadoopInputFormatIO- MongoDBIO- JdbcIO (in this case test >>>> was there, but was disabled; we fixed it and enabled)- HCatalogIO >>>> (currently in PR <https://github.com/apache/beam/pull/5097>)While currently >>>> all the tests are maven-based, we responded to ongoing Gradle migration and >>>> created PR <https://github.com/apache/beam/pull/5003> that allows running >>>> them via Gradle.All of those tests are executed on daily basis on Apache >>>> Jenkins <https://builds.apache.org/> and their results are published to >>>> individual BigQuery tables. There is also a dashboard on which tests >>>> results may be viewed and >>>> compared:https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688 >>>> <https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688>As >>>> we have some amount of tests already, we’re currently working on a tool >>>> that will analyze the results and search for anomalies, so devs are >>>> notified if degraded performance is observed. You can find proposal >>>> document >>>> here:https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA >>>> <https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA>We >>>> welcome you to share your thoughts on performance tests in general as well >>>> as proposed solution for anomaly detection.Best,Dariusz Aniszewski* >>>> >>>> >>> -- > Got feedback? go/pabloem-feedback >
