Great suggestions -- added some comments. Do you have plans to add more
sophisticated analysis past just analyzing runtime relative to the 6d
average?

On Mon, Apr 16, 2018 at 10:21 AM Łukasz Gajowy <lukasz.gaj...@gmail.com>
wrote:

> That is correct - for now, we're running tests on Dataflow only. There
> were plans to run them on Spark and Flink (and possibly more runners) but
> we had some difficulties on the way. We decided to focus on Dataflow, at
> least for now. Currently the tests are quite flaky, so the priority is to
> make them more stable. Meanwhile we're providing the all necessary
> "infrastructure" (hence the anomaly detection proposal).
>
> If anyone is willing to contribute in this area, those seem to be the
> biggest blockers for Spark and Flink:
> https://issues.apache.org/jira/browse/BEAM-3370
> https://issues.apache.org/jira/browse/BEAM-3371
>
> Best regards,
> Łukasz
>
> 2018-04-16 18:28 GMT+02:00 Pablo Estrada <pabl...@google.com>:
>
>> This is very cool!
>> Are these dashboards for tests running on Dataflow only? Are there plans
>> for other runners? : )
>> -P.
>>
>> On Mon, Apr 16, 2018 at 9:23 AM Chamikara Jayalath <chamik...@google.com>
>> wrote:
>>
>>> Thanks Dariusz. This sounds great. Added some comments.
>>> Also, +Jeff Gardner <gardn...@google.com>  who has experience on
>>> performance regression analysis of integration tests.
>>>
>>> Thanks,
>>> Cham
>>>
>>>
>>> On Mon, Apr 16, 2018 at 2:58 AM Łukasz Gajowy <lukasz.gaj...@gmail.com>
>>> wrote:
>>>
>>>> @Etienne +1 to doing that! :) if we have both results (Nexmark and
>>>> IOITs) in BQ we could use the same (similar?) tools to detect anomalies
>>>> captured by Nexmark (if there's need for doing that).
>>>>
>>>> 2018-04-16 11:17 GMT+02:00 Etienne Chauchot <echauc...@apache.org>:
>>>>
>>>>> Very nice to see the dashboards !
>>>>>
>>>>> Regarding Kenn's comment:  Nexmark supports outputing the results to
>>>>> Bigquery so it could be easily integrated into the dashboards. We're, with
>>>>> Kenn, scheduling Nexmark runs. We could configure the output to bigquery
>>>>> dashboard tables ?
>>>>> WDYT?
>>>>>
>>>>> Etienne
>>>>> Le samedi 14 avril 2018 à 23:20 +0000, Kenneth Knowles a écrit :
>>>>>
>>>>> This is very cool. So is it easy for someone to integrate the proposal
>>>>> to regularly run Nexmark benchmarks and get those on the dashboard? (or a
>>>>> separate one to keep IOs in their own page)
>>>>>
>>>>> Kenn
>>>>>
>>>>> On Fri, Apr 13, 2018 at 9:02 AM Dariusz Aniszewski <
>>>>> dariusz.aniszew...@polidea.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *Hello Beam devs!As you might already noticed, together with Łukasz
>>>>> Gajowy, Kamil Szewczyk and Katarzyna Kucharczyk (all directly cc’d here)
>>>>> we’re working on adding some performance tests to the project. We were
>>>>> following directions from the Testing I/O Transforms in Apache Beam
>>>>> <https://beam.apache.org/documentation/io/testing/> site (which we plan to
>>>>> update in near future).We started from testing various FileBasedIOs as 
>>>>> part
>>>>> of BEAM-3060 <https://issues.apache.org/jira/browse/BEAM-3060>. So far we
>>>>> have tests for: - TextIO (with and without compression)- AvroIO- XmlIO-
>>>>> TFRecordIOthat may run on following filesystems: - local- GCS- HDFS 
>>>>> (except
>>>>> for TFRecordIO, see BEAM-3945
>>>>> <https://issues.apache.org/jira/browse/BEAM-3945>)Besides FileBasedIOs we
>>>>> also covered: - HadoopInputFormatIO- MongoDBIO- JdbcIO (in this case test
>>>>> was there, but was disabled; we fixed it and enabled)- HCatalogIO
>>>>> (currently in PR <https://github.com/apache/beam/pull/5097>)While 
>>>>> currently
>>>>> all the tests are maven-based, we responded to ongoing Gradle migration 
>>>>> and
>>>>> created PR <https://github.com/apache/beam/pull/5003> that allows running
>>>>> them via Gradle.All of those tests are executed on daily basis on Apache
>>>>> Jenkins <https://builds.apache.org/> and their results are published to
>>>>> individual BigQuery tables. There is also a dashboard on which tests
>>>>> results may be viewed and
>>>>> compared:https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688
>>>>> <https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688>As
>>>>> we have some amount of tests already, we’re currently working on a tool
>>>>> that will analyze the results and search for anomalies, so devs are
>>>>> notified if degraded performance is observed. You can find proposal
>>>>> document
>>>>> here:https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA
>>>>> <https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA>We
>>>>> welcome you to share your thoughts on performance tests in general as well
>>>>> as proposed solution for anomaly detection.Best,Dariusz Aniszewski*
>>>>>
>>>>>
>>>> --
>> Got feedback? go/pabloem-feedback
>> <https://goto.google.com/pabloem-feedback>
>>
>
>

-- 
-------
Jason Kuster
Apache Beam / Google Cloud Dataflow

See something? Say something. go/jasonkuster-feedback

Reply via email to