Re: Flaky tests

[Stanislav Kozlovski]

Thank you for clarifying everybody, this is valuable expertise.

On 2019/02/27 09:01:00, Ilya Kasnacheev  wrote: 
> Hello!> 
> 
> We also have this wonderful tool: https://mtcga.gridgain.com/> 
> 
> It keeps track of tests which are currently flaky or under investigation,> 
> and given your change it will report non-flaky tests that failed, skipping> 
> all problematic tests.> 
> 
> It talks to TeamCity so it can run tests for you, check their results and> 
> comment JIRA with "Bot Visa", see:> 
> https://issues.apache.org/jira/browse/IGNITE-11299 comment from "Ignite TC> 
> Bot"> 
> 
> Since we don't have too many flaky tests thanks to Make TeamCity Green> 
> Again movement, it works reasonably well.> 
> 
> A year ago we did not have any of this but now we do, and we were able to> 
> shave 80% of test-checking busy work. This is mostly thanks to Dmitry> 
> Pavlov.> 
> 
> Regards,> 
> -- > 
> Ilya Kasnacheev> 
> 
> 
> ср, 27 февр. 2019 г. в 01:39, Stanislav Kozlovski <> 
> stanislav_kozlov...@outlook.com>:> 
> 
> > Hey there Ignite community,> 
> >> 
> > I contribute to a fellow open-source project - Apache Kafka - and there we> 
> > have been fighting flaky tests a lot. We run Java 8 and Java 11 builds on> 
> > every Pull Request and due to test flakiness, almost all of them turn out> 
> > red with 1 or 2 tests (completely unrelated to the change in the PR)> 
> > failing. This has resulted in committers either ignoring them and merging> 
> > the changes or in the worst case rerunning the hour-long build until it> 
> > becomes green.> 
> > This test flakiness has also slowed down our releases significantly.> 
> >> 
> > In general, I was just curious to understand if this is a problem that> 
> > your project faces as well. Does your project have a lot of intermittently> 
> > failing tests, do you have any active process of addressing such tests> 
> > (during the initial review, after realizing it is flaky, etc). Any 
> > pointers> 
> > will be greatly appreciated!> 
> >> 
> > Thanks,> 
> > Stanislav> 
> >> 
> >> 
> > <http://aka.ms/weboutlook>> 
> >> 
> 

Flaky tests

[Stanislav Kozlovski]

Hey there Ignite community,

I contribute to a fellow open-source project - Apache Kafka - and there we have 
been fighting flaky tests a lot. We run Java 8 and Java 11 builds on every Pull 
Request and due to test flakiness, almost all of them turn out red with 1 or 2 
tests (completely unrelated to the change in the PR) failing. This has resulted 
in committers either ignoring them and merging the changes or in the worst case 
rerunning the hour-long build until it becomes green.
This test flakiness has also slowed down our releases significantly.

In general, I was just curious to understand if this is a problem that your 
project faces as well. Does your project have a lot of intermittently failing 
tests, do you have any active process of addressing such tests (during the 
initial review, after realizing it is flaky, etc). Any pointers will be greatly 
appreciated!

Thanks,
Stanislav