+1 "Please do NOT just @Ignore any existing tests mixed in as part of your larger PR."
This seems like a key point to bring up to a higher level in the code review. i.e. should there be a search for @Ignore when doing a code review? On Sun, Apr 26, 2020 at 2:30 AM Michael Vorburger <m...@vorburger.ch> wrote: > Hello everyone, > > I propose https://github.com/apache/fineract/pull/782 as our New Policy > about how to deal with unstable tests (text proposed to be added to the > README in that PR is copy/pasted below for your reading convenience). > > Tx, > M. > > If your PR is failing to pass our CI build due to a test failure which you > suspect is a "flaky" test, and not due to a change in your PR, then please > do not simply wait for an active maintainer to come and help you, but > instead be a proactive contributor to the project, like this: Search for > the name of the failed test on https://issues.apache.org/jira/, e.g. for > AccountingScenarioIntegrationTest you would find FINERACT-899 > <https://issues.apache.org/jira/browse/FINERACT-899>. If you find > previous comments "proving" that the same test has arbitrarily failed in at > least 3 past PRs, then please do yourself raise a small separate new PR > proposing to add an @Ignore to the respective unstable test (e.g. #774 > <https://github.com/apache/fineract/pull/774>) with the commit message > mentioning said JIRA (as always). Please do NOT just @Ignore any existing > tests mixed in as part of your larger PR. If there is no existing JIRA for > the test, then first please evaluate whether the failure couldn't be a > (perhaps strange) impact of the change you are proposing after all. If it's > not, then please raise a new JIRA to document the suspected Flaky Test, and > link it to FINERACT-850 > <https://issues.apache.org/jira/browse/FINERACT-850>. This will allow the > next person coming along hitting the same test failure to easily find it, > and eventually propose to ignore the unstable test. Then (only) Close and > Reopen your PR, which will cause a new build, to see if it passes. > > _______________________ > Michael Vorburger > http://www.vorburger.ch >
