Hello everyone, I propose https://github.com/apache/fineract/pull/782 as our New Policy about how to deal with unstable tests (text proposed to be added to the README in that PR is copy/pasted below for your reading convenience).
Tx, M. If your PR is failing to pass our CI build due to a test failure which you suspect is a "flaky" test, and not due to a change in your PR, then please do not simply wait for an active maintainer to come and help you, but instead be a proactive contributor to the project, like this: Search for the name of the failed test on https://issues.apache.org/jira/, e.g. for AccountingScenarioIntegrationTest you would find FINERACT-899 <https://issues.apache.org/jira/browse/FINERACT-899>. If you find previous comments "proving" that the same test has arbitrarily failed in at least 3 past PRs, then please do yourself raise a small separate new PR proposing to add an @Ignore to the respective unstable test (e.g. #774 <https://github.com/apache/fineract/pull/774>) with the commit message mentioning said JIRA (as always). Please do NOT just @Ignore any existing tests mixed in as part of your larger PR. If there is no existing JIRA for the test, then first please evaluate whether the failure couldn't be a (perhaps strange) impact of the change you are proposing after all. If it's not, then please raise a new JIRA to document the suspected Flaky Test, and link it to FINERACT-850 <https://issues.apache.org/jira/browse/FINERACT-850>. This will allow the next person coming along hitting the same test failure to easily find it, and eventually propose to ignore the unstable test. Then (only) Close and Reopen your PR, which will cause a new build, to see if it passes. _______________________ Michael Vorburger http://www.vorburger.ch
