Hi Simon and all, > On 26 Mar 2019, at 10:31, Simon Urli <simon.u...@xwiki.com> wrote: > > Hi everyone, > > I was checking our list of flickering tests in JIRA > (https://jira.xwiki.org/issues/?jql=labels%20%3D%20flickering%20AND%20status%20%3D%20Open%20ORDER%20BY%20updated%20DESC) > and I noticed that we had somehow old flickering test issue concerning test > that I've never seen failing. > > So I propose we close some of them as inactive: the ones that we don't > remember having seen for a while. The ideal would be to have a mechanism to > update the issue when the CI fails on a flicker, but it takes time to do > properly and it's not a priority. > > On the contrary I propose to trust our memory: if we're wrong because we have > closed a flicker that is still happening, it will allow us to remind that we > have this flicker to fix and we can easily reopen the issue. > > As Thomas mentioned on the chat, we should also update the release plan to > include the inactive flickers in the list of issue to check. > > So for now I propose to close the following list of issues as inactive: > > * XWIKI-14399: AddRemoveTagsTest#addAndDeleteTagFromTagPage is flickering > (https://jira.xwiki.org/browse/XWIKI-14399) > * XWIKI-14396: AnnotationsTest#addAndDeleteAnnotations is flickering > (https://jira.xwiki.org/browse/XWIKI-14396) > * XWIKI-14394: SectionTest.testSectionEditInWikiEditorWhenSyntax2x > (xwiki-enterprise-test-ui) is flaky > (https://jira.xwiki.org/browse/XWIKI-14394) > * XWIKI-14386: appwithinminutes.AppsLiveTableTest.testEditApplication is > possibly flaky (https://jira.xwiki.org/browse/XWIKI-14386) > * XWIKI-14835: DeletePageTest#deletePageIsImpossibleWhenNoDeleteRights is > flickering (https://jira.xwiki.org/browse/XWIKI-14835) > * XWIKI-14860: LoginTest#testDataIsPreservedAfterLogin is flickering > (https://jira.xwiki.org/browse/XWIKI-14860) > > And I propose in general to close the flickers we don't remember having seen > after a cycle as inactive. > > WDYT?
Thanks for looking into this. I’d feel more comfortable if we can do some local testing before closing any test issue. For ex, I’d propose to run it N times (can also be done on a docker container agent, it’s very easy to start a new container on ks* or locally) and if it doesn’t fail at all then close it if it’s old enough. Note: To repeat N times a test: * For junit5: @RepeatedTest(N) * For junit4: @Intermittent(repetition=N) - see https://dev.xwiki.org/xwiki/bin/view/Community/Testing/ * At the shell level: for i in {1..50} do mvn test -D… done Some rules I propose: * N = 50 (for a test taking 3 minutes, that means 2.5 hours) * Last update date of the test jira issue < Now() - 6 months WDYT? Too hard? Thanks -Vincent > > Simon > -- > Simon Urli > Software Engineer at XWiki SAS > simon.u...@xwiki.com > More about us at http://www.xwiki.com
