> On 26 Mar 2019, at 10:31, Simon Urli <simon.u...@xwiki.com> wrote:
>
> Hi everyone,
>
> I was checking our list of flickering tests in JIRA
> (https://jira.xwiki.org/issues/?jql=labels%20%3D%20flickering%20AND%20status%20%3D%20Open%20ORDER%20BY%20updated%20DESC)
> and I noticed that we had somehow old flickering test issue concerning test
> that I've never seen failing.
>
> So I propose we close some of them as inactive: the ones that we don't
> remember having seen for a while. The ideal would be to have a mechanism to
> update the issue when the CI fails on a flicker, but it takes time to do
> properly and it's not a priority.
>
> On the contrary I propose to trust our memory: if we're wrong because we have
> closed a flicker that is still happening, it will allow us to remind that we
> have this flicker to fix and we can easily reopen the issue.
>
> As Thomas mentioned on the chat, we should also update the release plan to
> include the inactive flickers in the list of issue to check.
I should be able to easily create a report when any test fails inside our
jenkins pipeline and make it available similar to our clover report. I could
indicate if it’s a known flicker or not too in this report. That could
compensate for the fact that we only keep 7 days of records in our jobs.
Would need to define the report format, whether it’s the same file updated at
each run or a different one. If the same one, then either:
* I’d need to parse it first in memory, add the new tests and overwrite the file
* or add to the bottom of the file which will grow quite large quickly
WDYT?
Thanks
-Vincent
>
> So for now I propose to close the following list of issues as inactive:
>
> * XWIKI-14399: AddRemoveTagsTest#addAndDeleteTagFromTagPage is flickering
> (https://jira.xwiki.org/browse/XWIKI-14399)
> * XWIKI-14396: AnnotationsTest#addAndDeleteAnnotations is flickering
> (https://jira.xwiki.org/browse/XWIKI-14396)
> * XWIKI-14394: SectionTest.testSectionEditInWikiEditorWhenSyntax2x
> (xwiki-enterprise-test-ui) is flaky
> (https://jira.xwiki.org/browse/XWIKI-14394)
> * XWIKI-14386: appwithinminutes.AppsLiveTableTest.testEditApplication is
> possibly flaky (https://jira.xwiki.org/browse/XWIKI-14386)
> * XWIKI-14835: DeletePageTest#deletePageIsImpossibleWhenNoDeleteRights is
> flickering (https://jira.xwiki.org/browse/XWIKI-14835)
> * XWIKI-14860: LoginTest#testDataIsPreservedAfterLogin is flickering
> (https://jira.xwiki.org/browse/XWIKI-14860)
>
> And I propose in general to close the flickers we don't remember having seen
> after a cycle as inactive.
>
> WDYT?
>
> Simon
> --
> Simon Urli
> Software Engineer at XWiki SAS
> simon.u...@xwiki.com
> More about us at http://www.xwiki.com
