On 06/09/2019 10:35, Thomas Mortagne wrote:
On Fri, Sep 6, 2019 at 10:32 AM Vincent Massol <vinc...@massol.net> wrote:
Hi Simon,
On 6 Sep 2019, at 10:27, Simon Urli <simon.u...@xwiki.com> wrote:
Hi all,
On 05/09/2019 17:40, Simon Urli wrote:
On 05/09/2019 17:24, Thomas Mortagne wrote:
On Thu, Sep 5, 2019 at 3:43 PM Simon Urli <simon.u...@xwiki.com> wrote:
Hi everyone,
reopening this thread since I started to close some flicker issues as
part of BFD and got comments for those.
So the last mails on this threads suggested to close the flicker issues
if we didn't manage to reproduce them locally after a repeated tests,
and that we didn't see them after a while.
We didn't vote for those suggestion and I assumed a bit quick that I
could close some flicker issues that I personally don't remember about
on the CI after having tested them locally.
My point for doing that is the same as for the first mail I posted on
this thread: those flickers are old, and the code did change enough for
those to be fixed in a way or another.
Being old does not always means the code leading to those failures
changed that much.
Now I might be completely wrong, and the flicker to happen again, but I
don't think it's a problem since we can really easily open back the
issues if it's the case.
The other solution IMO is to indeed keep the issue open and in fact to
never really close them, because we just don't have time to investigate
each of them properly.
I really don't see any value of keeping things open and don't act on
them, that's why I suggest to close them after doing the checks we
suggested before:
1. try to repeat locally the failure;
This is totally useless IMO unless you make sure that your computer is
made super slow some way since that's the reason for most of the
flickering tests.
2. check that we didn't encounter those flickers since last cycle.
This one is enough for me but the hard part is to knowing that.
Ok, so the proposal is now to check only the age since last time we saw them of
the open flickers before closing them.
So first question, do we all agree on that?
Then for the second check, Vincent suggested to add some tooling: it
will be best, but it takes time to do. So on the meantime, as Thomas
also suggested, we could add a check in the release plan to create or
update all jira issues that concerns flickers. It would allow us to keep
some information about the liveness of our flickers.
So second question, do you agree on that?
Depends what it exactly means. Have some dedicated jira field to
indicate when you saw it last ? Comment that you just saw that test
failing again ?
My suggestion was about a dedicated JIRA field if possible.
So, ok if I create a new custom field in JIRA for flickers, called "Date of
last failure for flicker”?
[snip]
I don’t see how it’ll help since it’ll never be up to date, and the old value
will remain making us think it’s not been flickering for a long time.
In my mind the idea is not so much to use this field as a criteria to
close an issue but as a criteria to not close it.
That and I also wanted to propose an update in the release plan for the
first point:
Verify that no tests are failing on the CI Server (or that failures are
understood and update their date of last seen failure, see known
flickering tests).
Thanks
-Vincent
--
Simon Urli
Software Engineer at XWiki SAS
simon.u...@xwiki.com
More about us at http://www.xwiki.com
