Thank you for this discussion. I think disabling flaky tests in given way is reasonable.
I have one related question, once similar flaky test is disabled, and a ticket is reported. Who will be assigned to this ticket? Will it be the test author? My point is, that usually only test author and few other engineers working on the same component understand completely the given test, but probably they do not check the codebase daily if the test is still enabled. So, will be these people notified? Otherwise, most of these tickets will be probably closed after the proposed period without any test update/fix. Jozef Marko Software Developer jozef.ma...@ibm.com -----Original Message----- From: Toni Rikkola <trikk...@redhat.com> Sent: Wednesday, January 24, 2024 3:58 PM To: dev@kie.apache.org Subject: [EXTERNAL] Re: [PROPOSAL] Workflow for ignoring long failing tests The reason I mentioned this was that we had this same decision made 8 years ago and shared it to the entire team. It stated that anyone can disable any flaky test and make a ticket for it. It was even more strict, you just had to have one flaky failure. So to me it looks like we are just stating the same thing again hoping for a different outcome. But yes, to make it clear for everyone we should write it down and we can see how it goes and act on it later if needed. Toni On Wed, Jan 24, 2024 at 2:09 PM Tibor Zimányi <tzima...@apache.org> wrote: > I think we don't need to enforce it or gatekeep this any way. The > thing is, if we have the agreement that we want to have this official > workflow, it should fix the problem you are mentioning Toni, that > people are afraid to do so. If it will be officially stated, that > failing tests can be disabled in the code, people should just do it. > They can always point to this potential agreement. The best thing > would be to even document it on Confluence. I will do that if we have an > agreement on this. > > Tibor > > Dňa st 24. 1. 2024, 12:40 Toni Rikkola <trikk...@redhat.com> napísal(a): > > > Not informing the author was my mistake on that PR. I should have > > asked > the > > original author to review it. > > > > But the problem is not that we lack this procedure. The KIE team > > agreed > on > > this in 2015. It is that maybe we lack the strength to do it, > > especially when it is someone else's area of code. This needs some > > sort of > gatekeeper > > or supervisor and time frames when the build quality and if these > problems > > were acted on are validated and measured. > > > > Meanwhile I offer myself to work on these and disable them, if > > anyone > wants > > to point me to some flaky test I can act on them. I could even make > > some system for measurement. We can then, or now, decide what we > > then do with that data. > > > > Toni > > > > On Tue, Jan 23, 2024 at 8:04 PM Tibor Zimányi <tzima...@apache.org> > wrote: > > > > > I think the notification of the involved commiter could be done as > > > part > > of > > > creating of the issue to fix the tests. The person involved can be > > assigned > > > as an asignee. > > > > > > T. > > > > > > Dňa ut 23. 1. 2024, 18:50 Francisco Javier Tirado Sarti < > > > ftira...@redhat.com> > > > napísal(a): > > > > > > > Ok, but I think we first need to establish a way (my apologies > > > > is > that > > > way > > > > is already there) to notify committerthat a test that they were > > involved > > > > with is failing > > > > That way, we will avoid disable tests like this one > > > > INVALID URI REMOVED > > > > apache_incubator-2Dkie-2Dkogito-2Dexamples_issues_1831&d=DwIFaQ& > > > > c=jf_iaSHvJObTbx-siA1ZOg&r=Tvtr6trQuIEn20iSHtFwcpynBt8ViCybz4kYp > > > > jmDDUQ&m=C6gAvciwgXz7uJX__01AVZ3YHxSotIhuktwbnBnNuTvnLRU-NHMRqOd > > > > NENKhsqKe&s=8p3WYlpcXfcLl1n4BnrSpBWNIdPIUhslTPsb41eE4IM&e= , > > > which > > > > are quite important to ensure Springboot messaging is working. > > > > > > > > On Tue, Jan 23, 2024 at 6:38 PM ricardo zanini fernandes < > > > > ricardozan...@gmail.com> wrote: > > > > > > > > > +1 > > > > > > > > > > On Tue, Jan 23, 2024 at 1:03 PM Martin Cimbálek > > > > > <cim...@gmail.com> > > > > wrote: > > > > > > > > > > > +1 > > > > > > > > > > > > Regards, > > > > > > Martin Cimbalek > > > > > > > > > > > > On Tue, Jan 23, 2024 at 3:36 PM Tibor Zimányi < > tzima...@apache.org > > > > > > > > wrote: > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > I want to propose a workflow for situations, when some > > > > > > > tests > are > > > > > failing > > > > > > > for a longer time. In such cases, my proposed workflow is: > > > > > > > - If a test or a set of tests is failing for two days > (nightlies > > or > > > > PR > > > > > > > checks), ignore those tests in the codebase, so they are > > > > > > > not > > > > executed. > > > > > > > - File an issue in kie-issues repository, reporting the > > > > > > > test > > > > failures. > > > > > > > - If the issue is not resolved for half a year, delete > > > > > > > those > > tests > > > > from > > > > > > the > > > > > > > codebase. > > > > > > > > > > > > > > What do you think please? This should make sure all > > > > > > > failures > that > > > > don't > > > > > > get > > > > > > > fixed immediately after they occur get logged in the issue > > tracker, > > > > so > > > > > > they > > > > > > > can be appropriately handled and don't block unrelated PR > checks > > > and > > > > > > > builds. It will also make sure that tests that are not > > > > > > > fixed > for > > a > > > > long > > > > > > > time (therefore they could be perceived as not important) > > > > > > > are > > not a > > > > > > > maintenance burden for the future. > > > > > > > > > > > > > > Best regards, > > > > > > > Tibor > > > > > > > > > > > > > > > > > > > > > > > > > > > >
