I understood this policy refers just to flaky tests. We should not merge any PR on red.
On Fri, Jan 26, 2024 at 8:00 AM Toni Rikkola <trikk...@redhat.com> wrote: > What happened with the SpringBoot. I made the change and actually thought > the PR reviews would alert the right people and you could not get a PR in > without approval from the devs responsible for that area. This is just > since I don't know who is working on what and on what level. I don't know > how the wording in this rule could be set so that the right person has > actually been contacted. > > One other thing to take note of is that this policy applies for all the > broken tests, not just flaky ones. So if you manage to get a bad commit in, > it is then ok to ignore all the tests that that commit broke and now the > responsibility of fixing the codes are on the person that wrote the test > that picks up the bugs or changes someone else introduced. > > I know I like to over think these regulations, but due to the structure at > Apache we agree on rules and some can be taken advantage of. If the thing > mentioned above happens, there is not one great leader who can give an > order to fix the codes. Also if a community member gets a commit in, there > is no handle for us to get them to fix anything. It is the community's > problem after that. > > I have to give this a -1, this has to target just flaky tests. If a commit > breaks the test, the right thing would be to revert ASAP. If we really need > to get something in that breaks tests in another location, and those tests > need to be redone, then we should vote for that change to go in as an > exception and ignore the broken tests in the same commit bundle. > Acknowledging there is no guarantee the tests commented out will ever be > fixed, even if they show a bug was introduced. > > So for example in the current case. If the build is broken and the tests > are not flaky, we can vote for disabling those tests. I think that vote > will pass. > > I hope we understand what this project is now. One person can make a big > mess if the guards are not set up correctly. Five more companies can come > in and start working on this, out ruling everyone we have now. The base has > to be solid for many reasons. Letting this change in as a rule opens us to > too much abuse, intentional or unintentional. > > Toni > > On Thu, Jan 25, 2024 at 2:59 PM Francisco Javier Tirado Sarti < > ftira...@redhat.com> wrote: > > > +1 > > Thanks Tibor > > > > On Thu, Jan 25, 2024 at 9:37 AM Tibor Zimányi <tzima...@apache.org> > wrote: > > > > > Hi, > > > > > > as I wrote in one of the previous e-mails in this thread: > > > "I think the notification of the involved commiter could be done as > part > > of > > > creating of the issue to fix the tests. The person involved can be > > assigned > > > as an asignee. " > > > > > > To clarify, by involved commiter I meant either author of the test or > the > > > person who is maintaining that part of the codebase. > > > > > > Hopefully this clarifies this. > > > > > > Tibor > > > > > > Dňa št 25. 1. 2024, 9:22 Francisco Javier Tirado Sarti < > > > ftira...@redhat.com> > > > napísal(a): > > > > > > > I agree with Josef, I realize the SpringBoot listener example test > was > > > > disabled while fixing an issue related to it. If not, the test would > > have > > > > remained disabled. > > > > > > > > On Thu, Jan 25, 2024 at 5:51 AM Jozef Marko > > <jozef.ma...@ibm.com.invalid > > > > > > > > wrote: > > > > > > > > > Thank you for this discussion. I think disabling flaky tests in > given > > > way > > > > > is reasonable. > > > > > > > > > > I have one related question, once similar flaky test is disabled, > > and a > > > > > ticket is reported. Who will be assigned to this ticket? Will it be > > the > > > > > test author? > > > > > > > > > > My point is, that usually only test author and few other engineers > > > > working > > > > > on the same component understand completely the given test, but > > > probably > > > > > they do not check the codebase daily if the test is still enabled. > > > > > > > > > > So, will be these people notified? Otherwise, most of these tickets > > > will > > > > > be probably closed after the proposed period without any test > > > update/fix. > > > > > > > > > > Jozef Marko > > > > > Software Developer > > > > > jozef.ma...@ibm.com > > > > > > > > > > -----Original Message----- > > > > > From: Toni Rikkola <trikk...@redhat.com> > > > > > Sent: Wednesday, January 24, 2024 3:58 PM > > > > > To: dev@kie.apache.org > > > > > Subject: [EXTERNAL] Re: [PROPOSAL] Workflow for ignoring long > failing > > > > tests > > > > > > > > > > The reason I mentioned this was that we had this same decision > made 8 > > > > > years ago and shared it to the entire team. > > > > > It stated that anyone can disable any flaky test and make a ticket > > for > > > > it. > > > > > It was even more strict, you just had to have one flaky failure. > > > > > > > > > > So to me it looks like we are just stating the same thing again > > hoping > > > > for > > > > > a different outcome. > > > > > > > > > > But yes, to make it clear for everyone we should write it down and > we > > > can > > > > > see how it goes and act on it later if needed. > > > > > > > > > > Toni > > > > > > > > > > On Wed, Jan 24, 2024 at 2:09 PM Tibor Zimányi <tzima...@apache.org > > > > > > wrote: > > > > > > > > > > > I think we don't need to enforce it or gatekeep this any way. The > > > > > > thing is, if we have the agreement that we want to have this > > official > > > > > > workflow, it should fix the problem you are mentioning Toni, that > > > > > > people are afraid to do so. If it will be officially stated, that > > > > > > failing tests can be disabled in the code, people should just do > > it. > > > > > > They can always point to this potential agreement. The best thing > > > > > > would be to even document it on Confluence. I will do that if we > > have > > > > an > > > > > agreement on this. > > > > > > > > > > > > Tibor > > > > > > > > > > > > Dňa st 24. 1. 2024, 12:40 Toni Rikkola <trikk...@redhat.com> > > > > napísal(a): > > > > > > > > > > > > > Not informing the author was my mistake on that PR. I should > have > > > > > > > asked > > > > > > the > > > > > > > original author to review it. > > > > > > > > > > > > > > But the problem is not that we lack this procedure. The KIE > team > > > > > > > agreed > > > > > > on > > > > > > > this in 2015. It is that maybe we lack the strength to do it, > > > > > > > especially when it is someone else's area of code. This needs > > some > > > > > > > sort of > > > > > > gatekeeper > > > > > > > or supervisor and time frames when the build quality and if > these > > > > > > problems > > > > > > > were acted on are validated and measured. > > > > > > > > > > > > > > Meanwhile I offer myself to work on these and disable them, if > > > > > > > anyone > > > > > > wants > > > > > > > to point me to some flaky test I can act on them. I could even > > make > > > > > > > some system for measurement. We can then, or now, decide what > we > > > > > > > then do with that data. > > > > > > > > > > > > > > Toni > > > > > > > > > > > > > > On Tue, Jan 23, 2024 at 8:04 PM Tibor Zimányi < > > tzima...@apache.org > > > > > > > > > > wrote: > > > > > > > > > > > > > > > I think the notification of the involved commiter could be > done > > > as > > > > > > > > part > > > > > > > of > > > > > > > > creating of the issue to fix the tests. The person involved > can > > > be > > > > > > > assigned > > > > > > > > as an asignee. > > > > > > > > > > > > > > > > T. > > > > > > > > > > > > > > > > Dňa ut 23. 1. 2024, 18:50 Francisco Javier Tirado Sarti < > > > > > > > > ftira...@redhat.com> > > > > > > > > napísal(a): > > > > > > > > > > > > > > > > > Ok, but I think we first need to establish a way (my > > apologies > > > > > > > > > is > > > > > > that > > > > > > > > way > > > > > > > > > is already there) to notify committerthat a test that they > > > were > > > > > > > involved > > > > > > > > > with is failing > > > > > > > > > That way, we will avoid disable tests like this one > > > > > > > > > INVALID URI REMOVED > > > > > > > > > > > > apache_incubator-2Dkie-2Dkogito-2Dexamples_issues_1831&d=DwIFaQ& > > > > > > > > > > > > c=jf_iaSHvJObTbx-siA1ZOg&r=Tvtr6trQuIEn20iSHtFwcpynBt8ViCybz4kYp > > > > > > > > > > > > jmDDUQ&m=C6gAvciwgXz7uJX__01AVZ3YHxSotIhuktwbnBnNuTvnLRU-NHMRqOd > > > > > > > > > NENKhsqKe&s=8p3WYlpcXfcLl1n4BnrSpBWNIdPIUhslTPsb41eE4IM&e= > , > > > > > > > > which > > > > > > > > > are quite important to ensure Springboot messaging is > > working. > > > > > > > > > > > > > > > > > > On Tue, Jan 23, 2024 at 6:38 PM ricardo zanini fernandes < > > > > > > > > > ricardozan...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > > > > > On Tue, Jan 23, 2024 at 1:03 PM Martin Cimbálek > > > > > > > > > > <cim...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > Martin Cimbalek > > > > > > > > > > > > > > > > > > > > > > On Tue, Jan 23, 2024 at 3:36 PM Tibor Zimányi < > > > > > > tzima...@apache.org > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > I want to propose a workflow for situations, when > some > > > > > > > > > > > > tests > > > > > > are > > > > > > > > > > failing > > > > > > > > > > > > for a longer time. In such cases, my proposed > workflow > > > is: > > > > > > > > > > > > - If a test or a set of tests is failing for two days > > > > > > (nightlies > > > > > > > or > > > > > > > > > PR > > > > > > > > > > > > checks), ignore those tests in the codebase, so they > > are > > > > > > > > > > > > not > > > > > > > > > executed. > > > > > > > > > > > > - File an issue in kie-issues repository, reporting > the > > > > > > > > > > > > test > > > > > > > > > failures. > > > > > > > > > > > > - If the issue is not resolved for half a year, > delete > > > > > > > > > > > > those > > > > > > > tests > > > > > > > > > from > > > > > > > > > > > the > > > > > > > > > > > > codebase. > > > > > > > > > > > > > > > > > > > > > > > > What do you think please? This should make sure all > > > > > > > > > > > > failures > > > > > > that > > > > > > > > > don't > > > > > > > > > > > get > > > > > > > > > > > > fixed immediately after they occur get logged in the > > > issue > > > > > > > tracker, > > > > > > > > > so > > > > > > > > > > > they > > > > > > > > > > > > can be appropriately handled and don't block > unrelated > > PR > > > > > > checks > > > > > > > > and > > > > > > > > > > > > builds. It will also make sure that tests that are > not > > > > > > > > > > > > fixed > > > > > > for > > > > > > > a > > > > > > > > > long > > > > > > > > > > > > time (therefore they could be perceived as not > > important) > > > > > > > > > > > > are > > > > > > > not a > > > > > > > > > > > > maintenance burden for the future. > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Tibor > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: dev-unsubscr...@kie.apache.org > > > > > For additional commands, e-mail: dev-h...@kie.apache.org > > > > > > > > > > > > > > > > > > > >
