You need to mark a test you recently added as flaky as it has failed for me twice on Windows.
[INFO] [ERROR] Failures: [ERROR] OnPropertyConditionTest.whenPropertyMatches:80 expected: <truth> but was: <goodbye> [ERROR] OnPropertyConditionTest.whenPropertyPresent:73 expected: <hello> but was: <goodbye> Ralph > On May 28, 2022, at 12:52 PM, Matt Sicker <[email protected]> wrote: > > Well, let’s take a look at the commits that have failed CI builds lately. Of > course CI was green before, MutableThreadContextMapFilterTest was only > introduced recently. Since then, the following completely unrelated commits > have CI build failures: > > * > https://github.com/apache/logging-log4j2/commit/22382a42bb888cdfb371553ec74c1da1f343ea02 > > <https://github.com/apache/logging-log4j2/commit/22382a42bb888cdfb371553ec74c1da1f343ea02> > (unrelated dependency change) > * > https://github.com/apache/logging-log4j2/commit/a4d562b0363d7f4ceef22aa7ab3d0ff11c2d6376 > > <https://github.com/apache/logging-log4j2/commit/a4d562b0363d7f4ceef22aa7ab3d0ff11c2d6376> > (change in manual) > * > https://github.com/apache/logging-log4j2/commit/26efbc801b8e87088c0924abde17001cff34a88e > > <https://github.com/apache/logging-log4j2/commit/26efbc801b8e87088c0924abde17001cff34a88e> > (unrelated test updates for arbiters) > * > https://github.com/apache/logging-log4j2/commit/ecd2d7783090683d3b7ba0ceb26e5c953f5c5a44 > > <https://github.com/apache/logging-log4j2/commit/ecd2d7783090683d3b7ba0ceb26e5c953f5c5a44> > (modifying the Dependabot config) > * several Dependabot PRs because fuck it at that point > > So a test was added that flakes on Windows CI which causes churn in trying to > merge PRs. The fact that this test was left enabled in production like this > for as long as it was is surprising. > > The GitHub action build failure spam sent directly to committers doesn’t help > the case, either. Once I start getting those after I manually verify my > commits before pushing them, unless the test failures are related to changes > I just made (which does happen once in a while due to platform-specific > differences or forgetting to commit a file), these emails are simply just > spam. Thus, I go and fix the source of spam. If you prefer the spam, then > change the action config to email yourself instead for all build failures and > re-enable all the flaky tests. > — > Matt Sicker > >> On May 28, 2022, at 14:26, Ralph Goers <[email protected]> wrote: >> >> There is only one flaw with that argument. When I did release 2.17.2 we >> weren’t having random test failures like this. So something changed. >> >> Ralph >> >>> On May 28, 2022, at 11:34 AM, Matt Sicker <[email protected]> wrote: >>> >>> Oh, and if these tests used to work, they wouldn’t be flaking from >>> unrelated changes such as when someone updates a README file or other >>> non-code area that somehow results in a failed CI run. >>> — >>> Matt Sicker >>> >>>> On May 28, 2022, at 10:29, Matt Sicker <[email protected]> wrote: >>>> >>>> I sent a separate email complaining about how Dependabot does this. >>>> Anyways, the disabled tests are flakes. As I’ve said before, I run the >>>> full build and suite of tests locally before pushing commits, but I’ve >>>> been getting tons of build failure emails regardless. So instead of >>>> ignoring CI failures as seems to be standard right now, I disabled the >>>> flaky tests where applicable until someone cares enough to fix them. I >>>> filed Jira issues so we don’t forget, either. It was also the only real >>>> feasible way to get through dependency upgrade PRs without them randomly >>>> failing due to unrelated flaky tests. >>>> >>>> — >>>> Matt Sicker >>>> >>>>> On May 28, 2022, at 03:46, Piotr P. Karwasz <[email protected]> >>>>> wrote: >>>>> >>>>> Hi Ralph, >>>>> >>>>> On Sat, 28 May 2022 at 10:17, Ralph Goers <[email protected]> >>>>> wrote: >>>>> >>>>>> What I don’t understand is why several of the Jira issues seemingly have >>>>>> 100 commits on various branches and flooded my inbox with email. >>>>>> >>>>>> This is Dependabot-related: every time a commit with "LOG4J2" appears in >>>>> *any* branch, JIRA sends an e-mail. Rebasing many Dependabot branches >>>>> caused a storm of e-mails (I had some 150 e-mails this morning). >>>>> >>>>> Piotr >>> >> >
