RE: BadApple report

Hi Erick, The teste-only jobs @ ASF and Policeman Jenkins jobs of master branch were all converted to Gradle. It should have no effect on the Hossman Badapples analysis, but maybe have an extra look next week to find outlyers. The statistics about failed jobs in the XML output should be the

Re: BadApple report, but please read the first bit

Thanks Kevin; clearly I missed the link to that which I can now see at fucit. I was worried I may have worked on something that could have perturbed this recent issue but no -- I don't think so. ~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Wed,

Re: BadApple report, but please read the first bit

Didn’t think at first (only one cup of coffee). Here’s the Emails that test appears in, the formatting is poor… After that is the raw data from Hoss’ rollups that might be easier to ingest. I have 1.3G of this kind of historical data, I’ve had vague thoughts about putting it someplace

Re: BadApple report, but please read the first bit

http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test David for that specific test you asked the failures are recent with as far as I know no change to HDFS stuff. Starting June/July failing regularly. Kevin

Re: BadApple report, but please read the first bit

I have the weekly rollups (with a few gaps) going back to about April 2018, but nothing’s been done to try to make them generally available. Each BadApple report has rates for the last 4 weeks in the attached file, just below "Failures over the last 4 weeks, but not every week. Ordered

Re: BadApple report, but please read the first bit

Do we have any long term (aka "longitudinal") pass/fail rates for tests? SharedFSAutoReplicaFailoverTest in particular is kinda-sorta tied to HDFS, and that's going away to a plug-in for 9.0. The shared file system notion isn't well supported in SolrCloud, I think. ~ David Smiley Apache

Re: Badapple report

Merged (thanks Mike D!). Atri On Tue, Aug 11, 2020 at 5:32 PM Erick Erickson wrote: > > Great, thanks! Let me know when you push it, I can beast the test again. > > > On Aug 11, 2020, at 3:48 AM, Atri Sharma wrote: > > > > I investigated testRequestRateLimiters and hardened the tests up: > > >

Re: Badapple report

Great, thanks! Let me know when you push it, I can beast the test again. > On Aug 11, 2020, at 3:48 AM, Atri Sharma wrote: > > I investigated testRequestRateLimiters and hardened the tests up: > > https://github.com/apache/lucene-solr/pull/1736 > > This will stop testConcurrentRequests from

Re: Badapple report

I investigated testRequestRateLimiters and hardened the tests up: https://github.com/apache/lucene-solr/pull/1736 This will stop testConcurrentRequests from failing and should hopefully stop testSlotBorrowing as well. If testSlotBorrowing continues to fail, I will have to rethink the test. On

Re: Badapple report

OK, thanks. I’m not really annotating things at this point, although occasionally removing some that haven’t failed in a long time. > On Aug 10, 2020, at 1:44 PM, Tomás Fernández Löbbe > wrote: > > Hi Erick, > I've introduced and later fixed a bug in TestConfig. It hasn't failed since, > so

Re: Badapple report

Hi Erick, I've introduced and later fixed a bug in TestConfig. It hasn't failed since, so please don't annotate it. On Mon, Aug 10, 2020 at 7:47 AM Erick Erickson wrote: > We’re backsliding some. I encourage people to look at: > http://fucit.org/solr-jenkins-reports/failure-report.html, we have

Re: BadApple report

Megan: There are a number of tests that have been flagged by some devs that, no matter what, should _not_ be annotated with BadApple or AwaitsFix and that’s just a list to remind me what they are. It’s not much of a deal, though, because I’m not doing much annotating lately. The original

Re: BadApple report

Hi Erick, I'm wondering what is meant by "DO NOT ANNOTATE LIST" at the start of the report? Better yet, can you please link to the scraping tool used to generate the report? Thank you! Megan On Mon, Jul 6, 2020 at 8:07 AM Erick Erickson wrote: > Holding fairly steady, but IDK whether Hoss’

Re: BadApple report

Thanks for letting me know Tomás As useful as Hoss’ rollups are, there’s always a lag to deal with, sounds like this is one. > On Jun 8, 2020, at 2:26 PM, Tomás Fernández Löbbe > wrote: > > Thanks for keeping an eye Erick. I took a quick look at the > "TestIndexSearcher" failures and I

Re: BadApple report

Thanks for keeping an eye Erick. I took a quick look at the "TestIndexSearcher" failures and I think they're related to SOLR-14525. Should be fixed after this[1] commit by Noble. [1] https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=5827ddf On Mon, Jun 8, 2020 at 7:52 AM Erick Erickson

Re: BadApple report. It's worth reviewing the SuppressWarnings section even if you ignore the rest.

If you go to Hoss’ rollups here: http://fucit.org/solr-jenkins-reports/ Click on "Failures rates for the last 24h/7days” then click on one of the tests you’ll get a popup with a link to the output. IDK how long the output is kept around though. > On Jun 2, 2020, at 4:08 AM, Noble Paul wrote:

Re: BadApple report. It's worth reviewing the SuppressWarnings section even if you ignore the rest.

Is there a way to see the failures and their logs? On Tue, Jun 2, 2020 at 12:02 AM Erick Erickson wrote: > > This week is a significant improvement. Short form: > > > Raw fail count by week totals, most recent week first (corresponds to bits): > Week: 0 had 68 failures > Week: 1 had 113

Re: BadApple report

> Hoss’s rollups are here: > http://fucit.org/solr-jenkins-reports/failure-report.html which show the > rates, but not where they came from. If I click on a particular test entry on "failure-report.html", I'm presented with dialog with links for each failure. Clicking that link takes me to a

Re: BadApple report

Thanks that helps. I'll try to have a look at some of the failures related to areas I know. Ilan On Mon, May 25, 2020 at 7:07 PM Erick Erickson wrote: > Ilan: > > That’s, unfortunately, not an easy question. Hoss’s rollups are here: > http://fucit.org/solr-jenkins-reports/failure-report.html

Re: BadApple report

Ilan: That’s, unfortunately, not an easy question. Hoss’s rollups are here: http://fucit.org/solr-jenkins-reports/failure-report.html which show the rates, but not where they came from. Here’s an example of a failure from Jenkins, if you follow the link you can see the full output, (click

Re: BadApple report

Where are the test failure details? On Mon, May 25, 2020 at 4:47 PM Erick Erickson wrote: > Here’s the summary: > > Raw fail count by week totals, most recent week first (corresponds to > bits): > Week: 0 had 113 failures > Week: 1 had 103 failures > Week: 2 had 102 failures > Week: 3

Re: BadApple report

> > 0123 59.4 195 92 HdfsSyncSliceTest.test I'm looking into this HdfsSyncSliceTest failure. Jira https://issues.apache.org/jira/browse/SOLR-13886 Kevin Risden Kevin Risden On Mon, Apr 13, 2020 at 8:35 AM Erick Erickson wrote: > We’re backsliding a bit. Note that over the

Re: BadApple report

Will do. Actually, won’t do (disable that is)…. One of the things that’s kind of a pain is that the report doesn’t distinguish between different JVMs so there’s no really convenient way to ignore this kind of thing. Anyway, I’ve put both of them in my list, and I have to say I’m not actively

Re: BadApple report

Same goes for TestPackedInts. Currently test runs containing ZGC or Shenandoah garbage collectors don't reflect the test itself. Please don't disable them. On Mon, Jan 6, 2020 at 12:38 PM Robert Muir wrote: > We shouldn't disable Test2BPostings since there is nothing wrong with the > test: this

Re: BadApple report

We shouldn't disable Test2BPostings since there is nothing wrong with the test: this is one impacted by bugs in the Shenandoah and ZGC garbage collectors. See the other threads on the dev-list about them. On Mon, Jan 6, 2020 at 10:47 AM Erick Erickson wrote: > Short form: > > There were 1480

Re: BadApple report

HdfsAutoAddReplicasIntegrationTest.testSimple I am going to awaitsfix this test - https://issues.apache.org/jira/browse/SOLR-13338. I haven't had time to look into recent failures. I thought the Jetty upgrade would have helped. It had very similar timeout waiting exception. Kevin Risden On

Re: BadApple report, PLEASE CHECK THE FIRST PART.

Hi Erick, Le lun. 10 sept. 2018 à 20:06, Erick Erickson a écrit : > First, I have these two lists, are they still current? > > DO NOT ENABLE LIST: > 'TestControlledRealTimeReopenThread.testCRTReopen' > 'TestICUNormalizer2CharFilter.testRandomStrings' > 'TestICUTokenizerCJK' > +1 to

Re: BadApple report TestPolicy, TestCollectionStateWatchers TestWithCollection

Sure, won't BadApple TestWithCollection. On Mon, Aug 27, 2018 at 10:01 PM Shalin Shekhar Mangar wrote: > > Thanks Erick. I'm working on fixing TestWithCollection so please do not > BadApple it this week. > > On Tue, Aug 28, 2018 at 1:04 AM Erick Erickson > wrote: >> >> On the plus side, the

Re: BadApple report TestPolicy, TestCollectionStateWatchers TestWithCollection

Thanks Erick. I'm working on fixing TestWithCollection so please do not BadApple it this week. On Tue, Aug 28, 2018 at 1:04 AM Erick Erickson wrote: > On the plus side, the CDCR tests (except BiDir) seem to be fixed. > > Also on the plus side, there are quite a number of tests that have > _not_

Re: BadApple report. Seems like I'm wasting my time.

I still think it’s a mistake to try and use all the Jenkins results to drive ignoring tests. It needs to be an objective measure in a good env. We also should not be ignoring tests in mass.l without individual consideration. Critical test coverage should be treated differently than any random

Re: BadApple report. Seems like I'm wasting my time.

Alexandre: Feel free! What I'm struggling with is not that someone checked in some code that all the sudden started breaking things. Rather that a test that's been working perfectly will fail once the won't reproducibly fail again and does _not_ appear to be related to recent code changes. In

Re: BadApple report. Seems like I'm wasting my time.

Just a completely random thought that I do not have deep knowledge for (still learning my way around Solr tests). Is this something that Machine Learning could help with? The Github repo/history is a fantastic source of learning on who worked on which file, how often, etc. We certainly should be

Re: BadApple report. Seems like I'm wasting my time.

Shawn: Trouble is there were 945 tests that failed at least once in the last 4 weeks. And the trend is all over the map on a weekly basis. e-mail-2018-06-11.txt: There were 989 unannotated tests that failed e-mail-2018-06-18.txt: There were 689 unannotated tests that failed

Re: BadApple report. Seems like I'm wasting my time.

On 7/30/2018 11:52 AM, Erick Erickson wrote: Is anybody paying the least attention to this or should I just stop bothering? The job you're doing is thankless.  That's the nature of the work.  I'd love to have the time to really help you out. If only my employer didn't expect me to spend so

Re: BadApple report. Seems like I'm wasting my time.

I was thinking of the challenge with sporadic/random failures the other day and what would help. I think more and smarter notifications of failures could help a lot. (A) Using Git history, a Jenkins plugin could send an email to anyone who touched the failing test in the last 4 weeks. If that

Re: BadApple report. Seems like I'm wasting my time.

Hi Erick, > Is anybody paying the least attention to this or should I just stop bothering? I think your effort is invaluable, although if not backed by actions to fix those bugs it's pointless. I'm paying attention to the Lucene part. As for Solr tests I admit I gave up hope a longer while ago.

Re: BadApple report. Seems like I'm wasting my time.

Steve: Ok, InfixSuggestersTest.testShutdownDuringBuild is in my "Do not annotate" list. On Mon, Jul 30, 2018 at 7:33 PM, Steve Rowe wrote: > Hi Erick, > > I think it’s valuable to continue the BadApple process as you’re currently > running it. I’m guessing most people will not engage, but

Re: BadApple report. Seems like I'm wasting my time.

Hi Erick, I think it’s valuable to continue the BadApple process as you’re currently running it. I’m guessing most people will not engage, but some will, myself included (though I don’t claim to read the list every week). I’m working on fixing InfixSuggestersTest.testShutdownDuringBuild

Re: BadApple report

ShardSplitTest seems to be failing a ton. The test report at fucit.org shows this too -- 74%! ant test -Dtestcase=InfixSuggestersTest -Dtests.method=testShutdownDuringBuild -Dtests.seed=4D1D84B71946707A -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=en-ZA -Dtests.timezone=Asia/Bishkek

Re: BadApple report

Thanks! Le mar. 12 juin 2018 à 10:41, Andrzej Białecki < andrzej.biale...@lucidworks.com> a écrit : > > On 12 Jun 2018, at 09:01, Adrien Grand wrote: > > Hi Andrzej, > > I'm seeing a fix commit for SolrRrdBackendFactoryTest but not for > IndexSizeTriggerTest. The reason I'm looking is that the

Re: BadApple report

> On 12 Jun 2018, at 09:01, Adrien Grand wrote: > > Hi Andrzej, > > I'm seeing a fix commit for SolrRrdBackendFactoryTest but not for > IndexSizeTriggerTest. The reason I'm looking is that the 7.4 smokerelease > build failed on IndexSizeTriggerTest so I wanted to check whether the fix had

Re: BadApple report

Hi Andrzej, I'm seeing a fix commit for SolrRrdBackendFactoryTest but not for IndexSizeTriggerTest. The reason I'm looking is that the 7.4 smokerelease build failed on IndexSizeTriggerTest so I wanted to check whether the fix had been backported there. Le lun. 11 juin 2018 à 22:33, Andrzej

Re: BadApple report

Great! I won't BadApple those of course. Erick On Mon, Jun 11, 2018 at 1:33 PM, Andrzej Białecki wrote: > > > On 11 Jun 2018, at 17:51, Erick Erickson wrote: > > IndexSizeTriggerTest.testMixedBounds > > > I committed a change today that should fix this. > > SearchRateTriggerTest.testTrigger >

Re: BadApple report

> On 11 Jun 2018, at 17:51, Erick Erickson wrote: > > IndexSizeTriggerTest.testMixedBounds I committed a change today that should fix this. > SearchRateTriggerTest.testTrigger > TestLargeCluster.testBasic > TestLargeCluster.testNodeLost On June 6 I made a change that should fix these

Re: Badapple report, rehabilitating annotated tests. Please read the intro and comment.

Got it, thanks Adrien. Anyone else: If you have tests you don't want enabled, let me know. I'll add to a permanent list and add to the report. On Tue, Jun 5, 2018 at 10:06 AM, Adrien Grand wrote: > Hi Erick, > > Le mar. 5 juin 2018 à 18:19, Erick Erickson a > écrit : >> >>

Re: Badapple report, rehabilitating annotated tests. Please read the intro and comment.

Hi Erick, Le mar. 5 juin 2018 à 18:19, Erick Erickson a écrit : > TestControlledRealTimeReopenThread.testCRTReopen This test relies on wall clock time, it needs to be refactored before being enabled again. TestICUNormalizer2CharFilter.testRandomStrings > We did several ICU4J upgrades

Re: BadApple report

Great! I won't BadApple TestDocTermOrds on Thursday. Thanks! On Tue, Apr 17, 2018 at 1:20 AM, Alan Woodward wrote: > TestDocTermOrds should be fixed now, as should TestIndexSorting (I > un-badappled the latter yesterday) > >> On 16 Apr 2018, at 21:59, Erick Erickson

Re: BadApple report

TestDocTermOrds should be fixed now, as should TestIndexSorting (I un-badappled the latter yesterday) > On 16 Apr 2018, at 21:59, Erick Erickson wrote: > > We have a much smaller list of _consistently_ failing tests this week, i.e. > tests that are in Hoss' rollups

Re: BadApple report this week

Cool, I'll reconcile when it happens, perhaps this evening On Mon, Apr 2, 2018 at 7:57 AM, Mark Miller wrote: > 4th report is coming today in a couple hours. Still a manual snag involved > because I haven’t had a chance to figure out what happened with command line >

Re: BadApple report this week

4th report is coming today in a couple hours. Still a manual snag involved because I haven’t had a chance to figure out what happened with command line bitballoon site publishing. Mark On Mon, Apr 2, 2018 at 10:53 AM Erick Erickson wrote: > I don't have any candidates