Re: Our test failure rate is unacceptable.

2020-09-21 Thread Noble Paul
I tried in 8x of course. The ant command is gone from master anyway On Mon, Sep 21, 2020 at 1:32 AM Erick Erickson wrote: > > Can’t seem to get that to fail on master. Note the reproducible bit was 8x. > > > On Sep 19, 2020, at 5:13 PM, Erick Erickson wrote: > > > > ant test -Dtestcase=TestPack

Re: Our test failure rate is unacceptable.

2020-09-20 Thread Erick Erickson
Can’t seem to get that to fail on master. Note the reproducible bit was 8x. > On Sep 19, 2020, at 5:13 PM, Erick Erickson wrote: > > ant test -Dtestcase=TestPackages -Dtests.seed=C29471044D369FD3 > -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=et-EE > -Dtests.timezone=Europe/Mariehamn

Re: Our test failure rate is unacceptable.

2020-09-19 Thread Noble Paul
It passes for me all the time Erick Can you please test with the branch https://github.com/apache/lucene-solr/tree/jira/solr14879 On Sun, Sep 20, 2020 at 7:14 AM Erick Erickson wrote: > > This seems to be a reproducing seed, at least 2/2 > > ant test -Dtestcase=TestPackages -Dtests.seed=C29471

Re: Our test failure rate is unacceptable.

2020-09-19 Thread Erick Erickson
This seems to be a reproducing seed, at least 2/2 ant test -Dtestcase=TestPackages -Dtests.seed=C29471044D369FD3 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=et-EE -Dtests.timezone=Europe/Mariehamn -Dtests.asserts=true -Dtests.file.encoding=UTF-8 > On Sep 19, 2020, at 6:40 AM, Eric P

Re: Our test failure rate is unacceptable.

2020-09-19 Thread Eric Pugh
I’ll try and help with testing this feature more, as I have a specific package that needs this feature. We are somewhat stuck in a weird time, as we’re doing some great stuff, like the packages, to make core Solr foot print smaller, which means we need to add more complexity to core Solr, ye

Re: Our test failure rate is unacceptable.

2020-09-18 Thread Noble Paul
I shall revert the changes and work on a solution On Sat, Sep 19, 2020, 6:54 AM Jason Gerlowski wrote: > > I don't think it is along the Apache way to revert somebody's commit > without an explicit permission to do so > Interesting, I made the Devil's Advocate argument above with the > Apache Wa

Re: Our test failure rate is unacceptable.

2020-09-18 Thread Jason Gerlowski
> I don't think it is along the Apache way to revert somebody's commit without > an explicit permission to do so Interesting, I made the Devil's Advocate argument above with the Apache Way specifically in mind. "Community over Code" comes up most often in terms of navigating interpersonal conflic

Re: Our test failure rate is unacceptable.

2020-09-18 Thread Tomás Fernández Löbbe
I thought we were talking about reverting your own commits, not someone else’s... On Fri, Sep 18, 2020 at 12:31 PM Dawid Weiss wrote: > I don't think it is along the Apache way to revert somebody's commit > > without an explicit permission to do so... Not that I would personally > > mind if some

Re: Our test failure rate is unacceptable.

2020-09-18 Thread Dawid Weiss
I don't think it is along the Apache way to revert somebody's commit without an explicit permission to do so... Not that I would personally mind if somebody did it for me. On Fri, Sep 18, 2020 at 9:06 PM Tomás Fernández Löbbe wrote: > > Sometimes Jenkins may take hours to take your commit, may fa

Re: Our test failure rate is unacceptable.

2020-09-18 Thread Erick Erickson
Jason: I wouldn’t object to reverting immediately. It’s a courtesy to give someone a chance to fix it themselves rather than unilaterally revert is all. > On Sep 18, 2020, at 3:05 PM, Tomás Fernández Löbbe > wrote: > > Sometimes Jenkins may take hours to take your commit, may fail in the midd

Re: Our test failure rate is unacceptable.

2020-09-18 Thread Tomás Fernández Löbbe
Sometimes Jenkins may take hours to take your commit, may fail in the middle of your night, may not fail consistently, etc. That's why I don't think giving specific timeframes makes sense, but yes, as soon as you notice it's failing, it's either fix immediately or revert IMO. On Fri, Sep 18, 2020

Re: Our test failure rate is unacceptable.

2020-09-18 Thread Jason Gerlowski
> If it’s inadvertently added, we either fix it within an hour or so or revert > the offending commit > I don't want to set specific time frames, To play Devil's Advocate here: why wait even an hour to revert a 100% test failure? Reverts are usually trivial to do, unblock others immediately, an

Re: Our test failure rate is unacceptable.

2020-09-18 Thread Tomás Fernández Löbbe
I believe these failures are associated to https://issues.apache.org/jira/browse/SOLR-14151 • FAILED: org.apache.solr.pkg.TestPackages.classMethod • FAILED: org.apache.solr.schema.PreAnalyzedFieldManagedSchemaCloudTest.testAdd2Fields • FAILED: org.apache.solr.schema.ManagedSchemaRoundRobinCloudTe

Re: Our test failure rate is unacceptable.

2020-09-18 Thread Atri Sharma
When I said temporary, I meant 3-4 hours. Definitely not more than that. IMO we should just roll back offending commits if they are easily identifiable. I agree with you — we all have been guilty of breaking builds (mea culpa as well). The bad part here is the longevity of the failures. On Fri,

Re: Our test failure rate is unacceptable.

2020-09-18 Thread Kevin Risden
HdfsAutoAddReplicasTest is just wrapping AutoAddReplicas tests with HDFS last I checked. There haven't been any HDFS changes lately that I've seen. So what changed in regards to AutoAddReplicas? Based on http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.s

Re: Our test failure rate is unacceptable.

2020-09-18 Thread Erick Erickson
bq. IMO if a temporary instability is to be introduced deliberately, it should be published on the list Actually, I disagree. Having anything in the tests that fail 100% of the time is just unacceptable since it becomes a barrier for everyone else. AFAIK, if the problem can be identified to a p

Re: Our test failure rate is unacceptable.

2020-09-18 Thread Atri Sharma
IMO if a temporary instability is to be introduced deliberately, it should be published on the list. If it’s inadvertently added, we either fix it within an hour or so or revert the offending commit. On Fri, 18 Sep 2020 at 20:26, Erick Erickson wrote: > http://fucit.org/solr-jenkins-reports/fail

Our test failure rate is unacceptable.

2020-09-18 Thread Erick Erickson
http://fucit.org/solr-jenkins-reports/failure-report.html HdfsAutoAddReplicasTest failing 100% of the time. TestPackages.classMethod failing 100% of the time 3-4 AutoAddReplicas tests failing 98% of the time. Is anyone looking at these? I realize the code base is changing a lot, and some tempora