There is an okay chance I'm going to start making some improvements here as well. I've been working on a very stable set of tests on my starburst branch and will slowly bring in test fixes over time (I've already been making some on that branch for important tests). We should currently be defaulting to tests.badapples=false on all solr test runs - it's a joke to try and get a clean run otherwise, and even then somehow 4 or 5 tests that fail somewhat commonly have so far avoided Erick's @BadApple hack and slash. They are bad appled on my dev branch now, but that is currently where any time I have is spent rather than on the main dev branches.
Also, too many flakey tests are introduced because devs are not beasting or beasting well before committing new heavy tests. Perhaps we could add some docs around that. We have built in beasting support, we need to emphasize that a couple passes on a new test is not sufficient to test it's quality. - Mark On Fri, Jun 15, 2018 at 9:46 AM Erick Erickson <[email protected]> wrote: > (Siiiiggggghhhh) All very true. You're not alone in your frustration. > > I've been trying to at least BadApple tests that fail consistently, so > another option could be to disable BadApple'd tests. My hope has been > to get to the point of being able to reliably get clean runs, at least > when BadApple'd tests are disabled. > > From that point I want to draw a line in the sand and immediately > address tests that fail that are _not_ BadApple'd. At least then we'll > stop getting _worse_. And then we can work on the BadApple'd tests. > But as David says, that's not going to be any time soon. It's been a > couple of months that I've been trying to just get the tests > BadApple'd without even trying to fix any of them. > > It's particularly pernicious because with all the noise we don't see > failures we _should_ see. > > So I don't have any good short-term answer either. We've built up a > very large technical debt in the testing. The first step is to stop > adding more debt, which is what I've been working on so far. And > that's the easy part.... > > Siiiiiiiiiiiiiigggggggggghhhhhhhhhh > > Erick > > > On Fri, Jun 15, 2018 at 5:29 AM, David Smiley <[email protected]> > wrote: > > (Sigh) I sympathize with your points Simon. I'm +1 to modify the > > Lucene-side JIRA QA bot (Yetus) to not execute Solr tests. We can and > are > > trying to improve the stability of the Solr tests but even optimistically > > the practical reality is that it won't be good enough anytime soon. > When we > > get there, we can reverse this. > > > > On Fri, Jun 15, 2018 at 3:32 AM Simon Willnauer < > [email protected]> > > wrote: > >> > >> folks, > >> > >> I got more active working on IndexWriter and Soft-Deletes etc. in the > >> last couple of weeks. It's a blast again and I really enjoy it. The > >> one thing that is IMO not acceptable is the status of solr tests. I > >> tried so many times to get them passing on several different OSs but > >> it seems this is pretty hopepless. It's get's even worse the > >> Lucene/Solr QA job literally marks every ticket I attach a patch to as > >> `-1` because of arbitrary solr tests, here is an example: > >> > >> || Reason || Tests || > >> | Failed junit tests | solr.rest.TestManagedResourceStorage | > >> | | solr.cloud.autoscaling.SearchRateTriggerIntegrationTest | > >> | | solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest | > >> | | solr.client.solrj.impl.CloudSolrClientTest | > >> | | solr.common.util.TestJsonRecordReader | > >> > >> Speaking to other committers I hear we should just disable this job. > >> Sorry, WTF? > >> > >> These tests seem to fail all the time, randomly and over and over > >> again. This renders the test as entirely useless to me. I even invest > >> time (wrong, I invested) looking into it if they are caused by me or > >> if I can do something about it. Yet, someone could call me out for > >> being responsible for them as a commiter, yes I am hence this email. I > >> don't think I am obliged to fix them. These projects have 50+ > >> committers and having a shared codebase doesn't mean everybody has to > >> take care of everything. I think we are at the point where if I work > >> on Lucene I won't run solr tests at all otherwise there won't be any > >> progress. On the other hand solr tests never pass I wonder if the solr > >> code-base gets changes nevertheless? That is again a terrible > >> situation. > >> > >> I spoke to varun and anshum during buzzwords if they can give me some > >> hints what I am doing wrong but it seems like the way it is. I feel > >> terrible pushing stuff to our repo still seeing our tests fail. I get > >> ~15 build failures from solr tests a day I am not the only one that > >> has mail filters to archive them if there isn't a lucene tests in the > >> failures. > >> > >> This is a terrible state folks, how do we fix it? It's the lucene land > >> that get much love on the testing end but that also requires more work > >> on it, I expect solr to do the same. That at the same time requires > >> stop pushing new stuff until the situation is under control. The > >> effort of marking stuff as bad apples isn't the answer, this requires > >> effort from the drivers behind this project. > >> > >> simon > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > >> > > -- > > Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker > > LinkedIn: http://linkedin.com/in/davidwsmiley | Book: > > http://www.solrenterprisesearchserver.com > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > -- - Mark about.me/markrmiller
