It's a recurring theme, huh, Erick? :) http://markmail.org/message/7eykbuyyaxbxn364
I agree with your opinion and I have expressed it more than once -- a test that is failing for longer while and cannot be identified or fixed is a candidate for removal. The noise in Solr tests have increased to a degree that I stopped looking (a long time ago), unless somebody explicitly pings me about something. I tried to fix some of those tests, but it's beyond my capabilities in many cases (and my time budget in others). I also recall some folks had a different take on the subject; see Mark Miller's opinion in the thread above, for example (there were other threads too, but I can't find them now). Dawid On Wed, Feb 21, 2018 at 5:36 PM, Tommaso Teofili <[email protected]> wrote: > +1, agree with Adrien, thanks for bringing this up Erick! > > > > Il giorno mer 21 feb 2018 alle ore 17:15 Adrien Grand <[email protected]> ha > scritto: >> >> Thanks for bringing this up Erick. I agree with you we should silence >> those frequent failures. Like you said, the side-effects of not silencing >> them are even worse. I'll add that these flaky tests also make releasing >> harder, it took me three runs last time (Lucene/Solr 7.2) for the release >> build to succeed because of failed tests. >> >> Le mer. 21 févr. 2018 à 16:52, Erick Erickson <[email protected]> a >> écrit : >>> >>> There's an elephant in the room, and it's that failing tests are being >>> ignored. Mind you, Solr and Lucene are progressing at a furious pace >>> with lots of great functionality being added. That said, we're >>> building up a considerable "technical debt" when it comes to testing. >>> >>> And I should say up front that major new functionality is expected to >>> take a while to shake out (e.g. autoscaling, streaming, V2 API etc.), >>> and noise from tests of new functionality is expected while things >>> bake. >>> >>> Below is a list of tests that have failed at least once since just >>> last night. This has been getting worse as time passes, the broken >>> window problem. Some e-mails have 10 failing tests (+/-) so unless I >>> go through each and every one I don't know whether something I've done >>> is causing a problem or not. >>> >>> I'm as guilty of letting things slide as anyone else, there's been a >>> long-standing issue with TestLazyCores that I work on sporadically for >>> instance that's _probably_ "something in the test framework".... >>> >>> Several folks have spent some time digging into test failures and >>> identifying at least some of the causes, kudos to them. It seems >>> they're voices crying out in the wilderness though. >>> >>> There is so much noise at this point that tests are becoming >>> irrelevant. I'm trying to work on SOLR-10809 for instance, where >>> there's a pretty good possibility that I'll close at least one thing >>> that shouldn't be closed. So I ran the full suite 10 times and >>> gathered all the failures. Now I have to try to separate the failures >>> caused by that JIRA from the ones that aren't related to it so I beast >>> each of the failing tests 100 times against master. If I get a failure >>> on master too for a particular test, I'll assume it's "not my problem" >>> and drive on. >>> >>> I freely acknowledge that this is poor practice. It's driven by >>> frustration and the desire to make progress. While it's poor practice, >>> it's not as bad as only looking at tests that I _think_ are related or >>> ignoring all tests failures I can't instantly recognize as "my fault". >>> >>> So what's our stance on this? Mark Miller had a terrific program at >>> one point allowing categorization of tests that failed at a glance, >>> but it hasn't been updated in a while. Steve Rowe is working on the >>> problem too. Hoss and Cassandra have both added to the efforts as >>> well. And I'm sure I'm leaving out others. >>> >>> Then there's the @Ignore and @BadApple annotations.... >>> >>> So, as a community, are we going to devote some energy to this? Or >>> shall we just start ignoring all of the frequently failing tests? >>> Frankly we'd be farther ahead at this point marking failing tests that >>> aren't getting any work with @Ignore or @BadApple and getting >>> compulsive about not letting any _new_ tests fail than continuing our >>> current path. I don't _like_ this option mind you, but it's better >>> than letting these accumulate forever and tests become more and more >>> difficult to use. As tests become more difficult to use, they're used >>> less and the problem gets worse. >>> >>> Note, I made no effort to separate suite .vs. individual reports >>> here..... >>> >>> Erick >>> >>> FAILED: >>> junit.framework.TestSuite.org.apache.lucene.index.TestBagOfPositions >>> FAILED: >>> junit.framework.TestSuite.org.apache.lucene.index.TestIndexWriterDeleteByQuery >>> FAILED: >>> junit.framework.TestSuite.org.apache.lucene.store.TestSleepingLockWrapper >>> FAILED: >>> junit.framework.TestSuite.org.apache.solr.analytics.legacy.facet.LegacyFieldFacetCloudTest >>> FAILED: >>> junit.framework.TestSuite.org.apache.solr.analytics.legacy.facet.LegacyFieldFacetExtrasCloudTest >>> FAILED: >>> junit.framework.TestSuite.org.apache.solr.analytics.legacy.facet.LegacyQueryFacetCloudTest >>> FAILED: >>> junit.framework.TestSuite.org.apache.solr.client.solrj.TestLBHttpSolrClient >>> FAILED: >>> junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation >>> FAILED: >>> junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest >>> FAILED: >>> junit.framework.TestSuite.org.apache.solr.core.AlternateDirectoryTest >>> FAILED: junit.framework.TestSuite.org.apache.solr.core.TestLazyCores >>> FAILED: >>> junit.framework.TestSuite.org.apache.solr.handler.component.DistributedFacetPivotSmallAdvancedTest >>> FAILED: >>> junit.framework.TestSuite.org.apache.solr.ltr.TestSelectiveWeightCreation >>> FAILED: >>> junit.framework.TestSuite.org.apache.solr.ltr.store.rest.TestModelManager >>> FAILED: >>> junit.framework.TestSuite.org.apache.solr.rest.schema.analysis.TestManagedSynonymFilterFactory >>> FAILED: >>> junit.framework.TestSuite.org.apache.solr.search.join.BlockJoinFacetDistribTest >>> FAILED: >>> junit.framework.TestSuite.org.apache.solr.security.TestAuthorizationFramework >>> FAILED: >>> junit.framework.TestSuite.org.apache.solr.update.processor.TestOpenNLPExtractNamedEntitiesUpdateProcessorFactory >>> FAILED: org.apache.lucene.index.TestStressNRT.test >>> FAILED: org.apache.solr.cloud.AddReplicaTest.test >>> FAILED: org.apache.solr.cloud.DeleteShardTest.test >>> FAILED: org.apache.solr.cloud.PeerSyncReplicationTest.test >>> FAILED: org.apache.solr.cloud.ReplaceNodeNoTargetTest.test >>> FAILED: org.apache.solr.cloud.TestUtilizeNode.test >>> FAILED: >>> org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testCollectionsAPI >>> FAILED: >>> org.apache.solr.cloud.api.collections.ShardSplitTest.testSplitAfterFailedSplit >>> FAILED: >>> org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple >>> FAILED: >>> org.apache.solr.cloud.autoscaling.ComputePlanActionTest.testNodeWithMultipleReplicasLost >>> FAILED: >>> org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple >>> FAILED: org.apache.solr.cloud.autoscaling.SystemLogListenerTest.test >>> FAILED: >>> org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventQueue >>> FAILED: >>> org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger >>> FAILED: >>> org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testSearchRate >>> FAILED: >>> org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testSearchRate >>> FAILED: >>> org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigReplication >>> FAILED: >>> org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.testHistory >>> FAILED: >>> org.apache.solr.rest.schema.analysis.TestManagedSynonymFilterFactory.testCanHandleDecodingAndEncodingForSynonyms >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
