+1, agree with Adrien, thanks for bringing this up Erick!
Il giorno mer 21 feb 2018 alle ore 17:15 Adrien Grand <[email protected]> ha scritto: > Thanks for bringing this up Erick. I agree with you we should silence > those frequent failures. Like you said, the side-effects of not silencing > them are even worse. I'll add that these flaky tests also make releasing > harder, it took me three runs last time (Lucene/Solr 7.2) for the release > build to succeed because of failed tests. > > Le mer. 21 févr. 2018 à 16:52, Erick Erickson <[email protected]> a > écrit : > >> There's an elephant in the room, and it's that failing tests are being >> ignored. Mind you, Solr and Lucene are progressing at a furious pace >> with lots of great functionality being added. That said, we're >> building up a considerable "technical debt" when it comes to testing. >> >> And I should say up front that major new functionality is expected to >> take a while to shake out (e.g. autoscaling, streaming, V2 API etc.), >> and noise from tests of new functionality is expected while things >> bake. >> >> Below is a list of tests that have failed at least once since just >> last night. This has been getting worse as time passes, the broken >> window problem. Some e-mails have 10 failing tests (+/-) so unless I >> go through each and every one I don't know whether something I've done >> is causing a problem or not. >> >> I'm as guilty of letting things slide as anyone else, there's been a >> long-standing issue with TestLazyCores that I work on sporadically for >> instance that's _probably_ "something in the test framework".... >> >> Several folks have spent some time digging into test failures and >> identifying at least some of the causes, kudos to them. It seems >> they're voices crying out in the wilderness though. >> >> There is so much noise at this point that tests are becoming >> irrelevant. I'm trying to work on SOLR-10809 for instance, where >> there's a pretty good possibility that I'll close at least one thing >> that shouldn't be closed. So I ran the full suite 10 times and >> gathered all the failures. Now I have to try to separate the failures >> caused by that JIRA from the ones that aren't related to it so I beast >> each of the failing tests 100 times against master. If I get a failure >> on master too for a particular test, I'll assume it's "not my problem" >> and drive on. >> >> I freely acknowledge that this is poor practice. It's driven by >> frustration and the desire to make progress. While it's poor practice, >> it's not as bad as only looking at tests that I _think_ are related or >> ignoring all tests failures I can't instantly recognize as "my fault". >> >> So what's our stance on this? Mark Miller had a terrific program at >> one point allowing categorization of tests that failed at a glance, >> but it hasn't been updated in a while. Steve Rowe is working on the >> problem too. Hoss and Cassandra have both added to the efforts as >> well. And I'm sure I'm leaving out others. >> >> Then there's the @Ignore and @BadApple annotations.... >> >> So, as a community, are we going to devote some energy to this? Or >> shall we just start ignoring all of the frequently failing tests? >> Frankly we'd be farther ahead at this point marking failing tests that >> aren't getting any work with @Ignore or @BadApple and getting >> compulsive about not letting any _new_ tests fail than continuing our >> current path. I don't _like_ this option mind you, but it's better >> than letting these accumulate forever and tests become more and more >> difficult to use. As tests become more difficult to use, they're used >> less and the problem gets worse. >> >> Note, I made no effort to separate suite .vs. individual reports here..... >> >> Erick >> >> FAILED: junit.framework.TestSuite.org >> .apache.lucene.index.TestBagOfPositions >> FAILED: junit.framework.TestSuite.org >> .apache.lucene.index.TestIndexWriterDeleteByQuery >> FAILED: junit.framework.TestSuite.org >> .apache.lucene.store.TestSleepingLockWrapper >> FAILED: junit.framework.TestSuite.org >> .apache.solr.analytics.legacy.facet.LegacyFieldFacetCloudTest >> FAILED: junit.framework.TestSuite.org >> .apache.solr.analytics.legacy.facet.LegacyFieldFacetExtrasCloudTest >> FAILED: junit.framework.TestSuite.org >> .apache.solr.analytics.legacy.facet.LegacyQueryFacetCloudTest >> FAILED: junit.framework.TestSuite.org >> .apache.solr.client.solrj.TestLBHttpSolrClient >> FAILED: junit.framework.TestSuite.org >> .apache.solr.cloud.TestSolrCloudWithSecureImpersonation >> FAILED: junit.framework.TestSuite.org >> .apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest >> FAILED: junit.framework.TestSuite.org >> .apache.solr.core.AlternateDirectoryTest >> FAILED: junit.framework.TestSuite.org.apache.solr.core.TestLazyCores >> FAILED: junit.framework.TestSuite.org >> .apache.solr.handler.component.DistributedFacetPivotSmallAdvancedTest >> FAILED: junit.framework.TestSuite.org >> .apache.solr.ltr.TestSelectiveWeightCreation >> FAILED: junit.framework.TestSuite.org >> .apache.solr.ltr.store.rest.TestModelManager >> FAILED: junit.framework.TestSuite.org >> .apache.solr.rest.schema.analysis.TestManagedSynonymFilterFactory >> FAILED: junit.framework.TestSuite.org >> .apache.solr.search.join.BlockJoinFacetDistribTest >> FAILED: junit.framework.TestSuite.org >> .apache.solr.security.TestAuthorizationFramework >> FAILED: junit.framework.TestSuite.org >> .apache.solr.update.processor.TestOpenNLPExtractNamedEntitiesUpdateProcessorFactory >> FAILED: org.apache.lucene.index.TestStressNRT.test >> FAILED: org.apache.solr.cloud.AddReplicaTest.test >> FAILED: org.apache.solr.cloud.DeleteShardTest.test >> FAILED: org.apache.solr.cloud.PeerSyncReplicationTest.test >> FAILED: org.apache.solr.cloud.ReplaceNodeNoTargetTest.test >> FAILED: org.apache.solr.cloud.TestUtilizeNode.test >> FAILED: >> org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testCollectionsAPI >> FAILED: >> org.apache.solr.cloud.api.collections.ShardSplitTest.testSplitAfterFailedSplit >> FAILED: >> org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple >> FAILED: >> org.apache.solr.cloud.autoscaling.ComputePlanActionTest.testNodeWithMultipleReplicasLost >> FAILED: >> org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple >> FAILED: org.apache.solr.cloud.autoscaling.SystemLogListenerTest.test >> FAILED: >> org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventQueue >> FAILED: >> org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger >> FAILED: >> org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testSearchRate >> FAILED: >> org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testSearchRate >> FAILED: >> org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigReplication >> FAILED: >> org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.testHistory >> FAILED: >> org.apache.solr.rest.schema.analysis.TestManagedSynonymFilterFactory.testCanHandleDecodingAndEncodingForSynonyms >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >>
