Thanks for bringing this up Erick. I agree with you we should silence those frequent failures. Like you said, the side-effects of not silencing them are even worse. I'll add that these flaky tests also make releasing harder, it took me three runs last time (Lucene/Solr 7.2) for the release build to succeed because of failed tests.
Le mer. 21 févr. 2018 à 16:52, Erick Erickson <[email protected]> a écrit : > There's an elephant in the room, and it's that failing tests are being > ignored. Mind you, Solr and Lucene are progressing at a furious pace > with lots of great functionality being added. That said, we're > building up a considerable "technical debt" when it comes to testing. > > And I should say up front that major new functionality is expected to > take a while to shake out (e.g. autoscaling, streaming, V2 API etc.), > and noise from tests of new functionality is expected while things > bake. > > Below is a list of tests that have failed at least once since just > last night. This has been getting worse as time passes, the broken > window problem. Some e-mails have 10 failing tests (+/-) so unless I > go through each and every one I don't know whether something I've done > is causing a problem or not. > > I'm as guilty of letting things slide as anyone else, there's been a > long-standing issue with TestLazyCores that I work on sporadically for > instance that's _probably_ "something in the test framework".... > > Several folks have spent some time digging into test failures and > identifying at least some of the causes, kudos to them. It seems > they're voices crying out in the wilderness though. > > There is so much noise at this point that tests are becoming > irrelevant. I'm trying to work on SOLR-10809 for instance, where > there's a pretty good possibility that I'll close at least one thing > that shouldn't be closed. So I ran the full suite 10 times and > gathered all the failures. Now I have to try to separate the failures > caused by that JIRA from the ones that aren't related to it so I beast > each of the failing tests 100 times against master. If I get a failure > on master too for a particular test, I'll assume it's "not my problem" > and drive on. > > I freely acknowledge that this is poor practice. It's driven by > frustration and the desire to make progress. While it's poor practice, > it's not as bad as only looking at tests that I _think_ are related or > ignoring all tests failures I can't instantly recognize as "my fault". > > So what's our stance on this? Mark Miller had a terrific program at > one point allowing categorization of tests that failed at a glance, > but it hasn't been updated in a while. Steve Rowe is working on the > problem too. Hoss and Cassandra have both added to the efforts as > well. And I'm sure I'm leaving out others. > > Then there's the @Ignore and @BadApple annotations.... > > So, as a community, are we going to devote some energy to this? Or > shall we just start ignoring all of the frequently failing tests? > Frankly we'd be farther ahead at this point marking failing tests that > aren't getting any work with @Ignore or @BadApple and getting > compulsive about not letting any _new_ tests fail than continuing our > current path. I don't _like_ this option mind you, but it's better > than letting these accumulate forever and tests become more and more > difficult to use. As tests become more difficult to use, they're used > less and the problem gets worse. > > Note, I made no effort to separate suite .vs. individual reports here..... > > Erick > > FAILED: junit.framework.TestSuite.org > .apache.lucene.index.TestBagOfPositions > FAILED: junit.framework.TestSuite.org > .apache.lucene.index.TestIndexWriterDeleteByQuery > FAILED: junit.framework.TestSuite.org > .apache.lucene.store.TestSleepingLockWrapper > FAILED: junit.framework.TestSuite.org > .apache.solr.analytics.legacy.facet.LegacyFieldFacetCloudTest > FAILED: junit.framework.TestSuite.org > .apache.solr.analytics.legacy.facet.LegacyFieldFacetExtrasCloudTest > FAILED: junit.framework.TestSuite.org > .apache.solr.analytics.legacy.facet.LegacyQueryFacetCloudTest > FAILED: junit.framework.TestSuite.org > .apache.solr.client.solrj.TestLBHttpSolrClient > FAILED: junit.framework.TestSuite.org > .apache.solr.cloud.TestSolrCloudWithSecureImpersonation > FAILED: junit.framework.TestSuite.org > .apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest > FAILED: junit.framework.TestSuite.org > .apache.solr.core.AlternateDirectoryTest > FAILED: junit.framework.TestSuite.org.apache.solr.core.TestLazyCores > FAILED: junit.framework.TestSuite.org > .apache.solr.handler.component.DistributedFacetPivotSmallAdvancedTest > FAILED: junit.framework.TestSuite.org > .apache.solr.ltr.TestSelectiveWeightCreation > FAILED: junit.framework.TestSuite.org > .apache.solr.ltr.store.rest.TestModelManager > FAILED: junit.framework.TestSuite.org > .apache.solr.rest.schema.analysis.TestManagedSynonymFilterFactory > FAILED: junit.framework.TestSuite.org > .apache.solr.search.join.BlockJoinFacetDistribTest > FAILED: junit.framework.TestSuite.org > .apache.solr.security.TestAuthorizationFramework > FAILED: junit.framework.TestSuite.org > .apache.solr.update.processor.TestOpenNLPExtractNamedEntitiesUpdateProcessorFactory > FAILED: org.apache.lucene.index.TestStressNRT.test > FAILED: org.apache.solr.cloud.AddReplicaTest.test > FAILED: org.apache.solr.cloud.DeleteShardTest.test > FAILED: org.apache.solr.cloud.PeerSyncReplicationTest.test > FAILED: org.apache.solr.cloud.ReplaceNodeNoTargetTest.test > FAILED: org.apache.solr.cloud.TestUtilizeNode.test > FAILED: > org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testCollectionsAPI > FAILED: > org.apache.solr.cloud.api.collections.ShardSplitTest.testSplitAfterFailedSplit > FAILED: > org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple > FAILED: > org.apache.solr.cloud.autoscaling.ComputePlanActionTest.testNodeWithMultipleReplicasLost > FAILED: > org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple > FAILED: org.apache.solr.cloud.autoscaling.SystemLogListenerTest.test > FAILED: > org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventQueue > FAILED: > org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger > FAILED: > org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testSearchRate > FAILED: > org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testSearchRate > FAILED: > org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigReplication > FAILED: > org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.testHistory > FAILED: > org.apache.solr.rest.schema.analysis.TestManagedSynonymFilterFactory.testCanHandleDecodingAndEncodingForSynonyms > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
