There's an elephant in the room, and it's that failing tests are being ignored. Mind you, Solr and Lucene are progressing at a furious pace with lots of great functionality being added. That said, we're building up a considerable "technical debt" when it comes to testing.
And I should say up front that major new functionality is expected to take a while to shake out (e.g. autoscaling, streaming, V2 API etc.), and noise from tests of new functionality is expected while things bake. Below is a list of tests that have failed at least once since just last night. This has been getting worse as time passes, the broken window problem. Some e-mails have 10 failing tests (+/-) so unless I go through each and every one I don't know whether something I've done is causing a problem or not. I'm as guilty of letting things slide as anyone else, there's been a long-standing issue with TestLazyCores that I work on sporadically for instance that's _probably_ "something in the test framework".... Several folks have spent some time digging into test failures and identifying at least some of the causes, kudos to them. It seems they're voices crying out in the wilderness though. There is so much noise at this point that tests are becoming irrelevant. I'm trying to work on SOLR-10809 for instance, where there's a pretty good possibility that I'll close at least one thing that shouldn't be closed. So I ran the full suite 10 times and gathered all the failures. Now I have to try to separate the failures caused by that JIRA from the ones that aren't related to it so I beast each of the failing tests 100 times against master. If I get a failure on master too for a particular test, I'll assume it's "not my problem" and drive on. I freely acknowledge that this is poor practice. It's driven by frustration and the desire to make progress. While it's poor practice, it's not as bad as only looking at tests that I _think_ are related or ignoring all tests failures I can't instantly recognize as "my fault". So what's our stance on this? Mark Miller had a terrific program at one point allowing categorization of tests that failed at a glance, but it hasn't been updated in a while. Steve Rowe is working on the problem too. Hoss and Cassandra have both added to the efforts as well. And I'm sure I'm leaving out others. Then there's the @Ignore and @BadApple annotations.... So, as a community, are we going to devote some energy to this? Or shall we just start ignoring all of the frequently failing tests? Frankly we'd be farther ahead at this point marking failing tests that aren't getting any work with @Ignore or @BadApple and getting compulsive about not letting any _new_ tests fail than continuing our current path. I don't _like_ this option mind you, but it's better than letting these accumulate forever and tests become more and more difficult to use. As tests become more difficult to use, they're used less and the problem gets worse. Note, I made no effort to separate suite .vs. individual reports here..... Erick FAILED: junit.framework.TestSuite.org.apache.lucene.index.TestBagOfPositions FAILED: junit.framework.TestSuite.org.apache.lucene.index.TestIndexWriterDeleteByQuery FAILED: junit.framework.TestSuite.org.apache.lucene.store.TestSleepingLockWrapper FAILED: junit.framework.TestSuite.org.apache.solr.analytics.legacy.facet.LegacyFieldFacetCloudTest FAILED: junit.framework.TestSuite.org.apache.solr.analytics.legacy.facet.LegacyFieldFacetExtrasCloudTest FAILED: junit.framework.TestSuite.org.apache.solr.analytics.legacy.facet.LegacyQueryFacetCloudTest FAILED: junit.framework.TestSuite.org.apache.solr.client.solrj.TestLBHttpSolrClient FAILED: junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation FAILED: junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest FAILED: junit.framework.TestSuite.org.apache.solr.core.AlternateDirectoryTest FAILED: junit.framework.TestSuite.org.apache.solr.core.TestLazyCores FAILED: junit.framework.TestSuite.org.apache.solr.handler.component.DistributedFacetPivotSmallAdvancedTest FAILED: junit.framework.TestSuite.org.apache.solr.ltr.TestSelectiveWeightCreation FAILED: junit.framework.TestSuite.org.apache.solr.ltr.store.rest.TestModelManager FAILED: junit.framework.TestSuite.org.apache.solr.rest.schema.analysis.TestManagedSynonymFilterFactory FAILED: junit.framework.TestSuite.org.apache.solr.search.join.BlockJoinFacetDistribTest FAILED: junit.framework.TestSuite.org.apache.solr.security.TestAuthorizationFramework FAILED: junit.framework.TestSuite.org.apache.solr.update.processor.TestOpenNLPExtractNamedEntitiesUpdateProcessorFactory FAILED: org.apache.lucene.index.TestStressNRT.test FAILED: org.apache.solr.cloud.AddReplicaTest.test FAILED: org.apache.solr.cloud.DeleteShardTest.test FAILED: org.apache.solr.cloud.PeerSyncReplicationTest.test FAILED: org.apache.solr.cloud.ReplaceNodeNoTargetTest.test FAILED: org.apache.solr.cloud.TestUtilizeNode.test FAILED: org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testCollectionsAPI FAILED: org.apache.solr.cloud.api.collections.ShardSplitTest.testSplitAfterFailedSplit FAILED: org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple FAILED: org.apache.solr.cloud.autoscaling.ComputePlanActionTest.testNodeWithMultipleReplicasLost FAILED: org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple FAILED: org.apache.solr.cloud.autoscaling.SystemLogListenerTest.test FAILED: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventQueue FAILED: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger FAILED: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testSearchRate FAILED: org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testSearchRate FAILED: org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigReplication FAILED: org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.testHistory FAILED: org.apache.solr.rest.schema.analysis.TestManagedSynonymFilterFactory.testCanHandleDecodingAndEncodingForSynonyms --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
