BadApple report
Still noisy, waiting for the reference impl to untangle. Short form: Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 136 failures Week: 1 had 185 failures Week: 2 had 210 failures Week: 3 had 112 failures Failures in Hoss' reports in every one of the last 4 rollups. There were 380 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 0.7 2080 10 CachingDirectoryFactoryTest.stressTest 0123 18.3 666134 ClusterEventProducerTest.testEvents 0123 0.3 1471 5 DistributedFacetPivotLongTailTest.test 0123 2.5 2056 17 DistributedQueryComponentCustomSortTest.test 0123 100.0 57 57 DocValuesNotIndexedTest.classMethod 0123 0.8 1467 7 HttpPartitionOnCommitTest.test 0123 0.5 1896 10 ManagedSchemaRoundRobinCloudTest.testAddFieldsRoundRobin 0123 4.5 1516 29 MultiThreadedOCPTest.test 0123 0.3 1478 7 RollingRestartTest.test 0123 50.07 4 SharedFSAutoReplicaFailoverTest.test 0123 1.0 1526 20 TestCircuitBreaker.testResponseWithCBTiming 0123 11.3 1467129 TestContainerPlugin.testApi 0123 2.6 1584 41 TestDistributedStatsComponentCardinality.test 0123 0.9 1264 11 TestHdfsCloudBackupRestore.test 0123 1.3 1477 14 TestLocalFSCloudBackupRestore.test 0123 1.5 1526 32 TestPackages.testPluginLoading 0123 1.5 1801 33 TestPullReplicaErrorHandling.testCantConnectToLeader 0123 2.3 1801 40 TestPullReplicaErrorHandling.testPullReplicaDisconnectsFromZooKeeper Full report: DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate SuppressWarnings count: last week: 4,491, this week: 4,491, delta 0 *** Files with increased @SuppressWarnings annotations: Suppress count increase in: solr/core/src/java/org/apache/solr/schema/IndexSchemaFactory.java. Was: 0, now: 1 *** Files with decreased @SuppressWarnings annotations: Suppress count decrease in: lucene/core/src/test/org/apache/lucene/util/hnsw/TestHnsw.java. Was: 1, now: 0 Processing file (History bit 3): HOSS-2020-12-21.csv Processing file (History bit 2): HOSS-2020-12-07.csv Processing file (History bit 1): HOSS-2020-11-23.csv Processing file (History bit 0): HOSS-2020-11-09.csv Number of AwaitsFix: 31 Number of BadApples: 3 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 136 failures Week: 1 had 185 failures Week: 2 had 210 failures Week: 3 had 112 failures Failures in Hoss' reports in every one of the last 4 rollups. There were 380 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 0.7 2080 10 CachingDirectoryFactoryTest.stressTest 0123 18.3 666134 ClusterEventProducerTest.testEvents 0123 0.3 1471 5 DistributedFacetPivotLongTailTest.test 0123 2.5 2056 17 DistributedQueryComponentCustomSortTest.test 0123 100.0 57 57
BadApple report
Unfortunately, the reference impl is creating quite a bit of noise in Hoss’ rollups. That said, I have a mail filter for test failures that puts the reference impl tests in a different mail folder and my sense is that the regular branch is getting an increasing number of failures. If I have the energy, I’ll try to collect some of them. Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 210 failures Week: 1 had 112 failures Week: 2 had 110 failures Week: 3 had 150 failures Failures in Hoss' reports in every one of the last 4 rollups. There were 390 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 0.5 1745 10 CachingDirectoryFactoryTest.stressTest 0123 100.0 159159 CollectionsAPIAsyncDistributedZkTest.classMethod 0123 2.9 1759 49 CollectionsAPIAsyncDistributedZkTest.testAsyncIdRaceCondition 0123 3.0 1744 50 CollectionsAPIDistributedZkTest.testDeleteNonExistentCollection 0123 3.0 1772 53 CollectionsAPIDistributedZkTest.testNoConfigSetExist 0123 100.0 209209 JsonRequestApiHeatmapFacetingTest.classMethod 0123 100.0 209209 JsonRequestApiTest.classMethod 0123 0.4 1708 7 ManagedSchemaRoundRobinCloudTest.testAddFieldsRoundRobin 0123 3.1 1737 32 MoveReplicaTest.test 0123 1.9 1366 23 TestCircuitBreaker.testResponseWithCBTiming 0123 8.5 1275 94 TestContainerPlugin.testApi 0123 2.5 1368 36 TestDistributedStatsComponentCardinality.test 0123 0.3 1069 8 TestHdfsCloudBackupRestore.test 0123 0.2 1277 9 TestLocalFSCloudBackupRestore.test 0123 1.9 1313 17 TestPackages.testPluginLoading 0123 1.7 1575 20 TestPullReplicaErrorHandling.testCantConnectToLeader 0123 1.9 1575 31 TestPullReplicaErrorHandling.testPullReplicaDisconnectsFromZooKeeper 0123 100.0 209209 UsingSolrJRefGuideExamplesTest.classMethod 0123 100.0 192192 ZkConfigFilesTest.classMethod DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate SuppressWarnings count: last week: 4,481, this week: 4,487, delta 6 *** Files with increased @SuppressWarnings annotations: Suppress count increase in: lucene/analysis/common/src/java/org/tartarus/snowball/ext/YiddishStemmer.java. Was: null, now: 1 Suppress count increase in: lucene/core/src/test/org/apache/lucene/util/hnsw/TestHnsw.java. Was: null, now: 1 Suppress count increase in: lucene/grouping/src/test/org/apache/lucene/search/grouping/TestAllGroupHeadsCollector.java. Was: null, now: 1 Suppress count increase in: lucene/grouping/src/test/org/apache/lucene/search/grouping/TestDistinctValuesCollector.java. Was: null, now: 5 Suppress count increase in: lucene/grouping/src/test/org/apache/lucene/search/grouping/TestTopGroups.java. Was: null, now: 2 Suppress count increase in: lucene/misc/src/java/org/apache/lucene/misc/index/MultiPassIndexSplitter.java. Was: null, now: 1 Suppress count increase in: lucene/misc/src/java/org/apache/lucene/misc/util/fst/ListOfOutputs.java. Was: null, now: 1 Suppress count increase in: lucene/misc/src/java/org/apache/lucene/misc/util/fst/UpToTwoPositiveIntOutputs.java. Was: null, now: 1 Suppress count increase in: lucene/replicator/src/test/org/apache/lucene/replicator/TestIndexAndTaxonomyReplicationClient.java. Was: null, now: 2 Suppress count increase in: lucene/replicator/src/test/org/apache/lucene/replicator/TestIndexAndTaxonomyRevision.java. Was: null, now: 1 Suppress count increase in:
BadApple report
Still seeing quite a bit of noise due to the reference impl. That said, we do have a reproducible error for TestRandomDVFaceting both 8x and master, see SOLR-14990. Meanwhile, here’s the report for this week. Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 112 failures Week: 1 had 110 failures Week: 2 had 150 failures Week: 3 had 174 failures Failures in Hoss' reports in every one of the last 4 rollups. There were 342 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 0.4 1656 10 CachingDirectoryFactoryTest.stressTest 0123 100.0 159159 CollectionsAPIDistributedZkTest.classMethod 0123 2.1 1709 53 CollectionsAPIDistributedZkTest.testBadActionNames 0123 2.1 1709 53 CollectionsAPIDistributedZkTest.testMissingNumShards 0123 2.1 1709 53 CollectionsAPIDistributedZkTest.testMissingRequiredParameters 0123 2.1 1709 53 CollectionsAPIDistributedZkTest.testNoConfigSetExist 0123 2.1 1709 53 CollectionsAPIDistributedZkTest.testZeroNumShards 0123 100.0 260260 ConcurrentUpdateSolrClientMultiCollectionTest.classMethod 0123 100.0 260260 JsonRequestApiHeatmapFacetingTest.classMethod 0123 100.0 260260 JsonRequestApiTest.classMethod 0123 0.6 1510 6 ManagedSchemaRoundRobinCloudTest.testAddFieldsRoundRobin 0123 2.4 1504 29 MoveReplicaTest.test 0123 0.3 1185 19 TestCircuitBreaker.testResponseWithCBTiming 0123 0.7 1024 22 TestHdfsCloudBackupRestore.test 0123 0.9 1232 25 TestLocalFSCloudBackupRestore.test 0123 0.6 1259 28 TestPackages.testPluginLoading 0123 1.3 1409 16 TestPullReplicaErrorHandling.testCantConnectToLeader 0123 2.1 1409 23 TestPullReplicaErrorHandling.testPullReplicaDisconnectsFromZooKeeper 0123 0.7 1506 10 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0123 13.1 1726246 TestSTUniformSplitPostingFormat.testCheckIntegrityReadsAllBytes 0123 28.7 1723482 TestSynonymFilterFactory.testFormat 0123 28.7 1723482 TestSynonymFilterFactory.testSynonyms 0123 28.6 1726483 TestSysoutsLimits.OverHardLimit 0123 28.6 1726483 TestSysoutsLimits.testOverSoftLimit 0123 13.1 1726246 TestUniformSplitPostingFormat.testCheckIntegrityReadsAllBytes 0123 100.0 260260 UsingSolrJRefGuideExamplesTest.classMethod 0123 100.0 260260 ZkConfigFilesTest.classMethod DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate SuppressWarnings count: last week: 4,484, this week: 4,481, delta -3 *** Files with increased @SuppressWarnings annotations: Suppress count increase in: lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/idversion/IDVersionSegmentTermsEnum.java. Was: null, now: 4 Suppress count increase in: lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/idversion/VersionBlockTreeTermsWriter.java. Was: null, now: 2 Suppress count increase in: lucene/sandbox/src/java/org/apache/lucene/sandbox/search/IndexSortSortedNumericDocValuesRangeQuery.java. Was: null, now: 1 Suppress count increase in: lucene/sandbox/src/java/org/apache/lucene/sandbox/search/PhraseWildcardQuery.java. Was: null, now: 1 Suppress count increase in: solr/contrib/clustering/src/java/org/apache/solr/handler/clustering/ClusteringComponent.java. Was: 2, now: 5 Suppress count increase in:
BadApple report
Not much change this week, still getting considerable noise from the reference impl. Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 110 failures Week: 1 had 150 failures Week: 2 had 174 failures Week: 3 had 142 failures Failures in Hoss' reports in every one of the last 4 rollups. There were 368 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 100.0 265265 AsyncCallRequestStatusResponseTest.classMethod 0123 0.6 1765 14 CachingDirectoryFactoryTest.stressTest 0123 100.0 185185 CollectionsAPIDistributedZkTest.classMethod 0123 3.3 1826 61 CollectionsAPIDistributedZkTest.testBadActionNames 0123 3.3 1826 61 CollectionsAPIDistributedZkTest.testMissingNumShards 0123 3.3 1826 61 CollectionsAPIDistributedZkTest.testMissingRequiredParameters 0123 3.3 1826 61 CollectionsAPIDistributedZkTest.testNoConfigSetExist 0123 3.3 1826 61 CollectionsAPIDistributedZkTest.testZeroNumShards 0123 100.0 300300 ConcurrentUpdateSolrClientMultiCollectionTest.classMethod 0123 100.0 300300 JsonRequestApiHeatmapFacetingTest.classMethod 0123 100.0 300300 JsonRequestApiTest.classMethod 0123 0.4 1330 20 ShardSplitTest.testSplitMixedReplicaTypesLink 0123 3.0 1043 19 TestCircuitBreaker.testResponseWithCBTiming 0123 0.9 1076 21 TestHdfsCloudBackupRestore.test 0123 0.8 1295 23 TestLocalFSCloudBackupRestore.test 0123 1.1 1327 27 TestPackages.testPluginLoading 0123 1.0 1338 11 TestPullReplicaErrorHandling.testCantConnectToLeader 0123 2.6 1338 17 TestPullReplicaErrorHandling.testPullReplicaDisconnectsFromZooKeeper 0123 14.4 1844276 TestSTUniformSplitPostingFormat.testCheckIntegrityReadsAllBytes 0123 26.6 1842530 TestSynonymFilterFactory.testFormat 0123 26.6 1842530 TestSynonymFilterFactory.testSynonyms 0123 26.6 1844531 TestSysoutsLimits.OverHardLimit 0123 26.6 1844531 TestSysoutsLimits.testOverSoftLimit 0123 14.4 1844276 TestUniformSplitPostingFormat.testCheckIntegrityReadsAllBytes 0123 100.0 300300 UsingSolrJRefGuideExamplesTest.classMethod 0123 100.0 300300 ZkConfigFilesTest.classMethod DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate SuppressWarnings count: last week: 4,484, this week: 4,484, delta 0 *** Files with increased @SuppressWarnings annotations: *** Files with decreased @SuppressWarnings annotations: Processing file (History bit 3): HOSS-2020-11-02.csv Processing file (History bit 2): HOSS-2020-10-26.csv Processing file (History bit 1): HOSS-2020-10-19.csv Processing file (History bit 0): HOSS-2020-10-12.csv Number of AwaitsFix: 31 Number of BadApples: 3 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 110 failures Week: 1 had 150 failures Week: 2 had 174 failures Week: 3 had 142 failures Failures in Hoss' reports in every one of the last 4 rollups. There were 368 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file,
BadApple report
Still working through the failures on the reference impl, so AFAIK, the tests failing large percentages of the time are on that branch. Processing file (History bit 3): HOSS-2020-10-26.csv Processing file (History bit 2): HOSS-2020-10-19.csv Processing file (History bit 1): HOSS-2020-10-12.csv Processing file (History bit 0): HOSS-2020-10-05.csv Number of AwaitsFix: 31 Number of BadApples: 3 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 150 failures Week: 1 had 174 failures Week: 2 had 142 failures Week: 3 had 153 failures Failures in Hoss' reports in every one of the last 4 rollups. There were 397 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 100.0 35 35 AssignTest.classMethod 0123 100.0 255255 AsyncCallRequestStatusResponseTest.classMethod 0123 0.8 1916 13 CachingDirectoryFactoryTest.stressTest 0123 100.0 155155 CollectionsAPIDistributedZkTest.classMethod 0123 3.8 1960 51 CollectionsAPIDistributedZkTest.testBadActionNames 0123 3.8 1960 51 CollectionsAPIDistributedZkTest.testMissingNumShards 0123 3.8 1960 51 CollectionsAPIDistributedZkTest.testMissingRequiredParameters 0123 3.8 1960 51 CollectionsAPIDistributedZkTest.testNoConfigSetExist 0123 3.8 1960 51 CollectionsAPIDistributedZkTest.testZeroNumShards 0123 100.0 205205 CollectionsAPISolrJTest.classMethod 0123 100.0 250250 ConcurrentUpdateSolrClientMultiCollectionTest.classMethod 0123 100.0 205205 DeleteNodeTest.classMethod 0123 1.8 1565 56 HttpPartitionOnCommitTest.test 0123 1.1 1546 33 HttpPartitionTest.test 0123 100.0 250250 JsonRequestApiHeatmapFacetingTest.classMethod 0123 100.0 250250 JsonRequestApiTest.classMethod 0123 2.2 1576 53 MultiThreadedOCPTest.test 0123 100.0 205205 OverseerModifyCollectionTest.classMethod 0123 1.7 988 11 TestCircuitBreaker.testResponseWithCBTiming 0123 0.7 1521 6 TestCustomStream.testDynamicLoadingCustomStream 0123 1.3 1256 25 TestHdfsCloudBackupRestore.test 0123 1.1 1509 26 TestLocalFSCloudBackupRestore.test 0123 1.4 1513 25 TestPackages.testPluginLoading 0123 13.7 1982233 TestSTUniformSplitPostingFormat.testCheckIntegrityReadsAllBytes 0123 26.0 1981451 TestSynonymFilterFactory.testFormat 0123 26.0 1981451 TestSynonymFilterFactory.testSynonyms 0123 26.1 1983452 TestSysoutsLimits.OverHardLimit 0123 26.1 1983452 TestSysoutsLimits.testOverSoftLimit 0123 0.4 1519 6 TestSystemCollAutoCreate.testAutoCreate 0123 13.7 1982233 TestUniformSplitPostingFormat.testCheckIntegrityReadsAllBytes 0123 100.0 250250 UsingSolrJRefGuideExamplesTest.classMethod 0123 100.0 250250 ZkConfigFilesTest.classMethod DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate SuppressWarnings count: last week: 4,484, this week: 4,484, delta 0 *** Files with increased @SuppressWarnings annotations: Suppress count increase in:
BadApple report
The BadApple report remains skewed as the results include the reference impl so this is mostly in case people are curious…. I expect next week to see an uptick in the number of tests that have failed each of the last 4 weeks, that’ll be when the reference-impl parts of the report kick in. We’ll see how things progress after that. There were 354 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 1.4 1662 54 HttpPartitionOnCommitTest.test 0123 4.8 1624 25 HttpPartitionWithTlogReplicasTest.test 0123 0.3 1608 4 LBSolrClientTest.testServerIteratorTimeAllowed 0123 2.7 1684 53 MultiThreadedOCPTest.test 0123 50.08 4 SharedFSAutoReplicaFailoverTest.test 0123 5.0 1350 27 TestHdfsCloudBackupRestore.test 0123 4.8 1604 29 TestLocalFSCloudBackupRestore.test 0123 0.6 1610 10 TestSolrConfigHandlerCloud.test Full results: DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate SuppressWarnings count: last week: 4,520, this week: 4,484, delta -36 *** Files with increased @SuppressWarnings annotations: Suppress count increase in: solr/core/src/java/org/apache/solr/metrics/MetricSuppliers.java. Was: 5, now: 6 Suppress count increase in: solr/core/src/test/org/apache/solr/cloud/ReplaceNodeTest.java. Was: 0, now: 1 Suppress count increase in: solr/core/src/test/org/apache/solr/cloud/TestConfigSetsAPI.java. Was: 13, now: 15 *** Files with decreased @SuppressWarnings annotations: Suppress count decrease in: solr/core/src/java/org/apache/solr/cloud/api/collections/Assign.java. Was: 5, now: 1 Suppress count decrease in: solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java. Was: 15, now: 14 Suppress count decrease in: solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java. Was: 11, now: 8 Processing file (History bit 3): HOSS-2020-10-19.csv Processing file (History bit 2): HOSS-2020-10-12.csv Processing file (History bit 1): HOSS-2020-10-05.csv Processing file (History bit 0): HOSS-2020-09-28.csv Number of AwaitsFix: 31 Number of BadApples: 3 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 174 failures Week: 1 had 142 failures Week: 2 had 153 failures Week: 3 had 51 failures Failures in Hoss' reports in every one of the last 4 rollups. There were 354 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 1.4 1662 54 HttpPartitionOnCommitTest.test 0123 4.8 1624 25 HttpPartitionWithTlogReplicasTest.test 0123 0.3 1608 4 LBSolrClientTest.testServerIteratorTimeAllowed 0123 2.7 1684 53 MultiThreadedOCPTest.test 0123 50.08 4 SharedFSAutoReplicaFailoverTest.test 0123 5.0 1350 27 TestHdfsCloudBackupRestore.test 0123 4.8 1604 29 TestLocalFSCloudBackupRestore.test 0123 0.6 1610 10 TestSolrConfigHandlerCloud.test Failures over the last 4 weeks, but not every week. Ordered most-recent
BadApple report
Mostly for historical context for a while, It includes the reference impl so the stats will be skewed from now until we integrate it all. Short form: Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 142 failures Week: 1 had 153 failures Week: 2 had 51 failures Week: 3 had 82 failures Failures in Hoss' reports in every one of the last 4 rollups. There were 301 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 4.8 1808 93 HttpPartitionOnCommitTest.test 0123 0.5 1748 11 HttpPartitionWithTlogReplicasTest.test 0123 5.2 1789 51 MultiThreadedOCPTest.test 0123 50.08 4 SharedFSAutoReplicaFailoverTest.test 0123 1.5 1829102 TestExportWriter.testExpr 0123 0.3 1435 15 TestHdfsCloudBackupRestore.test 0123 1.0 1716 9 TestInPlaceUpdatesDistrib.test 0123 0.2 1721 16 TestLocalFSCloudBackupRestore.test 0123 1.0 1731 12 TestSolrConfigHandlerCloud.test Failures over the last 4 weeks, but not every week. Ordered most-recent first: Full report: DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate SuppressWarnings count: last week: 4,528, this week: 4,520, delta -8 *** Files with increased @SuppressWarnings annotations: Suppress count increase in: solr/core/src/java/org/apache/solr/util/stats/MetricUtils.java. Was: 3, now: 5 Suppress count increase in: solr/solrj/src/java/org/apache/solr/common/util/DOMUtil.java. Was: null, now: 5 *** Files with decreased @SuppressWarnings annotations: Suppress count decrease in: solr/core/src/java/org/apache/solr/handler/admin/MetricsHandler.java. Was: 6, now: 4 Suppress count decrease in: solr/core/src/test/org/apache/solr/handler/admin/MetricsHandlerTest.java. Was: 13, now: 11 Suppress count decrease in: solr/core/src/test/org/apache/solr/util/stats/MetricUtilsTest.java. Was: 10, now: 4 Processing file (History bit 3): HOSS-2020-10-12.csv Processing file (History bit 2): HOSS-2020-10-05.csv Processing file (History bit 1): HOSS-2020-09-28.csv Processing file (History bit 0): HOSS-2020-09-21.csv Number of AwaitsFix: 31 Number of BadApples: 3 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 142 failures Week: 1 had 153 failures Week: 2 had 51 failures Week: 3 had 82 failures Failures in Hoss' reports in every one of the last 4 rollups. There were 301 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 4.8 1808 93 HttpPartitionOnCommitTest.test 0123 0.5 1748 11 HttpPartitionWithTlogReplicasTest.test 0123 5.2 1789 51 MultiThreadedOCPTest.test 0123 50.08 4 SharedFSAutoReplicaFailoverTest.test 0123 1.5 1829102 TestExportWriter.testExpr 0123 0.3 1435 15 TestHdfsCloudBackupRestore.test 0123 1.0 1716 9 TestInPlaceUpdatesDistrib.test 0123 0.2 1721 16 TestLocalFSCloudBackupRestore.test 0123 1.0 1731 12 TestSolrConfigHandlerCloud.test
RE: BadApple report
Hi Erick, The teste-only jobs @ ASF and Policeman Jenkins jobs of master branch were all converted to Gradle. It should have no effect on the Hossman Badapples analysis, but maybe have an extra look next week to find outlyers. The statistics about failed jobs in the XML output should be the same. Uwe - Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Erick Erickson > Sent: Monday, August 24, 2020 3:59 PM > To: dev@lucene.apache.org > Subject: BadApple report > > We have some pretty frequent failures, see: > > http://fucit.org/solr-jenkins-reports/failure-report.html > > I’m pretty sure LBSolrClientTest has been addressed. I’m looking at what > commit caused TestConfigOverlay to start failing… > > This can be a little hard to interpret since it includes tests that have been > fixed > over the last week, not to mention that many of them are intermittent. > > The raw count of SupressAnnotations hasn’t changed, one was removed and > one added. > > Raw fail count by week totals, most recent week first (corresponds to bits): > Week: 0 had 119 failures > Week: 1 had 113 failures > Week: 2 had 100 failures > Week: 3 had 82 failures > > > Failures in Hoss' reports in every one of the last 4 rollups. > > There were 257 unannotated tests that failed in Hoss' rollups. Ordered by the > date I downloaded the rollup file, newest->oldest. See above for the dates the > files were collected > These tests were NOT BadApple'd or AwaitsFix'd > > Failures in the last 4 reports.. >Report Pct runsfails test > 0123 3.2 1719 86 CloudExitableDirectoryReaderTest.test > 0123 1.8 8552297 > CloudExitableDirectoryReaderTest.testCreepThenBite > 0123 1.9 1700 41 > CloudExitableDirectoryReaderTest.testWhitebox > 0123 9.8 1687125 HttpPartitionOnCommitTest.test > 0123 0.6 1571 19 HttpPartitionTest.test > 0123 3.5 1565 25 HttpPartitionWithTlogReplicasTest.test > 0123 0.3 1604 54 MultiThreadedOCPTest.test > 0123 2.0 825 8 SearchRateTriggerTest.testWaitForElapsed > 0123 0.3 1556 4 ShardSplitTest.testSplitShardWithRule > 0123 3.2 839 16 > TestCircuitBreaker.testResponseWithCBTiming > 0123 6.2 1824100 TestContainerPlugin.testApiFromPackage > 0123 2.3 1677 42 TestDistributedGrouping.test > 0123 3.4 1590 88 TestExportWriter.testExpr > 0123 6.8 1302 96 TestHdfsCloudBackupRestore.test > 0123 6.8 1646128 TestLocalFSCloudBackupRestore.test > 0123 0.6 1591 21 TestPackages.testPluginLoading > 0123 0.6 1550 9 > TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast > 0123 1.7 1538 9 TestReplicaProperties.test > 0123 0.3 1524 5 > TestSolrCloudWithDelegationTokens.testDelegationTokenRenew > 0123 0.6 1534 10 TestSolrConfigHandlerCloud.test > > > > Full output: - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
BadApple report
We have some pretty frequent failures, see: http://fucit.org/solr-jenkins-reports/failure-report.html I’m pretty sure LBSolrClientTest has been addressed. I’m looking at what commit caused TestConfigOverlay to start failing… This can be a little hard to interpret since it includes tests that have been fixed over the last week, not to mention that many of them are intermittent. The raw count of SupressAnnotations hasn’t changed, one was removed and one added. Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 119 failures Week: 1 had 113 failures Week: 2 had 100 failures Week: 3 had 82 failures Failures in Hoss' reports in every one of the last 4 rollups. There were 257 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 3.2 1719 86 CloudExitableDirectoryReaderTest.test 0123 1.8 8552297 CloudExitableDirectoryReaderTest.testCreepThenBite 0123 1.9 1700 41 CloudExitableDirectoryReaderTest.testWhitebox 0123 9.8 1687125 HttpPartitionOnCommitTest.test 0123 0.6 1571 19 HttpPartitionTest.test 0123 3.5 1565 25 HttpPartitionWithTlogReplicasTest.test 0123 0.3 1604 54 MultiThreadedOCPTest.test 0123 2.0 825 8 SearchRateTriggerTest.testWaitForElapsed 0123 0.3 1556 4 ShardSplitTest.testSplitShardWithRule 0123 3.2 839 16 TestCircuitBreaker.testResponseWithCBTiming 0123 6.2 1824100 TestContainerPlugin.testApiFromPackage 0123 2.3 1677 42 TestDistributedGrouping.test 0123 3.4 1590 88 TestExportWriter.testExpr 0123 6.8 1302 96 TestHdfsCloudBackupRestore.test 0123 6.8 1646128 TestLocalFSCloudBackupRestore.test 0123 0.6 1591 21 TestPackages.testPluginLoading 0123 0.6 1550 9 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0123 1.7 1538 9 TestReplicaProperties.test 0123 0.3 1524 5 TestSolrCloudWithDelegationTokens.testDelegationTokenRenew 0123 0.6 1534 10 TestSolrConfigHandlerCloud.test Full output: DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate SuppressWarnings count: last week: 4,818, this week: 4,818, delta 0 *** Files with increased @SuppressWarnings annotations: Suppress count increase in: solr/core/src/test/org/apache/solr/util/TestCircuitBreaker.java. Was: 0, now: 1 *** Files with decreased @SuppressWarnings annotations: Suppress count decrease in: solr/core/src/test/org/apache/solr/cloud/api/collections/SimpleCollectionCreateDeleteTest.java. Was: 1, now: 0 Processing file (History bit 3): HOSS-2020-08-24.csv Processing file (History bit 2): HOSS-2020-08-17.csv Processing file (History bit 1): HOSS-2020-08-10.csv Processing file (History bit 0): HOSS-2020-08-03.csv Number of AwaitsFix: 33 Number of BadApples: 4 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 119 failures Week: 1 had 113 failures Week: 2 had 100 failures Week: 3 had 82 failures Failures in Hoss' reports in every one of the last 4 rollups. There were 257 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See
BadApple report
Failures in Hoss' reports for the last 4 rollups. There were 242 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 6.4 1757 94 CloudExitableDirectoryReaderTest.test 0123 5.3 8740325 CloudExitableDirectoryReaderTest.testCreepThenBite 0123 2.6 1734 42 CloudExitableDirectoryReaderTest.testWhitebox 0123 9.5 1688107 HttpPartitionOnCommitTest.test 0123 2.7 1604 18 HttpPartitionTest.test 0123 1.8 1580 14 HttpPartitionWithTlogReplicasTest.test 0123 0.3 1567 10 LeaderFailoverAfterPartitionTest.test 0123 3.6 1639 57 MultiThreadedOCPTest.test 0123 0.3 1564 5 ReplaceNodeTest.test 0123 0.3 1584 4 ShardSplitTest.testSplitShardWithRule 0123 93.3 46 43 SharedFSAutoReplicaFailoverTest.test 0123 2.3 837 18 TestCircuitBreaker.testBuildingMemoryPressure 0123 0.9 837 12 TestCircuitBreaker.testResponseWithCBTiming 0123 3.6 1853101 TestContainerPlugin.testApiFromPackage 0123 2.8 1683 37 TestDistributedGrouping.test 0123 4.2 1629 89 TestExportWriter.testExpr 0123 11.7 1326 87 TestHdfsCloudBackupRestore.test 0123 9.3 1672121 TestLocalFSCloudBackupRestore.test 0123 1.2 1623 25 TestPackages.testPluginLoading 0123 0.3 1586 9 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0123 8.3 1629 82 TestReRankQParserPlugin.testMinExactCount 0123 0.3 1556 4 TestReplicaProperties.test 0123 0.3 1557 5 TestSolrCloudWithDelegationTokens.testDelegationTokenRenew 0123 1.5 1564 10 TestSolrConfigHandlerCloud.test Full report attached: DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate SuppressWarnings count: last week: 4,819, this week: 4,818, delta -1 *** Files with increased @SuppressWarnings annotations: Suppress count increase in: solr/solrj/src/java/org/apache/solr/common/LazySolrCluster.java. Was: null, now: 1 *** Files with decreased @SuppressWarnings annotations: Suppress count decrease in: solr/core/src/test/org/apache/solr/search/facet/TestCloudJSONFacetJoinDomain.java. Was: 7, now: 6 Processing file (History bit 3): HOSS-2020-08-17.csv Processing file (History bit 2): HOSS-2020-08-10.csv Processing file (History bit 1): HOSS-2020-08-03.csv Processing file (History bit 0): HOSS-2020-07-27.csv Number of AwaitsFix: 33 Number of BadApples: 4 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 113 failures Week: 1 had 100 failures Week: 2 had 82 failures Week: 3 had 94 failures Failures in Hoss' reports for the last 4 rollups. There were 242 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 6.4 1757 94 CloudExitableDirectoryReaderTest.test 0123 5.3 8740325 CloudExitableDirectoryReaderTest.testCreepThenBite 0123 2.6 1734 42
Re: BadApple report, but please read the first bit
Thanks Kevin; clearly I missed the link to that which I can now see at fucit. I was worried I may have worked on something that could have perturbed this recent issue but no -- I don't think so. ~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Wed, Aug 12, 2020 at 9:08 AM Kevin Risden wrote: > > http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test > > David for that specific test you asked the failures are recent with as far > as I know no change to HDFS stuff. Starting June/July failing regularly. > > Kevin Risden > > > > On Wed, Aug 12, 2020 at 9:03 AM Erick Erickson > wrote: > >> I have the weekly rollups (with a few gaps) going back to about April >> 2018, but nothing’s been done to try to make them generally available. Each >> BadApple report has rates for the last 4 weeks in the attached file, just >> below "Failures over the last 4 weeks, but not every week. Ordered >> most-recent first:” >> >> >> >> > On Aug 12, 2020, at 2:06 AM, David Smiley wrote: >> > >> > Do we have any long term (aka "longitudinal") pass/fail rates for tests? >> > >> > SharedFSAutoReplicaFailoverTest in particular is kinda-sorta tied to >> HDFS, and that's going away to a plug-in for 9.0. The shared file system >> notion isn't well supported in SolrCloud, I think. >> > >> > ~ David Smiley >> > Apache Lucene/Solr Search Developer >> > http://www.linkedin.com/in/davidwsmiley >> > >> > >> > On Mon, Aug 3, 2020 at 7:26 AM Erick Erickson >> wrote: >> > There are several tests that are causing a lot of noise: >> > >> > SharedFSAutoReplicaFailoverTest is failing 90%+ of the time. >> > TestBulkSchemaConcurrent 31% >> > StressHdfsTest 16% >> > SchemaApiFailureTest 13.88% >> > >> > I encourage people to look at: >> http://fucit.org/solr-jenkins-reports/failure-report.html and see if >> anything looks like it is affected by recent work. TestBulkSchemaConcurrent >> has been failing off and on for a long time, but the failure rate picked up >> dramatically in the last couple of weeks. Ditto SchemaApiFailureTest. >> > >> > Do we even care about Hdfs? Are we deprecating it or not? >> > >> > Holding relatively steady otherwise: >> > >> > Raw fail count by week totals, most recent week first (corresponds to >> bits): >> > Week: 0 had 82 failures >> > Week: 1 had 94 failures >> > Week: 2 had 502 failures >> > Week: 3 had 19 failures >> > >> > >> > Failures in Hoss' reports for the last 4 rollups. >> > >> > There were 562 unannotated tests that failed in Hoss' rollups. Ordered >> by the date I downloaded the rollup file, newest->oldest. See above for the >> dates the files were collected >> > These tests were NOT BadApple'd or AwaitsFix'd >> > >> > Failures in the last 4 reports.. >> >Report Pct runsfails test >> > 0123 0.3 1271 8 RollingRestartTest.test >> > 0123 93.3 41 36 >> SharedFSAutoReplicaFailoverTest.test >> > 0123 3.5 627 16 >> TestCircuitBreaker.testBuildingMemoryPressure >> > 0123 1.0 627 8 >> TestCircuitBreaker.testResponseWithCBTiming >> > 0123 5.8 1483 79 >> TestContainerPlugin.testApiFromPackage >> > 0123 2.3 1335 23 TestDistributedGrouping.test >> > >> > >> > >> > Full report: >> > >> > - >> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> > For additional commands, e-mail: dev-h...@lucene.apache.org >> >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> >>
Re: BadApple report, but please read the first bit
Didn’t think at first (only one cup of coffee). Here’s the Emails that test appears in, the formatting is poor… After that is the raw data from Hoss’ rollups that might be easier to ingest. I have 1.3G of this kind of historical data, I’ve had vague thoughts about putting it someplace accessible to others but haven’t done anything with it. I suppose, wrapped around this, is the entire question of how much value it’ll have depending on what happens with Mark’s reference impl... “Suite” fails are things like object tracker failures. e-mail-2018-03-26.txt:SharedFSAutoReplicaFailoverTest.java e-mail-2018-04-02.txt:SharedFSAutoReplicaFailoverTest.java e-mail-2018-04-09.txt:SharedFSAutoReplicaFailoverTest.java e-mail-2018-04-16.txt:SharedFSAutoReplicaFailoverTest.java e-mail-2018-04-30.txt:SharedFSAutoReplicaFailoverTest.java e-mail-2018-05-21.txt:SharedFSAutoReplicaFailoverTest.java e-mail-2018-06-11.txt: SharedFSAutoReplicaFailoverTest.test e-mail-2018-06-11.txt:3 100.02 2 SharedFSAutoReplicaFailoverTest(suite) e-mail-2018-06-11.txt:SharedFSAutoReplicaFailoverTest.test e-mail-2018-06-18.txt: 0100.02 2 SharedFSAutoReplicaFailoverTest(suite) e-mail-2018-06-25.txt: 0174.1 29 22 SharedFSAutoReplicaFailoverTest(suite) e-mail-2018-06-25.txt: 0 5.9 34 2 SharedFSAutoReplicaFailoverTest.test e-mail-2018-07-02.txt: 012 74.1 56 42 SharedFSAutoReplicaFailoverTest(suite) e-mail-2018-07-02.txt: 01 5.1 73 4 SharedFSAutoReplicaFailoverTest.test e-mail-2018-07-09.txt: 0123 74.1 83 62 SharedFSAutoReplicaFailoverTest(suite) e-mail-2018-07-09.txt: 0122.3 117 5 SharedFSAutoReplicaFailoverTest.test e-mail-2018-07-16.txt: 0123 74.1 108 80 SharedFSAutoReplicaFailoverTest(suite) e-mail-2018-07-16.txt: 0123 17.6 151 11 SharedFSAutoReplicaFailoverTest.test e-mail-2018-07-23.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite e-mail-2018-07-23.txt:SharedFSAutoReplicaFailoverTest.test e-mail-2018-07-30.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite e-mail-2018-07-30.txt:SharedFSAutoReplicaFailoverTest.test e-mail-2018-08-06.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite e-mail-2018-08-06.txt:SharedFSAutoReplicaFailoverTest.test e-mail-2018-08-14.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite e-mail-2018-08-14.txt:SharedFSAutoReplicaFailoverTest.test e-mail-2018-08-20.txt: SharedFSAutoReplicaFailoverTest.test e-mail-2018-08-20.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite e-mail-2018-08-20.txt:SharedFSAutoReplicaFailoverTest.test e-mail-2018-08-27.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite e-mail-2018-09-03.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite e-mail-2018-09-10.txt: 0 20.05 1 SharedFSAutoReplicaFailoverTest.test e-mail-2018-09-10.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite e-mail-2018-09-18.txt: 0133.38 2 SharedFSAutoReplicaFailoverTest.test e-mail-2018-09-18.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite e-mail-2018-10-08.txt:3 33.33 1 SharedFSAutoReplicaFailoverTest.test e-mail-2018-10-08.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite e-mail-2018-12-24.txt: 0 3 33.36 2 SharedFSAutoReplicaFailoverTest.test e-mail-2018-12-24.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite e-mail-2019-01-08.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite e-mail-2019-01-15.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite e-mail-2019-02-12.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite e-mail-2019-02-18.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite e-mail-2019-03-04.txt: 133.33 1 SharedFSAutoReplicaFailoverTest.test e-mail-2019-03-04.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite e-mail-2019-03-11.txt: 2 33.33 1 SharedFSAutoReplicaFailoverTest.test e-mail-2019-03-11.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite e-mail-2019-03-18.txt:3 33.33 1 SharedFSAutoReplicaFailoverTest.test e-mail-2019-03-18.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite e-mail-2019-03-25.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite e-mail-2019-04-01.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite e-mail-2019-04-08.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
Re: BadApple report, but please read the first bit
http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test David for that specific test you asked the failures are recent with as far as I know no change to HDFS stuff. Starting June/July failing regularly. Kevin Risden On Wed, Aug 12, 2020 at 9:03 AM Erick Erickson wrote: > I have the weekly rollups (with a few gaps) going back to about April > 2018, but nothing’s been done to try to make them generally available. Each > BadApple report has rates for the last 4 weeks in the attached file, just > below "Failures over the last 4 weeks, but not every week. Ordered > most-recent first:” > > > > > On Aug 12, 2020, at 2:06 AM, David Smiley wrote: > > > > Do we have any long term (aka "longitudinal") pass/fail rates for tests? > > > > SharedFSAutoReplicaFailoverTest in particular is kinda-sorta tied to > HDFS, and that's going away to a plug-in for 9.0. The shared file system > notion isn't well supported in SolrCloud, I think. > > > > ~ David Smiley > > Apache Lucene/Solr Search Developer > > http://www.linkedin.com/in/davidwsmiley > > > > > > On Mon, Aug 3, 2020 at 7:26 AM Erick Erickson > wrote: > > There are several tests that are causing a lot of noise: > > > > SharedFSAutoReplicaFailoverTest is failing 90%+ of the time. > > TestBulkSchemaConcurrent 31% > > StressHdfsTest 16% > > SchemaApiFailureTest 13.88% > > > > I encourage people to look at: > http://fucit.org/solr-jenkins-reports/failure-report.html and see if > anything looks like it is affected by recent work. TestBulkSchemaConcurrent > has been failing off and on for a long time, but the failure rate picked up > dramatically in the last couple of weeks. Ditto SchemaApiFailureTest. > > > > Do we even care about Hdfs? Are we deprecating it or not? > > > > Holding relatively steady otherwise: > > > > Raw fail count by week totals, most recent week first (corresponds to > bits): > > Week: 0 had 82 failures > > Week: 1 had 94 failures > > Week: 2 had 502 failures > > Week: 3 had 19 failures > > > > > > Failures in Hoss' reports for the last 4 rollups. > > > > There were 562 unannotated tests that failed in Hoss' rollups. Ordered > by the date I downloaded the rollup file, newest->oldest. See above for the > dates the files were collected > > These tests were NOT BadApple'd or AwaitsFix'd > > > > Failures in the last 4 reports.. > >Report Pct runsfails test > > 0123 0.3 1271 8 RollingRestartTest.test > > 0123 93.3 41 36 SharedFSAutoReplicaFailoverTest.test > > 0123 3.5 627 16 > TestCircuitBreaker.testBuildingMemoryPressure > > 0123 1.0 627 8 > TestCircuitBreaker.testResponseWithCBTiming > > 0123 5.8 1483 79 > TestContainerPlugin.testApiFromPackage > > 0123 2.3 1335 23 TestDistributedGrouping.test > > > > > > > > Full report: > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
Re: BadApple report, but please read the first bit
I have the weekly rollups (with a few gaps) going back to about April 2018, but nothing’s been done to try to make them generally available. Each BadApple report has rates for the last 4 weeks in the attached file, just below "Failures over the last 4 weeks, but not every week. Ordered most-recent first:” > On Aug 12, 2020, at 2:06 AM, David Smiley wrote: > > Do we have any long term (aka "longitudinal") pass/fail rates for tests? > > SharedFSAutoReplicaFailoverTest in particular is kinda-sorta tied to HDFS, > and that's going away to a plug-in for 9.0. The shared file system notion > isn't well supported in SolrCloud, I think. > > ~ David Smiley > Apache Lucene/Solr Search Developer > http://www.linkedin.com/in/davidwsmiley > > > On Mon, Aug 3, 2020 at 7:26 AM Erick Erickson wrote: > There are several tests that are causing a lot of noise: > > SharedFSAutoReplicaFailoverTest is failing 90%+ of the time. > TestBulkSchemaConcurrent 31% > StressHdfsTest 16% > SchemaApiFailureTest 13.88% > > I encourage people to look at: > http://fucit.org/solr-jenkins-reports/failure-report.html and see if anything > looks like it is affected by recent work. TestBulkSchemaConcurrent has been > failing off and on for a long time, but the failure rate picked up > dramatically in the last couple of weeks. Ditto SchemaApiFailureTest. > > Do we even care about Hdfs? Are we deprecating it or not? > > Holding relatively steady otherwise: > > Raw fail count by week totals, most recent week first (corresponds to bits): > Week: 0 had 82 failures > Week: 1 had 94 failures > Week: 2 had 502 failures > Week: 3 had 19 failures > > > Failures in Hoss' reports for the last 4 rollups. > > There were 562 unannotated tests that failed in Hoss' rollups. Ordered by the > date I downloaded the rollup file, newest->oldest. See above for the dates > the files were collected > These tests were NOT BadApple'd or AwaitsFix'd > > Failures in the last 4 reports.. >Report Pct runsfails test > 0123 0.3 1271 8 RollingRestartTest.test > 0123 93.3 41 36 SharedFSAutoReplicaFailoverTest.test > 0123 3.5 627 16 > TestCircuitBreaker.testBuildingMemoryPressure > 0123 1.0 627 8 > TestCircuitBreaker.testResponseWithCBTiming > 0123 5.8 1483 79 TestContainerPlugin.testApiFromPackage > 0123 2.3 1335 23 TestDistributedGrouping.test > > > > Full report: > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: BadApple report, but please read the first bit
Do we have any long term (aka "longitudinal") pass/fail rates for tests? SharedFSAutoReplicaFailoverTest in particular is kinda-sorta tied to HDFS, and that's going away to a plug-in for 9.0. The shared file system notion isn't well supported in SolrCloud, I think. ~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Mon, Aug 3, 2020 at 7:26 AM Erick Erickson wrote: > There are several tests that are causing a lot of noise: > > SharedFSAutoReplicaFailoverTest is failing 90%+ of the time. > TestBulkSchemaConcurrent 31% > StressHdfsTest 16% > SchemaApiFailureTest 13.88% > > I encourage people to look at: > http://fucit.org/solr-jenkins-reports/failure-report.html and see if > anything looks like it is affected by recent work. TestBulkSchemaConcurrent > has been failing off and on for a long time, but the failure rate picked up > dramatically in the last couple of weeks. Ditto SchemaApiFailureTest. > > Do we even care about Hdfs? Are we deprecating it or not? > > Holding relatively steady otherwise: > > Raw fail count by week totals, most recent week first (corresponds to > bits): > Week: 0 had 82 failures > Week: 1 had 94 failures > Week: 2 had 502 failures > Week: 3 had 19 failures > > > Failures in Hoss' reports for the last 4 rollups. > > There were 562 unannotated tests that failed in Hoss' rollups. Ordered by > the date I downloaded the rollup file, newest->oldest. See above for the > dates the files were collected > These tests were NOT BadApple'd or AwaitsFix'd > > Failures in the last 4 reports.. >Report Pct runsfails test > 0123 0.3 1271 8 RollingRestartTest.test > 0123 93.3 41 36 SharedFSAutoReplicaFailoverTest.test > 0123 3.5 627 16 > TestCircuitBreaker.testBuildingMemoryPressure > 0123 1.0 627 8 > TestCircuitBreaker.testResponseWithCBTiming > 0123 5.8 1483 79 TestContainerPlugin.testApiFromPackage > 0123 2.3 1335 23 TestDistributedGrouping.test > > > > Full report: > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Badapple report
Merged (thanks Mike D!). Atri On Tue, Aug 11, 2020 at 5:32 PM Erick Erickson wrote: > > Great, thanks! Let me know when you push it, I can beast the test again. > > > On Aug 11, 2020, at 3:48 AM, Atri Sharma wrote: > > > > I investigated testRequestRateLimiters and hardened the tests up: > > > > https://github.com/apache/lucene-solr/pull/1736 > > > > This will stop testConcurrentRequests from failing and should > > hopefully stop testSlotBorrowing as well. If testSlotBorrowing > > continues to fail, I will have to rethink the test. > > > > On Mon, Aug 10, 2020 at 8:17 PM Erick Erickson > > wrote: > >> > >> We’re backsliding some. I encourage people to look at: > >> http://fucit.org/solr-jenkins-reports/failure-report.html, we have a > >> number of ill-behaved tests, particularly TestRequestRateLimiter, > >> TestBulkSchemaConcurrent, TestConfig, SchemaApiFailureTest and > >> TestIndexingSequenceNumbers… > >> > >> > >> Raw fail count by week totals, most recent week first (corresponds to > >> bits): > >> Week: 0 had 100 failures > >> Week: 1 had 82 failures > >> Week: 2 had 94 failures > >> Week: 3 had 502 failures > >> > >> > >> Failures in Hoss' reports for the last 4 rollups. > >> > >> There were 585 unannotated tests that failed in Hoss' rollups. Ordered by > >> the date I downloaded the rollup file, newest->oldest. See above for the > >> dates the files were collected > >> These tests were NOT BadApple'd or AwaitsFix'd > >> > >> Failures in the last 4 reports.. > >> Report Pct runsfails test > >> 0123 4.4 1583 37 BasicDistributedZkTest.test > >> 0123 4.3 1727 77 CloudExitableDirectoryReaderTest.test > >> 0123 2.5 8598248 > >> CloudExitableDirectoryReaderTest.testCreepThenBite > >> 0123 1.9 1712 36 > >> CloudExitableDirectoryReaderTest.testWhitebox > >> 0123 0.5 1587 11 > >> DocValuesNotIndexedTest.testGroupingDVOnlySortLast > >> 0123 2.2 1679 82 HttpPartitionOnCommitTest.test > >> 0123 0.5 1592 16 HttpPartitionTest.test > >> 0123 1.0 1578 9 HttpPartitionWithTlogReplicasTest.test > >> 0123 1.3 1569 13 LeaderFailoverAfterPartitionTest.test > >> 0123 7.4 1643 59 MultiThreadedOCPTest.test > >> 0123 0.3 1567 8 ReplaceNodeTest.test > >> 0123 0.2 1588 6 ShardSplitTest.testSplitShardWithRule > >> 0123 100.0 38 33 SharedFSAutoReplicaFailoverTest.test > >> 0123 2.1 818 19 > >> TestCircuitBreaker.testBuildingMemoryPressure > >> 0123 2.6 818 13 > >> TestCircuitBreaker.testResponseWithCBTiming > >> 0123 6.2 1848104 TestContainerPlugin.testApiFromPackage > >> 0123 2.5 1662 33 TestDistributedGrouping.test > >> 0123 0.4 1448 6 TestDynamicLoading.testDynamicLoading > >> 0123 6.4 1614 74 TestExportWriter.testExpr > >> 0123 8.6 1356 70 TestHdfsCloudBackupRestore.test > >> 0123 9.1 1697136 TestLocalFSCloudBackupRestore.test > >> 0123 0.5 1607 26 TestPackages.testPluginLoading > >> 0123 0.7 1596 15 > >> TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast > >> 0123 1.5 1610 59 > >> TestReRankQParserPlugin.testMinExactCount > >> 0123 0.3 1552 4 TestReplicaProperties.test > >> 0123 0.3 1556 5 > >> TestSolrCloudWithDelegationTokens.testDelegationTokenRenew > >> 0123 0.3 1565 9 TestSolrConfigHandlerCloud.test > >> > >> > >> > >> - > >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > >> For additional commands, e-mail: dev-h...@lucene.apache.org > > > > -- > > Regards, > > > > Atri > > Apache Concerted > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > -- Regards, Atri Apache Concerted - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Badapple report
Great, thanks! Let me know when you push it, I can beast the test again. > On Aug 11, 2020, at 3:48 AM, Atri Sharma wrote: > > I investigated testRequestRateLimiters and hardened the tests up: > > https://github.com/apache/lucene-solr/pull/1736 > > This will stop testConcurrentRequests from failing and should > hopefully stop testSlotBorrowing as well. If testSlotBorrowing > continues to fail, I will have to rethink the test. > > On Mon, Aug 10, 2020 at 8:17 PM Erick Erickson > wrote: >> >> We’re backsliding some. I encourage people to look at: >> http://fucit.org/solr-jenkins-reports/failure-report.html, we have a number >> of ill-behaved tests, particularly TestRequestRateLimiter, >> TestBulkSchemaConcurrent, TestConfig, SchemaApiFailureTest and >> TestIndexingSequenceNumbers… >> >> >> Raw fail count by week totals, most recent week first (corresponds to bits): >> Week: 0 had 100 failures >> Week: 1 had 82 failures >> Week: 2 had 94 failures >> Week: 3 had 502 failures >> >> >> Failures in Hoss' reports for the last 4 rollups. >> >> There were 585 unannotated tests that failed in Hoss' rollups. Ordered by >> the date I downloaded the rollup file, newest->oldest. See above for the >> dates the files were collected >> These tests were NOT BadApple'd or AwaitsFix'd >> >> Failures in the last 4 reports.. >> Report Pct runsfails test >> 0123 4.4 1583 37 BasicDistributedZkTest.test >> 0123 4.3 1727 77 CloudExitableDirectoryReaderTest.test >> 0123 2.5 8598248 >> CloudExitableDirectoryReaderTest.testCreepThenBite >> 0123 1.9 1712 36 >> CloudExitableDirectoryReaderTest.testWhitebox >> 0123 0.5 1587 11 >> DocValuesNotIndexedTest.testGroupingDVOnlySortLast >> 0123 2.2 1679 82 HttpPartitionOnCommitTest.test >> 0123 0.5 1592 16 HttpPartitionTest.test >> 0123 1.0 1578 9 HttpPartitionWithTlogReplicasTest.test >> 0123 1.3 1569 13 LeaderFailoverAfterPartitionTest.test >> 0123 7.4 1643 59 MultiThreadedOCPTest.test >> 0123 0.3 1567 8 ReplaceNodeTest.test >> 0123 0.2 1588 6 ShardSplitTest.testSplitShardWithRule >> 0123 100.0 38 33 SharedFSAutoReplicaFailoverTest.test >> 0123 2.1 818 19 >> TestCircuitBreaker.testBuildingMemoryPressure >> 0123 2.6 818 13 >> TestCircuitBreaker.testResponseWithCBTiming >> 0123 6.2 1848104 TestContainerPlugin.testApiFromPackage >> 0123 2.5 1662 33 TestDistributedGrouping.test >> 0123 0.4 1448 6 TestDynamicLoading.testDynamicLoading >> 0123 6.4 1614 74 TestExportWriter.testExpr >> 0123 8.6 1356 70 TestHdfsCloudBackupRestore.test >> 0123 9.1 1697136 TestLocalFSCloudBackupRestore.test >> 0123 0.5 1607 26 TestPackages.testPluginLoading >> 0123 0.7 1596 15 >> TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast >> 0123 1.5 1610 59 TestReRankQParserPlugin.testMinExactCount >> 0123 0.3 1552 4 TestReplicaProperties.test >> 0123 0.3 1556 5 >> TestSolrCloudWithDelegationTokens.testDelegationTokenRenew >> 0123 0.3 1565 9 TestSolrConfigHandlerCloud.test >> >> >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org > > -- > Regards, > > Atri > Apache Concerted > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Badapple report
I investigated testRequestRateLimiters and hardened the tests up: https://github.com/apache/lucene-solr/pull/1736 This will stop testConcurrentRequests from failing and should hopefully stop testSlotBorrowing as well. If testSlotBorrowing continues to fail, I will have to rethink the test. On Mon, Aug 10, 2020 at 8:17 PM Erick Erickson wrote: > > We’re backsliding some. I encourage people to look at: > http://fucit.org/solr-jenkins-reports/failure-report.html, we have a number > of ill-behaved tests, particularly TestRequestRateLimiter, > TestBulkSchemaConcurrent, TestConfig, SchemaApiFailureTest and > TestIndexingSequenceNumbers… > > > Raw fail count by week totals, most recent week first (corresponds to bits): > Week: 0 had 100 failures > Week: 1 had 82 failures > Week: 2 had 94 failures > Week: 3 had 502 failures > > > Failures in Hoss' reports for the last 4 rollups. > > There were 585 unannotated tests that failed in Hoss' rollups. Ordered by the > date I downloaded the rollup file, newest->oldest. See above for the dates > the files were collected > These tests were NOT BadApple'd or AwaitsFix'd > > Failures in the last 4 reports.. >Report Pct runsfails test > 0123 4.4 1583 37 BasicDistributedZkTest.test > 0123 4.3 1727 77 CloudExitableDirectoryReaderTest.test > 0123 2.5 8598248 > CloudExitableDirectoryReaderTest.testCreepThenBite > 0123 1.9 1712 36 > CloudExitableDirectoryReaderTest.testWhitebox > 0123 0.5 1587 11 > DocValuesNotIndexedTest.testGroupingDVOnlySortLast > 0123 2.2 1679 82 HttpPartitionOnCommitTest.test > 0123 0.5 1592 16 HttpPartitionTest.test > 0123 1.0 1578 9 HttpPartitionWithTlogReplicasTest.test > 0123 1.3 1569 13 LeaderFailoverAfterPartitionTest.test > 0123 7.4 1643 59 MultiThreadedOCPTest.test > 0123 0.3 1567 8 ReplaceNodeTest.test > 0123 0.2 1588 6 ShardSplitTest.testSplitShardWithRule > 0123 100.0 38 33 SharedFSAutoReplicaFailoverTest.test > 0123 2.1 818 19 > TestCircuitBreaker.testBuildingMemoryPressure > 0123 2.6 818 13 > TestCircuitBreaker.testResponseWithCBTiming > 0123 6.2 1848104 TestContainerPlugin.testApiFromPackage > 0123 2.5 1662 33 TestDistributedGrouping.test > 0123 0.4 1448 6 TestDynamicLoading.testDynamicLoading > 0123 6.4 1614 74 TestExportWriter.testExpr > 0123 8.6 1356 70 TestHdfsCloudBackupRestore.test > 0123 9.1 1697136 TestLocalFSCloudBackupRestore.test > 0123 0.5 1607 26 TestPackages.testPluginLoading > 0123 0.7 1596 15 > TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast > 0123 1.5 1610 59 TestReRankQParserPlugin.testMinExactCount > 0123 0.3 1552 4 TestReplicaProperties.test > 0123 0.3 1556 5 > TestSolrCloudWithDelegationTokens.testDelegationTokenRenew > 0123 0.3 1565 9 TestSolrConfigHandlerCloud.test > > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org -- Regards, Atri Apache Concerted - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Badapple report
OK, thanks. I’m not really annotating things at this point, although occasionally removing some that haven’t failed in a long time. > On Aug 10, 2020, at 1:44 PM, Tomás Fernández Löbbe > wrote: > > Hi Erick, > I've introduced and later fixed a bug in TestConfig. It hasn't failed since, > so please don't annotate it. > > On Mon, Aug 10, 2020 at 7:47 AM Erick Erickson > wrote: > We’re backsliding some. I encourage people to look at: > http://fucit.org/solr-jenkins-reports/failure-report.html, we have a number > of ill-behaved tests, particularly TestRequestRateLimiter, > TestBulkSchemaConcurrent, TestConfig, SchemaApiFailureTest and > TestIndexingSequenceNumbers… > > > Raw fail count by week totals, most recent week first (corresponds to bits): > Week: 0 had 100 failures > Week: 1 had 82 failures > Week: 2 had 94 failures > Week: 3 had 502 failures > > > Failures in Hoss' reports for the last 4 rollups. > > There were 585 unannotated tests that failed in Hoss' rollups. Ordered by the > date I downloaded the rollup file, newest->oldest. See above for the dates > the files were collected > These tests were NOT BadApple'd or AwaitsFix'd > > Failures in the last 4 reports.. >Report Pct runsfails test > 0123 4.4 1583 37 BasicDistributedZkTest.test > 0123 4.3 1727 77 CloudExitableDirectoryReaderTest.test > 0123 2.5 8598248 > CloudExitableDirectoryReaderTest.testCreepThenBite > 0123 1.9 1712 36 > CloudExitableDirectoryReaderTest.testWhitebox > 0123 0.5 1587 11 > DocValuesNotIndexedTest.testGroupingDVOnlySortLast > 0123 2.2 1679 82 HttpPartitionOnCommitTest.test > 0123 0.5 1592 16 HttpPartitionTest.test > 0123 1.0 1578 9 HttpPartitionWithTlogReplicasTest.test > 0123 1.3 1569 13 LeaderFailoverAfterPartitionTest.test > 0123 7.4 1643 59 MultiThreadedOCPTest.test > 0123 0.3 1567 8 ReplaceNodeTest.test > 0123 0.2 1588 6 ShardSplitTest.testSplitShardWithRule > 0123 100.0 38 33 SharedFSAutoReplicaFailoverTest.test > 0123 2.1 818 19 > TestCircuitBreaker.testBuildingMemoryPressure > 0123 2.6 818 13 > TestCircuitBreaker.testResponseWithCBTiming > 0123 6.2 1848104 TestContainerPlugin.testApiFromPackage > 0123 2.5 1662 33 TestDistributedGrouping.test > 0123 0.4 1448 6 TestDynamicLoading.testDynamicLoading > 0123 6.4 1614 74 TestExportWriter.testExpr > 0123 8.6 1356 70 TestHdfsCloudBackupRestore.test > 0123 9.1 1697136 TestLocalFSCloudBackupRestore.test > 0123 0.5 1607 26 TestPackages.testPluginLoading > 0123 0.7 1596 15 > TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast > 0123 1.5 1610 59 TestReRankQParserPlugin.testMinExactCount > 0123 0.3 1552 4 TestReplicaProperties.test > 0123 0.3 1556 5 > TestSolrCloudWithDelegationTokens.testDelegationTokenRenew > 0123 0.3 1565 9 TestSolrConfigHandlerCloud.test > > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Badapple report
Hi Erick, I've introduced and later fixed a bug in TestConfig. It hasn't failed since, so please don't annotate it. On Mon, Aug 10, 2020 at 7:47 AM Erick Erickson wrote: > We’re backsliding some. I encourage people to look at: > http://fucit.org/solr-jenkins-reports/failure-report.html, we have a > number of ill-behaved tests, particularly TestRequestRateLimiter, > TestBulkSchemaConcurrent, TestConfig, SchemaApiFailureTest and > TestIndexingSequenceNumbers… > > > Raw fail count by week totals, most recent week first (corresponds to > bits): > Week: 0 had 100 failures > Week: 1 had 82 failures > Week: 2 had 94 failures > Week: 3 had 502 failures > > > Failures in Hoss' reports for the last 4 rollups. > > There were 585 unannotated tests that failed in Hoss' rollups. Ordered by > the date I downloaded the rollup file, newest->oldest. See above for the > dates the files were collected > These tests were NOT BadApple'd or AwaitsFix'd > > Failures in the last 4 reports.. >Report Pct runsfails test > 0123 4.4 1583 37 BasicDistributedZkTest.test > 0123 4.3 1727 77 CloudExitableDirectoryReaderTest.test > 0123 2.5 8598248 > CloudExitableDirectoryReaderTest.testCreepThenBite > 0123 1.9 1712 36 > CloudExitableDirectoryReaderTest.testWhitebox > 0123 0.5 1587 11 > DocValuesNotIndexedTest.testGroupingDVOnlySortLast > 0123 2.2 1679 82 HttpPartitionOnCommitTest.test > 0123 0.5 1592 16 HttpPartitionTest.test > 0123 1.0 1578 9 HttpPartitionWithTlogReplicasTest.test > 0123 1.3 1569 13 LeaderFailoverAfterPartitionTest.test > 0123 7.4 1643 59 MultiThreadedOCPTest.test > 0123 0.3 1567 8 ReplaceNodeTest.test > 0123 0.2 1588 6 ShardSplitTest.testSplitShardWithRule > 0123 100.0 38 33 SharedFSAutoReplicaFailoverTest.test > 0123 2.1 818 19 > TestCircuitBreaker.testBuildingMemoryPressure > 0123 2.6 818 13 > TestCircuitBreaker.testResponseWithCBTiming > 0123 6.2 1848104 TestContainerPlugin.testApiFromPackage > 0123 2.5 1662 33 TestDistributedGrouping.test > 0123 0.4 1448 6 TestDynamicLoading.testDynamicLoading > 0123 6.4 1614 74 TestExportWriter.testExpr > 0123 8.6 1356 70 TestHdfsCloudBackupRestore.test > 0123 9.1 1697136 TestLocalFSCloudBackupRestore.test > 0123 0.5 1607 26 TestPackages.testPluginLoading > 0123 0.7 1596 15 > TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast > 0123 1.5 1610 59 > TestReRankQParserPlugin.testMinExactCount > 0123 0.3 1552 4 TestReplicaProperties.test > 0123 0.3 1556 5 > TestSolrCloudWithDelegationTokens.testDelegationTokenRenew > 0123 0.3 1565 9 TestSolrConfigHandlerCloud.test > > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org
Badapple report
We’re backsliding some. I encourage people to look at: http://fucit.org/solr-jenkins-reports/failure-report.html, we have a number of ill-behaved tests, particularly TestRequestRateLimiter, TestBulkSchemaConcurrent, TestConfig, SchemaApiFailureTest and TestIndexingSequenceNumbers… Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 100 failures Week: 1 had 82 failures Week: 2 had 94 failures Week: 3 had 502 failures Failures in Hoss' reports for the last 4 rollups. There were 585 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 4.4 1583 37 BasicDistributedZkTest.test 0123 4.3 1727 77 CloudExitableDirectoryReaderTest.test 0123 2.5 8598248 CloudExitableDirectoryReaderTest.testCreepThenBite 0123 1.9 1712 36 CloudExitableDirectoryReaderTest.testWhitebox 0123 0.5 1587 11 DocValuesNotIndexedTest.testGroupingDVOnlySortLast 0123 2.2 1679 82 HttpPartitionOnCommitTest.test 0123 0.5 1592 16 HttpPartitionTest.test 0123 1.0 1578 9 HttpPartitionWithTlogReplicasTest.test 0123 1.3 1569 13 LeaderFailoverAfterPartitionTest.test 0123 7.4 1643 59 MultiThreadedOCPTest.test 0123 0.3 1567 8 ReplaceNodeTest.test 0123 0.2 1588 6 ShardSplitTest.testSplitShardWithRule 0123 100.0 38 33 SharedFSAutoReplicaFailoverTest.test 0123 2.1 818 19 TestCircuitBreaker.testBuildingMemoryPressure 0123 2.6 818 13 TestCircuitBreaker.testResponseWithCBTiming 0123 6.2 1848104 TestContainerPlugin.testApiFromPackage 0123 2.5 1662 33 TestDistributedGrouping.test 0123 0.4 1448 6 TestDynamicLoading.testDynamicLoading 0123 6.4 1614 74 TestExportWriter.testExpr 0123 8.6 1356 70 TestHdfsCloudBackupRestore.test 0123 9.1 1697136 TestLocalFSCloudBackupRestore.test 0123 0.5 1607 26 TestPackages.testPluginLoading 0123 0.7 1596 15 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0123 1.5 1610 59 TestReRankQParserPlugin.testMinExactCount 0123 0.3 1552 4 TestReplicaProperties.test 0123 0.3 1556 5 TestSolrCloudWithDelegationTokens.testDelegationTokenRenew 0123 0.3 1565 9 TestSolrConfigHandlerCloud.test DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate SuppressWarnings count: last week: 4,825, this week: 4,819, delta -6 *** Files with increased @SuppressWarnings annotations: Suppress count increase in: solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java. Was: 13, now: 15 Suppress count increase in: solr/core/src/java/org/apache/solr/packagemanager/PackageManager.java. Was: 7, now: 8 Suppress count increase in: solr/core/src/test/org/apache/solr/core/TestSolrConfigHandler.java. Was: 14, now: 17 Suppress count increase in: solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpSolrClient.java. Was: 12, now: 13 *** Files with decreased @SuppressWarnings annotations: Suppress count decrease in: solr/core/src/java/org/apache/solr/core/PluginBag.java. Was: 6, now: 5 Processing file (History bit 3): HOSS-2020-08-10.csv Processing file (History bit 2): HOSS-2020-08-03.csv Processing file (History bit 1): HOSS-2020-07-27.csv Processing file (History bit 0): HOSS-2020-07-20.csv Number of AwaitsFix: 33 Number of BadApples: 4 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists
BadApple report, but please read the first bit
There are several tests that are causing a lot of noise: SharedFSAutoReplicaFailoverTest is failing 90%+ of the time. TestBulkSchemaConcurrent 31% StressHdfsTest 16% SchemaApiFailureTest 13.88% I encourage people to look at: http://fucit.org/solr-jenkins-reports/failure-report.html and see if anything looks like it is affected by recent work. TestBulkSchemaConcurrent has been failing off and on for a long time, but the failure rate picked up dramatically in the last couple of weeks. Ditto SchemaApiFailureTest. Do we even care about Hdfs? Are we deprecating it or not? Holding relatively steady otherwise: Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 82 failures Week: 1 had 94 failures Week: 2 had 502 failures Week: 3 had 19 failures Failures in Hoss' reports for the last 4 rollups. There were 562 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 0.3 1271 8 RollingRestartTest.test 0123 93.3 41 36 SharedFSAutoReplicaFailoverTest.test 0123 3.5 627 16 TestCircuitBreaker.testBuildingMemoryPressure 0123 1.0 627 8 TestCircuitBreaker.testResponseWithCBTiming 0123 5.8 1483 79 TestContainerPlugin.testApiFromPackage 0123 2.3 1335 23 TestDistributedGrouping.test Full report: DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate SuppressWarnings count: last week: 4,825, this week: 4,825, delta 0 *** Files with increased @SuppressWarnings annotations: *** Files with decreased @SuppressWarnings annotations: Processing file (History bit 3): HOSS-2020-08-03.csv Processing file (History bit 2): HOSS-2020-07-27.csv Processing file (History bit 1): HOSS-2020-07-20.csv Processing file (History bit 0): HOSS-2020-07-13.csv Number of AwaitsFix: 33 Number of BadApples: 4 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 82 failures Week: 1 had 94 failures Week: 2 had 502 failures Week: 3 had 19 failures Failures in Hoss' reports for the last 4 rollups. There were 562 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 0.3 1271 8 RollingRestartTest.test 0123 93.3 41 36 SharedFSAutoReplicaFailoverTest.test 0123 3.5 627 16 TestCircuitBreaker.testBuildingMemoryPressure 0123 1.0 627 8 TestCircuitBreaker.testResponseWithCBTiming 0123 5.8 1483 79 TestContainerPlugin.testApiFromPackage 0123 2.3 1335 23 TestDistributedGrouping.test Failures over the last 4 weeks, but not every week. Ordered most-recent first: Report Pct runsfails test 0121.3 1174 19 BasicDistributedZkTest.test 0126.0 1261 57 CloudExitableDirectoryReaderTest.test 0124.2 6274189 CloudExitableDirectoryReaderTest.testCreepThenBite 0123.3 1246 27 CloudExitableDirectoryReaderTest.testWhitebox 0120.5 1189 9
BadApple report
Short form: Processing file (History bit 3): HOSS-2020-07-27.csv Processing file (History bit 2): HOSS-2020-07-20.csv Processing file (History bit 1): HOSS-2020-07-13.csv Processing file (History bit 0): HOSS-2020-07-06.csv Number of AwaitsFix: 33 Number of BadApples: 4 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 94 failures Week: 1 had 502 failures Week: 2 had 19 failures Week: 3 had 24 failures Failures in Hoss' reports for the last 4 rollups. There were 553 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 93.3 30 26 SharedFSAutoReplicaFailoverTest.test 0123 6.0 1141 59 TestContainerPlugin.testApiFromPackage 0123 1.6 1000 17 TestInPlaceUpdatesDistrib.test Full results attached: DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate SuppressWarnings count: last week: 4,835, this week: 4,825, delta -10 *** Files with increased @SuppressWarnings annotations: Suppress count increase in: solr/core/src/java/org/apache/solr/packagemanager/PackageManager.java. Was: 6, now: 7 *** Files with decreased @SuppressWarnings annotations: Suppress count decrease in: solr/core/src/java/org/apache/solr/search/Grouping.java. Was: 28, now: 27 Suppress count decrease in: solr/core/src/java/org/apache/solr/search/grouping/endresulttransformer/GroupedEndResultTransformer.java. Was: 2, now: 1 Suppress count decrease in: solr/core/src/test/org/apache/hadoop/fs/FileUtil.java. Was: 3, now: 2 Suppress count decrease in: solr/core/src/test/org/apache/solr/util/tracing/TestHttpServletCarrier.java. Was: 1, now: 0 Suppress count decrease in: solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/SignificantTermsStream.java. Was: 15, now: 12 Suppress count decrease in: solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/eval/ConversionEvaluatorsTest.java. Was: 2, now: 1 Suppress count decrease in: solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/eval/TemporalEvaluatorsTest.java. Was: 1, now: 0 Suppress count decrease in: solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/ops/ConcatOperationTest.java. Was: 1, now: 0 Suppress count decrease in: solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/ops/OperationsTest.java. Was: 1, now: 0 Processing file (History bit 3): HOSS-2020-07-27.csv Processing file (History bit 2): HOSS-2020-07-20.csv Processing file (History bit 1): HOSS-2020-07-13.csv Processing file (History bit 0): HOSS-2020-07-06.csv Number of AwaitsFix: 33 Number of BadApples: 4 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 94 failures Week: 1 had 502 failures Week: 2 had 19 failures Week: 3 had 24 failures Failures in Hoss' reports for the last 4 rollups. There were 553 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports..
BadApple report
Well, that’s one way to reduce the number of SuppressWarnings… cut out massive amounts of code ;)…. SuppressWarnings count: last week: 5,353, this week: 4,835, delta -518 We had quite a spike in the raw number of tests that have failed at least once in the last week: Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 502 failures Week: 1 had 19 failures Week: 2 had 24 failures Week: 3 had 26 failures IDK whether this reflects a temporary glitch or whether we’re now scanning more builds. At any rate we’ll see what next week brings. This bit is encouraging, very few tests have failed every week for the last 4. Failures in Hoss' reports for the last 4 rollups. There were 536 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 57.1 25 21 SharedFSAutoReplicaFailoverTest.test 0123 4.4 741 33 TestContainerPlugin.testApiFromPackage 0123 2.3 732 13 TestInPlaceUpdatesDistrib.test DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate SuppressWarnings count: last week: 5,353, this week: 4,835, delta -518 *** Files with increased @SuppressWarnings annotations: Suppress count increase in: solr/core/src/java/org/apache/solr/core/SolrClassLoader.java. Was: null, now: 1 Suppress count increase in: solr/core/src/java/org/apache/solr/handler/SchemaHandler.java. Was: 6, now: 7 Suppress count increase in: solr/core/src/java/org/apache/solr/pkg/PackageListeningClassLoader.java. Was: null, now: 1 Suppress count increase in: solr/core/src/test/org/apache/solr/pkg/TestPackages.java. Was: 5, now: 7 Suppress count increase in: solr/solrj/src/java/org/apache/solr/client/solrj/cloud/DelegatingCloudManager.java. Was: null, now: 1 Suppress count increase in: solr/solrj/src/java/org/apache/solr/client/solrj/impl/SolrClientNodeStateProvider.java. Was: 4, now: 6 Suppress count increase in: solr/solrj/src/java/org/apache/solr/common/cloud/Replica.java. Was: 0, now: 1 *** Files with decreased @SuppressWarnings annotations: Suppress count decrease in: solr/core/src/java/org/apache/solr/cloud/api/collections/Assign.java. Was: 6, now: 5 Suppress count decrease in: solr/core/src/test/org/apache/solr/cloud/rule/RulesTest.java. Was: 7, now: 5 Suppress count decrease in: solr/core/src/test/org/apache/solr/util/TestSolrCLIRunExample.java. Was: 1, now: 0 Suppress count decrease in: solr/solrj/src/java/org/apache/solr/client/solrj/impl/ZkDistribStateManager.java. Was: 1, now: 0 Suppress count decrease in: solr/solrj/src/java/org/apache/solr/common/cloud/ZkStateReader.java. Was: 7, now: 6 Processing file (History bit 3): HOSS-2020-07-20.csv Processing file (History bit 2): HOSS-2020-07-13.csv Processing file (History bit 1): HOSS-2020-07-06.csv Processing file (History bit 0): HOSS-2020-06-29.csv Number of AwaitsFix: 33 Number of BadApples: 4 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 502 failures Week: 1 had 19 failures Week: 2 had 24 failures Week: 3 had 26 failures Failures in Hoss' reports for the last 4 rollups. There were 536 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails
BadApple report
Actaully, pretty good. The attached file has a lot of noise in it that’s a listing of the files that have more or less SuppressWarnings annotations than last week, the delta is -19. It’s a crude measure, I can replace N SuppressWarnings in a class with one for the entire class, but it’s also easy to count. Down is the right direction though. NamedList accounts for a huge number of SuppressWarnings. I do wonder if we can figure out better ways to avoid warnings with that class. Other than replace it. Wholesale surgery to replace it just to avoid warnings is a pretty bad idea of course…. SuppressWarnings count: last week: 5,372, this week: 5,353, delta -19 Processing file (History bit 3): HOSS-2020-07-13.csv Processing file (History bit 2): HOSS-2020-07-06.csv Processing file (History bit 1): HOSS-2020-06-29.csv Processing file (History bit 0): HOSS-2020-06-22.csv Number of AwaitsFix: 46 Number of BadApples: 4 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 19 failures Week: 1 had 24 failures Week: 2 had 26 failures Week: 3 had 26 failures Failures in Hoss' reports for the last 4 rollups. There were 71 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 0.9 447 5 TestInPlaceUpdatesDistrib.test DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate SuppressWarnings count: last week: 5,372, this week: 5,353, delta -19 *** Files with increased @SuppressWarnings annotations: Suppress count increase in: solr/core/src/java/org/apache/solr/search/facet/SlotAcc.java. Was: 3, now: 5 Suppress count increase in: solr/core/src/test/org/apache/solr/search/facet/TestCloudJSONFacetSKGEquiv.java. Was: 5, now: 11 Suppress count increase in: solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java. Was: 12, now: 13 *** Files with decreased @SuppressWarnings annotations: Suppress count decrease in: solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/ContextImpl.java. Was: 5, now: 4 Suppress count decrease in: solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/XPathEntityProcessor.java. Was: 8, now: 7 Suppress count decrease in: solr/core/src/java/org/apache/solr/handler/IndexFetcher.java. Was: 19, now: 13 Suppress count decrease in: solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java. Was: 14, now: 13 Suppress count decrease in: solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java. Was: 1, now: 0 Suppress count decrease in: solr/core/src/java/org/apache/solr/handler/component/HttpShardHandlerFactory.java. Was: 7, now: 6 Suppress count decrease in: solr/core/src/java/org/apache/solr/search/function/distance/GeoDistValueSourceParser.java. Was: 2, now: 1 Suppress count decrease in: solr/core/src/java/org/apache/solr/security/AuthorizationContext.java. Was: 1, now: 0 Suppress count decrease in: solr/core/src/java/org/apache/solr/security/KerberosPlugin.java. Was: 1, now: 0 Suppress count decrease in: solr/core/src/java/org/apache/solr/security/RuleBasedAuthorizationPluginBase.java. Was: 4, now: 3 Suppress count decrease in: solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java. Was: 5, now: 4 Suppress count decrease in: solr/core/src/test/org/apache/solr/cloud/api/collections/CollectionsAPIDistributedZkTest.java. Was: 4, now: 3 Suppress count decrease in:
Re: BadApple report
Megan: There are a number of tests that have been flagged by some devs that, no matter what, should _not_ be annotated with BadApple or AwaitsFix and that’s just a list to remind me what they are. It’s not much of a deal, though, because I’m not doing much annotating lately. The original process was that I’d annotate tests that had failed every week for the last 4 weeks. Partly to get people’s attention, partly to make a record. There were tests that would come and go, so you’ll see in places\ a bunch of dates associated with an annotation. Those indicate that it’d be bad, then OK for 4 or more weeks, then bad again which I thought was useful to see just how rarely some tests failed. Best, Erick > On Jul 6, 2020, at 1:47 PM, Megan Carey wrote: > > Hi Erick, > > I'm wondering what is meant by "DO NOT ANNOTATE LIST" at the start of the > report? Better yet, can you please link to the scraping tool used to generate > the report? > > Thank you! > Megan > > On Mon, Jul 6, 2020 at 8:07 AM Erick Erickson wrote: > Holding fairly steady, but IDK whether Hoss’ scraping is getting data from > Uwe’s machines, thought I saw an e-mail go by about that. > > this is the first report where the suppresswarnings stats mean anything. > > Full report attached: > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: BadApple report
Hi Erick, I'm wondering what is meant by "DO NOT ANNOTATE LIST" at the start of the report? Better yet, can you please link to the scraping tool used to generate the report? Thank you! Megan On Mon, Jul 6, 2020 at 8:07 AM Erick Erickson wrote: > Holding fairly steady, but IDK whether Hoss’ scraping is getting data from > Uwe’s machines, thought I saw an e-mail go by about that. > > this is the first report where the suppresswarnings stats mean anything. > > Full report attached: > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org
BadApple report
Holding fairly steady, but IDK whether Hoss’ scraping is getting data from Uwe’s machines, thought I saw an e-mail go by about that. this is the first report where the suppresswarnings stats mean anything. Full report attached: DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate SuppressWarnings count: last week: 5,373, this week: 5,372, delta -1 *** Files with increased @SuppressWarnings annotations: Suppress count increase in: solr/core/src/java/org/apache/solr/api/AnnotatedApi.java. Was: 4, now: 5 Suppress count increase in: solr/core/src/java/org/apache/solr/packagemanager/PackageManager.java. Was: 4, now: 6 Suppress count increase in: solr/solrj/src/java/org/apache/solr/common/util/Utils.java. Was: 28, now: 30 *** Files with decreased @SuppressWarnings annotations: Suppress count decrease in: solr/core/src/java/org/apache/solr/handler/export/ExportWriter.java. Was: 1, now: 0 Suppress count decrease in: solr/core/src/test/org/apache/solr/handler/export/TestExportWriter.java. Was: 6, now: 2 Processing file (History bit 3): HOSS-2020-07-06.csv Processing file (History bit 2): HOSS-2020-06-29.csv Processing file (History bit 1): HOSS-2020-06-22.csv Processing file (History bit 0): HOSS-2020-06-15.csv Number of AwaitsFix: 45 Number of BadApples: 4 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 24 failures Week: 1 had 26 failures Week: 2 had 26 failures Week: 3 had 34 failures Failures in Hoss' reports for the last 4 rollups. There were 84 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0120.9 340 4 TestInPlaceUpdatesDistrib.test 012 15.8 305 32 TestReRankQParserPlugin.testMinExactCount 01 3 0.8 345 3 DocValuesNotIndexedTest.testGroupingDVOnlySortFirst 01 3 100.0 16 14 SharedFSAutoReplicaFailoverTest.test 01 7.6 420194 DebugComponentTest.testBasicInterface 01 7.6 420194 DebugComponentTest.testPerItemInterface 0110.3 55 5 ShardSplitTest.testSplitWithChaosMonkey 01 5.8 186 9 TestContainerPlugin.testApiFromPackage 0 20.8 230 2 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0 3 0.8 230 2 TestSimScenario.testSuggestions 0 1.6 122 2 PeerSyncWithLeaderTest.test 0 0.8 131 1 ShardSplitTest.testSplitShardWithRule 0 0.8 126 1 TestBlockJoin.testMultiChildQueriesOfDiffParentLevels 0 1.6 127 2 TestDemoParallelLeafReader.testBasic 0 1.6 127 2 TestDemoParallelLeafReader.testBasicMultipleSchemaGens 0 1.6 127 2 TestDemoParallelLeafReader.testRandom 0 1.6 127 2 TestDemoParallelLeafReader.testRandomMultipleSchemaGens 0 0.8 126 1 TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField 0 52.6 38 20 TestStressThreadBackup.testCoreAdminHandler 0 52.6 38 20 TestStressThreadBackup.testReplicationHandler 0 0.9 117 1 TestTlogReplica.testRemoveLeader 123 17.2 81 15 HdfsSyncSliceTest.test 123 4.7 355 11 RollingRestartTest.test 120.9 221 2 AutoScalingHandlerTest.testReadApi 12
BadApple report
Holding fairly steady. Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 26 failures Week: 1 had 26 failures Week: 2 had 34 failures Week: 3 had 128 failures This week’s report includes the SuppressWarnings summary. This is really the baseline, I added a few more that are counted in this as part of getting clean compiles, included here so people can see what they look like. Only one test has failed every week over the last 4: Failures in the last 4 reports.. Report Pct runsfails test 0123 4.7 639 17 RollingRestartTest.test Full report attached: DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate SuppressWarnings count: last week: 5,377, this week: 5,373, delta -4 *** Files with increased @SuppressWarnings annotations: Suppress count increase in: solr/contrib/dataimporthandler/src/test/org/apache/solr/handler/dataimport/TestZKPropertiesWriter.java. Was: 2, now: 5 Suppress count increase in: solr/core/src/java/org/apache/solr/api/CustomContainerPlugins.java. Was: null, now: 4 Suppress count increase in: solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java. Was: 13, now: 14 Suppress count increase in: solr/core/src/java/org/apache/solr/handler/admin/ContainerPluginsApi.java. Was: null, now: 3 Suppress count increase in: solr/core/src/java/org/apache/solr/search/JoinQParserPlugin.java. Was: 0, now: 2 Suppress count increase in: solr/core/src/java/org/apache/solr/search/join/CrossCollectionJoinQParser.java. Was: null, now: 1 Suppress count increase in: solr/core/src/test/org/apache/solr/handler/TestContainerPlugin.java. Was: null, now: 2 Suppress count increase in: solr/solrj/src/test/org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy.java. Was: 109, now: 115 Suppress count increase in: solr/solrj/src/test/org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy2.java. Was: 22, now: 23 *** Files with decreased @SuppressWarnings annotations: Suppress count decrease in: solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/AutoScalingConfig.java. Was: 10, now: 6 Suppress count decrease in: solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java. Was: 8, now: 7 Suppress count decrease in: solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Preference.java. Was: 4, now: 3 Suppress count decrease in: solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/ReplicaCount.java. Was: 1, now: 0 Suppress count decrease in: solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/ReplicaInfo.java. Was: 3, now: 2 Suppress count decrease in: solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/VersionedData.java. Was: 1, now: 0 Suppress count decrease in: solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/CloudSolrStream.java. Was: 3, now: 2 Suppress count decrease in: solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/DeepRandomStream.java. Was: 1, now: 0 Suppress count decrease in: solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/expr/StreamExpression.java. Was: 1, now: 0 Suppress count decrease in: solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/expr/StreamExpressionNamedParameter.java. Was: 1, now: 0 Suppress count decrease in: solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/expr/StreamExpressionValue.java. Was: 1, now: 0 Suppress count decrease in: solr/solrj/src/java/org/apache/solr/common/cloud/DocCollection.java. Was: 1, now: 0 Suppress count decrease in: solr/solrj/src/java/org/apache/solr/common/cloud/Replica.java. Was: 1, now: 0 Suppress count decrease in: solr/solrj/src/java/org/apache/solr/common/cloud/ZkNodeProps.java. Was: 2, now: 1 Suppress count decrease in: solr/solrj/src/java/org/apache/solr/common/util/JsonSchemaValidator.java. Was: 21, now: 15 Suppress count decrease in: solr/solrj/src/java/org/apache/solr/common/util/ValidatingJsonMap.java. Was: 12, now: 11
BadApple report
Not a bad week all told, but something seems a little odd, I remember a lot more e-mails going by, but perhaps it’s just these 26 tests failing repeatedly. Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 26 failures Week: 1 had 34 failures Week: 2 had 128 failures Week: 3 had 68 failures Failures in Hoss' reports for the last 4 rollups. There were 208 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 2.7 893 15 RollingRestartTest.test 0123 1.8 872 9 SystemCollectionCompatTest.testBackCompat Full report attached (less suppresswarnings data). DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate SuppressWarnings count: last week: 3,617, this week: 5,377, delta 1,760 Processing file (History bit 3): HOSS-2020-06-22.csv Processing file (History bit 2): HOSS-2020-06-15.csv Processing file (History bit 1): HOSS-2020-06-08.csv Processing file (History bit 0): HOSS-2020-06-01.csv Number of AwaitsFix: 46 Number of BadApples: 4 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 26 failures Week: 1 had 34 failures Week: 2 had 128 failures Week: 3 had 68 failures Failures in Hoss' reports for the last 4 rollups. There were 208 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 2.7 893 15 RollingRestartTest.test 0123 1.8 872 9 SystemCollectionCompatTest.testBackCompat Failures over the last 4 weeks, but not every week. Ordered most-recent first: Report Pct runsfails test 0122.2 52 10 HdfsSyncSliceTest.test 01 9.2 233 12 TestIndexWriterOnVMError.testOOM 0 23 0.9 761 5 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0 21.4 341 2 LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud 0 20.9 379 3 TestInPlaceUpdatesDistrib.test 0 3 0.9 488 3 AutoScalingHandlerTest.testReadApi 0 3 1.8 487 3 TestOfflineSorter.testThreadSafety 0 3 4.5 57 12 TestXYMultiPolygonShapeQueries.testRandomBig 0 0.9 108 1 AutoscalingHistoryHandlerTest.testHistory 0 27.3 161 44 CurrencyRangeFacetCloudTest.testJsonRangeFacetWithSubFacet 0 4.0 25 1 ForceLeaderTest.testReplicasInLowerTerms 0 0.9 111 1 HttpPartitionWithTlogReplicasTest.test 0 15.2 112 17 TestAllFilesDetectTruncation.test 0 12.1 91 11 TestCloudJSONFacetSKGEquiv.testRandom 0 0.9 110 1 TestNRTReaderWithThreads.testIndexing 0 0.9 108 1 TestPullReplicaErrorHandling.testCantConnectToLeader 0 2.6 78 2 TestReRankQParserPlugin.testMinExactCount 0 0.9 108 1 TestSimDistributedQueue.testPeekElements 0 0.9 111 1 TestStressNRT.test 123 0.9 757 4
BadApple report
The number of chronically failing tests dropped considerably this past week, whether that’s an anomaly or not is a good question. I’ve finished the SuppressWarnings annotations, so next week I _should_ be able to include how many new SuppressWarnings have been added to the code and have it mean something. I _strongly_ urge people to see if they can remove these annotations when they’re working on the area of code anyway. The second thing I urge people to do is use their IDE well. IntelliJ does a series of automatic “inspections” for instance that can point to issues. It’ll highlight C-style array declarations which isn’t really a bug, but... I’m _not_ saying we should fix everything the inspections highlight, for instance it doesn’t like if (a == false) want’s to “simplify” it to if (!a) That’s one inspection I want to turn off; I find it too easy to overlook the “!”. However, another thing that’s highlighted is something like if (object.getName().someMethod) where getName may return null. Again, I’m not saying each and every one of these should be changed. Just look at it and see if it’s really something that could happen and guard if so (how many NPEs have we had to be fixed later?). Oh, and do be aware that IntelliJ can annotate inspections, but don’t do that. There’s no reason to pollute the code with IntelliJ-specific annotations. OK, here’s the regular report. Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 34 failures Week: 1 had 128 failures Week: 2 had 68 failures Week: 3 had 113 failures Failures in Hoss' reports for the last 4 rollups. There were 264 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 1.7 1186 15 RollingRestartTest.test 0123 0.9 1161 9 SystemCollectionCompatTest.testBackCompat 0123 0.9 1190 11 TestSimScenario.testSuggestions DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2020-06-15.csv Processing file (History bit 2): HOSS-2020-06-08.csv Processing file (History bit 1): HOSS-2020-06-01.csv Processing file (History bit 0): HOSS-2020-05-25.csv Number of AwaitsFix: 44 Number of BadApples: 4 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 34 failures Week: 1 had 128 failures Week: 2 had 68 failures Week: 3 had 113 failures Failures in Hoss' reports for the last 4 rollups. There were 264 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 1.7 1186 15 RollingRestartTest.test 0123 0.9 1161 9 SystemCollectionCompatTest.testBackCompat 0123 0.9 1190 11 TestSimScenario.testSuggestions Failures over the last 4 weeks, but not every week. Ordered most-recent first: 0120.9 757 4 DocValuesNotIndexedTest.testGroupingDVOnlySortFirst 01 3.4 91 5 BasicAuthOnSingleNodeTest.testDeleteSecurityJsonZnode 01 7.6 391 13 DistribCursorPagingTest.test 01 7.4 45 3 HdfsWriteToMultipleCollectionsTest.test 0
Re: BadApple report
Thanks for letting me know Tomás As useful as Hoss’ rollups are, there’s always a lag to deal with, sounds like this is one. > On Jun 8, 2020, at 2:26 PM, Tomás Fernández Löbbe > wrote: > > Thanks for keeping an eye Erick. I took a quick look at the > "TestIndexSearcher" failures and I think they're related to SOLR-14525. > Should be fixed after this[1] commit by Noble. > > [1] https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=5827ddf > > On Mon, Jun 8, 2020 at 7:52 AM Erick Erickson wrote: > If people don’t know about: > http://fucit.org/solr-jenkins-reports/suspicious-failure-report.html, I > strongly recommend you periodically check it. It reports tests that have > changed their failure rates lately. There are three currently: > > "org.apache.solr.search.TestIndexSearcher","testSearcherListeners" > "org.apache.solr.update.processor.DocExpirationUpdateProcessorFactoryTest","testAutomaticDeletes" > "org.apache.solr.cloud.PackageManagerCLITest","testPackageManager > > Short form: > > Raw fail count by week totals, most recent week first (corresponds to bits): > Week: 0 had 128 failures > Week: 1 had 68 failures > Week: 2 had 113 failures > Week: 3 had 103 failures > > > Failures in Hoss' reports for the last 4 rollups. > > There were 298 unannotated tests that failed in Hoss' rollups. Ordered by the > date I downloaded the rollup file, newest->oldest. See above for the dates > the files were collected > These tests were NOT BadApple'd or AwaitsFix'd > > Failures in the last 4 reports.. >Report Pct runsfails test > 0123 0.4 1461 5 > DeleteReplicaTest.deleteReplicaAndVerifyDirectoryCleanup > 0123 0.7 1464 9 > MetricTriggerIntegrationTest.testMetricTrigger > 0123 1.6 1377 29 MultiThreadedOCPTest.test > 0123 0.7 1455 5 > NodeMarkersRegistrationTest.testNodeMarkersRegistration > 0123 2.1 1481 17 RollingRestartTest.test > 0123 0.4 1537 55 > ScheduledTriggerIntegrationTest.testScheduledTrigger > 0123 7.7 98 6 ShardSplitTest.testSplitWithChaosMonkey > 0123 0.4 1455 9 SystemCollectionCompatTest.testBackCompat > 0123 0.7 1456 14 TestPackages.testPluginLoading > 0123 1.1 1460 9 > TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast > 0123 0.7 1498 13 TestSimScenario.testSuggestions > > I took the SuppressWarnings count section out, it’s ridiculously big. > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: BadApple report
Thanks for keeping an eye Erick. I took a quick look at the "TestIndexSearcher" failures and I think they're related to SOLR-14525. Should be fixed after this[1] commit by Noble. [1] https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=5827ddf On Mon, Jun 8, 2020 at 7:52 AM Erick Erickson wrote: > If people don’t know about: > http://fucit.org/solr-jenkins-reports/suspicious-failure-report.html, I > strongly recommend you periodically check it. It reports tests that have > changed their failure rates lately. There are three currently: > > "org.apache.solr.search.TestIndexSearcher","testSearcherListeners" > > "org.apache.solr.update.processor.DocExpirationUpdateProcessorFactoryTest","testAutomaticDeletes" > "org.apache.solr.cloud.PackageManagerCLITest","testPackageManager > > Short form: > > Raw fail count by week totals, most recent week first (corresponds to > bits): > Week: 0 had 128 failures > Week: 1 had 68 failures > Week: 2 had 113 failures > Week: 3 had 103 failures > > > Failures in Hoss' reports for the last 4 rollups. > > There were 298 unannotated tests that failed in Hoss' rollups. Ordered by > the date I downloaded the rollup file, newest->oldest. See above for the > dates the files were collected > These tests were NOT BadApple'd or AwaitsFix'd > > Failures in the last 4 reports.. >Report Pct runsfails test > 0123 0.4 1461 5 > DeleteReplicaTest.deleteReplicaAndVerifyDirectoryCleanup > 0123 0.7 1464 9 > MetricTriggerIntegrationTest.testMetricTrigger > 0123 1.6 1377 29 MultiThreadedOCPTest.test > 0123 0.7 1455 5 > NodeMarkersRegistrationTest.testNodeMarkersRegistration > 0123 2.1 1481 17 RollingRestartTest.test > 0123 0.4 1537 55 > ScheduledTriggerIntegrationTest.testScheduledTrigger > 0123 7.7 98 6 > ShardSplitTest.testSplitWithChaosMonkey > 0123 0.4 1455 9 > SystemCollectionCompatTest.testBackCompat > 0123 0.7 1456 14 TestPackages.testPluginLoading > 0123 1.1 1460 9 > TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast > 0123 0.7 1498 13 TestSimScenario.testSuggestions > > I took the SuppressWarnings count section out, it’s ridiculously big. > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org
BadApple report
If people don’t know about: http://fucit.org/solr-jenkins-reports/suspicious-failure-report.html, I strongly recommend you periodically check it. It reports tests that have changed their failure rates lately. There are three currently: "org.apache.solr.search.TestIndexSearcher","testSearcherListeners" "org.apache.solr.update.processor.DocExpirationUpdateProcessorFactoryTest","testAutomaticDeletes" "org.apache.solr.cloud.PackageManagerCLITest","testPackageManager Short form: Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 128 failures Week: 1 had 68 failures Week: 2 had 113 failures Week: 3 had 103 failures Failures in Hoss' reports for the last 4 rollups. There were 298 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 0.4 1461 5 DeleteReplicaTest.deleteReplicaAndVerifyDirectoryCleanup 0123 0.7 1464 9 MetricTriggerIntegrationTest.testMetricTrigger 0123 1.6 1377 29 MultiThreadedOCPTest.test 0123 0.7 1455 5 NodeMarkersRegistrationTest.testNodeMarkersRegistration 0123 2.1 1481 17 RollingRestartTest.test 0123 0.4 1537 55 ScheduledTriggerIntegrationTest.testScheduledTrigger 0123 7.7 98 6 ShardSplitTest.testSplitWithChaosMonkey 0123 0.4 1455 9 SystemCollectionCompatTest.testBackCompat 0123 0.7 1456 14 TestPackages.testPluginLoading 0123 1.1 1460 9 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0123 0.7 1498 13 TestSimScenario.testSuggestions I took the SuppressWarnings count section out, it’s ridiculously big. DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate SuppressWarnings count: last week: 1,226, this week: 2,385, delta 1,159 Processing file (History bit 3): HOSS-2020-06-08.csv Processing file (History bit 2): HOSS-2020-06-01.csv Processing file (History bit 1): HOSS-2020-05-25.csv Processing file (History bit 0): HOSS-2020-05-18.csv Number of AwaitsFix: 42 Number of BadApples: 4 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 128 failures Week: 1 had 68 failures Week: 2 had 113 failures Week: 3 had 103 failures Failures in Hoss' reports for the last 4 rollups. There were 298 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 0.4 1461 5 DeleteReplicaTest.deleteReplicaAndVerifyDirectoryCleanup 0123 0.7 1464 9 MetricTriggerIntegrationTest.testMetricTrigger 0123 1.6 1377 29 MultiThreadedOCPTest.test 0123 0.7 1455 5 NodeMarkersRegistrationTest.testNodeMarkersRegistration 0123 2.1 1481 17 RollingRestartTest.test 0123 0.4 1537 55 ScheduledTriggerIntegrationTest.testScheduledTrigger 0123 7.7 98 6 ShardSplitTest.testSplitWithChaosMonkey 0123 0.4 1455 9 SystemCollectionCompatTest.testBackCompat 0123 0.7 1456 14 TestPackages.testPluginLoading 0123 1.1 1460 9
Re: BadApple report. It's worth reviewing the SuppressWarnings section even if you ignore the rest.
If you go to Hoss’ rollups here: http://fucit.org/solr-jenkins-reports/ Click on "Failures rates for the last 24h/7days” then click on one of the tests you’ll get a popup with a link to the output. IDK how long the output is kept around though. > On Jun 2, 2020, at 4:08 AM, Noble Paul wrote: > > Is there a way to see the failures and their logs? > > On Tue, Jun 2, 2020 at 12:02 AM Erick Erickson > wrote: >> >> This week is a significant improvement. Short form: >> >> >> Raw fail count by week totals, most recent week first (corresponds to bits): >> Week: 0 had 68 failures >> Week: 1 had 113 failures >> Week: 2 had 103 failures >> Week: 3 had 102 failures >> >> >> Failures in Hoss' reports for the last 4 rollups. >> >> There were 273 unannotated tests that failed in Hoss' rollups. Ordered by >> the date I downloaded the rollup file, newest->oldest. See above for the >> dates the files were collected >> These tests were NOT BadApple'd or AwaitsFix'd >> >> Failures in the last 4 reports.. >> Report Pct runsfails test >> 0123 3.1 1601 41 BasicDistributedZkTest.test >> 0123 1.7 1495 28 MultiThreadedOCPTest.test >> 0123 1.0 1587 14 RollingRestartTest.test >> 0123 3.1 1653 55 >> ScheduledTriggerIntegrationTest.testScheduledTrigger >> 0123 1.3 1574 13 SystemCollectionCompatTest.testBackCompat >> 0123 1.6 1571 14 TestPackages.testPluginLoading >> 0123 0.3 1570 7 >> TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast >> >> >> =SuppressWarnings== >> >> In the attached report there’s a new section counting SuppressWarnings. For >> the nonce, ignore it. Eventually, when all the warnings are fixed or >> suppressed, I will be advocating for _not_ introducing new warnings at least >> on Master. To encourage this, I want un-suppressed warnings to become >> compile-time errors. >> >> That’ll tempt people to just add @SuppressWarnings, and I don’t think that’s >> a proper fix, so the BadApple report will flag files that have more >> @SuppressWarnings than they did last week and I’ll complain ;) There’ll be >> exceptions of course... >> >> Yes, that flies counter to the zillion SuppressWarnings I’m putting in the >> code right now, but I’m not about to try to fix on the order of 5,000 >> warnings in our code all at once. that’s where the SuppressWarnings data is >> coming from in the attached report, I expect the counts to increase until we >> get clean compilations. Martin Fowler talks about rewriting working code for >> no good reason being a bad idea in “Refactoring”... >> >> My goal currently is to get the compilations clean, stop getting worse, and >> then we can make things better. Along about 2040, all the code that >> currently has SuppressWarnings will have been rewritten and they’ll all be >> gone... >> >> == >> Full report: >> >> >> >> >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org > > > > -- > - > Noble Paul > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: BadApple report. It's worth reviewing the SuppressWarnings section even if you ignore the rest.
Is there a way to see the failures and their logs? On Tue, Jun 2, 2020 at 12:02 AM Erick Erickson wrote: > > This week is a significant improvement. Short form: > > > Raw fail count by week totals, most recent week first (corresponds to bits): > Week: 0 had 68 failures > Week: 1 had 113 failures > Week: 2 had 103 failures > Week: 3 had 102 failures > > > Failures in Hoss' reports for the last 4 rollups. > > There were 273 unannotated tests that failed in Hoss' rollups. Ordered by the > date I downloaded the rollup file, newest->oldest. See above for the dates > the files were collected > These tests were NOT BadApple'd or AwaitsFix'd > > Failures in the last 4 reports.. >Report Pct runsfails test > 0123 3.1 1601 41 BasicDistributedZkTest.test > 0123 1.7 1495 28 MultiThreadedOCPTest.test > 0123 1.0 1587 14 RollingRestartTest.test > 0123 3.1 1653 55 > ScheduledTriggerIntegrationTest.testScheduledTrigger > 0123 1.3 1574 13 SystemCollectionCompatTest.testBackCompat > 0123 1.6 1571 14 TestPackages.testPluginLoading > 0123 0.3 1570 7 > TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast > > > =SuppressWarnings== > > In the attached report there’s a new section counting SuppressWarnings. For > the nonce, ignore it. Eventually, when all the warnings are fixed or > suppressed, I will be advocating for _not_ introducing new warnings at least > on Master. To encourage this, I want un-suppressed warnings to become > compile-time errors. > > That’ll tempt people to just add @SuppressWarnings, and I don’t think that’s > a proper fix, so the BadApple report will flag files that have more > @SuppressWarnings than they did last week and I’ll complain ;) There’ll be > exceptions of course... > > Yes, that flies counter to the zillion SuppressWarnings I’m putting in the > code right now, but I’m not about to try to fix on the order of 5,000 > warnings in our code all at once. that’s where the SuppressWarnings data is > coming from in the attached report, I expect the counts to increase until we > get clean compilations. Martin Fowler talks about rewriting working code for > no good reason being a bad idea in “Refactoring”... > > My goal currently is to get the compilations clean, stop getting worse, and > then we can make things better. Along about 2040, all the code that currently > has SuppressWarnings will have been rewritten and they’ll all be gone... > > == > Full report: > > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org -- - Noble Paul - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
BadApple report. It's worth reviewing the SuppressWarnings section even if you ignore the rest.
This week is a significant improvement. Short form: Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 68 failures Week: 1 had 113 failures Week: 2 had 103 failures Week: 3 had 102 failures Failures in Hoss' reports for the last 4 rollups. There were 273 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 3.1 1601 41 BasicDistributedZkTest.test 0123 1.7 1495 28 MultiThreadedOCPTest.test 0123 1.0 1587 14 RollingRestartTest.test 0123 3.1 1653 55 ScheduledTriggerIntegrationTest.testScheduledTrigger 0123 1.3 1574 13 SystemCollectionCompatTest.testBackCompat 0123 1.6 1571 14 TestPackages.testPluginLoading 0123 0.3 1570 7 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast =SuppressWarnings== In the attached report there’s a new section counting SuppressWarnings. For the nonce, ignore it. Eventually, when all the warnings are fixed or suppressed, I will be advocating for _not_ introducing new warnings at least on Master. To encourage this, I want un-suppressed warnings to become compile-time errors. That’ll tempt people to just add @SuppressWarnings, and I don’t think that’s a proper fix, so the BadApple report will flag files that have more @SuppressWarnings than they did last week and I’ll complain ;) There’ll be exceptions of course... Yes, that flies counter to the zillion SuppressWarnings I’m putting in the code right now, but I’m not about to try to fix on the order of 5,000 warnings in our code all at once. that’s where the SuppressWarnings data is coming from in the attached report, I expect the counts to increase until we get clean compilations. Martin Fowler talks about rewriting working code for no good reason being a bad idea in “Refactoring”... My goal currently is to get the compilations clean, stop getting worse, and then we can make things better. Along about 2040, all the code that currently has SuppressWarnings will have been rewritten and they’ll all be gone... == Full report: DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate SuppressWarnings count: last week: 1,130, this week: 1,226, delta 96 Suppress count increase in: solr/core/src/java/org/apache/solr/cloud/autoscaling/AutoScaling.java. Was: 0, now: 2 Suppress count increase in: solr/core/src/java/org/apache/solr/cloud/autoscaling/AutoScalingHandler.java. Was: 0, now: 10 Suppress count increase in: solr/core/src/java/org/apache/solr/cloud/autoscaling/ComputePlanAction.java. Was: 0, now: 7 Suppress count increase in: solr/core/src/java/org/apache/solr/cloud/autoscaling/ExecutePlanAction.java. Was: 0, now: 2 Suppress count increase in: solr/core/src/java/org/apache/solr/cloud/autoscaling/InactiveShardPlanAction.java. Was: 0, now: 1 Suppress count increase in: solr/core/src/java/org/apache/solr/cloud/autoscaling/IndexSizeTrigger.java. Was: 0, now: 2 Suppress count increase in: solr/core/src/java/org/apache/solr/cloud/autoscaling/MetricTrigger.java. Was: 0, now: 1 Suppress count increase in: solr/core/src/java/org/apache/solr/cloud/autoscaling/NodeAddedTrigger.java. Was: 0, now: 2 Suppress count increase in: solr/core/src/java/org/apache/solr/cloud/autoscaling/NodeLostTrigger.java. Was: 0, now: 2 Suppress count increase in: solr/core/src/java/org/apache/solr/cloud/autoscaling/ScheduledTriggers.java. Was: 0, now: 3 Suppress count increase in: solr/core/src/java/org/apache/solr/cloud/autoscaling/SearchRateTrigger.java. Was: 0, now: 5 Suppress count increase in: solr/core/src/java/org/apache/solr/cloud/autoscaling/SystemLogListener.java. Was: 0,
Re: BadApple report
> Hoss’s rollups are here: > http://fucit.org/solr-jenkins-reports/failure-report.html which show the > rates, but not where they came from. If I click on a particular test entry on "failure-report.html", I'm presented with dialog with links for each failure. Clicking that link takes me to a file listing page (e.g. http://fucit.org/solr-jenkins-reports/job-data/apache/Lucene-Solr-Tests-8.x/1569/), with Jenkins logs, etc. for that particular failure. Notably, it also has a file called "url.txt" with a link to the actual failure in Jenkins (e.g. http://fucit.org/solr-jenkins-reports/job-data/apache/Lucene-Solr-Tests-8.x/1569/url.txt). Just mentioning what I've seen with a few I've clicked on. The rollups might not have that for all failures, or for all different source-Jenkins. Just wanted to mention that you can get back to the Jenkins job in at least _some_ cases with a bit of clicking. On Mon, May 25, 2020 at 1:27 PM Ilan Ginzburg wrote: > > Thanks that helps. I'll try to have a look at some of the failures related to > areas I know. > > Ilan > > On Mon, May 25, 2020 at 7:07 PM Erick Erickson > wrote: >> >> Ilan: >> >> That’s, unfortunately, not an easy question. Hoss’s rollups are here: >> http://fucit.org/solr-jenkins-reports/failure-report.html which show the >> rates, but not where they came from. >> >> Here’s an example of a failure from Jenkins, if you follow the link you can >> see the full output, (click “console output”, then “full log”): >> https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-Linux/3181/. I usually see >> the individual ones go by by subscribing to “bui...@lucene.apache.org”. >> >> Otherwise, what I often do is use Mark Miller’s “beasting” script to see if >> I can get it to reproduce locally and go from there: >> >> https://gist.github.com/markrmiller/dbdb792216dc98b018ad >> >> It’s all complicated by the fact that the failures are intermittent. >> >> Best, >> Erick >> >> > On May 25, 2020, at 11:22 AM, Ilan Ginzburg wrote: >> > >> > Where are the test failure details? >> > >> > On Mon, May 25, 2020 at 4:47 PM Erick Erickson >> > wrote: >> > Here’s the summary: >> > >> > Raw fail count by week totals, most recent week first (corresponds to >> > bits): >> > Week: 0 had 113 failures >> > Week: 1 had 103 failures >> > Week: 2 had 102 failures >> > Week: 3 had 343 failures >> > >> > >> > Failures in Hoss' reports for the last 4 rollups. >> > >> > There were 511 unannotated tests that failed in Hoss' rollups. Ordered by >> > the date I downloaded the rollup file, newest->oldest. See above for the >> > dates the files were collected >> > These tests were NOT BadApple'd or AwaitsFix'd >> > >> > Failures in the last 4 reports.. >> >Report Pct runsfails test >> > 0123 0.7 1593 40 BasicDistributedZkTest.test >> > 0123 2.1 1518 28 MultiThreadedOCPTest.test >> > 0123 0.7 1613 14 RollingRestartTest.test >> > 0123 7.1 1635 44 >> > ScheduledTriggerIntegrationTest.testScheduledTrigger >> > 0123 2.4 1614 17 >> > SearchRateTriggerTest.testWaitForElapsed >> > 0123 0.2 1614 6 >> > ShardSplitTest.testSplitShardWithRuleLink >> > 0123 0.5 1577 5 >> > SolrCloudReportersTest.testExplicitConfiguration >> > 0123 0.7 1560 19 TestInPlaceUpdatesDistrib.test >> > 0123 1.0 1566 17 TestPackages.testPluginLoading >> > 0123 0.8 1598 7 >> > TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast >> > 0123 0.7 1598 8 TestSimScenario.testAutoAddReplicas >> > >> > >> > >> > Full report: >> > >> > - >> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> > For additional commands, e-mail: dev-h...@lucene.apache.org >> >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: BadApple report
Thanks that helps. I'll try to have a look at some of the failures related to areas I know. Ilan On Mon, May 25, 2020 at 7:07 PM Erick Erickson wrote: > Ilan: > > That’s, unfortunately, not an easy question. Hoss’s rollups are here: > http://fucit.org/solr-jenkins-reports/failure-report.html which show the > rates, but not where they came from. > > Here’s an example of a failure from Jenkins, if you follow the link you > can see the full output, (click “console output”, then “full log”): > https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-Linux/3181/. I usually > see the individual ones go by by subscribing to “bui...@lucene.apache.org > ”. > > Otherwise, what I often do is use Mark Miller’s “beasting” script to see > if I can get it to reproduce locally and go from there: > > https://gist.github.com/markrmiller/dbdb792216dc98b018ad > > It’s all complicated by the fact that the failures are intermittent. > > Best, > Erick > > > On May 25, 2020, at 11:22 AM, Ilan Ginzburg wrote: > > > > Where are the test failure details? > > > > On Mon, May 25, 2020 at 4:47 PM Erick Erickson > wrote: > > Here’s the summary: > > > > Raw fail count by week totals, most recent week first (corresponds to > bits): > > Week: 0 had 113 failures > > Week: 1 had 103 failures > > Week: 2 had 102 failures > > Week: 3 had 343 failures > > > > > > Failures in Hoss' reports for the last 4 rollups. > > > > There were 511 unannotated tests that failed in Hoss' rollups. Ordered > by the date I downloaded the rollup file, newest->oldest. See above for the > dates the files were collected > > These tests were NOT BadApple'd or AwaitsFix'd > > > > Failures in the last 4 reports.. > >Report Pct runsfails test > > 0123 0.7 1593 40 BasicDistributedZkTest.test > > 0123 2.1 1518 28 MultiThreadedOCPTest.test > > 0123 0.7 1613 14 RollingRestartTest.test > > 0123 7.1 1635 44 > ScheduledTriggerIntegrationTest.testScheduledTrigger > > 0123 2.4 1614 17 > SearchRateTriggerTest.testWaitForElapsed > > 0123 0.2 1614 6 > ShardSplitTest.testSplitShardWithRuleLink > > 0123 0.5 1577 5 > SolrCloudReportersTest.testExplicitConfiguration > > 0123 0.7 1560 19 TestInPlaceUpdatesDistrib.test > > 0123 1.0 1566 17 TestPackages.testPluginLoading > > 0123 0.8 1598 7 > TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast > > 0123 0.7 1598 8 TestSimScenario.testAutoAddReplicas > > > > > > > > Full report: > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
Re: BadApple report
Ilan: That’s, unfortunately, not an easy question. Hoss’s rollups are here: http://fucit.org/solr-jenkins-reports/failure-report.html which show the rates, but not where they came from. Here’s an example of a failure from Jenkins, if you follow the link you can see the full output, (click “console output”, then “full log”): https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-Linux/3181/. I usually see the individual ones go by by subscribing to “bui...@lucene.apache.org”. Otherwise, what I often do is use Mark Miller’s “beasting” script to see if I can get it to reproduce locally and go from there: https://gist.github.com/markrmiller/dbdb792216dc98b018ad It’s all complicated by the fact that the failures are intermittent. Best, Erick > On May 25, 2020, at 11:22 AM, Ilan Ginzburg wrote: > > Where are the test failure details? > > On Mon, May 25, 2020 at 4:47 PM Erick Erickson > wrote: > Here’s the summary: > > Raw fail count by week totals, most recent week first (corresponds to bits): > Week: 0 had 113 failures > Week: 1 had 103 failures > Week: 2 had 102 failures > Week: 3 had 343 failures > > > Failures in Hoss' reports for the last 4 rollups. > > There were 511 unannotated tests that failed in Hoss' rollups. Ordered by the > date I downloaded the rollup file, newest->oldest. See above for the dates > the files were collected > These tests were NOT BadApple'd or AwaitsFix'd > > Failures in the last 4 reports.. >Report Pct runsfails test > 0123 0.7 1593 40 BasicDistributedZkTest.test > 0123 2.1 1518 28 MultiThreadedOCPTest.test > 0123 0.7 1613 14 RollingRestartTest.test > 0123 7.1 1635 44 > ScheduledTriggerIntegrationTest.testScheduledTrigger > 0123 2.4 1614 17 SearchRateTriggerTest.testWaitForElapsed > 0123 0.2 1614 6 ShardSplitTest.testSplitShardWithRuleLink > 0123 0.5 1577 5 > SolrCloudReportersTest.testExplicitConfiguration > 0123 0.7 1560 19 TestInPlaceUpdatesDistrib.test > 0123 1.0 1566 17 TestPackages.testPluginLoading > 0123 0.8 1598 7 > TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast > 0123 0.7 1598 8 TestSimScenario.testAutoAddReplicas > > > > Full report: > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: BadApple report
Where are the test failure details? On Mon, May 25, 2020 at 4:47 PM Erick Erickson wrote: > Here’s the summary: > > Raw fail count by week totals, most recent week first (corresponds to > bits): > Week: 0 had 113 failures > Week: 1 had 103 failures > Week: 2 had 102 failures > Week: 3 had 343 failures > > > Failures in Hoss' reports for the last 4 rollups. > > There were 511 unannotated tests that failed in Hoss' rollups. Ordered by > the date I downloaded the rollup file, newest->oldest. See above for the > dates the files were collected > These tests were NOT BadApple'd or AwaitsFix'd > > Failures in the last 4 reports.. >Report Pct runsfails test > 0123 0.7 1593 40 BasicDistributedZkTest.test > 0123 2.1 1518 28 MultiThreadedOCPTest.test > 0123 0.7 1613 14 RollingRestartTest.test > 0123 7.1 1635 44 > ScheduledTriggerIntegrationTest.testScheduledTrigger > 0123 2.4 1614 17 > SearchRateTriggerTest.testWaitForElapsed > 0123 0.2 1614 6 > ShardSplitTest.testSplitShardWithRuleLink > 0123 0.5 1577 5 > SolrCloudReportersTest.testExplicitConfiguration > 0123 0.7 1560 19 TestInPlaceUpdatesDistrib.test > 0123 1.0 1566 17 TestPackages.testPluginLoading > 0123 0.8 1598 7 > TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast > 0123 0.7 1598 8 TestSimScenario.testAutoAddReplicas > > > > Full report: > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org
BadApple report
Here’s the summary: Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 113 failures Week: 1 had 103 failures Week: 2 had 102 failures Week: 3 had 343 failures Failures in Hoss' reports for the last 4 rollups. There were 511 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 0.7 1593 40 BasicDistributedZkTest.test 0123 2.1 1518 28 MultiThreadedOCPTest.test 0123 0.7 1613 14 RollingRestartTest.test 0123 7.1 1635 44 ScheduledTriggerIntegrationTest.testScheduledTrigger 0123 2.4 1614 17 SearchRateTriggerTest.testWaitForElapsed 0123 0.2 1614 6 ShardSplitTest.testSplitShardWithRuleLink 0123 0.5 1577 5 SolrCloudReportersTest.testExplicitConfiguration 0123 0.7 1560 19 TestInPlaceUpdatesDistrib.test 0123 1.0 1566 17 TestPackages.testPluginLoading 0123 0.8 1598 7 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0123 0.7 1598 8 TestSimScenario.testAutoAddReplicas Full report: DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate SuppressWarnings count: last week: 1,130, this week: 1,130, delta 0 Processing file (History bit 3): HOSS-2020-05-25.csv Processing file (History bit 2): HOSS-2020-05-18.csv Processing file (History bit 1): HOSS-2020-05-11.csv Processing file (History bit 0): HOSS-2020-05-04.csv Number of AwaitsFix: 43 Number of BadApples: 4 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 113 failures Week: 1 had 103 failures Week: 2 had 102 failures Week: 3 had 343 failures Failures in Hoss' reports for the last 4 rollups. There were 511 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 0.7 1593 40 BasicDistributedZkTest.test 0123 2.1 1518 28 MultiThreadedOCPTest.test 0123 0.7 1613 14 RollingRestartTest.test 0123 7.1 1635 44 ScheduledTriggerIntegrationTest.testScheduledTrigger 0123 2.4 1614 17 SearchRateTriggerTest.testWaitForElapsed 0123 0.2 1614 6 ShardSplitTest.testSplitShardWithRuleLink 0123 0.5 1577 5 SolrCloudReportersTest.testExplicitConfiguration 0123 0.7 1560 19 TestInPlaceUpdatesDistrib.test 0123 1.0 1566 17 TestPackages.testPluginLoading 0123 0.8 1598 7 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0123 0.7 1598 8 TestSimScenario.testAutoAddReplicas Failures over the last 4 weeks, but not every week. Ordered most-recent first: 0120.2 1203 3 ShardSplitTest.testSplitShardWithRule 0120.5 1196 8 SystemCollectionCompatTest.testBackCompat 01 3 0.3 1217 9 DeleteReplicaTest.deleteReplicaAndVerifyDirectoryCleanup 01 3 0.3 1220 6 LeaderFailoverAfterPartitionTest.test 01 3 0.3 1210 6
BadApple report
Short form: Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 103 failures Week: 1 had 102 failures Week: 2 had 343 failures Week: 3 had 86 failures Failures in Hoss' reports for the last 4 rollups. There were 493 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 2.8 1471 23 MultiThreadedOCPTest.test 0123 1.0 1578 13 RollingRestartTest.test 0123 2.4 1519 13 ScheduledTriggerIntegrationTest.testScheduledTrigger 0123 0.2 1569 8 SearchRateTriggerTest.testWaitForElapsed 0123 2.9 1493 18 TestInPlaceUpdatesDistrib.test 0123 0.5 1503 15 TestPackages.testPluginLoading We seem to have gotten past the big bump caused by the disk full situation. Still, we’re up a number of tests since 3 weeks ago. Full report attached: DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate SuppressWarnings count: last week: 965, this week: 965, delta 0 Processing file (History bit 3): HOSS-2020-05-18.csv Processing file (History bit 2): HOSS-2020-05-11.csv Processing file (History bit 1): HOSS-2020-05-04.csv Processing file (History bit 0): HOSS-2020-04-27.csv Number of AwaitsFix: 42 Number of BadApples: 4 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 103 failures Week: 1 had 102 failures Week: 2 had 343 failures Week: 3 had 86 failures Failures in Hoss' reports for the last 4 rollups. There were 493 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 2.8 1471 23 MultiThreadedOCPTest.test 0123 1.0 1578 13 RollingRestartTest.test 0123 2.4 1519 13 ScheduledTriggerIntegrationTest.testScheduledTrigger 0123 0.2 1569 8 SearchRateTriggerTest.testWaitForElapsed 0123 2.9 1493 18 TestInPlaceUpdatesDistrib.test 0123 0.5 1503 15 TestPackages.testPluginLoading Failures over the last 4 weeks, but not every week. Ordered most-recent first: 0122.7 1191 37 BasicDistributedZkTest.test 0120.2 1153 4 ComputePlanActionTest.testNodeAdded 0120.7 1204 10 HttpPartitionTest.test 0120.7 1183 13 HttpPartitionWithTlogReplicasTest.test 0120.5 1200 4 LeaderElectionIntegrationTest.testSimpleSliceLeaderElection 0120.2 1207 5 ShardSplitTest.testSplitShardWithRuleLink 0120.2 1178 3 SolrCloudReportersTest.testExplicitConfiguration 0120.5 1199 4 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0120.2 1187 5 TestSimScenario.testAutoAddReplicas 01 3 0.2 1132 7 SystemCollectionCompatTest.testBackCompat 01 0.7 792 5 DocValuesNotIndexedTest.testGroupingDVOnlySortFirst 01 0.2 798 2 FullSolrCloudDistribCmdsTest.testConcurrentIndexing 01 0.2 798 2
BadApple report
Largely ignore the fact that weeks 0 and 1 had so many failures, that was due to Jenkins running out of space, which bled over into the week0 report. This is the first one that reports the number of SuppressWarnings annotations that we can use as a baseline. If I start adding SuppressWarnings through the code as per my other e-mail, this number will increase drastically over the next while, but ignore it for now. ** SuppressWarnings count: last week: 973, this week: 973, delta 0 Processing file (History bit 3): HOSS-2020-05-11.csv Processing file (History bit 2): HOSS-2020-05-04.csv Processing file (History bit 1): HOSS-2020-04-27.csv Processing file (History bit 0): HOSS-2020-04-20.csv Number of AwaitsFix: 42 Number of BadApples: 4 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 102 failures Week: 1 had 343 failures Week: 2 had 86 failures Week: 3 had 78 failures Failures in Hoss' reports for the last 4 rollups. There were 484 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 0.5 1566 10 ConnectionManagerTest.testReconnectWhenZkDisappeared 0123 0.3 1556 10 ExecutePlanActionTest.testTaskTimeout 0123 0.8 1360 20 MultiThreadedOCPTest.test 0123 0.8 1566 10 RollingRestartTest.test 0123 0.3 1567 11 SearchRateTriggerTest.testWaitForElapsed 0123 0.5 1557 10 TestCryptoKeys.test 0123 0.8 1474 8 TestInPlaceUpdatesDistrib.test 0123 0.3 1582 13 TestIndexWriterDelete.testDeleteAllNoDeadLock 0123 0.5 1500 18 TestPackages.testPluginLoading DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate SuppressWarnings count: last week: 973, this week: 973, delta 0 Processing file (History bit 3): HOSS-2020-05-11.csv Processing file (History bit 2): HOSS-2020-05-04.csv Processing file (History bit 1): HOSS-2020-04-27.csv Processing file (History bit 0): HOSS-2020-04-20.csv Number of AwaitsFix: 42 Number of BadApples: 4 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 102 failures Week: 1 had 343 failures Week: 2 had 86 failures Week: 3 had 78 failures Failures in Hoss' reports for the last 4 rollups. There were 484 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 0.5 1566 10 ConnectionManagerTest.testReconnectWhenZkDisappeared 0123 0.3 1556 10 ExecutePlanActionTest.testTaskTimeout 0123 0.8 1360 20 MultiThreadedOCPTest.test 0123 0.8 1566 10 RollingRestartTest.test 0123 0.3 1567 11 SearchRateTriggerTest.testWaitForElapsed 0123 0.5 1557 10 TestCryptoKeys.test 0123 0.8 1474 8 TestInPlaceUpdatesDistrib.test 0123 0.3 1582 13 TestIndexWriterDelete.testDeleteAllNoDeadLock 0123 0.5 1500 18 TestPackages.testPluginLoading Failures over the last 4 weeks, but not every week. Ordered most-recent first: 0120.3 1094 3
Re: PLEASE READ! BadApple report. Last week was horrible!
Phew! Thanks for digging Erick, and for producing these BadApple reports. Mike McCandless http://blog.mikemccandless.com On Wed, May 6, 2020 at 7:59 AM Erick Erickson wrote: > OK, this morning things are back to normal. I think the disk space issue > was to blame because checking after Mike’s fix didn’t look like it > cured the problem. > > Thanks all! > > > On May 5, 2020, at 1:41 PM, Chris Hostetter > wrote: > > > > > > : And FWIW, I beasted one of the failing suites last night _without_ > > : Mike’s changes and didn’t get any failures so I can’t say anything > about > > : whether Mike’s changes helped or not. > > > > IIUC McCandless's failure only affects you if you use the "jenkins" test > > data file (the really big wikipedia dump) ... see the jira he mentioned > > for details. > > > > > > > > -Hoss > > http://www.lucidworks.com/ > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
Re: PLEASE READ! BadApple report. Last week was horrible!
OK, this morning things are back to normal. I think the disk space issue was to blame because checking after Mike’s fix didn’t look like it cured the problem. Thanks all! > On May 5, 2020, at 1:41 PM, Chris Hostetter wrote: > > > : And FWIW, I beasted one of the failing suites last night _without_ > : Mike’s changes and didn’t get any failures so I can’t say anything about > : whether Mike’s changes helped or not. > > IIUC McCandless's failure only affects you if you use the "jenkins" test > data file (the really big wikipedia dump) ... see the jira he mentioned > for details. > > > > -Hoss > http://www.lucidworks.com/ > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: PLEASE READ! BadApple report. Last week was horrible!
OK, thanks Chris. The 24 hour rollup still shows many failures in the several classes, I’ll check tomorrow to see if that’s a consequence of the disk full problem. > On May 5, 2020, at 1:41 PM, Chris Hostetter wrote: > > > : And FWIW, I beasted one of the failing suites last night _without_ > : Mike’s changes and didn’t get any failures so I can’t say anything about > : whether Mike’s changes helped or not. > > IIUC McCandless's failure only affects you if you use the "jenkins" test > data file (the really big wikipedia dump) ... see the jira he mentioned > for details. > > > > -Hoss > http://www.lucidworks.com/ > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: PLEASE READ! BadApple report. Last week was horrible!
: And FWIW, I beasted one of the failing suites last night _without_ : Mike’s changes and didn’t get any failures so I can’t say anything about : whether Mike’s changes helped or not. IIUC McCandless's failure only affects you if you use the "jenkins" test data file (the really big wikipedia dump) ... see the jira he mentioned for details. -Hoss http://www.lucidworks.com/ - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: PLEASE READ! BadApple report. Last week was horrible!
UNLOAD) [n:127.0.0.1:49613_solrx:replicaTypesTestColl_shard1_replica_p4 ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1f2a6e95: rootName = solr_49613, domain = solr.core.replicaTypesTestColl.shard1.replica_p4, service url = null, agent id = null] for registry solr.core.replicaTypesTestColl.shard1.replica_p4/com.codahale.metrics.MetricRegistry@2edb03e2 [junit4] 2> 33770 ERROR (indexFetcher-621-thread-1) [n:127.0.0.1:49612_solr ] o.a.s.h.ReplicationHandler Index fetch failed :java.lang.NullPointerException [junit4] 2>at org.apache.solr.handler.IndexFetcher.getLeaderReplica(IndexFetcher.java:709) [junit4] 2>at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:387) [junit4] 2>at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:351) [junit4] 2>at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:422) [junit4] 2>at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1208) [junit4] 2>at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [junit4] 2>at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [junit4] 2>at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [junit4] 2>at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [junit4] 2>at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [junit4] 2>at java.base/java.lang.Thread.run(Thread.java:834) [junit4] 2> > On May 5, 2020, at 4:33 AM, Uwe Schindler wrote: > > Hi, > > there was also a problem with the Windows Node. It ran out of disk space, > because some test seem to have filled up all of the disk. All followup builds > failed. I cleaned all Workspaces (8.x, master) and it freed 20 Gigabytes! > > Uwe > > - > Uwe Schindler > Achterdiek 19, D-28357 Bremen > https://www.thetaphi.de > eMail: u...@thetaphi.de > >> -Original Message- >> From: Erick Erickson >> Sent: Monday, May 4, 2020 1:54 PM >> To: dev@lucene.apache.org >> Subject: PLEASE READ! BadApple report. Last week was horrible! >> >> I don’t know whether we had some temporary glitch that broke lots of tests >> and they’ve been fixed or we had a major regression, but this needs to be >> addressed ASAP if they’re still failing. See everything below the line "ALL >> OF >> THE TESTS BELOW HERE HAVE ONLY FAILED IN THE LAST WEEK!” in this e-mail. >> I’ll raise a JIRA if we can’t get some traction quickly here. >> >> Hey, stuff happens. there’s no problem with tests going totally weird for a >> while. If you can say “Oh, yeah, all those failures for class XYZ are >> probably >> fixed” that’s fine. >> >> Gosh-a-rooni, I hope my logging changes aren’t the culprit (gulp)…. >> >> Hoss’ rolllup for the last 24 hours is not encouraging in terms of the >> problem already being fixed. There are lots of failures in some >> classes, notably: >> >> CloudHttp2SolrClientTest >> CollectionsAPIDistributedZkTest >> DeleteReplicaTest >> TestDocCollectionWatcher >> >> Unfortunately, the failure rate is not very high so reliably >> reproducing is hard. >> >> I’ve reproduced the last week’s failure in this e-mail, full >> report attached. >> >> Here’s Hoss’ rollup: >> http://fucit.org/solr-jenkins-reports/failure-report.html >> >> Usual synopsis: >> >> Raw fail count by week totals, most recent week first (corresponds to bits): >> Week: 0 had 343 failures >> Week: 1 had 86 failures >> Week: 2 had 78 failures >> Week: 3 had 117 failures >> >> >> Failures in Hoss' reports for the last 4 rollups. >> >> There were 497 unannotated tests that failed in Hoss' rollups. Ordered by the >> date I downloaded the rollup file, newest->oldest. See above for the dates >> the >> files were collected >> These tests were NOT BadApple'd or AwaitsFix’d >> >> Failures in the last 4 reports.. >> Report Pct runsfails test >> 0123 0.7 1617 11 >> ConnectionManagerTest.testReconnectWhenZkDisappeared >> 0123 1.5 1606 12 ExecutePlanActionTest.testTaskTimeout >> 0123 1.6 1320 19 MultiThreadedOCPTest.test >> 0123 1.0
RE: PLEASE READ! BadApple report. Last week was horrible!
Hi, there was also a problem with the Windows Node. It ran out of disk space, because some test seem to have filled up all of the disk. All followup builds failed. I cleaned all Workspaces (8.x, master) and it freed 20 Gigabytes! Uwe - Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Erick Erickson > Sent: Monday, May 4, 2020 1:54 PM > To: dev@lucene.apache.org > Subject: PLEASE READ! BadApple report. Last week was horrible! > > I don’t know whether we had some temporary glitch that broke lots of tests > and they’ve been fixed or we had a major regression, but this needs to be > addressed ASAP if they’re still failing. See everything below the line "ALL OF > THE TESTS BELOW HERE HAVE ONLY FAILED IN THE LAST WEEK!” in this e-mail. > I’ll raise a JIRA if we can’t get some traction quickly here. > > Hey, stuff happens. there’s no problem with tests going totally weird for a > while. If you can say “Oh, yeah, all those failures for class XYZ are probably > fixed” that’s fine. > > Gosh-a-rooni, I hope my logging changes aren’t the culprit (gulp)…. > > Hoss’ rolllup for the last 24 hours is not encouraging in terms of the > problem already being fixed. There are lots of failures in some > classes, notably: > > CloudHttp2SolrClientTest > CollectionsAPIDistributedZkTest > DeleteReplicaTest > TestDocCollectionWatcher > > Unfortunately, the failure rate is not very high so reliably > reproducing is hard. > > I’ve reproduced the last week’s failure in this e-mail, full > report attached. > > Here’s Hoss’ rollup: > http://fucit.org/solr-jenkins-reports/failure-report.html > > Usual synopsis: > > Raw fail count by week totals, most recent week first (corresponds to bits): > Week: 0 had 343 failures > Week: 1 had 86 failures > Week: 2 had 78 failures > Week: 3 had 117 failures > > > Failures in Hoss' reports for the last 4 rollups. > > There were 497 unannotated tests that failed in Hoss' rollups. Ordered by the > date I downloaded the rollup file, newest->oldest. See above for the dates the > files were collected > These tests were NOT BadApple'd or AwaitsFix’d > > Failures in the last 4 reports.. >Report Pct runsfails test > 0123 0.7 1617 11 > ConnectionManagerTest.testReconnectWhenZkDisappeared > 0123 1.5 1606 12 ExecutePlanActionTest.testTaskTimeout > 0123 1.6 1320 19 MultiThreadedOCPTest.test > 0123 1.0 1620 13 RollingRestartTest.test > 0123 1.2 1617 12 SearchRateTriggerTest.testWaitForElapsed > 0123 3.8 119 7 ShardSplitTest.testSplitWithChaosMonkey > 0123 0.3 1519 7 TestInPlaceUpdatesDistrib.test > 0123 0.7 1629 14 > TestIndexWriterDelete.testDeleteAllNoDeadLock > 0123 2.4 1548 18 TestPackages.testPluginLoading > 0123 0.3 1587 4 UnloadDistributedZkTest.test > > > FAILURES IN THE LAST WEEK (343!) > Look particularly at the ones with only a zero in the “Report” column, those > are > failures that were _not_ in the previous 3 week’s rollups. > >Report Pct runsfails test > 0120.5 1165 4 CustomHighlightComponentTest.test > 0121.0 1168 6 > NodeMarkersRegistrationTest.testNodeMarkersRegistration > 0121.0 1170 8 TestCryptoKeys.test > 01 3 0.7 1233 11 LeaderFailoverAfterPartitionTest.test > 01 3 63.2 102 39 StressHdfsTest.test > 01 0.3 709 2 > ScheduledTriggerIntegrationTest.testScheduledTrigger > 01 0.2 768 2 ShardRoutingTest.test > 01 2.6 807 22 TestAllFilesHaveChecksumFooter.test > 01 2.6 808 22 TestAllFilesHaveCodecHeader.test > 01 0.2 769 2 TestCloudSchemaless.test > 01 0.2 769 2 TestDynamicLoading.testDynamicLoading > 01 0.3 707 2 > TestDynamicLoadingUrl.testDynamicLoadingUrl > 01 0.5 767 4 TestPointFields.testFloatPointStats > 0127.1 83 19 TestSQLHandler.doTest > 01 0.2 794 12 TestSameScoresWithThreads.test > 01 2.6 806 22 TestShardSearching.testSimple > 01 0.5 726 4 TestSimScenario.testSplitShard > 01 1.1 726 7 TestSimScenario.testSuggestions > 01 0.3 771 2 TestWithCollection.testAddRe
Re: PLEASE READ! BadApple report. Last week was horrible!
Mike: I saw the push. Hoss’ rollups go for “the last 24 hours”, so it’ll be Tuesday evening before things have had a chance to work their way through, I’ll look tomorrow. Meanwhile I’m beasting one of the failing test suites (without the change) and 280 iterations so far and no failures. That said, the failure rate was < 1% so it’s not conclusive. Only another 720 runs to go before I pull the latest changes and try again… ;) > On May 4, 2020, at 1:33 PM, Michael McCandless > wrote: > > Hi Erick, > > OK I pushed a fix! See if it decreases the failure rate for those newly bad > apples? > > Sorry and thanks :) > > Mike McCandless > > http://blog.mikemccandless.com > > > On Mon, May 4, 2020 at 1:06 PM Erick Erickson wrote: > Mike: > > I have no idea. Hoss’ rollups don’t link back to builds, they > just aggregate the results. > > Not a huge deal if it’s something like this of course. Let’s just > say I’ve had my share or “moments” ;). > > And unfortunately, the test failures are pretty rare on a > percentage basis, so it’s hard to tell. > > I’m watching LUCENE-9191 and I’ll look back at Hoss’ rollups > a day after you push it and see if the failures disappear. > > It’ll take a while for the fixes to roll through all the reporting. > > Tell you what. I’ll try beasting one of the classes that fails a lot and then > try it again after you push LUCENE-9191 and we’ll go from there. > > Thanks for getting into this so promptly! > > Erick > > > On May 4, 2020, at 9:10 AM, Michael McCandless > > wrote: > > > > Hi Erick, > > > > It's possible this was the root cause of many of the failures: > > https://issues.apache.org/jira/browse/LUCENE-9191 > > > > Do these transient failures look something like this? > > > >[junit4]> Throwable #1: java.nio.charset.MalformedInputException: > > Input length = 1 > >[junit4]>at > > __randomizedtesting.SeedInfo.seed([172C6414BE5E2A2C:E5829DFC005A1F0]:0) > >[junit4]>at > > java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:274) > >[junit4]>at > > java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339) > >[junit4]>at > > java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) > >[junit4]>at > > java.base/java.io.InputStreamReader.read(InputStreamReader.java:185) > >[junit4]>at > > java.base/java.io.BufferedReader.fill(BufferedReader.java:161) > >[junit4]>at > > java.base/java.io.BufferedReader.readLine(BufferedReader.java:326) > >[junit4]>at > > java.base/java.io.BufferedReader.readLine(BufferedReader.java:392) > >[junit4]>at > > org.apache.lucene.util.LineFileDocs.open(LineFileDocs.java:175) > >[junit4]>at > > org.apache.lucene.util.LineFileDocs.(LineFileDocs.java:65) > >[junit4]>at > > org.apache.lucene.util.LineFileDocs.(LineFileDocs.java:69) > > > > > > If so, then it is likely the root cause ... I'm working on a fix. Sorry! > > > > Mike McCandless > > > > http://blog.mikemccandless.com > > > > > > On Mon, May 4, 2020 at 7:54 AM Erick Erickson > > wrote: > > I don’t know whether we had some temporary glitch that broke lots of tests > > and they’ve been fixed or we had a major regression, but this needs to be > > addressed ASAP if they’re still failing. See everything below the line "ALL > > OF THE TESTS BELOW HERE HAVE ONLY FAILED IN THE LAST WEEK!” in this e-mail. > > I’ll raise a JIRA if we can’t get some traction quickly here. > > > > Hey, stuff happens. there’s no problem with tests going totally weird for a > > while. If you can say “Oh, yeah, all those failures for class XYZ are > > probably fixed” that’s fine. > > > > Gosh-a-rooni, I hope my logging changes aren’t the culprit (gulp)…. > > > > Hoss’ rolllup for the last 24 hours is not encouraging in terms of the > > problem already being fixed. There are lots of failures in some > > classes, notably: > > > > CloudHttp2SolrClientTest > > CollectionsAPIDistributedZkTest > > DeleteReplicaTest > > TestDocCollectionWatcher > > > > Unfortunately, the failure rate is not very high so reliably > > reproducing is hard. > > > > I’ve reproduced the last week’s failure in this e-mail, full > > report attached. > > > > Here’s Hoss’ rollup: > > http://fucit.org/solr-jenkins-reports/failure-report.html > > > > Usual synopsis: > > > > Raw fail count by week totals, most recent week first (corresponds to bits): > > Week: 0 had 343 failures > > Week: 1 had 86 failures > > Week: 2 had 78 failures > > Week: 3 had 117 failures > > > > > > Failures in Hoss' reports for the last 4 rollups. > > > > There were 497 unannotated tests that failed in Hoss' rollups. Ordered by > > the date I downloaded the rollup file, newest->oldest. See above for the > > dates the files were collected > > These tests were NOT BadApple'd or AwaitsFix’d >
Re: PLEASE READ! BadApple report. Last week was horrible!
Hi Erick, OK I pushed a fix! See if it decreases the failure rate for those newly bad apples? Sorry and thanks :) Mike McCandless http://blog.mikemccandless.com On Mon, May 4, 2020 at 1:06 PM Erick Erickson wrote: > Mike: > > I have no idea. Hoss’ rollups don’t link back to builds, they > just aggregate the results. > > Not a huge deal if it’s something like this of course. Let’s just > say I’ve had my share or “moments” ;). > > And unfortunately, the test failures are pretty rare on a > percentage basis, so it’s hard to tell. > > I’m watching LUCENE-9191 and I’ll look back at Hoss’ rollups > a day after you push it and see if the failures disappear. > > It’ll take a while for the fixes to roll through all the reporting. > > Tell you what. I’ll try beasting one of the classes that fails a lot and > then > try it again after you push LUCENE-9191 and we’ll go from there. > > Thanks for getting into this so promptly! > > Erick > > > On May 4, 2020, at 9:10 AM, Michael McCandless < > luc...@mikemccandless.com> wrote: > > > > Hi Erick, > > > > It's possible this was the root cause of many of the failures: > https://issues.apache.org/jira/browse/LUCENE-9191 > > > > Do these transient failures look something like this? > > > >[junit4]> Throwable #1: java.nio.charset.MalformedInputException: > Input length = 1 > >[junit4]>at > __randomizedtesting.SeedInfo.seed([172C6414BE5E2A2C:E5829DFC005A1F0]:0) > >[junit4]>at > java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:274) > >[junit4]>at > java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339) > >[junit4]>at > java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) > >[junit4]>at java.base/java.io > .InputStreamReader.read(InputStreamReader.java:185) > >[junit4]>at java.base/java.io > .BufferedReader.fill(BufferedReader.java:161) > >[junit4]>at java.base/java.io > .BufferedReader.readLine(BufferedReader.java:326) > >[junit4]>at java.base/java.io > .BufferedReader.readLine(BufferedReader.java:392) > >[junit4]>at > org.apache.lucene.util.LineFileDocs.open(LineFileDocs.java:175) > >[junit4]>at > org.apache.lucene.util.LineFileDocs.(LineFileDocs.java:65) > >[junit4]>at > org.apache.lucene.util.LineFileDocs.(LineFileDocs.java:69) > > > > > > If so, then it is likely the root cause ... I'm working on a fix. Sorry! > > > > Mike McCandless > > > > http://blog.mikemccandless.com > > > > > > On Mon, May 4, 2020 at 7:54 AM Erick Erickson > wrote: > > I don’t know whether we had some temporary glitch that broke lots of > tests and they’ve been fixed or we had a major regression, but this needs > to be addressed ASAP if they’re still failing. See everything below the > line "ALL OF THE TESTS BELOW HERE HAVE ONLY FAILED IN THE LAST WEEK!” in > this e-mail. I’ll raise a JIRA if we can’t get some traction quickly here. > > > > Hey, stuff happens. there’s no problem with tests going totally weird > for a while. If you can say “Oh, yeah, all those failures for class XYZ are > probably fixed” that’s fine. > > > > Gosh-a-rooni, I hope my logging changes aren’t the culprit (gulp)…. > > > > Hoss’ rolllup for the last 24 hours is not encouraging in terms of the > > problem already being fixed. There are lots of failures in some > > classes, notably: > > > > CloudHttp2SolrClientTest > > CollectionsAPIDistributedZkTest > > DeleteReplicaTest > > TestDocCollectionWatcher > > > > Unfortunately, the failure rate is not very high so reliably > > reproducing is hard. > > > > I’ve reproduced the last week’s failure in this e-mail, full > > report attached. > > > > Here’s Hoss’ rollup: > > http://fucit.org/solr-jenkins-reports/failure-report.html > > > > Usual synopsis: > > > > Raw fail count by week totals, most recent week first (corresponds to > bits): > > Week: 0 had 343 failures > > Week: 1 had 86 failures > > Week: 2 had 78 failures > > Week: 3 had 117 failures > > > > > > Failures in Hoss' reports for the last 4 rollups. > > > > There were 497 unannotated tests that failed in Hoss' rollups. Ordered > by the date I downloaded the rollup file, newest->oldest. See above for the > dates the files were collected > > These tests were NOT BadApple'd or AwaitsFix’d > > > > Failures in the last 4 reports.. > >Report Pct runsfails test > > 0123 0.7 1617 11 > ConnectionManagerTest.testReconnectWhenZkDisappeared > > 0123 1.5 1606 12 > ExecutePlanActionTest.testTaskTimeout > > 0123 1.6 1320 19 MultiThreadedOCPTest.test > > 0123 1.0 1620 13 RollingRestartTest.test > > 0123 1.2 1617 12 > SearchRateTriggerTest.testWaitForElapsed > > 0123 3.8 119 7 > ShardSplitTest.testSplitWithChaosMonkey > > 0123 0.3 1519 7
Re: PLEASE READ! BadApple report. Last week was horrible!
Mike: I have no idea. Hoss’ rollups don’t link back to builds, they just aggregate the results. Not a huge deal if it’s something like this of course. Let’s just say I’ve had my share or “moments” ;). And unfortunately, the test failures are pretty rare on a percentage basis, so it’s hard to tell. I’m watching LUCENE-9191 and I’ll look back at Hoss’ rollups a day after you push it and see if the failures disappear. It’ll take a while for the fixes to roll through all the reporting. Tell you what. I’ll try beasting one of the classes that fails a lot and then try it again after you push LUCENE-9191 and we’ll go from there. Thanks for getting into this so promptly! Erick > On May 4, 2020, at 9:10 AM, Michael McCandless > wrote: > > Hi Erick, > > It's possible this was the root cause of many of the failures: > https://issues.apache.org/jira/browse/LUCENE-9191 > > Do these transient failures look something like this? > >[junit4]> Throwable #1: java.nio.charset.MalformedInputException: > Input length = 1 >[junit4]>at > __randomizedtesting.SeedInfo.seed([172C6414BE5E2A2C:E5829DFC005A1F0]:0) >[junit4]>at > java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:274) >[junit4]>at > java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339) >[junit4]>at > java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) >[junit4]>at > java.base/java.io.InputStreamReader.read(InputStreamReader.java:185) >[junit4]>at > java.base/java.io.BufferedReader.fill(BufferedReader.java:161) >[junit4]>at > java.base/java.io.BufferedReader.readLine(BufferedReader.java:326) >[junit4]>at > java.base/java.io.BufferedReader.readLine(BufferedReader.java:392) >[junit4]>at > org.apache.lucene.util.LineFileDocs.open(LineFileDocs.java:175) >[junit4]>at > org.apache.lucene.util.LineFileDocs.(LineFileDocs.java:65) >[junit4]>at > org.apache.lucene.util.LineFileDocs.(LineFileDocs.java:69) > > > If so, then it is likely the root cause ... I'm working on a fix. Sorry! > > Mike McCandless > > http://blog.mikemccandless.com > > > On Mon, May 4, 2020 at 7:54 AM Erick Erickson wrote: > I don’t know whether we had some temporary glitch that broke lots of tests > and they’ve been fixed or we had a major regression, but this needs to be > addressed ASAP if they’re still failing. See everything below the line "ALL > OF THE TESTS BELOW HERE HAVE ONLY FAILED IN THE LAST WEEK!” in this e-mail. > I’ll raise a JIRA if we can’t get some traction quickly here. > > Hey, stuff happens. there’s no problem with tests going totally weird for a > while. If you can say “Oh, yeah, all those failures for class XYZ are > probably fixed” that’s fine. > > Gosh-a-rooni, I hope my logging changes aren’t the culprit (gulp)…. > > Hoss’ rolllup for the last 24 hours is not encouraging in terms of the > problem already being fixed. There are lots of failures in some > classes, notably: > > CloudHttp2SolrClientTest > CollectionsAPIDistributedZkTest > DeleteReplicaTest > TestDocCollectionWatcher > > Unfortunately, the failure rate is not very high so reliably > reproducing is hard. > > I’ve reproduced the last week’s failure in this e-mail, full > report attached. > > Here’s Hoss’ rollup: > http://fucit.org/solr-jenkins-reports/failure-report.html > > Usual synopsis: > > Raw fail count by week totals, most recent week first (corresponds to bits): > Week: 0 had 343 failures > Week: 1 had 86 failures > Week: 2 had 78 failures > Week: 3 had 117 failures > > > Failures in Hoss' reports for the last 4 rollups. > > There were 497 unannotated tests that failed in Hoss' rollups. Ordered by the > date I downloaded the rollup file, newest->oldest. See above for the dates > the files were collected > These tests were NOT BadApple'd or AwaitsFix’d > > Failures in the last 4 reports.. >Report Pct runsfails test > 0123 0.7 1617 11 > ConnectionManagerTest.testReconnectWhenZkDisappeared > 0123 1.5 1606 12 ExecutePlanActionTest.testTaskTimeout > 0123 1.6 1320 19 MultiThreadedOCPTest.test > 0123 1.0 1620 13 RollingRestartTest.test > 0123 1.2 1617 12 SearchRateTriggerTest.testWaitForElapsed > 0123 3.8 119 7 ShardSplitTest.testSplitWithChaosMonkey > 0123 0.3 1519 7 TestInPlaceUpdatesDistrib.test > 0123 0.7 1629 14 > TestIndexWriterDelete.testDeleteAllNoDeadLock > 0123 2.4 1548 18 TestPackages.testPluginLoading > 0123 0.3 1587 4 UnloadDistributedZkTest.test > > > FAILURES IN THE LAST WEEK (343!) > Look particularly at the ones with only a zero in
Re: PLEASE READ! BadApple report. Last week was horrible!
Hi Erick, It's possible this was the root cause of many of the failures: https://issues.apache.org/jira/browse/LUCENE-9191 Do these transient failures look something like this? [junit4]> Throwable #1: java.nio.charset.MalformedInputException: Input length = 1 [junit4]>at __randomizedtesting.SeedInfo.seed([172C6414BE5E2A2C:E5829DFC005A1F0]:0) [junit4]>at java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:274) [junit4]>at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339) [junit4]>at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) [junit4]>at java.base/java.io.InputStreamReader.read(InputStreamReader.java:185) [junit4]>at java.base/java.io.BufferedReader.fill(BufferedReader.java:161) [junit4]>at java.base/java.io.BufferedReader.readLine(BufferedReader.java:326) [junit4]>at java.base/java.io.BufferedReader.readLine(BufferedReader.java:392) [junit4]>at org.apache.lucene.util.LineFileDocs.open(LineFileDocs.java:175) [junit4]>at org.apache.lucene.util.LineFileDocs.(LineFileDocs.java:65) [junit4]>at org.apache.lucene.util.LineFileDocs.(LineFileDocs.java:69) If so, then it is likely the root cause ... I'm working on a fix. Sorry! Mike McCandless http://blog.mikemccandless.com On Mon, May 4, 2020 at 7:54 AM Erick Erickson wrote: > I don’t know whether we had some temporary glitch that broke lots of tests > and they’ve been fixed or we had a major regression, but this needs to be > addressed ASAP if they’re still failing. See everything below the line "ALL > OF THE TESTS BELOW HERE HAVE ONLY FAILED IN THE LAST WEEK!” in this e-mail. > I’ll raise a JIRA if we can’t get some traction quickly here. > > Hey, stuff happens. there’s no problem with tests going totally weird for > a while. If you can say “Oh, yeah, all those failures for class XYZ are > probably fixed” that’s fine. > > Gosh-a-rooni, I hope my logging changes aren’t the culprit (gulp)…. > > Hoss’ rolllup for the last 24 hours is not encouraging in terms of the > problem already being fixed. There are lots of failures in some > classes, notably: > > CloudHttp2SolrClientTest > CollectionsAPIDistributedZkTest > DeleteReplicaTest > TestDocCollectionWatcher > > Unfortunately, the failure rate is not very high so reliably > reproducing is hard. > > I’ve reproduced the last week’s failure in this e-mail, full > report attached. > > Here’s Hoss’ rollup: > http://fucit.org/solr-jenkins-reports/failure-report.html > > Usual synopsis: > > Raw fail count by week totals, most recent week first (corresponds to > bits): > Week: 0 had 343 failures > Week: 1 had 86 failures > Week: 2 had 78 failures > Week: 3 had 117 failures > > > Failures in Hoss' reports for the last 4 rollups. > > There were 497 unannotated tests that failed in Hoss' rollups. Ordered by > the date I downloaded the rollup file, newest->oldest. See above for the > dates the files were collected > These tests were NOT BadApple'd or AwaitsFix’d > > Failures in the last 4 reports.. >Report Pct runsfails test > 0123 0.7 1617 11 > ConnectionManagerTest.testReconnectWhenZkDisappeared > 0123 1.5 1606 12 ExecutePlanActionTest.testTaskTimeout > 0123 1.6 1320 19 MultiThreadedOCPTest.test > 0123 1.0 1620 13 RollingRestartTest.test > 0123 1.2 1617 12 > SearchRateTriggerTest.testWaitForElapsed > 0123 3.8 119 7 > ShardSplitTest.testSplitWithChaosMonkey > 0123 0.3 1519 7 TestInPlaceUpdatesDistrib.test > 0123 0.7 1629 14 > TestIndexWriterDelete.testDeleteAllNoDeadLock > 0123 2.4 1548 18 TestPackages.testPluginLoading > 0123 0.3 1587 4 UnloadDistributedZkTest.test > > > FAILURES IN THE LAST WEEK (343!) > Look particularly at the ones with only a zero in the “Report” column, > those are > failures that were _not_ in the previous 3 week’s rollups. > >Report Pct runsfails test > 0120.5 1165 4 CustomHighlightComponentTest.test > 0121.0 1168 6 > NodeMarkersRegistrationTest.testNodeMarkersRegistration > 0121.0 1170 8 TestCryptoKeys.test > 01 3 0.7 1233 11 LeaderFailoverAfterPartitionTest.test > 01 3 63.2 102 39 StressHdfsTest.test > 01 0.3 709 2 > ScheduledTriggerIntegrationTest.testScheduledTrigger > 01 0.2 768 2 ShardRoutingTest.test > 01 2.6 807 22 TestAllFilesHaveChecksumFooter.test > 01 2.6 808 22 TestAllFilesHaveCodecHeader.test > 01 0.2 769 2 TestCloudSchemaless.test > 01 0.2
PLEASE READ! BadApple report. Last week was horrible!
I don’t know whether we had some temporary glitch that broke lots of tests and they’ve been fixed or we had a major regression, but this needs to be addressed ASAP if they’re still failing. See everything below the line "ALL OF THE TESTS BELOW HERE HAVE ONLY FAILED IN THE LAST WEEK!” in this e-mail. I’ll raise a JIRA if we can’t get some traction quickly here. Hey, stuff happens. there’s no problem with tests going totally weird for a while. If you can say “Oh, yeah, all those failures for class XYZ are probably fixed” that’s fine. Gosh-a-rooni, I hope my logging changes aren’t the culprit (gulp)…. Hoss’ rolllup for the last 24 hours is not encouraging in terms of the problem already being fixed. There are lots of failures in some classes, notably: CloudHttp2SolrClientTest CollectionsAPIDistributedZkTest DeleteReplicaTest TestDocCollectionWatcher Unfortunately, the failure rate is not very high so reliably reproducing is hard. I’ve reproduced the last week’s failure in this e-mail, full report attached. Here’s Hoss’ rollup: http://fucit.org/solr-jenkins-reports/failure-report.html Usual synopsis: Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 343 failures Week: 1 had 86 failures Week: 2 had 78 failures Week: 3 had 117 failures Failures in Hoss' reports for the last 4 rollups. There were 497 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix’d Failures in the last 4 reports.. Report Pct runsfails test 0123 0.7 1617 11 ConnectionManagerTest.testReconnectWhenZkDisappeared 0123 1.5 1606 12 ExecutePlanActionTest.testTaskTimeout 0123 1.6 1320 19 MultiThreadedOCPTest.test 0123 1.0 1620 13 RollingRestartTest.test 0123 1.2 1617 12 SearchRateTriggerTest.testWaitForElapsed 0123 3.8 119 7 ShardSplitTest.testSplitWithChaosMonkey 0123 0.3 1519 7 TestInPlaceUpdatesDistrib.test 0123 0.7 1629 14 TestIndexWriterDelete.testDeleteAllNoDeadLock 0123 2.4 1548 18 TestPackages.testPluginLoading 0123 0.3 1587 4 UnloadDistributedZkTest.test FAILURES IN THE LAST WEEK (343!) Look particularly at the ones with only a zero in the “Report” column, those are failures that were _not_ in the previous 3 week’s rollups. Report Pct runsfails test 0120.5 1165 4 CustomHighlightComponentTest.test 0121.0 1168 6 NodeMarkersRegistrationTest.testNodeMarkersRegistration 0121.0 1170 8 TestCryptoKeys.test 01 3 0.7 1233 11 LeaderFailoverAfterPartitionTest.test 01 3 63.2 102 39 StressHdfsTest.test 01 0.3 709 2 ScheduledTriggerIntegrationTest.testScheduledTrigger 01 0.2 768 2 ShardRoutingTest.test 01 2.6 807 22 TestAllFilesHaveChecksumFooter.test 01 2.6 808 22 TestAllFilesHaveCodecHeader.test 01 0.2 769 2 TestCloudSchemaless.test 01 0.2 769 2 TestDynamicLoading.testDynamicLoading 01 0.3 707 2 TestDynamicLoadingUrl.testDynamicLoadingUrl 01 0.5 767 4 TestPointFields.testFloatPointStats 0127.1 83 19 TestSQLHandler.doTest 01 0.2 794 12 TestSameScoresWithThreads.test 01 2.6 806 22 TestShardSearching.testSimple 01 0.5 726 4 TestSimScenario.testSplitShard 01 1.1 726 7 TestSimScenario.testSuggestions 01 0.3 771 2 TestWithCollection.testAddReplicaSimple 0 23 0.3 1223 4 CdcrVersionReplicationTest.testCdcrDocVersions 0 23 0.8 1172 6 CloudHttp2SolrClientTest.testRetryUpdatesWhenClusterStateIsStale 0 23 1.4 1202 8 CollectionsAPISolrJTest.testColStatus 0 23 1.0 1249 11 HttpPartitionTest.test 0 23 1.1 1210 8 HttpPartitionWithTlogReplicasTest.test 0 23 0.5 1258 4 ShardSplitTest.testSplitShardWithRuleLink 0 23 0.2 1231 4 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0 23 0.2 1232 6 TestSolrConfigHandlerCloud.test 0 20.3 767 2 DocValuesNotIndexedTest.testGroupingDVOnlySortLast 0 20.3 750 2 TestLBHttp2SolrClient.testTwoServers 0 20.3 794 2 TestSolrCloudSnapshots.testSnapshots 0 2 40.7 51 12
BadApple report
Kevin: The good news is that no SyncSliceTest failures in the last week, cool! Number of AwaitsFix: 42 Number of BadApples: 4 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 86 failures Week: 1 had 78 failures Week: 2 had 117 failures Week: 3 had 99 failures ** **Failures in Hoss' reports for the last 4 rollups. There were 265 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 0.9 1355 19 MultiThreadedOCPTest.test 0123 0.5 1670 12 RollingRestartTest.test 0123 0.3 1663 8 SearchRateTriggerTest.testWaitForElapsed 0123 6.7 126 8 ShardSplitTest.testSplitWithChaosMonkey 0123 0.3 1666 28 SystemCollectionCompatTest.testBackCompat 0123 0.6 1615 10 TestInPlaceUpdatesDistrib.test 0123 0.6 1640 21 TestPackages.testPluginLoading 0123 0.3 1646 4 UnloadDistributedZkTest.test Full report attached: DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2020-04-27.csv Processing file (History bit 2): HOSS-2020-04-20.csv Processing file (History bit 1): HOSS-2020-04-13.csv Processing file (History bit 0): HOSS-2020-04-06.csv Number of AwaitsFix: 42 Number of BadApples: 4 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 86 failures Week: 1 had 78 failures Week: 2 had 117 failures Week: 3 had 99 failures ** **Failures in Hoss' reports for the last 4 rollups. There were 265 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 0.9 1355 19 MultiThreadedOCPTest.test 0123 0.5 1670 12 RollingRestartTest.test 0123 0.3 1663 8 SearchRateTriggerTest.testWaitForElapsed 0123 6.7 126 8 ShardSplitTest.testSplitWithChaosMonkey 0123 0.3 1666 28 SystemCollectionCompatTest.testBackCompat 0123 0.6 1615 10 TestInPlaceUpdatesDistrib.test 0123 0.6 1640 21 TestPackages.testPluginLoading 0123 0.3 1646 4 UnloadDistributedZkTest.test Failures over the last 4 weeks, but not every week. Ordered most-recent first: 0120.3 1196 5 ComputePlanActionTest.testSelectedCollections 0120.3 1211 8 ConnectionManagerTest.testReconnectWhenZkDisappeared 0120.3 1196 3 DaemonStreamApiTest.testAPIs 0120.5 1203 6 ExecutePlanActionTest.testTaskTimeout 012 80.0 21 15 SharedFSAutoReplicaFailoverTest.test 0121.3 1215 11 TestIndexWriterDelete.testDeleteAllNoDeadLock 01 4.0 50 2 CdcrReplicationHandlerTest.testReplicationWithBufferedUpdates 01 0.3 762 2 CustomHighlightComponentTest.test 01 3.8 58 7 HdfsUnloadDistributedZkTest.test 01 0.3 736 3 LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud 01 0.3 765 2
BadApple report
Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 78 failures Week: 1 had 117 failures Week: 2 had 99 failures Week: 3 had 69 failures Failures in Hoss' reports for the last 4 rollups. There were 243 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 0.3 1681 9 CdcrVersionReplicationTest.testCdcrDocVersions 0123 23.8 198 88 HdfsSyncSliceTest.test 0123 0.5 1694 10 HttpPartitionTest.test 0123 0.5 1698 10 HttpPartitionWithTlogReplicasTest.test 0123 6.7 130 10 ShardSplitTest.testSplitWithChaosMonkey 0123 0.3 1712 20 SyncSliceTest.test 0123 2.9 1739 36 SystemCollectionCompatTest.testBackCompat 0123 0.5 1676 12 TestInPlaceUpdatesDistrib.test 0123 1.2 1696 21 TestPackages.testPluginLoading 0123 0.3 1682 6 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0123 1.0 1679 7 TestSolrConfigHandlerCloud.test DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2020-04-20.csv Processing file (History bit 2): HOSS-2020-04-13.csv Processing file (History bit 1): HOSS-2020-04-06.csv Processing file (History bit 0): HOSS-2020-03-30.csv Number of AwaitsFix: 41 Number of BadApples: 4 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 78 failures Week: 1 had 117 failures Week: 2 had 99 failures Week: 3 had 69 failures Failures in Hoss' reports for the last 4 rollups. There were 243 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 0.3 1681 9 CdcrVersionReplicationTest.testCdcrDocVersions 0123 23.8 198 88 HdfsSyncSliceTest.test 0123 0.5 1694 10 HttpPartitionTest.test 0123 0.5 1698 10 HttpPartitionWithTlogReplicasTest.test 0123 6.7 130 10 ShardSplitTest.testSplitWithChaosMonkey 0123 0.3 1712 20 SyncSliceTest.test 0123 2.9 1739 36 SystemCollectionCompatTest.testBackCompat 0123 0.5 1676 12 TestInPlaceUpdatesDistrib.test 0123 1.2 1696 21 TestPackages.testPluginLoading 0123 0.3 1682 6 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0123 1.0 1679 7 TestSolrConfigHandlerCloud.test Failures over the last 4 weeks, but not every week. Ordered most-recent first: 0120.3 1264 4 CloudHttp2SolrClientTest.testRetryUpdatesWhenClusterStateIsStale 0122.9 1020 16 MultiThreadedOCPTest.test 0120.3 1301 10 RollingRestartTest.test 0121.0 1295 7 SearchRateTriggerTest.testWaitForElapsed 0120.3 1290 5 TestCloudRecovery2.test 0120.2 1295 4 TestReplicationHandler.doTestIndexAndConfigReplication 0120.3 1283 3 UnloadDistributedZkTest.test 01 3 1.0 1241
Re: BadApple report
> > 0123 59.4 195 92 HdfsSyncSliceTest.test I'm looking into this HdfsSyncSliceTest failure. Jira https://issues.apache.org/jira/browse/SOLR-13886 Kevin Risden Kevin Risden On Mon, Apr 13, 2020 at 8:35 AM Erick Erickson wrote: > We’re backsliding a bit. Note that over the last two weeks we’ve had > successively more failures, HdfsSyncSliceTest is failing over half the > time! Can we just nuke it? > > Here’s the short form > > aw fail count by week totals, most recent week first (corresponds to bits): > Week: 0 had 117 failures > Week: 1 had 99 failures > Week: 2 had 69 failures > Week: 3 had 65 failures > > > Failures in Hoss' reports for the last 4 rollups. > > There were 252 unannotated tests that failed in Hoss' rollups. Ordered by > the date I downloaded the rollup file, newest->oldest. See above for the > dates the files were collected > These tests were NOT BadApple'd or AwaitsFix'd > > Failures in the last 4 reports.. >Report Pct runsfails test > 0123 59.4 195 92 HdfsSyncSliceTest.test > 0123 0.5 1697 10 HttpPartitionWithTlogReplicasTest.test > 0123 6.1 133 12 > ShardSplitTest.testSplitWithChaosMonkey > 0123 1.8 1712 20 SyncSliceTest.test > 0123 2.5 1754 49 > SystemCollectionCompatTest.testBackCompat > 0123 0.5 1706 26 TestPackages.testPluginLoading > 0123 0.2 1676 4 TestSolrConfigHandlerCloud.test > > > > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org
BadApple report
We’re backsliding a bit. Note that over the last two weeks we’ve had successively more failures, HdfsSyncSliceTest is failing over half the time! Can we just nuke it? Here’s the short form aw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 117 failures Week: 1 had 99 failures Week: 2 had 69 failures Week: 3 had 65 failures Failures in Hoss' reports for the last 4 rollups. There were 252 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 59.4 195 92 HdfsSyncSliceTest.test 0123 0.5 1697 10 HttpPartitionWithTlogReplicasTest.test 0123 6.1 133 12 ShardSplitTest.testSplitWithChaosMonkey 0123 1.8 1712 20 SyncSliceTest.test 0123 2.5 1754 49 SystemCollectionCompatTest.testBackCompat 0123 0.5 1706 26 TestPackages.testPluginLoading 0123 0.2 1676 4 TestSolrConfigHandlerCloud.test DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2020-04-13.csv Processing file (History bit 2): HOSS-2020-04-06.csv Processing file (History bit 1): HOSS-2020-03-30.csv Processing file (History bit 0): HOSS-2020-03-24.csv Number of AwaitsFix: 41 Number of BadApples: 4 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 117 failures Week: 1 had 99 failures Week: 2 had 69 failures Week: 3 had 65 failures Failures in Hoss' reports for the last 4 rollups. There were 252 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 59.4 195 92 HdfsSyncSliceTest.test 0123 0.5 1697 10 HttpPartitionWithTlogReplicasTest.test 0123 6.1 133 12 ShardSplitTest.testSplitWithChaosMonkey 0123 1.8 1712 20 SyncSliceTest.test 0123 2.5 1754 49 SystemCollectionCompatTest.testBackCompat 0123 0.5 1706 26 TestPackages.testPluginLoading 0123 0.2 1676 4 TestSolrConfigHandlerCloud.test Failures over the last 4 weeks, but not every week. Ordered most-recent first: 0120.5 1286 8 CdcrVersionReplicationTest.testCdcrDocVersions 0123.7 83 5 HdfsBasicDistributedZkTest.test 0121.1 1290 8 HttpPartitionTest.test 0129.4 95 11 Test2BPostings.test 0123.7 81 3 TestDuelingCodecsAtNight.testBigEquals 0120.5 1281 10 TestInPlaceUpdatesDistrib.test 0120.5 1285 5 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0120.5 1281 6 TestSolrDeletionPolicy1.testNumCommitsConfigured 0120.7 1299 13 TestStressLiveNodes.testStress 01 3 1.4 1316 14 RollingRestartTest.test 01 3 0.5 1295 5 SearchRateTriggerTest.testWaitForElapsed 01 3 2.5 1316 23 TestRandomChains.testRandomChains 01 0.5 872 3 CloudHttp2SolrClientTest.testRetryUpdatesWhenClusterStateIsStale 01 0.5
BadApple report
Short form: We had a slight uptick in failures last week, root cause unknown. Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 99 failures Week: 1 had 69 failures Week: 2 had 65 failures Week: 3 had 129 failures Failures in Hoss' reports for the last 4 rollups. There were 252 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 45.2 208 99 HdfsSyncSliceTest.test 0123 0.9 1702 9 HttpPartitionWithTlogReplicasTest.test 0123 6.1 130 12 ShardSplitTest.testSplitWithChaosMonkey 0123 1.5 1717 16 SyncSliceTest.test 0123 0.9 1843 94 SystemCollectionCompatTest.testBackCompat 0123 2.6 1725 32 TestPackages.testPluginLoading 0123 0.2 1685 4 TestSolrConfigHandlerCloud.test Full report attched. DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2020-04-06.csv Processing file (History bit 2): HOSS-2020-03-30.csv Processing file (History bit 1): HOSS-2020-03-24.csv Processing file (History bit 0): HOSS-2020-03-16.csv Number of AwaitsFix: 41 Number of BadApples: 4 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 99 failures Week: 1 had 69 failures Week: 2 had 65 failures Week: 3 had 129 failures Failures in Hoss' reports for the last 4 rollups. There were 252 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 45.2 208 99 HdfsSyncSliceTest.test 0123 0.9 1702 9 HttpPartitionWithTlogReplicasTest.test 0123 6.1 130 12 ShardSplitTest.testSplitWithChaosMonkey 0123 1.5 1717 16 SyncSliceTest.test 0123 0.9 1843 94 SystemCollectionCompatTest.testBackCompat 0123 2.6 1725 32 TestPackages.testPluginLoading 0123 0.2 1685 4 TestSolrConfigHandlerCloud.test Failures over the last 4 weeks, but not every week. Ordered most-recent first: 0120.2 1244 3 TestCloudJSONFacetSKG.testRandom 0123.4 100 23 TestXYMultiPolygonShapeQueries.testRandomBig 01 3 1.1 1301 7 CdcrVersionReplicationTest.testCdcrDocVersions 01 3 0.4 1304 5 DocValuesNotIndexedTest.testGroupingDVOnlySortFirst 01 3 0.4 1304 5 DocValuesNotIndexedTest.testGroupingDVOnlySortLast 01 3 9.4 83 5 HdfsBasicDistributedZkTest.test 01 3 0.2 1295 4 HttpPartitionTest.test 01 3 15.4 91 9 Test2BPostings.test 01 3 0.9 1292 12 TestInPlaceUpdatesDistrib.test 01 3 0.2 1306 8 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 01 3 1.7 1307 13 TestStressLiveNodes.testStress 01 3 0.2 1300 3 TriggerCooldownIntegrationTest.testCooldown 01 0.2 853 2 LeaderElectionTest.testStressElection 01 0.2 843 3 PeerSyncWithLeaderTest.test 01 0.2 862
BadApple report
There are a couple of tests that can have BadApple removed, MultiThreadedOCPTest.test SolrZkClientTest.testSimpleUpdateACLs I’ll take care of those today or tomorrow. Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 69 failures Week: 1 had 65 failures Week: 2 had 129 failures Week: 3 had 87 failures Failures in Hoss' reports for the last 4 rollups. There were 251 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 40.0 160 73 HdfsSyncSliceTest.test 0123 0.5 1680 8 HttpPartitionWithTlogReplicasTest.test 0123 1.0 1685 12 SyncSliceTest.test 0123 2.2 1857113 SystemCollectionCompatTest.testBackCompat 0123 0.5 1691 21 TestPackages.testPluginLoading 0123 0.3 1681 6 TestSolrConfigHandlerCloud.test File attached. DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2020-03-30.csv Processing file (History bit 2): HOSS-2020-03-24.csv Processing file (History bit 1): HOSS-2020-03-16.csv Processing file (History bit 0): HOSS-2020-02-10.csv Number of AwaitsFix: 41 Number of BadApples: 6 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 2 MultiThreadedOCPTest.test SolrZkClientTest.testSimpleUpdateACLs Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 69 failures Week: 1 had 65 failures Week: 2 had 129 failures Week: 3 had 87 failures Failures in Hoss' reports for the last 4 rollups. There were 251 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 40.0 160 73 HdfsSyncSliceTest.test 0123 0.5 1680 8 HttpPartitionWithTlogReplicasTest.test 0123 1.0 1685 12 SyncSliceTest.test 0123 2.2 1857113 SystemCollectionCompatTest.testBackCompat 0123 0.5 1691 21 TestPackages.testPluginLoading 0123 0.3 1681 6 TestSolrConfigHandlerCloud.test Failures over the last 4 weeks, but not every week. Ordered most-recent first: 012 11.8 97 10 ShardSplitTest.testSplitWithChaosMonkey 0127.7 196 26 TestFactories.test 01 3 0.3 1223 3 TestCloudJSONFacetSKG.testRandom 01 3 30.6 90 24 TestXYMultiPolygonShapeQueries.testRandomBig 01 4.0 50 2 CdcrReplicationHandlerTest.testReplicationWithBufferedUpdates 01 0.3 799 2 ConnectionManagerTest.testReconnectWhenZkDisappeared 0 23 4.2 62 4 HdfsBasicDistributedZkTest.test 0 23 8.3 73 6 Test2BPostings.test 0 23 0.5 1289 9 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0 23 0.3 1268 3 TestSolrCloudWithDelegationTokens.testDelegationTokenRenew 0 23 0.5 1274 7 TestStressLiveNodes.testStress 0 20.3 837 2 CdcrVersionReplicationTest.testCdcrDocVersions 0 20.5 844 3 DocValuesNotIndexedTest.testGroupingDVOnlySortFirst 0 20.2 844 3
BadApple report
Short form: There were 287 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 0.3 1747 25 BasicDistributedZkTest.test 0123 35.9 155 67 HdfsSyncSliceTest.test 0123 0.5 1727 8 HttpPartitionWithTlogReplicasTest.test 0123 0.3 1728 10 SyncSliceTest.test 0123 5.8 1950142 SystemCollectionCompatTest.testBackCompat 0123 2.4 1743 24 TestPackages.testPluginLoading 0123 0.3 1730 7 TestSolrConfigHandlerCloud.test Interestingly, Test2BPostings.test didn’t fail last week, is that compiler issue fixed? Full report attached: DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2020-03-24.csv Processing file (History bit 2): HOSS-2020-03-16.csv Processing file (History bit 1): HOSS-2020-02-10.csv Processing file (History bit 0): HOSS-2020-02-03.csv Number of AwaitsFix: 41 Number of BadApples: 6 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 65 failures Week: 1 had 129 failures Week: 2 had 87 failures Week: 3 had 114 failures **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 2 MultiThreadedOCPTest.test SolrZkClientTest.testSimpleUpdateACLs Failures in Hoss' reports for the last 4 rollups. There were 287 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 0.3 1747 25 BasicDistributedZkTest.test 0123 35.9 155 67 HdfsSyncSliceTest.test 0123 0.5 1727 8 HttpPartitionWithTlogReplicasTest.test 0123 0.3 1728 10 SyncSliceTest.test 0123 5.8 1950142 SystemCollectionCompatTest.testBackCompat 0123 2.4 1743 24 TestPackages.testPluginLoading 0123 0.3 1730 7 TestSolrConfigHandlerCloud.test Failures over the last 4 weeks, but not every week. Ordered most-recent first: 0121.2 1301 14 RollingRestartTest.test 01 3 0.5 1293 4 SearchRateTriggerTest.testWaitForElapsed 01 3 12.1 87 8 ShardSplitTest.testSplitWithChaosMonkey 01 0.2 841 3 LeaderFailoverAfterPartitionTest.test 01 0.3 843 2 PeerSyncTest.test 01 1.7 843 15 StreamExpressionTest.testFacet2DStream 01 1.3 841 13 StreamExpressionTest.testFacetStream 01 1.5 842 14 StreamExpressionTest.testMultiCollection 01 1.3 842 14 StreamExpressionTest.testStatsStream 01 1.7 844 16 StreamExpressionTest.testSubFacetStream 01 1.0 841 13 StreamExpressionTest.testTimeSeriesStream 01 1.7 843 15 StreamExpressionTest.tooLargeForGetRequest 01 0.3 841 2 TestCloudSearcherWarming.testPeersyncFailureReplicationSuccess 01 0.3 685 2 TestDelegationWithHadoopAuth.testDelegationTokenRenew 01 0.8 845 5 TestDynamicLoading.testDynamicLoading 0126.1 170 24 TestFactories.test 01 1.4 898 23
BadApple report
I was on vacation the last couple of weeks so missed the BadApple reports. Full results attached Failures in Hoss' reports for the last 4 rollups. There were 373 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 2.4 1694 49 BasicDistributedZkTest.test 0123 0.2 1645 5 ExecutePlanActionTest.testTaskTimeout 0123 14.3 103 21 HdfsSyncSliceTest.test 0123 0.7 1647 8 HttpPartitionWithTlogReplicasTest.test 0123 0.7 1648 13 SyncSliceTest.test 0123 4.8 1744 71 SystemCollectionCompatTest.testBackCompat 0123 14.3 90 13 Test2BPostings.test (known compiler issue) 0123 0.2 1647 13 TestPackages.testPluginLoading 0123 0.5 1654 15 TestStressLiveNodes.testStress 0123 10.5 91 12 TestXYMultiPolygonShapeQueries.testRandomBig DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2020-02-10.csv Processing file (History bit 2): HOSS-2020-02-03.csv Processing file (History bit 1): HOSS-2020-01-27.csv Processing file (History bit 0): HOSS-2020-01-20.csv Number of AwaitsFix: 41 Number of BadApples: 6 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 87 failures Week: 1 had 114 failures Week: 2 had 125 failures Week: 3 had 191 failures **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 2 MultiThreadedOCPTest.test SolrZkClientTest.testSimpleUpdateACLs Failures in Hoss' reports for the last 4 rollups. There were 373 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 2.4 1694 49 BasicDistributedZkTest.test 0123 0.2 1645 5 ExecutePlanActionTest.testTaskTimeout 0123 14.3 103 21 HdfsSyncSliceTest.test 0123 0.7 1647 8 HttpPartitionWithTlogReplicasTest.test 0123 0.7 1648 13 SyncSliceTest.test 0123 4.8 1744 71 SystemCollectionCompatTest.testBackCompat 0123 14.3 90 13 Test2BPostings.test 0123 0.2 1647 13 TestPackages.testPluginLoading 0123 0.5 1654 15 TestStressLiveNodes.testStress 0123 10.5 91 12 TestXYMultiPolygonShapeQueries.testRandomBig Will BadApple all tests above this line except ones listed at the top** 0120.2 1246 4 BasicDistributedZk2Test.test 0120.2 1244 5 MetricTriggerIntegrationTest.testMetricTrigger 01 3 0.2 1279 3 LeaderElectionIntegrationTest.testSimpleSliceLeaderElection 01 3 0.2 1369 5 TestConcurrentMergeScheduler.testFlushExceptions 01 3 0.5 1277 4 TestLockTree.testLocks 01 3 0.4 1358 5 TestSearcherManager.testConcurrentIndexCloseSearchAndRefresh 01 3 0.2 1280 4 TestSolrCloudWithDelegationTokens.testDelegationTokenRenew 01 3 0.7 1287 8 TestSolrConfigHandlerCloud.test 01 0.2 885 2 ChaosMonkeyNothingIsSafeWithPullReplicasTest.test 01 0.2
Badapple report
Attached. Short form: **Haven't failed in the last 4 rollups. **Methods: 2 MultiThreadedOCPTest.test SolrZkClientTest.testSimpleUpdateACLs Failures in Hoss' reports for the last 4 rollups. There were 292 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 30.8 112 27 HdfsSyncSliceTest.test 0123 0.6 1760 10 HttpPartitionWithTlogReplicasTest.test 0123 1.0 1775 16 SyncSliceTest.test 0123 6.7 1929106 SystemCollectionCompatTest.testBackCompat 0123 23.3 95 16 Test2BPostings.test (Known compiler bug, don't annotate) 0123 1.7 1770 20 TestPackages.testPluginLoading 0123 1.5 1780 19 TestStressLiveNodes.testStress DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2020-02-24.csv Processing file (History bit 2): HOSS-2020-02-10.csv Processing file (History bit 1): HOSS-2020-02-03.csv Processing file (History bit 0): HOSS-2020-01-27.csv Number of AwaitsFix: 41 Number of BadApples: 6 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 87 failures Week: 1 had 87 failures Week: 2 had 114 failures Week: 3 had 125 failures **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 2 MultiThreadedOCPTest.test SolrZkClientTest.testSimpleUpdateACLs Failures in Hoss' reports for the last 4 rollups. There were 292 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 30.8 112 27 HdfsSyncSliceTest.test 0123 0.6 1760 10 HttpPartitionWithTlogReplicasTest.test 0123 1.0 1775 16 SyncSliceTest.test 0123 6.7 1929106 SystemCollectionCompatTest.testBackCompat 0123 23.3 95 16 Test2BPostings.test 0123 1.7 1770 20 TestPackages.testPluginLoading 0123 1.5 1780 19 TestStressLiveNodes.testStress Will BadApple all tests above this line except ones listed at the top** 0120.4 1391 4 LeaderElectionIntegrationTest.testSimpleSliceLeaderElection 012 28.2 77 17 TestIndexingSequenceNumbers.testStressConcurrentCommit 0120.4 1395 5 TestSolrCloudWithDelegationTokens.testDelegationTokenRenew 01 3 0.6 1316 9 AutoScalingHandlerTest.testReadApi 01 3 0.2 1316 5 AutoScalingHandlerTest.testSuggestionsWithPayload 01 3 5.0 62 9 HdfsBasicDistributedZkTest.test 01 3 0.2 1317 6 RollingRestartTest.test 01 3 0.2 1311 4 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 01 0.2 947 2 CustomHighlightComponentTest.test 01 0.2 954 2 LeaderVoteWaitTimeoutTest.basicTest 01 0.8 945 5 OverseerRolesTest.testOverseerRole 01 0.3 718 2 OverseerTest.testShardLeaderChange 01 3.7 49 4 TestLucene80DocValuesFormat.testNumericFieldJumpTables 0 23 4.3 73 13 HdfsWriteToMultipleCollectionsTest.test 0
BadApple report
Holding reasonable steady in terms of failures every week for the last 4: Failures in the last 4 reports.. Report Pct runsfails test 0123 2.4 1694 49 BasicDistributedZkTest.test 0123 0.2 1645 5 ExecutePlanActionTest.testTaskTimeout 0123 14.3 103 21 HdfsSyncSliceTest.test 0123 0.7 1647 8 HttpPartitionWithTlogReplicasTest.test 0123 0.7 1648 13 SyncSliceTest.test 0123 4.8 1744 71 SystemCollectionCompatTest.testBackCompat 0123 14.3 90 13 Test2BPostings.test * Compiler bug 0123 0.2 1647 13 TestPackages.testPluginLoading 0123 0.5 1654 15 TestStressLiveNodes.testStress 0123 10.5 91 12 TestXYMultiPolygonShapeQueries.testRandomBig And a nice steady decline in the total number of failures over the last 4 weeks. The number of awaitsfix and badapples have been constant over these 4 weeks. Raw fail count by week most recent first Week: 0 had 87 failures Week: 1 had 114 failures Week: 2 had 125 failures Week: 3 had 191 failures As a bonus, here’s are the AwaitsFix and BadApple counts since I’ve been collecting them: e-mail-2018-04-02.txt: Number of AwaitsFix: 15 Number of BadApples: 78 e-mail-2018-04-30.txt: Number of AwaitsFix: 21 Number of BadApples: 90 e-mail-2018-05-21.txt: Number of AwaitsFix: 21 Number of BadApples: 90 e-mail-2018-06-11.txt: Number of AwaitsFix: 16 Number of BadApples: 111 e-mail-2018-06-18.txt: Number of AwaitsFix: 17 Number of BadApples: 92 e-mail-2018-06-25.txt: Number of AwaitsFix: 18 Number of BadApples: 93 e-mail-2018-07-02.txt: Number of AwaitsFix: 18 Number of BadApples: 88 e-mail-2018-07-09.txt: Number of AwaitsFix: 18 Number of BadApples: 96 e-mail-2018-07-16.txt: Number of AwaitsFix: 17 Number of BadApples: 96 e-mail-2018-07-23.txt: Number of AwaitsFix: 18 Number of BadApples: 100 e-mail-2018-07-30.txt: Number of AwaitsFix: 18 Number of BadApples: 100 e-mail-2018-08-06.txt: Number of AwaitsFix: 18 Number of BadApples: 131 e-mail-2018-08-14.txt: Number of AwaitsFix: 18 Number of BadApples: 125 e-mail-2018-08-20.txt: Number of AwaitsFix: 18 Number of BadApples: 118 e-mail-2018-08-27.txt: Number of AwaitsFix: 18 Number of BadApples: 118 e-mail-2018-09-03.txt: Number of AwaitsFix: 18 Number of BadApples: 118 e-mail-2018-09-10.txt: Number of AwaitsFix: 18 Number of BadApples: 101 e-mail-2018-09-18.txt: Number of AwaitsFix: 18 Number of BadApples: 97 e-mail-2018-10-08.txt: Number of AwaitsFix: 19 Number of BadApples: 148 e-mail-2018-12-24.txt: Number of AwaitsFix: 52 Number of BadApples: 138 e-mail-2019-01-08.txt: Number of AwaitsFix: 49 Number of BadApples: 55 e-mail-2019-01-15.txt: Number of AwaitsFix: 48 Number of BadApples: 60 e-mail-2019-02-12.txt: Number of AwaitsFix: 48 Number of BadApples: 57 e-mail-2019-02-18.txt: Number of AwaitsFix: 48 Number of BadApples: 18 e-mail-2019-03-04.txt: Number of AwaitsFix: 44 Number of BadApples: 22 e-mail-2019-03-11.txt: Number of AwaitsFix: 44 Number of BadApples: 20 e-mail-2019-03-18.txt: Number of AwaitsFix: 44 Number of BadApples: 30 e-mail-2019-03-25.txt: Number of AwaitsFix: 46 Number of BadApples: 17 e-mail-2019-04-01.txt: Number of AwaitsFix: 46 Number of BadApples: 17 e-mail-2019-04-08.txt: Number of AwaitsFix: 46 Number of BadApples: 13 e-mail-2019-04-15.txt: Number of AwaitsFix: 48 Number of BadApples: 12 e-mail-2019-04-22.txt: Number of AwaitsFix: 48 Number of BadApples: 12 e-mail-2019-05-06.txt: Number of AwaitsFix: 48 Number of BadApples: 12 e-mail-2019-05-20.txt: Number of AwaitsFix: 48 Number of BadApples: 12 e-mail-2019-06-03.txt: Number of AwaitsFix: 43 Number of BadApples: 12 e-mail-2019-06-10.txt: Number of AwaitsFix: 45 Number of BadApples: 12 e-mail-2019-06-17.txt: Number of AwaitsFix: 43 Number of BadApples: 12 e-mail-2019-06-24.txt: Number of AwaitsFix: 43 Number of BadApples: 11 e-mail-2019-07-01.txt: Number of AwaitsFix: 39 Number of BadApples: 11 e-mail-2019-07-29.txt: Number of AwaitsFix: 38 Number of BadApples: 12 e-mail-2019-08-05.txt: Number of AwaitsFix: 38 Number of BadApples: 11 e-mail-2019-08-12.txt: Number of AwaitsFix: 38 Number of BadApples: 11 e-mail-2019-08-19.txt: Number of AwaitsFix: 38 Number of BadApples: 11 e-mail-2019-09-16.txt: Number of AwaitsFix: 38 Number of BadApples: 11 e-mail-2019-10-28.txt: Number of AwaitsFix: 40 Number of BadApples: 11 e-mail-2019-11-04.txt: Number of AwaitsFix: 39 Number of BadApples: 11 e-mail-2019-11-11.txt: Number of AwaitsFix: 38 Number of BadApples: 11 e-mail-2019-11-18.txt: Number of AwaitsFix: 38 Number of BadApples: 10 e-mail-2019-11-25.txt: Number of AwaitsFix: 40 Number of BadApples: 8 e-mail-2019-12-02.txt: Number of AwaitsFix: 40 Number of BadApples: 8 e-mail-2019-12-09.txt: Number of
BadApple report
Won’t add annotations. Here’s the failures in the last 4 runs: Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 114 failures Week: 1 had 125 failures Week: 2 had 191 failures Week: 3 had 118 failures Failures in the last 4 reports.. Report Pct runsfails test 0123 0.4 1612 54 BasicDistributedZkTest.test 0123 24.0 118 24 HdfsSyncSliceTest.test 0123 0.5 1567 11 HttpPartitionWithTlogReplicasTest.test 0123 0.2 1552 8 LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud 0123 0.5 1562 11 SyncSliceTest.test 0123 7.5 1636 53 SystemCollectionCompatTest.testBackCompat 0123 15.0 95 17 Test2BPostings.test * compiler issue 0123 0.7 1597 21 TestStressLiveNodes.testStress Full report attached: DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2020-02-03.csv Processing file (History bit 2): HOSS-2020-01-27.csv Processing file (History bit 1): HOSS-2020-01-20.csv Processing file (History bit 0): HOSS-2020-01-13.csv Number of AwaitsFix: 41 Number of BadApples: 6 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 114 failures Week: 1 had 125 failures Week: 2 had 191 failures Week: 3 had 118 failures **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 2 MultiThreadedOCPTest.test SolrZkClientTest.testSimpleUpdateACLs Failures in Hoss' reports for the last 4 rollups. There were 397 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 0.4 1612 54 BasicDistributedZkTest.test 0123 24.0 118 24 HdfsSyncSliceTest.test 0123 0.5 1567 11 HttpPartitionWithTlogReplicasTest.test 0123 0.2 1552 8 LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud 0123 0.5 1562 11 SyncSliceTest.test 0123 7.5 1636 53 SystemCollectionCompatTest.testBackCompat 0123 15.0 95 17 Test2BPostings.test 0123 0.7 1597 21 TestStressLiveNodes.testStress Will BadApple all tests above this line except ones listed at the top** 0120.4 1208 4 ExecutePlanActionTest.testTaskTimeout 012 36.7 70 13 HdfsWriteToMultipleCollectionsTest.test 0120.2 1195 3 SearchRateTriggerTest.testWaitForElapsed 0128.3 69 4 ShardSplitTest.testSplitWithChaosMonkey 0120.7 1235 8 TestOfflineSorter.testThreadSafety 0121.1 1213 12 TestPackages.testPluginLoading 0120.2 1193 3 TestSolrCLIRunExample.testInteractiveSolrCloudExampleWithAutoScalingPolicy 0124.8 72 10 TestXYMultiPolygonShapeQueries.testRandomBig 01 3 0.2 1156 3 TestPullReplicaErrorHandling.testPullReplicaDisconnectsFromZooKeeper 01 0.2 808 3 BasicDistributedZk2Test.test 01 0.2 810 4 ConnectionManagerTest.testReconnectWhenZkDisappeared 01 0.7 808 4 MetricTriggerIntegrationTest.testMetricTrigger 01 0.4 810 3 NodeMarkersRegistrationTest.testNodeMarkersRegistration 01 0.2 805 4
BadApple report
Failures in each of the last 4 reports.. Report Pct runsfails test 0123 0.3 1384 11 AutoScalingHandlerTest.testReadApi 0123 0.3 1402 8 HttpPartitionTest.test 0123 0.3 1393 11 HttpPartitionWithTlogReplicasTest.test 0123 0.3 1395 7 LeaderElectionIntegrationTest.testSimpleSliceLeaderElection 0123 1.0 1417 12 LeaderFailoverAfterPartitionTest.test 0123 0.5 1395 8 LeaderVoteWaitTimeoutTest.basicTest 0123 1.0 1402 30 LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud 0123 0.5 1395 14 RollingRestartTest.test 0123 0.5 1394 7 SyncSliceTest.test 0123 1.0 1422 19 SystemCollectionCompatTest.testBackCompat 0123 0.9 1455 8 TestBagOfPositions.test 0123 0.9 1464 12 TestBagOfPostings.test 0123 8.3 938 57 TestFuzzyQuery.testErrorMessage 0123 0.2 1456 5 TestLucene80DocValuesFormat.testSparseDocValuesVsStoredFields 0123 0.5 1396 8 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0123 0.2 1465 5 TestSearcherManager.testConcurrentIndexCloseSearchAndRefresh 0123 1.0 1456 28 TestStressLiveNodes.testStress Not actively annotating at this point. Full list attached. DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2020-01-20.csv Processing file (History bit 2): HOSS-2020-01-13.csv Processing file (History bit 1): HOSS-2020-01-06.csv Processing file (History bit 0): HOSS-2019-12-30.csv Number of AwaitsFix: 41 Number of BadApples: 6 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 191 failures Week: 1 had 118 failures Week: 2 had 298 failures Week: 3 had 84 failures **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 2 MultiThreadedOCPTest.test SolrZkClientTest.testSimpleUpdateACLs Failures in Hoss' reports for the last 4 rollups. There were 533 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 0.3 1384 11 AutoScalingHandlerTest.testReadApi 0123 0.3 1402 8 HttpPartitionTest.test 0123 0.3 1393 11 HttpPartitionWithTlogReplicasTest.test 0123 0.3 1395 7 LeaderElectionIntegrationTest.testSimpleSliceLeaderElection 0123 1.0 1417 12 LeaderFailoverAfterPartitionTest.test 0123 0.5 1395 8 LeaderVoteWaitTimeoutTest.basicTest 0123 1.0 1402 30 LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud 0123 0.5 1395 14 RollingRestartTest.test 0123 0.5 1394 7 SyncSliceTest.test 0123 1.0 1422 19 SystemCollectionCompatTest.testBackCompat 0123 16.0 118 25 Test2BPostings.test 0123 0.9 1455 8 TestBagOfPositions.test 0123 0.9 1464 12 TestBagOfPostings.test 0123 8.3 938 57 TestFuzzyQuery.testErrorMessage 0123 0.2 1456 5 TestLucene80DocValuesFormat.testSparseDocValuesVsStoredFields 0123 0.5 1396 8 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0123 0.2 1465 5
BadApple report
I’m not actively annotating anything at this point, the number of failed tests over each of the last 4 weeks is short enough that I’ll just echo those in these e-mails, the full report is attached for anyone who wants to track history. I’ll revise the wording to not make it look like I’ll annotate things. So things like Test2BPostings.test that are a problem with particular Java compilers will appear in the list but will not be annotated, never fear. Failures in the last 4 reports.. Report Pct runsfails test 0123 1.7 1248 12 HttpPartitionWithTlogReplicasTest.test 0123 1.4 1260 9 LeaderFailoverAfterPartitionTest.test 0123 0.3 1257 30 LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud 0123 0.6 1266 17 RollingRestartTest.test 0123 0.3 1249 7 SyncSliceTest.test 0123 1.4 1298 25 SystemCollectionCompatTest.testBackCompat 0123 26.9 130 28 Test2BPostings.test 0123 4.3 781 57 TestFuzzyQuery.testErrorMessage 0123 0.5 1302 5 TestLucene80DocValuesFormat.testSparseDocValuesVsStoredFields 0123 0.3 1251 8 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0123 2.1 1316 31 TestStressLiveNodes.testStress DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2020-13.csv Processing file (History bit 2): HOSS-2020-01-06.csv Processing file (History bit 1): HOSS-2019-12-30.csv Processing file (History bit 0): HOSS-2019-12-23.csv Number of AwaitsFix: 41 Number of BadApples: 6 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 118 failures Week: 1 had 298 failures Week: 2 had 84 failures Week: 3 had 108 failures **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 2 MultiThreadedOCPTest.test SolrZkClientTest.testSimpleUpdateACLs Failures in Hoss' reports for the last 4 rollups. There were 461 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 1.7 1248 12 HttpPartitionWithTlogReplicasTest.test 0123 1.4 1260 9 LeaderFailoverAfterPartitionTest.test 0123 0.3 1257 30 LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud 0123 0.6 1266 17 RollingRestartTest.test 0123 0.3 1249 7 SyncSliceTest.test 0123 1.4 1298 25 SystemCollectionCompatTest.testBackCompat 0123 26.9 130 28 Test2BPostings.test 0123 4.3 781 57 TestFuzzyQuery.testErrorMessage 0123 0.5 1302 5 TestLucene80DocValuesFormat.testSparseDocValuesVsStoredFields 0123 0.3 1251 8 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0123 2.1 1316 31 TestStressLiveNodes.testStress Will BadApple all tests above this line except ones listed at the top** 0120.8 990 10 AutoScalingHandlerTest.testReadApi 0120.6 990 5 AutoScalingHandlerTest.testSuggestionsWithPayload 0120.6 991 5 CollectionsAPISolrJTest.testCreateCollWithDefaultClusterPropertiesNewFormat 0120.8 1006 7 HttpPartitionTest.test 0120.6 997 6
Re: BadApple report
Will do. Actually, won’t do (disable that is)…. One of the things that’s kind of a pain is that the report doesn’t distinguish between different JVMs so there’s no really convenient way to ignore this kind of thing. Anyway, I’ve put both of them in my list, and I have to say I’m not actively annotating things at this point. > On Jan 6, 2020, at 12:40 PM, Robert Muir wrote: > > Same goes for TestPackedInts. Currently test runs containing ZGC or > Shenandoah garbage collectors don't reflect the test itself. Please don't > disable them. > > On Mon, Jan 6, 2020 at 12:38 PM Robert Muir wrote: > We shouldn't disable Test2BPostings since there is nothing wrong with the > test: this is one impacted by bugs in the Shenandoah and ZGC garbage > collectors. See the other threads on the dev-list about them. > > On Mon, Jan 6, 2020 at 10:47 AM Erick Erickson > wrote: > Short form: > > There were 1480 unannotated tests that failed in Hoss' rollups. Ordered by > the date I downloaded the rollup file, newest->oldest. See above for the > dates the files were collected > These tests were NOT BadApple'd or AwaitsFix'd > All tests that failed 4 weeks running will be BadApple'd unless there are > objections > > Failures in the last 4 reports.. >Report Pct runsfails test > 0123 2.4 1031 36 > LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud > 0123 0.9 1042 17 RollingRestartTest.test > 0123 0.9 1054 23 SystemCollectionCompatTest.testBackCompat > 0123 18.9 127 23 Test2BPostings.test > 0123 0.3 1037 36 > TestCloudSearcherWarming.testRepFactor1LeaderStartup > 0123 1.3 1090 51 > TestModelManagerPersistence.testFilePersistence > 0123 1.6 1089 50 > TestModelManagerPersistence.testWrapperModelPersistence > 0123 0.3 1123 4 TestPackedInts.testPackedLongValues > 0123 0.9 1029 9 > TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast > 0123 0.3 1036 12 > TestSkipOverseerOperations.testSkipLeaderOperations > 0123 2.3 1072 25 TestStressLiveNodes.testStress > 0123 52.3 155 50 > TestXYMultiPolygonShapeQueries.testRandomBig > Will BadApple all tests above this line except ones listed at > the top** > > > full report attached: > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: BadApple report
Same goes for TestPackedInts. Currently test runs containing ZGC or Shenandoah garbage collectors don't reflect the test itself. Please don't disable them. On Mon, Jan 6, 2020 at 12:38 PM Robert Muir wrote: > We shouldn't disable Test2BPostings since there is nothing wrong with the > test: this is one impacted by bugs in the Shenandoah and ZGC garbage > collectors. See the other threads on the dev-list about them. > > On Mon, Jan 6, 2020 at 10:47 AM Erick Erickson > wrote: > >> Short form: >> >> There were 1480 unannotated tests that failed in Hoss' rollups. Ordered >> by the date I downloaded the rollup file, newest->oldest. See above for the >> dates the files were collected >> These tests were NOT BadApple'd or AwaitsFix'd >> All tests that failed 4 weeks running will be BadApple'd unless there are >> objections >> >> Failures in the last 4 reports.. >>Report Pct runsfails test >> 0123 2.4 1031 36 >> LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud >> 0123 0.9 1042 17 RollingRestartTest.test >> 0123 0.9 1054 23 >> SystemCollectionCompatTest.testBackCompat >> 0123 18.9 127 23 Test2BPostings.test >> 0123 0.3 1037 36 >> TestCloudSearcherWarming.testRepFactor1LeaderStartup >> 0123 1.3 1090 51 >> TestModelManagerPersistence.testFilePersistence >> 0123 1.6 1089 50 >> TestModelManagerPersistence.testWrapperModelPersistence >> 0123 0.3 1123 4 TestPackedInts.testPackedLongValues >> 0123 0.9 1029 9 >> TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast >> 0123 0.3 1036 12 >> TestSkipOverseerOperations.testSkipLeaderOperations >> 0123 2.3 1072 25 TestStressLiveNodes.testStress >> 0123 52.3 155 50 >> TestXYMultiPolygonShapeQueries.testRandomBig >> Will BadApple all tests above this line except ones listed >> at the top** >> >> >> full report attached: >> >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org > >
Re: BadApple report
We shouldn't disable Test2BPostings since there is nothing wrong with the test: this is one impacted by bugs in the Shenandoah and ZGC garbage collectors. See the other threads on the dev-list about them. On Mon, Jan 6, 2020 at 10:47 AM Erick Erickson wrote: > Short form: > > There were 1480 unannotated tests that failed in Hoss' rollups. Ordered by > the date I downloaded the rollup file, newest->oldest. See above for the > dates the files were collected > These tests were NOT BadApple'd or AwaitsFix'd > All tests that failed 4 weeks running will be BadApple'd unless there are > objections > > Failures in the last 4 reports.. >Report Pct runsfails test > 0123 2.4 1031 36 > LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud > 0123 0.9 1042 17 RollingRestartTest.test > 0123 0.9 1054 23 > SystemCollectionCompatTest.testBackCompat > 0123 18.9 127 23 Test2BPostings.test > 0123 0.3 1037 36 > TestCloudSearcherWarming.testRepFactor1LeaderStartup > 0123 1.3 1090 51 > TestModelManagerPersistence.testFilePersistence > 0123 1.6 1089 50 > TestModelManagerPersistence.testWrapperModelPersistence > 0123 0.3 1123 4 TestPackedInts.testPackedLongValues > 0123 0.9 1029 9 > TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast > 0123 0.3 1036 12 > TestSkipOverseerOperations.testSkipLeaderOperations > 0123 2.3 1072 25 TestStressLiveNodes.testStress > 0123 52.3 155 50 > TestXYMultiPolygonShapeQueries.testRandomBig > Will BadApple all tests above this line except ones listed at > the top** > > > full report attached: > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org
BadApple report
Short form: There were 1480 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 2.4 1031 36 LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud 0123 0.9 1042 17 RollingRestartTest.test 0123 0.9 1054 23 SystemCollectionCompatTest.testBackCompat 0123 18.9 127 23 Test2BPostings.test 0123 0.3 1037 36 TestCloudSearcherWarming.testRepFactor1LeaderStartup 0123 1.3 1090 51 TestModelManagerPersistence.testFilePersistence 0123 1.6 1089 50 TestModelManagerPersistence.testWrapperModelPersistence 0123 0.3 1123 4 TestPackedInts.testPackedLongValues 0123 0.9 1029 9 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0123 0.3 1036 12 TestSkipOverseerOperations.testSkipLeaderOperations 0123 2.3 1072 25 TestStressLiveNodes.testStress 0123 52.3 155 50 TestXYMultiPolygonShapeQueries.testRandomBig Will BadApple all tests above this line except ones listed at the top** full report attached: DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey TestLatLonShapeQueries.testRandomBig TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2020-01-06.csv Processing file (History bit 2): HOSS-2019-12-30.csv Processing file (History bit 1): HOSS-2019-12-23.csv Processing file (History bit 0): HOSS-2019-12-09.csv Number of AwaitsFix: 41 Number of BadApples: 6 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 298 failures Week: 1 had 84 failures Week: 2 had 108 failures Week: 3 had 1170 failures **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file no tests removed **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 2 MultiThreadedOCPTest.test SolrZkClientTest.testSimpleUpdateACLs Failures in Hoss' reports for the last 4 rollups. There were 1480 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 2.4 1031 36 LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud 0123 0.9 1042 17 RollingRestartTest.test 0123 0.9 1054 23 SystemCollectionCompatTest.testBackCompat 0123 18.9 127 23 Test2BPostings.test 0123 0.3 1037 36 TestCloudSearcherWarming.testRepFactor1LeaderStartup 0123 1.3 1090 51 TestModelManagerPersistence.testFilePersistence 0123 1.6 1089 50 TestModelManagerPersistence.testWrapperModelPersistence 0123 0.3 1123 4 TestPackedInts.testPackedLongValues 0123 0.9 1029 9 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0123 0.3 1036 12 TestSkipOverseerOperations.testSkipLeaderOperations 0123 2.3 1072 25 TestStressLiveNodes.testStress 0123 52.3 155 50 TestXYMultiPolygonShapeQueries.testRandomBig Will BadApple all tests above this line except ones listed at the top** 0123.6 906 38 BasicAuthIntegrationTest.testBasicAuth 0120.6 887 6 HttpPartitionWithTlogReplicasTest.test 0120.3 893 4
BadApple report
As all the security stuff settles down, I’m still taking these snapshots but mostly to keep a complete record. The longer records, i.e. for the last 7 days contains a lot of noise comparatively. That said, it’s worth looking at Hoss’ last 7 day rollup, we do have a number of tests failing quite regularly although many of those are “suite” level: http://fucit.org/solr-jenkins-reports/failure-report.html Short form: Failures in Hoss' reports for the last 4 rollups. There were 1434 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 7.7 675 15 DimensionalRoutedAliasUpdateProcessorTest.testCatTime 0123 1.6 801 44 LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud 0123 1.9 805 14 RollingRestartTest.test 0123 2.9 99 5 ShardSplitTest.testSplitWithChaosMonkey 0123 3.6 807 18 SystemCollectionCompatTest.testBackCompat 0123 18.9 115 16 Test2BPostings.test 0123 3.6 862 51 TestModelManagerPersistence.testFilePersistence 0123 3.6 865 54 TestModelManagerPersistence.testWrapperModelPersistence 0123 0.8 781 6 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0123 2.6 809 16 TestStressLiveNodes.testStress Full output attached DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey TestLatLonShapeQueries.testRandomBig TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2019-12-23.csv Processing file (History bit 2): HOSS-2019-12-09.csv Processing file (History bit 1): HOSS-2019-12-02.csv Processing file (History bit 0): HOSS-2019-11-25.csv Number of AwaitsFix: 40 Number of BadApples: 7 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 108 failures Week: 1 had 1170 failures Week: 2 had 83 failures Week: 3 had 253 failures **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file no tests removed **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 2 SolrZkClientTest.testSimpleUpdateACLs TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader Failures in Hoss' reports for the last 4 rollups. There were 1434 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 7.7 675 15 DimensionalRoutedAliasUpdateProcessorTest.testCatTime 0123 1.6 801 44 LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud 0123 1.9 805 14 RollingRestartTest.test 0123 2.9 99 5 ShardSplitTest.testSplitWithChaosMonkey 0123 3.6 807 18 SystemCollectionCompatTest.testBackCompat 0123 18.9 115 16 Test2BPostings.test 0123 3.6 862 51 TestModelManagerPersistence.testFilePersistence 0123 3.6 865 54 TestModelManagerPersistence.testWrapperModelPersistence 0123 0.8 781 6 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0123 2.6 809 16 TestStressLiveNodes.testStress Will BadApple all tests above this line except ones listed at the top** 0120.8 600 6 MissingSegmentRecoveryTest.testLeaderRecovery 0120.4 658 3
Badapple report
Short form: Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 83 failures Week: 1 had 253 failures Week: 2 had 56 failures Week: 3 had 66 failures Failures in the last 4 reports.. Report Pct runsfails test 0123 16.7 839 82 BasicAuthIntegrationTest.testBasicAuth 0123 1.8 828 10 DimensionalRoutedAliasUpdateProcessorTest.testCatTime 0123 1.8 828 21 DimensionalRoutedAliasUpdateProcessorTest.testTimeCat 0123 8.0 94 5 HdfsBasicDistributedZkTest.test 0123 11.0 838 72 LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud 0123 77.8 144 72 MoveReplicaHDFSTest.test 0123 4.8 98 5 ShardSplitTest.testSplitWithChaosMonkey 0123 0.9 801 9 SystemCollectionCompatTest.testBackCompat 0123 1.9 817 57 TestModelManagerPersistence.testFilePersistence 0123 2.3 815 55 TestModelManagerPersistence.testWrapperModelPersistence 0123 2.2 795 10 TestStressLiveNodes.testStress Will BadApple all tests above this line except ones listed at the top** Full report attached: DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey TestLatLonShapeQueries.testRandomBig TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2019-12-02.csv Processing file (History bit 2): HOSS-2019-11-25.csv Processing file (History bit 1): HOSS-2019-11-18.csv Processing file (History bit 0): HOSS-2019-11-11.csv Number of AwaitsFix: 40 Number of BadApples: 8 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 83 failures Week: 1 had 253 failures Week: 2 had 56 failures Week: 3 had 66 failures **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file no tests removed **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 3 FullSolrCloudDistribCmdsTest.test SolrZkClientTest.testSimpleUpdateACLs TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader Failures in Hoss' reports for the last 4 rollups. There were 356 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 16.7 839 82 BasicAuthIntegrationTest.testBasicAuth 0123 1.8 828 10 DimensionalRoutedAliasUpdateProcessorTest.testCatTime 0123 1.8 828 21 DimensionalRoutedAliasUpdateProcessorTest.testTimeCat 0123 8.0 94 5 HdfsBasicDistributedZkTest.test 0123 11.0 838 72 LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud 0123 77.8 144 72 MoveReplicaHDFSTest.test 0123 4.8 98 5 ShardSplitTest.testSplitWithChaosMonkey 0123 0.9 801 9 SystemCollectionCompatTest.testBackCompat 0123 1.9 817 57 TestModelManagerPersistence.testFilePersistence 0123 2.3 815 55 TestModelManagerPersistence.testWrapperModelPersistence 0123 2.2 795 10 TestStressLiveNodes.testStress Will BadApple all tests above this line except ones listed at the top** 0120.5 577 3 ChaosMonkeyNothingIsSafeWithPullReplicasTest.test 0120.5 584 3 TestSimpleTextTermVectorsFormat.testRamBytesUsed 0120.5 487 8 TestSolrCachePerf.testGetPutCompute 0120.5 538 3 TestTlogReplica.testKillLeader 01 3 0.9 602 5 LeaderVoteWaitTimeoutTest.basicTest 01 3 1.8 630 14 RollingRestartTest.test 01
BadApple report, not a good week.
This is not a good week at all: Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 253 failures Most recent 7 days Week: 1 had 56 failures 7 days before that Week: 2 had 66 failures Week: 3 had 83 failures Going from 56 failures to 253 is A Very Bad Outcome. IDK whether this is an actual horrible regression or we’re reporting on more runs or what. This makes me fear the changes I made in SOLR-13952 since many of the failures are in the last day. I’ve decided to roll that back anyway, that effort has gone well past the point of diminishing returns so we’ll see if that magically fixes the failure rate. I’m still going to cull the gradle_8 changes for substantive changes and the suppress warnings zombie threads and push those in the next few days. Full report attached Erick DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey TestLatLonShapeQueries.testRandomBig TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2019-11-25.csv Processing file (History bit 2): HOSS-2019-11-18.csv Processing file (History bit 1): HOSS-2019-11-11.csv Processing file (History bit 0): HOSS-2019-11-04.csv Number of AwaitsFix: 40 Number of BadApples: 8 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 253 failures Week: 1 had 56 failures Week: 2 had 66 failures Week: 3 had 83 failures **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file no tests removed **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 3 FullSolrCloudDistribCmdsTest.test SolrZkClientTest.testSimpleUpdateACLs TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader Failures in Hoss' reports for the last 4 rollups. There were 345 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 8.7 910 52 BasicAuthIntegrationTest.testBasicAuth 0123 2.0 927 10 DimensionalRoutedAliasUpdateProcessorTest.testCatTime 0123 3.0 927 19 DimensionalRoutedAliasUpdateProcessorTest.testTimeCat 0123 4.0 95 4 HdfsBasicDistributedZkTest.test 0123 3.7 922 57 LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud 0123 37.5 180 78 MoveReplicaHDFSTest.test 0123 0.5 893 5 ReindexCollectionTest.testSameTargetReindexing 0123 7.1 105 6 ShardSplitTest.testSplitWithChaosMonkey 0123 1.6 901 9 SystemCollectionCompatTest.testBackCompat 0123 5.2 929 54 TestCloudSearcherWarming.testRepFactor1LeaderStartup 0123 9.6 931 73 TestModelManagerPersistence.testFilePersistence 0123 10.5 926 67 TestModelManagerPersistence.testWrapperModelPersistence 0123 1.6 912 18 TestSkipOverseerOperations.testSkipLeaderOperations 0123 1.1 895 12 TestStressLiveNodes.testStress Will BadApple all tests above this line except ones listed at the top** 0120.5 573 3 TestPullReplicaErrorHandling.testPullReplicaDisconnectsFromZooKeeper 01 3 0.5 684 6 HttpPartitionWithTlogReplicasTest.test 01 0.5 366 2 ChaosMonkeyNothingIsSafeWithPullReplicasTest.test 01 0.5 371 2 TestSimpleTextTermVectorsFormat.testRamBytesUsed 01 1.2 296 7 TestSolrCachePerf.testGetPutCompute 01 0.6 345 2 TestTlogReplica.testKillLeader 0 23 1.6 725 13 RollingRestartTest.test 0 23 3.5 733 16 SyncSliceTest.test 0 23 0.6 635 4
Badapple report. Please read the first 5 lines at least.
MoveReplicaHDFSTest.test LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud TestModelManagerPersistence all fail more than 10%, MoveReplicaHDFSTest 50%. BasicAuthIntegrationTest.testBasicAuth comes in at just under 10%. Short form: There were 147 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 8.3 913 44 BasicAuthIntegrationTest.testBasicAuth 0123 0.5 943 9 DimensionalRoutedAliasUpdateProcessorTest.testCatTime 0123 2.8 943 15 DimensionalRoutedAliasUpdateProcessorTest.testTimeCat 0123 10.4 954 84 LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud 0123 50.0 193 78 MoveReplicaHDFSTest.test 0123 3.2 911 13 RollingRestartTest.test 0123 3.6 103 5 ShardSplitTest.testSplitWithChaosMonkey 0123 6.8 929 45 TestCloudSearcherWarming.testRepFactor1LeaderStartup 0123 12.9 942 76 TestModelManagerPersistence.testFilePersistence 0123 11.4 938 71 TestModelManagerPersistence.testWrapperModelPersistence 0123 0.5 882 6 TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast 0123 1.4 913 20 TestSkipOverseerOperations.testSkipLeaderOperations 0123 1.0 899 12 TestStressLiveNodes.testStress Will BadApple all tests above this line except ones listed at the top** Full results: DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey TestLatLonShapeQueries.testRandomBig TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2019-11-11.csv Processing file (History bit 2): HOSS-2019-11-04.csv Processing file (History bit 1): HOSS-2019-10-28.csv Processing file (History bit 0): HOSS-2019-10-21.csv Number of AwaitsFix: 38 Number of BadApples: 11 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 66 failures Week: 1 had 83 failures Week: 2 had 56 failures Week: 3 had 49 failures **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file no tests removed **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 4 FullSolrCloudDistribCmdsTest.test SolrZkClientTest.testSimpleUpdateACLs TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader TestDistributedStatsComponentCardinality.test Failures in Hoss' reports for the last 4 rollups. There were 147 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 8.3 913 44 BasicAuthIntegrationTest.testBasicAuth 0123 0.5 943 9 DimensionalRoutedAliasUpdateProcessorTest.testCatTime 0123 2.8 943 15 DimensionalRoutedAliasUpdateProcessorTest.testTimeCat 0123 10.4 954 84 LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud 0123 50.0 193 78 MoveReplicaHDFSTest.test 0123 3.2 911 13 RollingRestartTest.test 0123 3.6 103 5 ShardSplitTest.testSplitWithChaosMonkey 0123 6.8 929 45 TestCloudSearcherWarming.testRepFactor1LeaderStartup 0123 12.9 942 76 TestModelManagerPersistence.testFilePersistence 0123 11.4 938 71 TestModelManagerPersistence.testWrapperModelPersistence 0123
BadApple report
It’s been a while. I think this is mostly informational. I was all excited when the reports were getting s much better, but that was an artifact of some test environments not being up and running. When Mark’s test work hits, we’ll probably have to start over. That said, people SHOULD LOOK HERE PERIODICALLY: http://fucit.org/solr-jenkins-reports/failure-report.html For instance, TestPackages has a 76% failure rate over the last week. Here’s the top failures. I’m not going to annotate for a while. There were 141 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 5.0 849 47 BasicAuthIntegrationTest.testBasicAuth 0123 0.5 884 13 DimensionalRoutedAliasUpdateProcessorTest.testCatTime 0123 2.3 884 15 DimensionalRoutedAliasUpdateProcessorTest.testTimeCat 0123 18.0 873 69 LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud 0123 0.5 832 5 MathExpressionTest.testGammaDistribution 0123 3.9 852 37 TestCloudSearcherWarming.testRepFactor1LeaderStartup 0123 10.8 855 64 TestModelManagerPersistence.testFilePersistence 0123 11.2 857 66 TestModelManagerPersistence.testWrapperModelPersistence **DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey TestLatLonShapeQueries.testRandomBig TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2019-10-28.csv Processing file (History bit 2): HOSS-2019-10-21.csv Processing file (History bit 1): HOSS-2019-10-15.csv Processing file (History bit 0): HOSS-2019-10-07.csv Number of AwaitsFix: 40 Number of BadApples: 11 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 56 failures Week: 1 had 49 failures Week: 2 had 42 failures Week: 3 had 69 failures **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file no tests removed **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 4 FullSolrCloudDistribCmdsTest.test SolrZkClientTest.testSimpleUpdateACLs TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader TestDistributedStatsComponentCardinality.test Failures in Hoss' reports for the last 4 rollups. There were 141 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 5.0 849 47 BasicAuthIntegrationTest.testBasicAuth 0123 0.5 884 13 DimensionalRoutedAliasUpdateProcessorTest.testCatTime 0123 2.3 884 15 DimensionalRoutedAliasUpdateProcessorTest.testTimeCat 0123 18.0 873 69 LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud 0123 0.5 832 5 MathExpressionTest.testGammaDistribution 0123 3.9 852 37 TestCloudSearcherWarming.testRepFactor1LeaderStartup 0123 10.8 855 64 TestModelManagerPersistence.testFilePersistence 0123 11.2 857 66 TestModelManagerPersistence.testWrapperModelPersistence Will BadApple all tests above this line except ones listed at the top** 0123.7 58 3 ShardSplitTest.testSplitWithChaosMonkey 0122.0 578 11 TestSkipOverseerOperations.testSkipLeaderOperations 01 3 4.3 49 4 Test2BPostings.test 01 3 0.5 619 5 TestStressLiveNodes.testStress 01 2.0 385 15
BadApple report
I’m going to suspend these until we build up a better backlog of tests since a number of machines weren’t being collected by Hoss’ rollups. I’ll continue to gather the rollups every week, but for a while I don’t think it’s worth cluttering your inbox. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
No BadApple report this week
I’ll probably just continue to gather Hoss’ rollups each week, but until we get the jenkins stuff back running it’s probably not worth the effort. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Badapple report
No annotation changes will happen this week. Summary: Processing file (History bit 3): HOSS-2019-088-05.csv Processing file (History bit 2): HOSS-2019-08-19.csv Processing file (History bit 1): HOSS-2019-08-12.csv Processing file (History bit 0): HOSS-2019-07-29.csv Number of AwaitsFix: 38 Number of BadApples: 11 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 28 failures Week: 1 had 18 failures Week: 2 had 21 failures Week: 3 had 47 failures **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 3 FullSolrCloudDistribCmdsTest.test SolrZkClientTest.testSimpleUpdateACLs TestDistributedStatsComponentCardinality.test Failures in Hoss' reports for the last 4 rollups. There were 80 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 1.3 365 8 AliasIntegrationTest.testClusterStateProviderAPI 0123 26.7 44 15 HdfsAutoAddReplicasIntegrationTest.testSimple Will BadApple all tests above this line except ones listed at the top** DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey TestLatLonShapeQueries.testRandomBig TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2019-088-05.csv Processing file (History bit 2): HOSS-2019-08-19.csv Processing file (History bit 1): HOSS-2019-08-12.csv Processing file (History bit 0): HOSS-2019-07-29.csv Number of AwaitsFix: 38 Number of BadApples: 11 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 28 failures Week: 1 had 18 failures Week: 2 had 21 failures Week: 3 had 47 failures **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 3 FullSolrCloudDistribCmdsTest.test SolrZkClientTest.testSimpleUpdateACLs TestDistributedStatsComponentCardinality.test Failures in Hoss' reports for the last 4 rollups. There were 80 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 1.3 365 8 AliasIntegrationTest.testClusterStateProviderAPI 0123 26.7 44 15 HdfsAutoAddReplicasIntegrationTest.testSimple Will BadApple all tests above this line except ones listed at the top** 01 3 6.2 428150 BasicAuthIntegrationTest.testBasicAuth 01 3 1.4 268 3 RollingRestartTest.test 0 23 1.0 351 6 HttpPartitionWithTlogReplicasTest.test 0 23 1.4 276 3 TestCloudJSONFacetJoinDomain.testRandom 0 3 16.7 19 3 CdcrReplicationHandlerTest.testReplicationWithBufferedUpdates 0 3 4.0 252 7 TestRandomChains.testRandomChains 0 3 3.0 252 7 TestRandomChains.testRandomChainsWithLargeStrings 0 3 1.5 201 3 TestStressLiveNodes.testStress 0 3 1.4 202 2 TestUseDocValuesAsStored.testDuplicateMultiValued 0 8.3 12 1
Badapple report
Continued improvement I think. Or at least the improvements 3 weeks ago are working their way through the system. Note that the number of tests that _only_ failed three weeks ago is almost half the total. So I have some optimism that next week we’ll see a further large drop. Here’s the synopsis, full report attached: Number of AwaitsFix: 38 Number of BadApples: 11 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 28 failures Week: 1 had 21 failures Week: 2 had 47 failures Week: 3 had 142 failures **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 3 SolrZkClientTest.testSimpleUpdateACLs TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader TestDistributedStatsComponentCardinality.test Failures in Hoss' reports for the last 4 rollups. There were 182 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 1.3 597 11 AliasIntegrationTest.testClusterStateProviderAPI 0123 26.7 65 17 HdfsAutoAddReplicasIntegrationTest.testSimple 0123 1.0 686 9 HttpPartitionWithTlogReplicasTest.test Will BadApple all tests above this line except ones listed at the top** DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey TestLatLonShapeQueries.testRandomBig TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2019-088-05.csv Processing file (History bit 2): HOSS-2019-08-12.csv Processing file (History bit 1): HOSS-2019-07-29.csv Processing file (History bit 0): HOSS-2019-07-08.csv Number of AwaitsFix: 38 Number of BadApples: 11 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 28 failures Week: 1 had 21 failures Week: 2 had 47 failures Week: 3 had 142 failures **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 3 SolrZkClientTest.testSimpleUpdateACLs TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader TestDistributedStatsComponentCardinality.test Failures in Hoss' reports for the last 4 rollups. There were 182 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 1.3 597 11 AliasIntegrationTest.testClusterStateProviderAPI 0123 26.7 65 17 HdfsAutoAddReplicasIntegrationTest.testSimple 0123 1.0 686 9 HttpPartitionWithTlogReplicasTest.test Will BadApple all tests above this line except ones listed at the top** 0121.4 276 3 TestCloudJSONFacetJoinDomain.testRandom 0 23 6.2 685176 BasicAuthIntegrationTest.testBasicAuth 0 23 16.7 49 5 CdcrReplicationHandlerTest.testReplicationWithBufferedUpdates 0 23 1.4 496 3 RollingRestartTest.test 0 23 1.5 500 10 TestStressLiveNodes.testStress 0 23 1.4 525 8 TestUseDocValuesAsStored.testDuplicateMultiValued 0 24.0 252 7
BadApple report
Interestingly, the numbers of failed test has gone down pretty radically over the last while. I skipped about 4 weeks of collecting the reports while moving, but if I compare the tests that failed during the last two weeks in the rollup from July 1 with the the last two weeks sollected today, the difference is stark: 161 .vs. 44. Note that this does not count annotated tests that fail. Here’s the short form of the current state, full report attached. Failures in Hoss' reports for the last 4 rollups. There were 252 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 1.3 741 12 AliasIntegrationTest.testClusterStateProviderAPI 0123 6.2 896177 BasicAuthIntegrationTest.testBasicAuth 0123 26.7 80 21 HdfsAutoAddReplicasIntegrationTest.testSimple 0123 1.0 809 6 HttpPartitionWithTlogReplicasTest.test 0123 1.4 711 5 RollingRestartTest.test 0123 1.4 732 9 TestUseDocValuesAsStored.testDuplicateMultiValued Will BadApple all tests above this line except ones listed at the top** DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey TestLatLonShapeQueries.testRandomBig TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2019-088-05.csv Processing file (History bit 2): HOSS-2019-07-29.csv Processing file (History bit 1): HOSS-2019-07-08.csv Processing file (History bit 0): HOSS-2019-07-01.csv Number of AwaitsFix: 38 Number of BadApples: 11 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 28 failures Week: 1 had 47 failures Week: 2 had 142 failures Week: 3 had 123 failures **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 2 SolrZkClientTest.testSimpleUpdateACLs TestDistributedStatsComponentCardinality.test Failures in Hoss' reports for the last 4 rollups. There were 252 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 1.3 741 12 AliasIntegrationTest.testClusterStateProviderAPI 0123 6.2 896177 BasicAuthIntegrationTest.testBasicAuth 0123 26.7 80 21 HdfsAutoAddReplicasIntegrationTest.testSimple 0123 1.0 809 6 HttpPartitionWithTlogReplicasTest.test 0123 1.4 711 5 RollingRestartTest.test 0123 1.4 732 9 TestUseDocValuesAsStored.testDuplicateMultiValued Will BadApple all tests above this line except ones listed at the top** 012 16.7 49 5 CdcrReplicationHandlerTest.testReplicationWithBufferedUpdates 0121.5 500 10 TestStressLiveNodes.testStress 01 1.4 203 2 TestCloudJSONFacetJoinDomain.testRandom 01 4.0 252 7 TestRandomChains.testRandomChains 01 3.0 252 7 TestRandomChains.testRandomChainsWithLargeStrings 0 23 1.0 653 5 HttpPartitionTest.test 0 23 2.7 604 34 RulesTest.doIntegrationTest 0 22.0 434 5 CollectionPropsTest.testWatcher 0 21.4 368 3 LeaderVoteWaitTimeoutTest.basicTest 0 21.3 374 2
BadApple report
Here it is after a hiatus. I have moved from California to South Orange, NJ… it’s a long story why. But I’ll be glad to tell y’all about driving a Chevy Bolt EV across country and how Wyoming has very few commercial charging options… But I did get to see Old Faithful erupt… Any, I won’t make any annotation changes this week. It’ll be a little strange for the next 3 weeks as I’ll pick up the last 4 summaries for the report and there’s a two week gap. So fixes in the last week won’t be reflected in the reports for up to 6 weeks after they were made. Full report attached. **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 1 SolrZkClientTest.testSimpleUpdateACLs Failures in Hoss' reports for the last 4 rollups. There were 338 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 2.2 896 21 AliasIntegrationTest.testClusterStateProviderAPI 0123 52.2 1046183 BasicAuthIntegrationTest.testBasicAuth 0123 0.7 921 5 CollectionPropsTest.testReadWriteCached 0123 38.5 89 23 HdfsAutoAddReplicasIntegrationTest.testSimple 0123 0.7 928 10 HttpPartitionWithTlogReplicasTest.test 0123 0.8 858 31 LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud 0123 0.8 870 9 RollingRestartTest.test 0123 21.1 112 17 ShardSplitTest.testSplitWithChaosMonkey 0123 0.8 877 9 SystemCollectionCompatTest.testBackCompat Will BadApple all tests above this line except ones listed at the top** DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey TestLatLonShapeQueries.testRandomBig TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2019-07-29.csv Processing file (History bit 2): HOSS-2019-07-08.csv Processing file (History bit 1): HOSS-2019-07-01.csv Processing file (History bit 0): HOSS-2019-06-24.csv Number of AwaitsFix: 38 Number of BadApples: 12 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 47 failures Week: 1 had 142 failures Week: 2 had 123 failures Week: 3 had 152 failures **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 1 SolrZkClientTest.testSimpleUpdateACLs Failures in Hoss' reports for the last 4 rollups. There were 338 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 2.2 896 21 AliasIntegrationTest.testClusterStateProviderAPI 0123 52.2 1046183 BasicAuthIntegrationTest.testBasicAuth 0123 0.7 921 5 CollectionPropsTest.testReadWriteCached 0123 38.5 89 23 HdfsAutoAddReplicasIntegrationTest.testSimple 0123 0.7 928 10 HttpPartitionWithTlogReplicasTest.test 0123 0.8 858 31 LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud 0123 0.8 870 9 RollingRestartTest.test 0123 21.1
Re: BadApple report
HdfsAutoAddReplicasIntegrationTest.testSimple I am going to awaitsfix this test - https://issues.apache.org/jira/browse/SOLR-13338. I haven't had time to look into recent failures. I thought the Jetty upgrade would have helped. It had very similar timeout waiting exception. Kevin Risden On Mon, Jul 1, 2019 at 12:13 PM Erick Erickson wrote: > Pretty steady, I won’t be doing anything with annotations this week: > > **Annotations will be removed from the following tests because they > haven't failed in the last 4 rollups. > > **Methods: 3 >FullSolrCloudDistribCmdsTest.test >MultiThreadedOCPTest.test >SolrZkClientTest.testSimpleUpdateACLs > > > Failures in Hoss' reports for the last 4 rollups. > > There were 585 unannotated tests that failed in Hoss' rollups. Ordered by > the date I downloaded the rollup file, newest->oldest. See above for the > dates the files were collected > These tests were NOT BadApple'd or AwaitsFix'd > All tests that failed 4 weeks running will be BadApple'd unless there are > objections > > Failures in the last 4 reports.. >Report Pct runsfails test > 0123 1.8 955 30 > AliasIntegrationTest.testClusterStateProviderAPI > 0123 0.5 972207 BasicAuthIntegrationTest.testBasicAuth > 0123 30.4 89 24 > HdfsAutoAddReplicasIntegrationTest.testSimple > 0123 0.4 921 10 HttpPartitionTest.test > 0123 0.9 924 12 NestedShardedAtomicUpdateTest.test > 0123 0.5 908 5 > ReindexCollectionTest.testBasicReindexing > 0123 0.9 928 12 RollingRestartTest.test > 0123 12.0 90 11 > ShardSplitTest.testSplitWithChaosMonkey > 0123 0.9 927 8 > SystemCollectionCompatTest.testBackCompat > 0123 0.5 926 23 > TestFieldCacheRewriteMethod.testRegexps > 0123 0.9 924 13 > TestSimpleSearchEquivalence.testBooleanBoostPropagation > 0123 0.9 924 15 > TestSimpleSearchEquivalence.testBoostQuerySimplification > 0123 0.4 924 8 > TestSimpleSearchEquivalence.testPhraseRelativePositions > 0123 0.4 924 9 > TestSimpleSearchEquivalence.testSloppyPhraseRelativePositions > 0123 1.8 908 14 TestTopDocsMerge.testSort_1 > Will BadApple all tests above this line except ones listed at > the top** > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
BadApple report
Pretty steady, I won’t be doing anything with annotations this week: **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 3 FullSolrCloudDistribCmdsTest.test MultiThreadedOCPTest.test SolrZkClientTest.testSimpleUpdateACLs Failures in Hoss' reports for the last 4 rollups. There were 585 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 1.8 955 30 AliasIntegrationTest.testClusterStateProviderAPI 0123 0.5 972207 BasicAuthIntegrationTest.testBasicAuth 0123 30.4 89 24 HdfsAutoAddReplicasIntegrationTest.testSimple 0123 0.4 921 10 HttpPartitionTest.test 0123 0.9 924 12 NestedShardedAtomicUpdateTest.test 0123 0.5 908 5 ReindexCollectionTest.testBasicReindexing 0123 0.9 928 12 RollingRestartTest.test 0123 12.0 90 11 ShardSplitTest.testSplitWithChaosMonkey 0123 0.9 927 8 SystemCollectionCompatTest.testBackCompat 0123 0.5 926 23 TestFieldCacheRewriteMethod.testRegexps 0123 0.9 924 13 TestSimpleSearchEquivalence.testBooleanBoostPropagation 0123 0.9 924 15 TestSimpleSearchEquivalence.testBoostQuerySimplification 0123 0.4 924 8 TestSimpleSearchEquivalence.testPhraseRelativePositions 0123 0.4 924 9 TestSimpleSearchEquivalence.testSloppyPhraseRelativePositions 0123 1.8 908 14 TestTopDocsMerge.testSort_1 Will BadApple all tests above this line except ones listed at the top** - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
BadApple report
I won’t change annotations again this week. Here’s the short from: **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 2 FullSolrCloudDistribCmdsTest.test SolrZkClientTest.testSimpleUpdateACLs Failures in Hoss' reports for the last 4 rollups. There were 543 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 4.3 961 38 AliasIntegrationTest.testClusterStateProviderAPI 0123 4.8 994213 BasicAuthIntegrationTest.testBasicAuth 0123 2.8 900 20 BasicDistributedZkTest.test 0123 2.8 900 14 DistributedFacetPivotLargeTest.test 0123 25.0 87 22 HdfsAutoAddReplicasIntegrationTest.testSimple 0123 0.9 911 11 HttpPartitionTest.test 0123 1.4 930 14 NestedShardedAtomicUpdateTest.test 0123 1.1 809 9 OverseerTest.testOverseerFailure 0123 0.9 915 5 ReindexCollectionTest.testBasicReindexing 0123 2.2 929 11 RollingRestartTest.test 0123 0.9 960 7 ShardSplitTest.testSplitShardWithRule 0123 13.6 88 9 ShardSplitTest.testSplitWithChaosMonkey 0123 0.9 911 7 SystemCollectionCompatTest.testBackCompat 0123 2.3 911 16 TestDocValuesRewriteMethod.testRegexps 0123 0.5 901 5 TestDynamicLoading.testDynamicLoading 0123 0.5 919 19 TestRegexpRandom2.testRegexps 0123 1.3 913 12 TestSimpleSearchEquivalence.testBooleanBoostPropagation 0123 1.3 913 14 TestSimpleSearchEquivalence.testBoostQuerySimplification 0123 0.9 913 9 TestSimpleSearchEquivalence.testSloppyPhraseRelativePositions 0123 1.9 899 13 TestTopDocsMerge.testSort_1 Full report attached: DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey TestLatLonShapeQueries.testRandomBig TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2019-06-24.csv Processing file (History bit 2): HOSS-2019-06-17.csv Processing file (History bit 1): HOSS-2019-06-10.csv Processing file (History bit 0): HOSS-2019-06-03.csv **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 2 FullSolrCloudDistribCmdsTest.test SolrZkClientTest.testSimpleUpdateACLs Failures in Hoss' reports for the last 4 rollups. There were 543 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 4.3 961 38 AliasIntegrationTest.testClusterStateProviderAPI 0123 4.8 994213 BasicAuthIntegrationTest.testBasicAuth 0123 2.8 900 20 BasicDistributedZkTest.test 0123 2.8 900 14 DistributedFacetPivotLargeTest.test 0123 25.0 87 22 HdfsAutoAddReplicasIntegrationTest.testSimple 0123 0.9 911 11 HttpPartitionTest.test 0123 1.4 930 14 NestedShardedAtomicUpdateTest.test 0123 1.1 809 9 OverseerTest.testOverseerFailure 0123 0.9 915 5 ReindexCollectionTest.testBasicReindexing 0123 2.2 929 11 RollingRestartTest.test 0123 0.9
BadApple report
Holding pretty steady, won’t remove annotations just yet. Full report attached. I _strongly_ urge people to take a quick glance at: http://fucit.org/solr-jenkins-reports/failure-report.html regularly. There are 5 tests that are failing 25% of the time or more currently. ——Report **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 3 FullSolrCloudDistribCmdsTest.test SolrZkClientTest.testSimpleUpdateACLs TestCollectionStateWatchers.testCanWaitForNonexistantCollection Failures in Hoss' reports for the last 4 rollups. There were 258 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 1.8 881 27 AliasIntegrationTest.testClusterStateProviderAPI 0123 25.0 97 27 HdfsAutoAddReplicasIntegrationTest.testSimple 0123 1.4 856 7 HttpPartitionTest.test 0123 1.8 850 13 NestedShardedAtomicUpdateTest.test 0123 11.1 88 6 ShardSplitTest.testSplitWithChaosMonkey 0123 1.4 841 15 SolrRrdBackendFactoryTest.testBasic 0123 0.5 818 5 SystemCollectionCompatTest.testBackCompat 0123 0.5 843 6 TestDynamicLoading.testDynamicLoading Will BadApple all tests above this line except ones listed at the top** DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey TestLatLonShapeQueries.testRandomBig TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2019-06-10.csv Processing file (History bit 2): HOSS-2019-06-03.csv Processing file (History bit 1): HOSS-2019-05-28.csv Processing file (History bit 0): HOSS-2019-05-20.csv **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 3 FullSolrCloudDistribCmdsTest.test SolrZkClientTest.testSimpleUpdateACLs TestCollectionStateWatchers.testCanWaitForNonexistantCollection Failures in Hoss' reports for the last 4 rollups. There were 258 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 1.8 881 27 AliasIntegrationTest.testClusterStateProviderAPI 0123 25.0 97 27 HdfsAutoAddReplicasIntegrationTest.testSimple 0123 1.4 856 7 HttpPartitionTest.test 0123 1.8 850 13 NestedShardedAtomicUpdateTest.test 0123 11.1 88 6 ShardSplitTest.testSplitWithChaosMonkey 0123 1.4 841 15 SolrRrdBackendFactoryTest.testBasic 0123 0.5 818 5 SystemCollectionCompatTest.testBackCompat 0123 0.5 843 6 TestDynamicLoading.testDynamicLoading Will BadApple all tests above this line except ones listed at the top** 012 66.1 722202 BasicAuthIntegrationTest.testBasicAuth 0120.5 616 4 DeleteReplicaTest.deleteLiveReplicaTest 0121.8 642 8 PeerSyncReplicationTest.test 0120.5 632 3 ReindexCollectionTest.testBasicReindexing 0120.4 662 4 ShardSplitTest.testSplitShardWithRule 0121.8 622 7 TestCloudRecovery2.test 0125.3 643 14 TestRegexpRandom2.testRegexps 01 3 1.0 557 5
BadApple report
I probably won’t remove the annotations indicated this week, kinda busy. Overall looks like we’re getting gradually better. Full report attached: **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 3 FullSolrCloudDistribCmdsTest.test SolrZkClientTest.testSimpleUpdateACLs TestCollectionStateWatchers.testCanWaitForNonexistantCollection Failures in Hoss' reports for the last 4 rollups. There were 199 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 5.2 902 30 AliasIntegrationTest.testClusterStateProviderAPI 0123 23.8 97 25 HdfsAutoAddReplicasIntegrationTest.testSimple 0123 0.5 848 8 LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud 0123 1.8 861 11 NestedShardedAtomicUpdateTest.test 0123 1.0 863 13 SolrRrdBackendFactoryTest.testBasic 0123 0.5 842 7 SystemCollectionCompatTest.testBackCompat Will BadApple all tests above this line except ones listed at the top** DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey TestLatLonShapeQueries.testRandomBig TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2019-06-03.csv Processing file (History bit 2): HOSS-2019-05-28.csv Processing file (History bit 1): HOSS-2019-05-20.csv Processing file (History bit 0): HOSS-2019-05-13.csv **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 3 FullSolrCloudDistribCmdsTest.test SolrZkClientTest.testSimpleUpdateACLs TestCollectionStateWatchers.testCanWaitForNonexistantCollection Failures in Hoss' reports for the last 4 rollups. There were 199 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 5.2 902 30 AliasIntegrationTest.testClusterStateProviderAPI 0123 23.8 97 25 HdfsAutoAddReplicasIntegrationTest.testSimple 0123 0.5 848 8 LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud 0123 1.8 861 11 NestedShardedAtomicUpdateTest.test 0123 1.0 863 13 SolrRrdBackendFactoryTest.testBasic 0123 0.5 842 7 SystemCollectionCompatTest.testBackCompat Will BadApple all tests above this line except ones listed at the top** 0120.9 637 4 HttpPartitionTest.test 0121.0 612 6 JWTAuthPluginIntegrationTest.testMetrics 0124.3 70 4 ShardSplitTest.testSplitWithChaosMonkey 0120.5 601 6 StreamDecoratorTest.testParallelCommitStream 0120.5 636 5 TestDynamicLoading.testDynamicLoading 01 3 3.0 684 16 BasicAuthIntegrationTest.testBasicAuth 01 3 1.0 643 4 DeleteReplicaTest.deleteLiveReplicaTest 01 3 0.9 659 7 PeerSyncReplicationTest.test 01 3 0.9 673 4 ShardSplitTest.testSplitShardWithRule 01 3 1.0 627 4 TestCloudRecovery2.test 01 3 0.6 542 3 TestDelegationWithHadoopAuth.testDelegationTokenRenew 01 0.5 420 2
BadApple report, things are changing
things are settled down quite a bit. So ongoing I’ll publish this each week, but will only periodically change the annotations. If/when we stop running 7x Jenkins jobs, I may start annotating with BadApple again, we’ll see. Meanwhile I’ll post the list of new test failures over the last 4 weeks and attach the full report, but won’t change the source for a while. Failures in the last 4 reports.. Report Pct runsfails test 0123 6.9 137 14 HdfsUnloadDistributedZkTest.test 0123 3.0 1334 32 LeaderTragicEventTest.test 0123 0.4 1306 11 MathExpressionTest.testGammaDistribution 0123 1.5 1321 10 MissingSegmentRecoveryTest.testLeaderRecovery 0123 0.8 1315 6 OverseerRolesTest.testOverseerRole 0123 0.4 1330 12 TestSimExtremeIndexing.testScaleUp Will BadApple all tests above this line except ones listed at the top** e-mail-2019-02-18.txt Description: application/applefile - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
BadApple report
Well, I didn't add stuff last week, slipped through the cracks. Anyway, here's the current list. NOTE: lots more tests are being un-annotated than annotated, which is good. Also, this last report has 421 total tests that failed sometime in the last 4 weeks. The report before had 655. Still quite a ways to go, but nice progress! **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 25 CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap ComputePlanActionTest.testNodeAdded ComputePlanActionTest.testNodeLostTriggerWithDeleteNodePreferredOp CustomCollectionTest.testRouteFieldForHashRouter DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplicaLegacy MathExpressionTest.testMultiVariateNormalDistribution ScheduledTriggerIntegrationTest.testScheduledTrigger ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitMixedReplicaTypesLink SolrRrdBackendFactoryTest.testBasic StreamDecoratorTest.testParallelExecutorStream StreamingTest.testParallelMergeStream StreamingTest.testZeroParallelReducerStream TestCloudRecovery.corruptedLogTest TestDistribIDF.testMultiCollectionQuery TestIndexWriterOnVMError.testCheckpoint TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName TestPullReplica.testCreateDelete TestSkipOverseerOperations.testSkipDownOperations TestStressInPlaceUpdates.stressTest TestTlogReplica.testCreateDelete TestWithCollection.testAddReplicaWithPolicy TestWithCollection.testNodeAdded TimeRoutedAliasUpdateProcessorTest.test ZkShardTermsTest.testParticipationOfReplicas Failures in Hoss' reports for the last 4 rollups. There were 421 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 28.6 74 25 LIROnShardRestartTest.testAllReplicasInLIR 0123 1.1 1682 21 TestSQLHandler.doTest 0123 0.4 670 12 TestSimTriggerIntegration.testCooldown 0123 0.3 1280 20 TestSimTriggerIntegration.testListeners 0123 0.2 2018 87 TestSimTriggerIntegration.testNodeLostTriggerRestoreState 0123 8.8 669179 TestSimTriggerIntegration.testNodeMarkersRegistration Will BadApple all tests above this line except ones listed at the top** Erick DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey TestLatLonShapeQueries.testRandomBig TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate Processing file (History bit 3): HOSS-2019-01-15.csv Processing file (History bit 2): HOSS-2019-01-08.csv Processing file (History bit 1): HOSS-2018-12-31.csv Processing file (History bit 0): HOSS-2018-12-24.csv **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 25 CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap ComputePlanActionTest.testNodeAdded ComputePlanActionTest.testNodeLostTriggerWithDeleteNodePreferredOp CustomCollectionTest.testRouteFieldForHashRouter DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplicaLegacy MathExpressionTest.testMultiVariateNormalDistribution ScheduledTriggerIntegrationTest.testScheduledTrigger ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitMixedReplicaTypesLink SolrRrdBackendFactoryTest.testBasic StreamDecoratorTest.testParallelExecutorStream StreamingTest.testParallelMergeStream StreamingTest.testZeroParallelReducerStream TestCloudRecovery.corruptedLogTest TestDistribIDF.testMultiCollectionQuery TestIndexWriterOnVMError.testCheckpoint TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName TestPullReplica.testCreateDelete
BadApple report for Monday
Well, I missed two weeks in a row. So sue me ;). This week fer sure Here's the condensed report. Let me know if there are any issues. Full report attached. DO NOT ENABLE LIST: 'TestControlledRealTimeReopenThread.testCRTReopen' 'TestICUNormalizer2CharFilter.testRandomStrings' 'TestICUTokenizerCJK' 'TestImpersonationWithHadoopAuth.testForwarding' 'TestLTRReRankingPipeline.testDifferentTopN' 'TestRandomChains' DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild MaxSizeAutoCommitTest ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey TestLatLonShapeQueries.testRandomBig TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate TestWithCollection **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 15 HdfsUnloadDistributedZkTest HdfsWriteToMultipleCollectionsTest LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud MetricsHistoryHandlerTest.testBasic MoveReplicaHDFSTest.testNormalFailedMove OverseerRolesTest.testOverseerRole RestartWhileUpdatingTest.test SolrJmxReporterCloudTest.testJmxReporter StreamDecoratorTest.testClassifyStream TestCollectionStateWatchers.testSimpleCollectionWatch TestCollectionStateWatchers.testWaitForStateWatcherIsRetainedOnPredicateFailure TestCollectionStateWatchers.testWatchesWorkForStateFormat1 TestLocalFSCloudBackupRestore.test TestWithCollection.testDeleteWithCollection TestWithCollection.testMoveReplicaWithCollection Will BadApple these Failures in the last 4 reports.. Report Pct runsfails test 0123 0.2 1868 7 BasicDistributedZkTest.test 0123 2.5 1909 59 CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap 0123 0.2 1846 4 CdcrOpsAndBoundariesTest.testOps 0123 0.4 1850 6 CdcrWithNodesRestartsTest.testReplicationAfterLeaderChange 0123 1.2 1873 16 CollectionsAPIAsyncDistributedZkTest.testAsyncIdRaceCondition 0123 1.2 1952 20 ComputePlanActionTest.testNodeAdded 0123 6.4 1952 79 ComputePlanActionTest.testNodeAddedTriggerWithAddReplicaPreferredOp_2Shard 0123 4.5 1952 49 ComputePlanActionTest.testNodeLostTriggerWithDeleteNodePreferredOp 0123 1.0 1862 16 CustomHighlightComponentTest.test 0123 0.6 1859 9 DeleteReplicaTest.deleteReplicaFromClusterState 0123 3.6 1897 43 DistributedMLTComponentTest.test 0123 0.4 1889 11 LargeVolumeBinaryJettyTest.testMultiThreaded 0123 1.2 1892 11 LargeVolumeJettyTest.testMultiThreaded 0123 3.1 1920 67 MetricTriggerIntegrationTest.testMetricTrigger 0123 14.4 1607187 MoveReplicaHDFSTest.testFailedMove 0123 2.3 1565 42 ScheduledTriggerIntegrationTest.testScheduledTrigger 0123 1.0 1938 13 ShardSplitTest.testSplitMixedReplicaTypesLink 0123 1.2 1918 15 StreamDecoratorTest.testParallelRollupStream 0123 0.4 1855 6 TestCloudRecovery.corruptedLogTest 0123 0.2 1865 10 TestDistribIDF.testMultiCollectionQuery 0123 0.4 1848 6 TestDistributedStatsComponentCardinality.test 0123 37.5 112 24 TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit 0123 6.5 118 6 TestIndexWriterOnVMError.testCheckpoint 0123 0.4 1862 11 TestLTROnSolrCloud.testSimpleQuery 0123 2.6 1925108 TestSimComputePlanAction.testNodeAdded 0123 0.8 1924 10 TestSimComputePlanAction.testNodeLost 0123 0.4 1860 6 TestSimExecutePlanAction.testIntegration 0123 2.9 1854 45 TestSimGenericDistributedQueue(suite) 0123 1.3 3909 30 TestSimGenericDistributedQueue.testDistributedQueue 0123 1.2 1950 21 TestSimGenericDistributedQueue.testDistributedQueueBlocking 0123 3.0 1957 55 TestSimLargeCluster.testNodeLost 0123 0.4 1957 19 TestSimLargeCluster.testSearchRate 0123 6.9 1918 58 TestSimPolicyCloud.testCreateCollectionAddReplica 0123 1.3 2071 13 TestSimTriggerIntegration.testEventFromRestoredState 0123 1.5 2069 47 TestSimTriggerIntegration.testEventQueue 0123 0.4 2071 7 TestSimTriggerIntegration.testNodeLostTrigger 0123 2.4 2071 35 TestSimTriggerIntegration.testNodeLostTriggerRestoreState 0123 8.0 2071106
BadApple report, 60+ tests to be annotated
This is a pretty bad week. 60+ tests to be annotated and only 4 to be un-annotated. Here's the culled list, full report attached. **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 4 MoveReplicaHDFSTest.testNormalFailedMove MultiThreadedOCPTest.test TestReplicationHandler.doTestStressReplication TestSolrCloudWithDelegationTokens.testDelegationTokenRenew Failures in Hoss' reports for the last 4 rollups. There were 624 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 0.2 1737 4 CloudSolrClientBuilderTest.test0Timeouts 0123 0.2 1737 4 CloudSolrClientBuilderTest.testByDefaultConfiguresClientToSendUpdatesOnlyToShardLeaders 0123 0.2 1737 4 CloudSolrClientBuilderTest.testIsDirectUpdatesToLeadersOnlyDefault 0123 0.2 1737 4 CloudSolrClientBuilderTest.testSeveralZkHostsSpecifiedSingly 0123 0.2 1737 4 CloudSolrClientBuilderTest.testSeveralZkHostsSpecifiedTogether 0123 0.2 1737 4 CloudSolrClientBuilderTest.testSingleZkHostSpecified 0123 0.2 1736 4 CloudSolrClientMultiConstructorTest.testBadChroot 0123 0.2 1736 5 CloudSolrClientMultiConstructorTest.testZkConnectionStringConstructorWithValidChroot 0123 0.2 1736 5 CloudSolrClientMultiConstructorTest.testZkConnectionStringSetterWithValidChroot 0123 0.8 1711 12 CollectionsAPIAsyncDistributedZkTest.testAsyncRequests 0123 0.2 1734 4 ConcurrentUpdateSolrClientBuilderTest.testMissingQueueSize 0123 0.6 1681 10 CustomHighlightComponentTest(suite) 0123 0.9 1710 21 DistributedDebugComponentTest.testCompareWithNonDistributedRequest 0123 0.8 1699 14 DocValuesNotIndexedTest(suite) 0123 40.0 90 40 HdfsBasicDistributedZkTest(suite) 0123 73.5 106 74 HdfsCollectionsAPIDistributedZkTest(suite) 0123 0.2 1735 4 HttpClientUtilTest.testSSLSystemProperties 0123 0.2 1735 4 HttpClientUtilTest.testToBooleanDefaultIfNull 0123 0.2 1735 4 HttpClientUtilTest.testToBooleanObject 0123 0.2 1731 5 HttpSolrClientBuilderTest(suite) 0123 0.2 1732 5 LBHttpSolrClientBuilderTest(suite) 0123 0.2 1736 5 LBHttpSolrClientTest.testLBHttpSolrClientHttpClientResponseParserStringArray 0123 0.2 1772 6 MathExpressionTest.testMultiVariateNormalDistribution 0123 4.2 1660 59 MoveReplicaHDFSTest.testFailedMove 0123 0.2 1735 4 NamedListTest.testRemoveArgs 0123 0.2 1735 4 NamedListTest.testShallowMap 0123 0.2 1734 4 QueryResponseTest.testGroupResponse 0123 0.2 1734 4 QueryResponseTest.testIntervalFacetsResponse 0123 0.2 1734 4 QueryResponseTest.testRangeFacets 0123 0.2 1734 4 QueryResponseTest.testSimpleGroupResponse 0123 0.7 1271 8 SaslZkACLProviderTest(suite) 0123 1.1 1553 19 ScheduledTriggerTest.testTrigger 0123 0.2 1737 4 ShardParamsTest.testGetShardsTolerantAsBool 0123 0.2 1737 5 SolrExceptionTest.testSolrException 0123 0.2 1736 4 SolrParamTest.testGetParams 0123 0.2 1734 4 StreamExpressionToExpessionTest.testDaemonStream 0123 0.2 1734 4 StreamExpressionToExpessionTest.testUpdateStream 0123 0.2 1734 4 StreamExpressionToExplanationTest.testDaemonStream 0123 0.2 1734 4 StreamExpressionToExplanationTest.testUpdateStream 0123 0.2 1735 4 TestCollectionAdminRequest.testInvalidAliasNameRejectedWhenCreatingAlias 0123 0.2 1735 4 TestCollectionAdminRequest.testInvalidCollectionNameRejectedWhenCreatingCollection 0123 0.2 1735 4 TestCollectionAdminRequest.testInvalidShardNameRejectedWhenCreatingShard 0123 0.2 1735 4 TestCollectionAdminRequest.testInvalidShardNamesRejectedWhenCallingSetShards 0123 0.2 1735 4 TestCollectionAdminRequest.testInvalidShardNamesRejectedWhenCreatingImplicitCollection 0123 0.2 1733 4 TestDelegationTokenResponse.testGetResponse 0123 0.2 1733 4 TestDelegationTokenResponse.testRenewResponse 0123 0.2 1733 4 TestDocumentObjectBinder.testDynamicFieldBinding 0123 0.2 1733 4 TestDocumentObjectBinder.testSimple 0123 0.2 1735
Re: BadApple report, PLEASE CHECK THE FIRST PART.
Hi Erick, Le lun. 10 sept. 2018 à 20:06, Erick Erickson a écrit : > First, I have these two lists, are they still current? > > DO NOT ENABLE LIST: > 'TestControlledRealTimeReopenThread.testCRTReopen' > 'TestICUNormalizer2CharFilter.testRandomStrings' > 'TestICUTokenizerCJK' > +1 to keep these tests disabled > 'TestRandomChains' > This suite doesn't look disabled today? > DO NOT ANNOTATE LIST > TestLatLonShapeQueries.testRandomBig > TestRandomChains.testRandomChainsWithLargeStrings > +1 to not disable those
BadApple report, PLEASE CHECK THE FIRST PART.
First, I have these two lists, are they still current? DO NOT ENABLE LIST: 'TestControlledRealTimeReopenThread.testCRTReopen' 'TestICUNormalizer2CharFilter.testRandomStrings' 'TestICUTokenizerCJK' 'TestImpersonationWithHadoopAuth.testForwarding' 'TestLTRReRankingPipeline.testDifferentTopN' 'TestRandomChains' DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild MaxSizeAutoCommitTest ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey TestLatLonShapeQueries.testRandomBig TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate TestWithCollection Second, I've gotten a bit more clarity on suite-level failures and may be un-BadApple-ing certain of them. Basically, if all _tests_ in a suite are annotated and we still get suite-level failures, that's valuable information as it implicates the framework and/or setup/teardown code in the class or superclass. *You can stop reading now. **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 13 AutoAddReplicasIntegrationTest.testSimple CloudSolrClientTest.preferReplicaTypesTest DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica DocValuesNotIndexedTest.testGroupingDVOnly FullSolrCloudDistribCmdsTest.test GraphTest.testShortestPathStream LIRRollingUpdatesTest.testNewLeaderAndMixedReplicas LIRRollingUpdatesTest.testNewLeaderOldReplica LIRRollingUpdatesTest.testNewReplicaOldLeader MoveReplicaHDFSTest.testNormalFailedMove ScheduledTriggerIntegrationTest.testScheduledTrigger SolrCloudReportersTest.testDefaultPlugins TestHdfsCloudBackupRestore.test **Suites: 0 Failures in Hoss' reports for the last 4 rollups. There were 605 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 36.4 78 34 HdfsBasicDistributedZkTest(suite) 0123 0.5 1578 6 MetricsHistoryHandlerTest.testBasic 0123 1.8 1308 46 MoveReplicaHDFSTest.testFailedMove 0123 0.7 1153 8 SaslZkACLProviderTest(suite) 0123 0.3 1156 6 SaslZkACLProviderTest.testSaslZkACLProvider 0123 1.0 1366 12 SearchRateTriggerIntegrationTest.testBelowSearchRate 0123 1.2 1571 15 ShardSplitTest.testSplitAfterFailedSplit 0123 0.7 1571 23 ShardSplitTest.testSplitMixedReplicaTypesLink 0123 4.1 1591 60 TestSQLHandler(suite) 0123 4.2 1653 63 TestSQLHandler.doTest 0123 0.3 1584 4 TestTlogReplica(suite) 0123 0.5 1738 6 TestWithCollection.testNodeAdded 0123 1.3 1619 19 ZkShardTermsTest.testParticipationOfReplicas 0123 2.3 1466 31 ZookeeperStatusHandlerTest(suite) Will BadApple all tests above this line except ones listed at the top** DO NOT ENABLE LIST: 'TestControlledRealTimeReopenThread.testCRTReopen' 'TestICUNormalizer2CharFilter.testRandomStrings' 'TestICUTokenizerCJK' 'TestImpersonationWithHadoopAuth.testForwarding' 'TestLTRReRankingPipeline.testDifferentTopN' 'TestRandomChains' DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild MaxSizeAutoCommitTest ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey TestLatLonShapeQueries.testRandomBig TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate TestWithCollection Processing file (History bit 3): HOSS-2018-09-03.csv Processing file (History bit 2): HOSS-2018-08-27.csv Processing file (History bit 1): HOSS-2018-08-20.csv Processing file (History bit 0): HOSS-2018-08-13.csv **Annotated tests/suites that didn't fail in the last 4 weeks. **Tests and suites removed from the next two lists because they were specified in 'doNotEnable' in the properties file no tests removed **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 33
Re: BadApple report TestPolicy, TestCollectionStateWatchers TestWithCollection
Sure, won't BadApple TestWithCollection. On Mon, Aug 27, 2018 at 10:01 PM Shalin Shekhar Mangar wrote: > > Thanks Erick. I'm working on fixing TestWithCollection so please do not > BadApple it this week. > > On Tue, Aug 28, 2018 at 1:04 AM Erick Erickson > wrote: >> >> On the plus side, the CDCR tests (except BiDir) seem to be fixed. >> >> Also on the plus side, there are quite a number of tests that have >> _not_ failed in the last 4 weeks and I'll un-annotate. >> >> On the minus side, TestPolicy has 39 tests that have failed at least >> once in the last 4 weeks. I'll beast this to try to produce some data >> as I hope that there's a single underlying cause. >> >> **Annotated tests/suites that didn't fail in the last 4 weeks. >> >> **Annotations will be removed from the following tests because they >> haven't failed in the last 4 rollups. >> >> **Methods: 30 >>CollectionsAPIAsyncDistributedZkTest.testAsyncIdRaceCondition >>DistributedMLTComponentTest.test >>GraphExpressionTest.testShortestPathStream >>LargeVolumeJettyTest >>LeaderElectionIntegrationTest.testSimpleSliceLeaderElection >>MathExpressionTest.testDistributions >>MoveReplicaHDFSTest.testNormalFailedMove >>SchemaApiFailureTest.testAddTheSameFieldTwice >>SearchRateTriggerTest.testTrigger >>TestDelegationWithHadoopAuth.testDelegationTokenRenew >>TestDistribIDF.testMultiCollectionQuery >>TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit >>TestManagedResourceStorage >>TestSimExecutePlanAction.testExecute >>TestSimGenericDistributedQueue >>TestSimGenericDistributedQueue.testDistributedQueue >>TestSimLargeCluster.testAddNode >>TestSimLargeCluster.testBasic >>TestSimLargeCluster.testNodeLost >>TestSimTriggerIntegration.testCooldown >>TestSimTriggerIntegration.testEventFromRestoredState >>TestSimTriggerIntegration.testEventQueue >>TestSimTriggerIntegration.testListeners >>TestSimTriggerIntegration.testNodeAddedTrigger >>TestSimTriggerIntegration.testNodeAddedTriggerRestoreState >>TestSimTriggerIntegration.testNodeLostTrigger >>TestSimTriggerIntegration.testNodeLostTriggerRestoreState >>TestSimTriggerIntegration.testNodeMarkersRegistration >>TestSimTriggerIntegration.testTriggerThrottling >>TestStressCloudBlindAtomicUpdates.test_dv_idx >> >> **Suites: 0 >> >> >> Failures in Hoss' reports for the last 4 rollups. >> >> There were 571 unannotated tests that failed in Hoss' rollups. Ordered >> by the date I downloaded the rollup file, newest->oldest. See above >> for the dates the files were collected >> These tests were NOT BadApple'd or AwaitsFix'd >> All tests that failed 4 weeks running will be BadApple'd unless there >> are objections >> >> Failures in the last 4 reports.. >>Report Pct runsfails test >> 0123 0.7 1749 8 >> CdcrBootstrapTest.testBootstrapWithContinousIndexingOnSourceCluster >> 0123 1.6 1751 18 CustomHighlightComponentTest.test >> 0123 0.5 1582 8 >> DeleteReplicaTest.deleteReplicaFromClusterState >> 0123 42.9 101 14 HdfsBasicDistributedZk2Test.test >> 0123 1.4 1741 21 JdbcTest(suite) >> 0123 10.3 96 6 >> LIROnShardRestartTest.testAllReplicasInLIR >> 0123 1.8 1801 29 LeaderVoteWaitTimeoutTest.basicTest >> 0123 1.8 1602 32 >> LeaderVoteWaitTimeoutTest.testMostInSyncReplicasCanWinElection >> 0123 4.8 849 46 MoveReplicaHDFSTest.testFailedMove >> 0123 4.5 1515 40 SchemaApiFailureTest(suite) >> 0123 0.7 1741 14 StreamingTest(suite) >> 0123 0.2 1764 11 StreamingTest.testParallelMergeStream >> 0123 0.2 1764 4 >> StreamingTest.testZeroParallelReducerStream >> 0123 0.5 1729 14 SystemLogListenerTest.test >> 0123 0.2 1537 4 >> TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader >> 0123 0.2 1770 21 >> TestCollectionStateWatchers.testCanWaitForNonexistantCollection >> 0123 0.2 1770 21 >> TestCollectionStateWatchers.testDeletionsTriggerWatches >> 0123 0.2 1770 11 >> TestCollectionStateWatchers.testWaitForStateChecksCurrentState >> 0123 5.8 286 9 TestLargeCluster.testBasic >> 0123 2.3 286 35 TestLargeCluster.testNodeLost >> 0123 4.9 1674 40 TestLargeCluster.testSearchRate >> 0123 1.4 1803 35 TestPolicy.testComputePlanAfterNodeAdded >> 0123 1.4 1802 34 TestPolicy.testConditionsSort >> 0123 1.4 1803 35 TestPolicy.testCoresSuggestions >> 0123 1.4 1800 32 TestPolicy.testDiskSpaceHint >> 0123 2.5 1808 40 TestPolicy.testDiskSpaceReqd >> 0123 1.4 1806 38
Re: BadApple report TestPolicy, TestCollectionStateWatchers TestWithCollection
Thanks Erick. I'm working on fixing TestWithCollection so please do not BadApple it this week. On Tue, Aug 28, 2018 at 1:04 AM Erick Erickson wrote: > On the plus side, the CDCR tests (except BiDir) seem to be fixed. > > Also on the plus side, there are quite a number of tests that have > _not_ failed in the last 4 weeks and I'll un-annotate. > > On the minus side, TestPolicy has 39 tests that have failed at least > once in the last 4 weeks. I'll beast this to try to produce some data > as I hope that there's a single underlying cause. > > **Annotated tests/suites that didn't fail in the last 4 weeks. > > **Annotations will be removed from the following tests because they > haven't failed in the last 4 rollups. > > **Methods: 30 >CollectionsAPIAsyncDistributedZkTest.testAsyncIdRaceCondition >DistributedMLTComponentTest.test >GraphExpressionTest.testShortestPathStream >LargeVolumeJettyTest >LeaderElectionIntegrationTest.testSimpleSliceLeaderElection >MathExpressionTest.testDistributions >MoveReplicaHDFSTest.testNormalFailedMove >SchemaApiFailureTest.testAddTheSameFieldTwice >SearchRateTriggerTest.testTrigger >TestDelegationWithHadoopAuth.testDelegationTokenRenew >TestDistribIDF.testMultiCollectionQuery >TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit >TestManagedResourceStorage >TestSimExecutePlanAction.testExecute >TestSimGenericDistributedQueue >TestSimGenericDistributedQueue.testDistributedQueue >TestSimLargeCluster.testAddNode >TestSimLargeCluster.testBasic >TestSimLargeCluster.testNodeLost >TestSimTriggerIntegration.testCooldown >TestSimTriggerIntegration.testEventFromRestoredState >TestSimTriggerIntegration.testEventQueue >TestSimTriggerIntegration.testListeners >TestSimTriggerIntegration.testNodeAddedTrigger >TestSimTriggerIntegration.testNodeAddedTriggerRestoreState >TestSimTriggerIntegration.testNodeLostTrigger >TestSimTriggerIntegration.testNodeLostTriggerRestoreState >TestSimTriggerIntegration.testNodeMarkersRegistration >TestSimTriggerIntegration.testTriggerThrottling >TestStressCloudBlindAtomicUpdates.test_dv_idx > > **Suites: 0 > > > Failures in Hoss' reports for the last 4 rollups. > > There were 571 unannotated tests that failed in Hoss' rollups. Ordered > by the date I downloaded the rollup file, newest->oldest. See above > for the dates the files were collected > These tests were NOT BadApple'd or AwaitsFix'd > All tests that failed 4 weeks running will be BadApple'd unless there > are objections > > Failures in the last 4 reports.. >Report Pct runsfails test > 0123 0.7 1749 8 > CdcrBootstrapTest.testBootstrapWithContinousIndexingOnSourceCluster > 0123 1.6 1751 18 CustomHighlightComponentTest.test > 0123 0.5 1582 8 > DeleteReplicaTest.deleteReplicaFromClusterState > 0123 42.9 101 14 HdfsBasicDistributedZk2Test.test > 0123 1.4 1741 21 JdbcTest(suite) > 0123 10.3 96 6 > LIROnShardRestartTest.testAllReplicasInLIR > 0123 1.8 1801 29 LeaderVoteWaitTimeoutTest.basicTest > 0123 1.8 1602 32 > LeaderVoteWaitTimeoutTest.testMostInSyncReplicasCanWinElection > 0123 4.8 849 46 MoveReplicaHDFSTest.testFailedMove > 0123 4.5 1515 40 SchemaApiFailureTest(suite) > 0123 0.7 1741 14 StreamingTest(suite) > 0123 0.2 1764 11 StreamingTest.testParallelMergeStream > 0123 0.2 1764 4 > StreamingTest.testZeroParallelReducerStream > 0123 0.5 1729 14 SystemLogListenerTest.test > 0123 0.2 1537 4 > TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader > 0123 0.2 1770 21 > TestCollectionStateWatchers.testCanWaitForNonexistantCollection > 0123 0.2 1770 21 > TestCollectionStateWatchers.testDeletionsTriggerWatches > 0123 0.2 1770 11 > TestCollectionStateWatchers.testWaitForStateChecksCurrentState > 0123 5.8 286 9 TestLargeCluster.testBasic > 0123 2.3 286 35 TestLargeCluster.testNodeLost > 0123 4.9 1674 40 TestLargeCluster.testSearchRate > 0123 1.4 1803 35 > TestPolicy.testComputePlanAfterNodeAdded > 0123 1.4 1802 34 TestPolicy.testConditionsSort > 0123 1.4 1803 35 TestPolicy.testCoresSuggestions > 0123 1.4 1800 32 TestPolicy.testDiskSpaceHint > 0123 2.5 1808 40 TestPolicy.testDiskSpaceReqd > 0123 1.4 1806 38 TestPolicy.testEmptyClusterState > 0123 2.5 1809 41 TestPolicy.testEqualFunction > 0123 1.8 1805 37 TestPolicy.testFreeDiskSuggestions > 0123 2.5 1805 37 TestPolicy.testFreediskPercentage > 0123
BadApple report TestPolicy, TestCollectionStateWatchers TestWithCollection
On the plus side, the CDCR tests (except BiDir) seem to be fixed. Also on the plus side, there are quite a number of tests that have _not_ failed in the last 4 weeks and I'll un-annotate. On the minus side, TestPolicy has 39 tests that have failed at least once in the last 4 weeks. I'll beast this to try to produce some data as I hope that there's a single underlying cause. **Annotated tests/suites that didn't fail in the last 4 weeks. **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 30 CollectionsAPIAsyncDistributedZkTest.testAsyncIdRaceCondition DistributedMLTComponentTest.test GraphExpressionTest.testShortestPathStream LargeVolumeJettyTest LeaderElectionIntegrationTest.testSimpleSliceLeaderElection MathExpressionTest.testDistributions MoveReplicaHDFSTest.testNormalFailedMove SchemaApiFailureTest.testAddTheSameFieldTwice SearchRateTriggerTest.testTrigger TestDelegationWithHadoopAuth.testDelegationTokenRenew TestDistribIDF.testMultiCollectionQuery TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit TestManagedResourceStorage TestSimExecutePlanAction.testExecute TestSimGenericDistributedQueue TestSimGenericDistributedQueue.testDistributedQueue TestSimLargeCluster.testAddNode TestSimLargeCluster.testBasic TestSimLargeCluster.testNodeLost TestSimTriggerIntegration.testCooldown TestSimTriggerIntegration.testEventFromRestoredState TestSimTriggerIntegration.testEventQueue TestSimTriggerIntegration.testListeners TestSimTriggerIntegration.testNodeAddedTrigger TestSimTriggerIntegration.testNodeAddedTriggerRestoreState TestSimTriggerIntegration.testNodeLostTrigger TestSimTriggerIntegration.testNodeLostTriggerRestoreState TestSimTriggerIntegration.testNodeMarkersRegistration TestSimTriggerIntegration.testTriggerThrottling TestStressCloudBlindAtomicUpdates.test_dv_idx **Suites: 0 Failures in Hoss' reports for the last 4 rollups. There were 571 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 0.7 1749 8 CdcrBootstrapTest.testBootstrapWithContinousIndexingOnSourceCluster 0123 1.6 1751 18 CustomHighlightComponentTest.test 0123 0.5 1582 8 DeleteReplicaTest.deleteReplicaFromClusterState 0123 42.9 101 14 HdfsBasicDistributedZk2Test.test 0123 1.4 1741 21 JdbcTest(suite) 0123 10.3 96 6 LIROnShardRestartTest.testAllReplicasInLIR 0123 1.8 1801 29 LeaderVoteWaitTimeoutTest.basicTest 0123 1.8 1602 32 LeaderVoteWaitTimeoutTest.testMostInSyncReplicasCanWinElection 0123 4.8 849 46 MoveReplicaHDFSTest.testFailedMove 0123 4.5 1515 40 SchemaApiFailureTest(suite) 0123 0.7 1741 14 StreamingTest(suite) 0123 0.2 1764 11 StreamingTest.testParallelMergeStream 0123 0.2 1764 4 StreamingTest.testZeroParallelReducerStream 0123 0.5 1729 14 SystemLogListenerTest.test 0123 0.2 1537 4 TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader 0123 0.2 1770 21 TestCollectionStateWatchers.testCanWaitForNonexistantCollection 0123 0.2 1770 21 TestCollectionStateWatchers.testDeletionsTriggerWatches 0123 0.2 1770 11 TestCollectionStateWatchers.testWaitForStateChecksCurrentState 0123 5.8 286 9 TestLargeCluster.testBasic 0123 2.3 286 35 TestLargeCluster.testNodeLost 0123 4.9 1674 40 TestLargeCluster.testSearchRate 0123 1.4 1803 35 TestPolicy.testComputePlanAfterNodeAdded 0123 1.4 1802 34 TestPolicy.testConditionsSort 0123 1.4 1803 35 TestPolicy.testCoresSuggestions 0123 1.4 1800 32 TestPolicy.testDiskSpaceHint 0123 2.5 1808 40 TestPolicy.testDiskSpaceReqd 0123 1.4 1806 38 TestPolicy.testEmptyClusterState 0123 2.5 1809 41 TestPolicy.testEqualFunction 0123 1.8 1805 37 TestPolicy.testFreeDiskSuggestions 0123 2.5 1805 37 TestPolicy.testFreediskPercentage 0123 1.8 1805 37 TestPolicy.testGreedyConditions 0123 1.4 1804 36 TestPolicy.testMerge 0123 1.4 1805 37 TestPolicy.testMoveReplica 0123 2.5 1811 43 TestPolicy.testMoveReplicaSuggester 0123 1.8 1803 35
Weekly BadApple report
**Annotated tests/suites that didn't fail in the last 4 weeks. **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 8 BasicAuthIntegrationTest.testBasicAuth CollectionsAPIAsyncDistributedZkTest.testAsyncRequests MoveReplicaHDFSTest.testNormalFailedMove MoveReplicaTest.testFailedMove SaslZkACLProviderTest.testSaslZkACLProvider SolrRrdBackendFactoryTest.testBasic TestLocalFSCloudBackupRestore TestPullReplicaErrorHandling.throws **Suites: 0 Failures in Hoss' reports for the last 4 rollups. These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 2.5 1475 34 ChaosMonkeyNothingIsSafeTest(suite) 0123 2.5 1660 22 CollectionsAPIDistributedZkTest.testCollectionsAPI 0123 0.3 1658 15 CustomCollectionTest.testRouteFieldForHashRouter 0123 0.5 1619 6 DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica 0123 0.5 1635 12 DeleteShardTest.testDirectoryCleanupAfterDeleteShard 0123 7.2 1475 42 GraphTest(suite) 0123 0.5 1618 8 LIRRollingUpdatesTest.testNewLeaderAndMixedReplicas 0123 0.3 1618 5 LIRRollingUpdatesTest.testNewLeaderOldReplica 0123 0.3 1618 5 LIRRollingUpdatesTest.testNewReplicaOldLeader 0123 2.2 1659 15 LeaderTragicEventTest(suite) 0123 5.0 1451 68 MoveReplicaHDFSTest.testFailedMove 0123 3.0 1587 45 SchemaApiFailureTest(suite) 0123 0.3 1596 10 TestCloudRecovery(suite) 0123 2.9 134 7 TestGenericDistributedQueue.testDistributedQueue 0123 3.1 1309 46 TestHdfsCloudBackupRestore.test 0123 1.7 1426 25 TestHdfsUpdateLog(suite) 0123 4.3 1474 27 TestLTROnSolrCloud(suite) 0123 0.4 1448 47 TestLocalFSCloudBackupRestore.test 0123 3.4 1462 98 TestSQLHandler(suite) 0123 0.3 1601 7 TestTlogReplica(suite) 0123 25.0 132 42 TestTlogReplica.testCreateDelete 0123 14.3 1280164 TestTriggerIntegration.testNodeLostTriggerRestoreState DO NOT ENABLE LIST: 'IndexSizeTriggerTest.testMergeIntegration' 'IndexSizeTriggerTest.testMixedBounds' 'IndexSizeTriggerTest.testSplitIntegration' 'IndexSizeTriggerTest.testTrigger' 'TestControlledRealTimeReopenThread.testCRTReopen' 'TestICUNormalizer2CharFilter.testRandomStrings' 'TestICUTokenizerCJK' 'TestImpersonationWithHadoopAuth.testForwarding' 'TestLTRReRankingPipeline.testDifferentTopN' 'TestRandomChains' DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey TestLatLonShapeQueries.testRandomBig TestRandomChains.testRandomChainsWithLargeStrings Processing file (History bit 3): HOSS-2018-08-06.csv Processing file (History bit 2): HOSS-2018-07-30.csv Processing file (History bit 1): HOSS-2018-07-23.csv Processing file (History bit 0): HOSS-2018-07-16.csv **Annotated tests/suites that didn't fail in the last 4 weeks. **Tests and suites removed from the next two lists because they were specified in 'doNotEnable' in the properties file no tests removed **Annotations will be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 8 BasicAuthIntegrationTest.testBasicAuth CollectionsAPIAsyncDistributedZkTest.testAsyncRequests MoveReplicaHDFSTest.testNormalFailedMove MoveReplicaTest.testFailedMove SaslZkACLProviderTest.testSaslZkACLProvider SolrRrdBackendFactoryTest.testBasic TestLocalFSCloudBackupRestore TestPullReplicaErrorHandling.throws **Suites: 0 Failures in Hoss' reports for the last 4 rollups. There were 830 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd All tests that failed 4 weeks running will be BadApple'd unless there are objections Failures in the last 4 reports.. Report Pct runsfails test 0123 9.5 1913248 CdcrBidirectionalTest.testBiDir 0123 2.5 1475 34 ChaosMonkeyNothingIsSafeTest(suite) 0123 2.5 1660 22 CollectionsAPIDistributedZkTest.testCollectionsAPI 0123 0.3 1658 15 CustomCollectionTest.testRouteFieldForHashRouter 0123 0.5 1619 6 DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica 0123 0.5 1635 12
Re: BadApple report. Seems like I'm wasting my time.
I still think it’s a mistake to try and use all the Jenkins results to drive ignoring tests. It needs to be an objective measure in a good env. We also should not be ignoring tests in mass.l without individual consideration. Critical test coverage should be treated differently than any random test, especially when stability is sometimes simple to achieve for that test. A decade+ of history says it’s unlikely you get much consistent help digging out of a huge test ignore hell. Beasting in a known good environment and a few very interested parties is the only path out of this if you ask me. We need to get clean in a known good env and then automate beasting defense, using Jenkins to find issues in other environments. Unfortunately, not something I can help out with in the short term anymore. Mark On Wed, Aug 1, 2018 at 8:10 AM Erick Erickson wrote: > Alexandre: > > Feel free! What I'm struggling with is not that someone checked in > some code that all the sudden started breaking things. Rather that a > test that's been working perfectly will fail once the won't > reproducibly fail again and does _not_ appear to be related to recent > code changes. > > In fact that's the crux of the matter, it's difficult/impossible to > tell at a glance when a test fails whether it is or is not related to > a recent code change. > > Erick > > On Wed, Aug 1, 2018 at 8:05 AM, Alexandre Rafalovitch > wrote: > > Just a completely random thought that I do not have deep knowledge for > > (still learning my way around Solr tests). > > > > Is this something that Machine Learning could help with? The Github > > repo/history is a fantastic source of learning on who worked on which > > file, how often, etc. We certainly should be able to get some 'most > > significant developer' stats out of that. > > > > Regards, > >Alex. > > > > On 1 August 2018 at 10:56, Erick Erickson > wrote: > >> Shawn: > >> > >> Trouble is there were 945 tests that failed at least once in the last > >> 4 weeks. And the trend is all over the map on a weekly basis. > >> > >> e-mail-2018-06-11.txt: There were 989 unannotated tests that failed > >> e-mail-2018-06-18.txt: There were 689 unannotated tests that failed > >> e-mail-2018-06-25.txt: There were 555 unannotated tests that failed > >> e-mail-2018-07-02.txt: There were 723 unannotated tests that failed > >> e-mail-2018-07-09.txt: There were 793 unannotated tests that failed > >> e-mail-2018-07-16.txt: There were 809 unannotated tests that failed > >> e-mail-2018-07-23.txt: There were 953 unannotated tests that failed > >> e-mail-2018-07-30.txt: There were 945 unannotated tests that failed > >> > >> I'm BadApple'ing tests that fail every week for the last 4 weeks on > >> the theory that those are not temporary issues (hey, we all commit > >> code that breaks something then have to figure out why and fix). > >> > >> I also have the feeling that somewhere, somehow, our test framework is > >> making some assumptions that are invalid. Or too strict. Or too fast. > >> Or there's some fundamental issue with some of our classes. Or... The > >> number of sporadic issues where the Object Tracker spits stuff out for > >> instance screams that some assumption we're making, either in the code > >> or in the test framework is flawed. > >> > >> What I don't know is how to make visible progress. It's discouraging > >> to fix something and then next week have more tests fail for unrelated > >> reasons. > >> > >> Visibility is the issue to me. We have no good way of saying "these > >> tests _just started failing for a reason. As a quick experiment, I > >> extended the triage to 10 weeks (no attempt to ascertain if these > >> tests even existed 10 weeks ago). Here are the tests that have _only_ > >> failed in the last week, not the previous 9. BadApple'ing anything > >> that's only failed once seems overkill > >> > >> Although the test that failed 77 times does just stand out > >> > >> week pctruns failstest > >> 00.2 460 1 > >> CloudSolrClientTest.testVersionsAreReturned > >> 00.2 466 1 > >> ComputePlanActionTest.testSelectedCollections > >> 00.2 464 1 > >> ConfusionMatrixGeneratorTest.testGetConfusionMatrixWithBM25NB > >> 08.1 37 3 IndexSizeTriggerTest(suite) > >> 00.2 454 1 > MBeansHandlerTest.testAddedMBeanDiff > >> 00.2 454 1 MBeansHandlerTest.testDiff > >> 00.2 455 1 MetricTriggerTest.test > >> 00.2 455 1 MetricsHandlerTest.test > >> 00.2 455 1 MetricsHandlerTest.testKeyMetrics > >> 00.2 453 1 RequestHandlersTest.testInitCount > >> 00.2 453 1 RequestHandlersTest.testStatistics > >> 00.2 453 1 > ScheduledTriggerIntegrationTest(suite) > >> 00.2 451 1 >
Re: BadApple report. Seems like I'm wasting my time.
Alexandre: Feel free! What I'm struggling with is not that someone checked in some code that all the sudden started breaking things. Rather that a test that's been working perfectly will fail once the won't reproducibly fail again and does _not_ appear to be related to recent code changes. In fact that's the crux of the matter, it's difficult/impossible to tell at a glance when a test fails whether it is or is not related to a recent code change. Erick On Wed, Aug 1, 2018 at 8:05 AM, Alexandre Rafalovitch wrote: > Just a completely random thought that I do not have deep knowledge for > (still learning my way around Solr tests). > > Is this something that Machine Learning could help with? The Github > repo/history is a fantastic source of learning on who worked on which > file, how often, etc. We certainly should be able to get some 'most > significant developer' stats out of that. > > Regards, >Alex. > > On 1 August 2018 at 10:56, Erick Erickson wrote: >> Shawn: >> >> Trouble is there were 945 tests that failed at least once in the last >> 4 weeks. And the trend is all over the map on a weekly basis. >> >> e-mail-2018-06-11.txt: There were 989 unannotated tests that failed >> e-mail-2018-06-18.txt: There were 689 unannotated tests that failed >> e-mail-2018-06-25.txt: There were 555 unannotated tests that failed >> e-mail-2018-07-02.txt: There were 723 unannotated tests that failed >> e-mail-2018-07-09.txt: There were 793 unannotated tests that failed >> e-mail-2018-07-16.txt: There were 809 unannotated tests that failed >> e-mail-2018-07-23.txt: There were 953 unannotated tests that failed >> e-mail-2018-07-30.txt: There were 945 unannotated tests that failed >> >> I'm BadApple'ing tests that fail every week for the last 4 weeks on >> the theory that those are not temporary issues (hey, we all commit >> code that breaks something then have to figure out why and fix). >> >> I also have the feeling that somewhere, somehow, our test framework is >> making some assumptions that are invalid. Or too strict. Or too fast. >> Or there's some fundamental issue with some of our classes. Or... The >> number of sporadic issues where the Object Tracker spits stuff out for >> instance screams that some assumption we're making, either in the code >> or in the test framework is flawed. >> >> What I don't know is how to make visible progress. It's discouraging >> to fix something and then next week have more tests fail for unrelated >> reasons. >> >> Visibility is the issue to me. We have no good way of saying "these >> tests _just started failing for a reason. As a quick experiment, I >> extended the triage to 10 weeks (no attempt to ascertain if these >> tests even existed 10 weeks ago). Here are the tests that have _only_ >> failed in the last week, not the previous 9. BadApple'ing anything >> that's only failed once seems overkill >> >> Although the test that failed 77 times does just stand out >> >> week pctruns failstest >> 00.2 460 1 >> CloudSolrClientTest.testVersionsAreReturned >> 00.2 466 1 >> ComputePlanActionTest.testSelectedCollections >> 00.2 464 1 >> ConfusionMatrixGeneratorTest.testGetConfusionMatrixWithBM25NB >> 08.1 37 3 IndexSizeTriggerTest(suite) >> 00.2 454 1 MBeansHandlerTest.testAddedMBeanDiff >> 00.2 454 1 MBeansHandlerTest.testDiff >> 00.2 455 1 MetricTriggerTest.test >> 00.2 455 1 MetricsHandlerTest.test >> 00.2 455 1 MetricsHandlerTest.testKeyMetrics >> 00.2 453 1 RequestHandlersTest.testInitCount >> 00.2 453 1 RequestHandlersTest.testStatistics >> 00.2 453 1 ScheduledTriggerIntegrationTest(suite) >> 00.2 451 1 >> SearchRateTriggerTest.testWaitForElapsed >> 00.2 425 1 >> SoftAutoCommitTest.testSoftCommitWithinAndHardCommitMaxTimeRapidAdds >> 0 14.7 525 77 >> StreamExpressionTest.testSignificantTermsStream >> 00.2 454 1 TestBadConfig(suite) >> 00.2 465 1 >> TestBlockJoin.testMultiChildQueriesOfDiffParentLevels >> 00.6 462 3 >> TestCloudCollectionsListeners.testCollectionDeletion >> 00.2 456 1 TestInfoStreamLogging(suite) >> 00.2 456 1 TestLazyCores.testLazySearch >> 00.2 473 1 >> TestLucene70DocValuesFormat.testSortedSetAroundBlockSize >> 0 15.4 26 4 >> TestMockDirectoryWrapper.testThreadSafetyInListAll >> 00.2 454 1 TestNodeLostTrigger.testTrigger >> 00.2 453 1 TestRecovery.stressLogReplay >> 00.2 505 1 >>
