Re: [DISCUSS] SIP-12: Incremental Backup and Restore
>, and implement the new imporved version as a V2-api only, and then deprecate >the v1 API? V2 only please On Tue, Dec 22, 2020 at 1:34 AM Jason Gerlowski wrote: > > Hey Jan, thanks for the review. > > I hadn't thought about the V2 API in connection to this work. You're > right though I think - the SIP proposes net-new APIs, so it should add > V2 equivalents at the very least. I'll draft tentative details for > these APIs on the SIP and we can refine things from there. > > I'm more up in the air on your specific suggestion to restrict the SIP > changes to these v2 APIs. It is an elegant approach to the > backcompat, and it provides a carrot for v2 adoption - both of which I > like. But it would let users create snapshot-based backups (and keep > us maintaining that code) longer than there's any strict need to. And > users are left on the less-efficient format by default. (By contrast, > the current SIP has snapshot-backup creation being replaced by > incremental-backup creation as soon as the latter is available.). Did > you have a particular lifespan in mind for snapshot-based creation if > we go with this approach? > > Jason > > On Thu, Dec 17, 2020 at 3:54 PM Jan Høydahl wrote: > > > > Much needed! Thanks for initiating this Jason! > > > > As we want to move away from v1 APIs where a HTTP GET is used for creation > > and deletion, would it be an idea to leave the old backup/resotre APIs > > as-is, and implement the new imporved version as a V2-api only, and then > > deprecate the v1 API? Then we don't need to worry about back-compat, and we > > get a head-start on converting the COLLECTION API to v2 style. > > > > Jan > > > > > 15. des. 2020 kl. 15:48 skrev Jason Gerlowski : > > > > > > Hey all, > > > > > > This morning I published SIP-12, which proposes an overhaul of Solr's > > > backup and restore functionality. While the "headline" improvement in > > > this SIP is a change to do backups incrementally, it bundles in a > > > number of other improvements as well, including the addition of > > > corruption checks, APIs to list and delete backups, and stronger > > > integration points with popular object storage APIs. > > > > > > The SIP can be found here: > > > https://cwiki.apache.org/confluence/display/SOLR/SIP-12%3A+Incremental+Backup+and+Restore > > > > > > Please read the SIP description and come back here for discussion. As > > > the discussion progresses we will update the SIP page with any > > > outcomes and eventually move things to a VOTE. > > > > > > Looking forward to hearing your feedback. > > > > > > Best, > > > > > > Jason > > > > > > - > > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > > > > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > -- - Noble Paul - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: [DISCUSS] SIP-12: Incremental Backup and Restore
Hey Jan, thanks for the review. I hadn't thought about the V2 API in connection to this work. You're right though I think - the SIP proposes net-new APIs, so it should add V2 equivalents at the very least. I'll draft tentative details for these APIs on the SIP and we can refine things from there. I'm more up in the air on your specific suggestion to restrict the SIP changes to these v2 APIs. It is an elegant approach to the backcompat, and it provides a carrot for v2 adoption - both of which I like. But it would let users create snapshot-based backups (and keep us maintaining that code) longer than there's any strict need to. And users are left on the less-efficient format by default. (By contrast, the current SIP has snapshot-backup creation being replaced by incremental-backup creation as soon as the latter is available.). Did you have a particular lifespan in mind for snapshot-based creation if we go with this approach? Jason On Thu, Dec 17, 2020 at 3:54 PM Jan Høydahl wrote: > > Much needed! Thanks for initiating this Jason! > > As we want to move away from v1 APIs where a HTTP GET is used for creation > and deletion, would it be an idea to leave the old backup/resotre APIs as-is, > and implement the new imporved version as a V2-api only, and then deprecate > the v1 API? Then we don't need to worry about back-compat, and we get a > head-start on converting the COLLECTION API to v2 style. > > Jan > > > 15. des. 2020 kl. 15:48 skrev Jason Gerlowski : > > > > Hey all, > > > > This morning I published SIP-12, which proposes an overhaul of Solr's > > backup and restore functionality. While the "headline" improvement in > > this SIP is a change to do backups incrementally, it bundles in a > > number of other improvements as well, including the addition of > > corruption checks, APIs to list and delete backups, and stronger > > integration points with popular object storage APIs. > > > > The SIP can be found here: > > https://cwiki.apache.org/confluence/display/SOLR/SIP-12%3A+Incremental+Backup+and+Restore > > > > Please read the SIP description and come back here for discussion. As > > the discussion progresses we will update the SIP page with any > > outcomes and eventually move things to a VOTE. > > > > Looking forward to hearing your feedback. > > > > Best, > > > > Jason > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
BadApple report
Still noisy, waiting for the reference impl to untangle. Short form: Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 136 failures Week: 1 had 185 failures Week: 2 had 210 failures Week: 3 had 112 failures Failures in Hoss' reports in every one of the last 4 rollups. There were 380 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 0.7 2080 10 CachingDirectoryFactoryTest.stressTest 0123 18.3 666134 ClusterEventProducerTest.testEvents 0123 0.3 1471 5 DistributedFacetPivotLongTailTest.test 0123 2.5 2056 17 DistributedQueryComponentCustomSortTest.test 0123 100.0 57 57 DocValuesNotIndexedTest.classMethod 0123 0.8 1467 7 HttpPartitionOnCommitTest.test 0123 0.5 1896 10 ManagedSchemaRoundRobinCloudTest.testAddFieldsRoundRobin 0123 4.5 1516 29 MultiThreadedOCPTest.test 0123 0.3 1478 7 RollingRestartTest.test 0123 50.07 4 SharedFSAutoReplicaFailoverTest.test 0123 1.0 1526 20 TestCircuitBreaker.testResponseWithCBTiming 0123 11.3 1467129 TestContainerPlugin.testApi 0123 2.6 1584 41 TestDistributedStatsComponentCardinality.test 0123 0.9 1264 11 TestHdfsCloudBackupRestore.test 0123 1.3 1477 14 TestLocalFSCloudBackupRestore.test 0123 1.5 1526 32 TestPackages.testPluginLoading 0123 1.5 1801 33 TestPullReplicaErrorHandling.testCantConnectToLeader 0123 2.3 1801 40 TestPullReplicaErrorHandling.testPullReplicaDisconnectsFromZooKeeper Full report: DO NOT ENABLE LIST: MoveReplicaHDFSTest.testFailedMove MoveReplicaHDFSTest.testNormalFailedMove TestControlledRealTimeReopenThread.testCRTReopen TestICUNormalizer2CharFilter.testRandomStrings TestICUTokenizerCJK TestImpersonationWithHadoopAuth.testForwarding TestLTRReRankingPipeline.testDifferentTopN TestRandomChains DO NOT ANNOTATE LIST CdcrBidirectionalTest.testBiDir IndexSizeTriggerTest.testMergeIntegration IndexSizeTriggerTest.testMixedBounds IndexSizeTriggerTest.testSplitIntegration IndexSizeTriggerTest.testTrigger InfixSuggestersTest.testShutdownDuringBuild ShardSplitTest.test ShardSplitTest.testSplitMixedReplicaTypes ShardSplitTest.testSplitWithChaosMonkey Test2BPostings.test TestLatLonShapeQueries.testRandomBig TestPackedInts.testPackedLongValues TestRandomChains.testRandomChainsWithLargeStrings TestTriggerIntegration.testSearchRate SuppressWarnings count: last week: 4,491, this week: 4,491, delta 0 *** Files with increased @SuppressWarnings annotations: Suppress count increase in: solr/core/src/java/org/apache/solr/schema/IndexSchemaFactory.java. Was: 0, now: 1 *** Files with decreased @SuppressWarnings annotations: Suppress count decrease in: lucene/core/src/test/org/apache/lucene/util/hnsw/TestHnsw.java. Was: 1, now: 0 Processing file (History bit 3): HOSS-2020-12-21.csv Processing file (History bit 2): HOSS-2020-12-07.csv Processing file (History bit 1): HOSS-2020-11-23.csv Processing file (History bit 0): HOSS-2020-11-09.csv Number of AwaitsFix: 31 Number of BadApples: 3 **Annotated tests that didn't fail in the last 4 weeks. **Tests removed from the next two lists because they were specified in 'doNotEnable' in the properties file MoveReplicaHDFSTest.testNormalFailedMove **Annotations can be removed from the following tests because they haven't failed in the last 4 rollups. **Methods: 0 Raw fail count by week totals, most recent week first (corresponds to bits): Week: 0 had 136 failures Week: 1 had 185 failures Week: 2 had 210 failures Week: 3 had 112 failures Failures in Hoss' reports in every one of the last 4 rollups. There were 380 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected These tests were NOT BadApple'd or AwaitsFix'd Failures in the last 4 reports.. Report Pct runsfails test 0123 0.7 2080 10 CachingDirectoryFactoryTest.stressTest 0123 18.3 666134 ClusterEventProducerTest.testEvents 0123 0.3 1471 5 DistributedFacetPivotLongTailTest.test 0123 2.5 2056 17 DistributedQueryComponentCustomSortTest.test 0123 100.0 57 57
