On 7/18/2017 11:53 AM, suresh pendap wrote: > After looking at the source code I see that the default values for > numRecordsToKeep is 100 and maxNumLogsToKeep is 10. > > So it seems by default the replica can only have 1000 document updates lag > before the replica goes for a Full recovery from the leader.
I don't think that's quite right. In many situations, the number of documents in the transaction log will likely be less than 1000. Enough logs will be kept that *at least* 100 documents are there, if that can be accomplished with ten logfiles or less. Based on a quick reading of the code, if the newest ten logs have less than 100 documents, then there will be less than 100 docs available. This would not end up being a problem for data integrity, because small infrequent updates would be the only way to end up with less than 100 docs, and in that situation, the small number of documents in the transaction log, when replayed at core startup, will be enough to ensure integrity. I think the reasons the default numbers are so small is an attempt to keep startup time low. I am pretty sure that anytime a core starts for *any* reason, all the transaction logs that are present will get replayed. I know for sure that this happens on Solr restart; I think it also happens on core reload. By keeping the required minimum documents at a low value like 100, there's a better chance that the transaction logs will be small, and therefore core startup will be fast. On a system where there are no hard commits, all updates end up going into a single super-large transaction log. This meets the default configuration numbers, because there are less than ten logs present, and what is present contains at least 100 documents. Unfortunately, this means that when the core starts, it will replay that HUGE transaction log, a process that could take hours. This situation is prevented by enabling autoCommit with a relatively short maxTime value. Setting openSearcher to false in the autoCommit ensures that document visibility behavior is not altered by autoCommit. Thanks, Shawn
