Thanks a bunch for the thorough reply, Shawn. Phew. We’d chosen to go w Master-slave replication instead of SolrCloud per the sudden need we had encountered and the desire to avoid the nuances and changes related to moving to SolrCloud. But so much for this being a more straightforward solution, huh?
Few questions: - should we try to bite the solrcloud bullet and be done w it? - is there some more config work we could put in place to avoid the soft commit issue and the ultra large merge dangers, keeping the replications happening quickly? - maybe for our initial need we use Master for writing and user access in NRT events, but slaves for the heavier backend processing. Thoughts? - anyone do consulting on this that would be interested in chatting? Thanks again! On Mon, Apr 9, 2018 at 18:18 Shawn Heisey <apa...@elyograg.org> wrote: > On 4/9/2018 12:15 PM, John Blythe wrote: > > we're starting to dive into master/slave replication architecture. we'll > > have 1 master w 4 slaves behind it. our app is NRT. if user performs an > > action in section A's data they may choose to jump to section B which > will > > be dependent on having the updates from their action in section A. as > such, > > we're thinking that the replication time should be set to 1-2s (the > chances > > of them arriving at section B quickly enough to catch the 2s gap is > highly > > unlikely at best). > > Once you start talking about master-slave replication, my assumption is > that you're not running SolrCloud. You would NOT want to try and mix > SolrCloud with replication. The features do not play well together. > SolrCloud with NRT replicas (this is the only replica type that exists > in 6.x and earlier) may be a better option than master-slave replication. > > > since the replicas will simply be looking for new files it seems like > this > > would be a lightweight operation even every couple seconds for 4 > replicas. > > that said, i'm going *entirely* off of assumption at this point and > wanted > > to check in w you all to see any nuances, gotchas, hidden landmines, etc. > > that we should be considering before rolling things out. > > Most of the time, you'd be correct to think that indexing is going to > create a new small segment and replication will have little work to do. > But as you create more and more segments, eventually Lucene is going to > start merging those segments. For discussion purposes, I'm going to > describe a situation where each new segment during indexing is about > 100KB in size, and the merge policy is left at the default settings. > I'm also going to assume that no documents are getting deleted or > reindexed (which will delete the old version). Deleted documents can > have an impact on merging, but it will usually only be a dramatic impact > if there are a LOT of deleted documents. > > The first ten segments created will be this 100KB size. Then Lucene is > going to see that there are enough segments to trigger the merge policy > - it's going to combine ten of those segments into one that's > approximately one megabyte. Repeat this ten times, and ten of those 1 > megabyte segments will be combined into one ten megabyte segment. > Repeat all of THAT ten times, and there will be a 100 megabyte segment. > And there will eventually be another level creating 1 gigabyte > segments. If the index is below 5GB in size, the entire thing *could* > be merged into one segment by this process. > > The end result of all this: Replication is not always going to be > super-quick. If merging creates a 1 gigabyte segment, then the amount > of time to transfer that new segment is going to depend on how fast your > disks are, and how fast your network is. If you're using commodity SATA > drives in the 4 to 10 terabyte range and a gigabit network, the network > is probably going to be the bottleneck -- assuming that the system has > plenty of memory and isn't under a high load. If the network is the > bottleneck in that situation, it's probably going to take close to ten > seconds to transfer a 1GB segment, and the greater part of a minute to > transfer a 5GB segment, which is the biggest one that the default merge > policy configuration will create without an optimize operation. > > Also, you should understand something that has come to my attention > recently (and is backed up by documentation): If the master does a soft > commit and the segment that was committed remains in memory (not flushed > to disk), that segment will NOT be replicated to the slaves. It has to > get flushed to disk before it can be replicated. > > Thanks, > Shawn > > -- John Blythe
