thanks, erick. great info. although you can't (yet) direct queries to one or the other. So just making > them all NRT and forgetting about it is reasonable.
are you simply flagging the fact that we wouldn't direct the queries to A v. B v. C since SolrCloud will make the decisions itself as to which part of the distro gets hit for the operation? if not, can you expound on this a bit more? The very nature of merging is such that you will _always_ get large merges > until you have 5G segments (by default) bummer Quite possible, but you have to route things yourself. But in that case > you're limited to one machine to handle all your NRT traffic. I skimmed > your post so don't know whether your NRT traffic load is high enough to > worry about. ok. i think we'll take a two-pronged approach. for the immediate purposes of trying to solve an issue we've begun encountering we will begin thoroughtesting the load between various operations in the master-slave setup we've set up. pending the results, we can roll forward w a temporary patch in which all end-user touch points route through the primary box for read/write while large scale operations/processing we do in the background will point to the ELB the slaves are sitting behind. we'll also begin setting up a simple solrcloud instance to toy with per your suggestion above. inb4 tons more questions on my part :) thanks! -- John Blythe On Tue, Apr 10, 2018 at 11:14 AM, Erick Erickson <erickerick...@gmail.com> wrote: > bq: should we try to bite the solrcloud bullet and be done w it > > that's what I'd do. As of 7.0 there are different "flavors", TLOG, > PULL and NRT so that's also a possibility, although you can't (yet) > direct queries to one or the other. So just making them all NRT and > forgetting about it is reasonable. > > bq: is there some more config work we could put in place to avoid ... > commit issue and the ultra large merge dangers > > No. The very nature of merging is such that you will _always_ get > large merges until you have 5G segments (by default). The max segment > size (outside "optimize/forceMerge/expungeDeletes" which you shouldn't > do) is 5G so the steady-state worst-case segment pull is limited to > that. > > bq: maybe for our initial need we use Master for writing and user > access in NRT events, but slaves for the heavier backend > > Quite possible, but you have to route things yourself. But in that > case you're limited to one machine to handle all your NRT traffic. I > skimmed your post so don't know whether your NRT traffic load is high > enough to worry about. > > The very first thing I'd do is set up a simple SolrCloud setup and > give it a spin. Unless your indexing load is quite heavy, the added > work the NRT replicas have in SolrCloud isn't a problem so worrying > about that is premature optimization unless you have a heavy load..... > > Best, > Erick > > On Mon, Apr 9, 2018 at 4:36 PM, John Blythe <johnbly...@gmail.com> wrote: > > Thanks a bunch for the thorough reply, Shawn. > > > > Phew. We’d chosen to go w Master-slave replication instead of SolrCloud > per > > the sudden need we had encountered and the desire to avoid the nuances > and > > changes related to moving to SolrCloud. But so much for this being a more > > straightforward solution, huh? > > > > Few questions: > > - should we try to bite the solrcloud bullet and be done w it? > > - is there some more config work we could put in place to avoid the soft > > commit issue and the ultra large merge dangers, keeping the replications > > happening quickly? > > - maybe for our initial need we use Master for writing and user access in > > NRT events, but slaves for the heavier backend processing. Thoughts? > > - anyone do consulting on this that would be interested in chatting? > > > > Thanks again! > > > > On Mon, Apr 9, 2018 at 18:18 Shawn Heisey <apa...@elyograg.org> wrote: > > > >> On 4/9/2018 12:15 PM, John Blythe wrote: > >> > we're starting to dive into master/slave replication architecture. > we'll > >> > have 1 master w 4 slaves behind it. our app is NRT. if user performs > an > >> > action in section A's data they may choose to jump to section B which > >> will > >> > be dependent on having the updates from their action in section A. as > >> such, > >> > we're thinking that the replication time should be set to 1-2s (the > >> chances > >> > of them arriving at section B quickly enough to catch the 2s gap is > >> highly > >> > unlikely at best). > >> > >> Once you start talking about master-slave replication, my assumption is > >> that you're not running SolrCloud. You would NOT want to try and mix > >> SolrCloud with replication. The features do not play well together. > >> SolrCloud with NRT replicas (this is the only replica type that exists > >> in 6.x and earlier) may be a better option than master-slave > replication. > >> > >> > since the replicas will simply be looking for new files it seems like > >> this > >> > would be a lightweight operation even every couple seconds for 4 > >> replicas. > >> > that said, i'm going *entirely* off of assumption at this point and > >> wanted > >> > to check in w you all to see any nuances, gotchas, hidden landmines, > etc. > >> > that we should be considering before rolling things out. > >> > >> Most of the time, you'd be correct to think that indexing is going to > >> create a new small segment and replication will have little work to do. > >> But as you create more and more segments, eventually Lucene is going to > >> start merging those segments. For discussion purposes, I'm going to > >> describe a situation where each new segment during indexing is about > >> 100KB in size, and the merge policy is left at the default settings. > >> I'm also going to assume that no documents are getting deleted or > >> reindexed (which will delete the old version). Deleted documents can > >> have an impact on merging, but it will usually only be a dramatic impact > >> if there are a LOT of deleted documents. > >> > >> The first ten segments created will be this 100KB size. Then Lucene is > >> going to see that there are enough segments to trigger the merge policy > >> - it's going to combine ten of those segments into one that's > >> approximately one megabyte. Repeat this ten times, and ten of those 1 > >> megabyte segments will be combined into one ten megabyte segment. > >> Repeat all of THAT ten times, and there will be a 100 megabyte segment. > >> And there will eventually be another level creating 1 gigabyte > >> segments. If the index is below 5GB in size, the entire thing *could* > >> be merged into one segment by this process. > >> > >> The end result of all this: Replication is not always going to be > >> super-quick. If merging creates a 1 gigabyte segment, then the amount > >> of time to transfer that new segment is going to depend on how fast your > >> disks are, and how fast your network is. If you're using commodity SATA > >> drives in the 4 to 10 terabyte range and a gigabit network, the network > >> is probably going to be the bottleneck -- assuming that the system has > >> plenty of memory and isn't under a high load. If the network is the > >> bottleneck in that situation, it's probably going to take close to ten > >> seconds to transfer a 1GB segment, and the greater part of a minute to > >> transfer a 5GB segment, which is the biggest one that the default merge > >> policy configuration will create without an optimize operation. > >> > >> Also, you should understand something that has come to my attention > >> recently (and is backed up by documentation): If the master does a soft > >> commit and the segment that was committed remains in memory (not flushed > >> to disk), that segment will NOT be replicated to the slaves. It has to > >> get flushed to disk before it can be replicated. > >> > >> Thanks, > >> Shawn > >> > >> -- > > John Blythe >
