On Sun, Mar 15, 2009 at 05:44:43PM -0400, Michael McCandless wrote: > So eg you could merge 2 sets of segments at once (like Lucene)?
Yes. >> These processes are forbidden from merging any files that pre-date the >> establishment of "consolidate.lock". > > Why? It seems like it needs to merge segments created before it acquired > that lock (that's why it was launched). I was unclear. By "these processes", I meant write processes launched while the consolidation process is active. >> It finishes that task, commits, releases "write.lock", releases >> "consolidate.lock",then exits. > That, and update the master "segments" file to actually record the merge, > and incRef/decRef to delete files. By "commits", I was referring to updating the master file; I probably should have used more precise language. You're right that I'd left off file deletion. > But, what if while a large merge is happening, and enough segments have > been written to warrant a small merge to kick off & finish? That should work OK. Write processes launched during consolidation will be allowed to performing any mods or merges they like on segments that were written *after* the the consolidator grabbed consolidate.lock. They just can't touch stuff that the consolidator might be operating on. The only thing I'm concerned about is the time that it takes to carry recent deletions against merged segments forward so that they apply against the new, consolidated segment. The consolidator process has to obtain the write.lock for that, blocking any new mods to the index. Hopefully that doesn't cause too big a blip. Marvin Humphrey
