Erick Erickson commented on LUCENE-7976:

Thanks for taking the time Mike! Going in reverse order:

bq: I wonder if we could just relax TMP to allow it to consider merges with 
fewer than maxMergeAtOnce, and then "improve" the scoring function to give a 
good score to cases that would reclaim > X% deletions?

Interesting. At this point in the change cycle I've got my head around most of 
what's going on and can think about how to do it better. We'd want to tweak 
things I should think so scoring could return "don't do this merge at all"? I'm 
thinking of the case where we have, say, 1 max-sized (or bigger) segment as a 
candidate with a few deletions, we wouldn't want to merge that at all, right? 
I'm thinking some threshold score above which we score it as "don't bother"....

The number of changes I'm introducing here does make me nervous, I wonder if 
taking a fresh look at it with an eye toward just doing the above would lead to 
less surgery.... I mean this has been working fine for years, I do worry that 
I'm introducing bugs... I don't mind throwing away a bunch of work if smaller 
changes can cure my problem.

bq: I think what you mean is you want to change the forceMerge and 
forceMergeDeletes APIs in IndexWriter

Right, that would have been the consequence. But I changed my mind on this 
yesterday, I don't think any Lucene API change is needed after all. What I did 
instead (not in the current patch) is default the Solr "update" command to 
Integer.MAX_VALUE for forceMerge. That just flows into the TMP code without 
changing the API and lets maxMergedSegmentBytes control how many segments are 
created. Anyone who wants the old behavior needs to pass 1, which is a change 
in behavior.

bq:  I think it's a bug that findForceMergeDeletes doesn't do the same thing....

OK, let me look this over again. Yesterday I started to see the differences 
between forceMerge and forceMergeDeletes and thought they should stay separate, 
but you seem to be saying the idea of combining them is worth exploring. I'll 
revisit this again this weekend. Wouldn't making that work require changing the 
findForceMergeDeletes interface? I'm perfectly willing but didn't want to do 
that without discussion. And it seems that then findForcedDeletesMerges and 
findForcedMerges would be very thin wrappers around the same code for both.... 
Or were you thinking of handling this differently?

Thanks again...

> Make TieredMergePolicy respect maxSegmentSizeMB and allow singleton merges of 
> very large segments
> -------------------------------------------------------------------------------------------------
>                 Key: LUCENE-7976
>                 URL: https://issues.apache.org/jira/browse/LUCENE-7976
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Erick Erickson
>            Assignee: Erick Erickson
>            Priority: Major
>         Attachments: LUCENE-7976.patch, LUCENE-7976.patch
> We're seeing situations "in the wild" where there are very large indexes (on 
> disk) handled quite easily in a single Lucene index. This is particularly 
> true as features like docValues move data into MMapDirectory space. The 
> current TMP algorithm allows on the order of 50% deleted documents as per a 
> dev list conversation with Mike McCandless (and his blog here:  
> https://www.elastic.co/blog/lucenes-handling-of-deleted-documents).
> Especially in the current era of very large indexes in aggregate, (think many 
> TB) solutions like "you need to distribute your collection over more shards" 
> become very costly. Additionally, the tempting "optimize" button exacerbates 
> the issue since once you form, say, a 100G segment (by 
> optimizing/forceMerging) it is not eligible for merging until 97.5G of the 
> docs in it are deleted (current default 5G max segment size).
> The proposal here would be to add a new parameter to TMP, something like 
> <maxAllowedPctDeletedInBigSegments> (no, that's not serious name, suggestions 
> welcome) which would default to 100 (or the same behavior we have now).
> So if I set this parameter to, say, 20%, and the max segment size stays at 
> 5G, the following would happen when segments were selected for merging:
> > any segment with > 20% deleted documents would be merged or rewritten NO 
> > MATTER HOW LARGE. There are two cases,
> >> the segment has < 5G "live" docs. In that case it would be merged with 
> >> smaller segments to bring the resulting segment up to 5G. If no smaller 
> >> segments exist, it would just be rewritten
> >> The segment has > 5G "live" docs (the result of a forceMerge or optimize). 
> >> It would be rewritten into a single segment removing all deleted docs no 
> >> matter how big it is to start. The 100G example above would be rewritten 
> >> to an 80G segment for instance.
> Of course this would lead to potentially much more I/O which is why the 
> default would be the same behavior we see now. As it stands now, though, 
> there's no way to recover from an optimize/forceMerge except to re-index from 
> scratch. We routinely see 200G-300G Lucene indexes at this point "in the 
> wild" with 10s of  shards replicated 3 or more times. And that doesn't even 
> include having these over HDFS.
> Alternatives welcome! Something like the above seems minimally invasive. A 
> new merge policy is certainly an alternative.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to