[
https://issues.apache.org/jira/browse/LUCENE-7976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16214347#comment-16214347
]
Erick Erickson commented on LUCENE-7976:
----------------------------------------
I completely agree that removing the Solr optimize button should be done, take
that as read. I've linked that JIRA here. I think these two issues are
interrelated. We need to give users some tools to control the percentage
deleted docs their index accumulates and make it much less tempting to back
themselves into a corner.
I do not and will not agree that all uses of forceMerge are invalid. Currently,
one thing that contributes to their being overused is the percentage of deleted
documents in the index. If a user notices that near 50% of the docs are
deleted, what else can they do? expungeDeletes doesn't help here, it still
creates a massive segment.
The other valid use case is an index that changes, say, once a day. forceMerge
makes perfect sense here since it can be run every time the index is built and
_does_ result in some improvements in throughput. People squeezing 1,000s of
QPS out of their system are pretty sensitive to any throughput increase they
can get.
Making optimize less attractive or harder to use does not address the problem
that TMP can (and does! see Mikes blog) accumulate up to 50% of the index as
deleted documents during the normal course of an indexes' lifetime.
As for removing "trappy behavior" like delete-by-query or atomic updates, there
are completely valid use cases where the entire index gets replaced gradually
over time that would get us back into this situation even if those features
were removed. And I can't imagine getting consensus that they should be removed.
> Add a parameter to TieredMergePolicy to merge segments that have more than X
> percent deleted documents
> ------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-7976
> URL: https://issues.apache.org/jira/browse/LUCENE-7976
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Erick Erickson
>
> We're seeing situations "in the wild" where there are very large indexes (on
> disk) handled quite easily in a single Lucene index. This is particularly
> true as features like docValues move data into MMapDirectory space. The
> current TMP algorithm allows on the order of 50% deleted documents as per a
> dev list conversation with Mike McCandless (and his blog here:
> https://www.elastic.co/blog/lucenes-handling-of-deleted-documents).
> Especially in the current era of very large indexes in aggregate, (think many
> TB) solutions like "you need to distribute your collection over more shards"
> become very costly. Additionally, the tempting "optimize" button exacerbates
> the issue since once you form, say, a 100G segment (by
> optimizing/forceMerging) it is not eligible for merging until 97.5G of the
> docs in it are deleted (current default 5G max segment size).
> The proposal here would be to add a new parameter to TMP, something like
> <maxAllowedPctDeletedInBigSegments> (no, that's not serious name, suggestions
> welcome) which would default to 100 (or the same behavior we have now).
> So if I set this parameter to, say, 20%, and the max segment size stays at
> 5G, the following would happen when segments were selected for merging:
> > any segment with > 20% deleted documents would be merged or rewritten NO
> > MATTER HOW LARGE. There are two cases,
> >> the segment has < 5G "live" docs. In that case it would be merged with
> >> smaller segments to bring the resulting segment up to 5G. If no smaller
> >> segments exist, it would just be rewritten
> >> The segment has > 5G "live" docs (the result of a forceMerge or optimize).
> >> It would be rewritten into a single segment removing all deleted docs no
> >> matter how big it is to start. The 100G example above would be rewritten
> >> to an 80G segment for instance.
> Of course this would lead to potentially much more I/O which is why the
> default would be the same behavior we see now. As it stands now, though,
> there's no way to recover from an optimize/forceMerge except to re-index from
> scratch. We routinely see 200G-300G Lucene indexes at this point "in the
> wild" with 10s of shards replicated 3 or more times. And that doesn't even
> include having these over HDFS.
> Alternatives welcome! Something like the above seems minimally invasive. A
> new merge policy is certainly an alternative.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
