[jira] (SOLR-6761) Ability to ignore commit and optimize requests from clients when running in SolrCloud mode.

[Erick Erickson (JIRA)]

Title: Message Title

Erick Erickson commented on SOLR-6761   Re: Ability to ignore commit and optimize requests from clients when running in SolrCloud mode. Azitabh: bq: In use cases where delete is used often, the size of index on disk will keep on growing unless optimize is called periodically This is not true. As the index changes, segments are merged. During the merge process, the space used by deleted documents is recovered. In fact, an "optimize" is really forcing the merge of all the segments (under the covers it's been renamed in fact to "forceMerge". If all you're doing is deleting, segments files can hang around for a while even if they're empty, but as you start adding docs again, they're merged according to your merge policy and space is reclaimed. Generally you'll have 10-15% of your index occupied by deleted docs, although that number can vary. For an excellent video, see: http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html. The third animation (TieredMergePolicy) is the default. And optimizing to a single segment is a very expensive operation as it rewrites the entire index. Worse, the single segment will accumulate deleted documents and since TieredMergePolicy tries to merge "like sized segments", the docs you're deleting after that will accumulate to an even greater percentage of your index than you'd have otherwise. So generally the recommendation is to not optimize unless you have a static index. By that I mean say you index say once a day during off hours. Go ahead and optimize, you'll get some performance gain. But for an index that's changing all the time it's counter-productive. If you absolutely must purge the deleted data, use expungeDeletes. That'll merge segments that have deleted documents into multiple segments. Unless you have evidence that you have outrageous numbers of deleted docs, though, I wouldn't even to that. And this is easily monitored on the admin UI by looking at the deletedDocs number. Note again, though, that if you exclusively do deletes that number will spike for a while. The take away here is that data for deleted documents accumulates for a while, but Lucene will take care of reclaiming that space in the normal course of events. Your concern that they will accumulate forever and segments will only be added is inaccurate. Add Comment   This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)