What Walter said. Although with Solr 7.6, unless you specify maxSegments
explicitly,
you won’t create segments over the default 5G maximum.
And if you have in the past specified maxSegments so you have segments over 5G,
optimize (again without specifying maxSegments) will do a “singleton merge”
From that short description, you should not be running optimize at all.
Just stop doing it. It doesn’t make that big a difference.
It may take your indexes a few weeks to get back to a normal state after the
forced merges.
wunder
Walter Underwood
wun...@wunderwood.org
Thank you David, Walt , Eric.
1. First time bloated index generated , there is no disk space issue. one copy
of index is 1/6 of disk capacity. we ran into disk capacity after more than 2
copies of bloated copies.2. Solr is upgraded from 5.*. in 5.* more than 5
segments is causing performance
It Depends (tm).
As of Solr 7.5, optimize is different. See:
https://lucidworks.com/post/solr-and-optimizing-your-index-take-ii/
So, assuming you have _not_ specified maxSegments=1, any very large
segment (near 5G) that has _zero_ deleted documents won’t be merged.
So there are two scenarios:
For a full forced merge (mistakenly named “optimize”), the worst case disk space
is 3X the size of the index. It is common to need 2X the size of the index.
When I worked on Ultraseek Server 20+ years ago, it had the same merge behavior.
I implemented a disk space check that would refuse to merge
I cant give you a 100% true answer but ive experienced this, and what
"seemed" to happen to me was that the optimize would start, and that will
drive the size up by 3 fold, and if you out of disk space in the process
the optimize will quit since, it cant optimize, and leave the live index
pieces
when optimize command is issued, the expectation after the completion of
optimization process is that the index size either decreases or at most remain
same. In solr 7.6 cluster with 50 plus shards, when optimize command is issued,
some of the shard's transient or older segment files are not
