10% free space is guaranteed to cause problems. That is a faulty installation.
Explain to ops that Solr needs double the minimum index size. This is required
for normal operation. That isn't "extra", it is required for merges. Solr makes
copies instead of doing record locking. The merge design is essential for speed.
If they don't provide that, it will break, and it will be their fault.
If they don't want to provide that, they need a different search engine.
Adapting the merge policy to work with only 10% free space is not reasonable.
When one segment is bigger than 10% (and it will be), merging that segment will
fail.
wunder
On Sep 9, 2013, at 12:24 PM, Manuel Le Normand wrote:
> I can only agree for the 50% free space recommendation. Unfortunately I do
> not have this for the current time, I'm standing on a 10% free disk (out of
> 300GB for each server). I'm aware it is very low.
>
> Does this seem reasonable adapting the current merge policy (or writing a
> new one) that would free up the transient disk space every merge instead of
> waiting for all of them to achieve? Where can I get such a answer ("people
> who wrote the code")?
>
> Thanks
>
>
> On Sun, Sep 8, 2013 at 9:30 PM, Erick Erickson <erickerick...@gmail.com>wrote:
>
>> Right, but you should have at least as much free space as your total index
>> size, and I don't see the total index size (but I'm just glancing).
>>
>> I'm not entirely sure you can precisely calculate the maximum free space
>> you have relative to the amount needed for merging, some of the people who
>> wrote that code can probably tell you more.
>>
>> I'd _really_ try to get more disk space. The amount of engineer time spent
>> trying to tune this is waaaaay more expensive than a disk...
>>
>> Best,
>> Erick
>>
>>
>> On Sun, Sep 8, 2013 at 11:51 AM, Manuel Le Normand <
>> manuel.lenorm...@gmail.com> wrote:
>>
>>> Hi,
>>> In order to delete part of my index I run a delete by query that intends
>> to
>>> erase 15% of the docs.
>>> I added this params to the solrconfig.xml
>>> <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>>> <int name="maxMergeAtOnce">2</int>
>>> <int name="maxMergeAtOnceExplicit">2</int>
>>> <double name="maxMergedSegmentMB">5000.0</double>
>>> <double name="reclaimDeletesWeight">10.0</double>
>>> <double name="segmentsPerTier">15.0</double>
>>> </mergePolicy>
>>>
>>> The extra params were added in order to promote merge of old segments but
>>> with restriction on the transient disk that can be used (as I have only
>>> 15GB per shard).
>>>
>>> This procedure failed on a no space left on device exception, although
>>> proper calculations show that these params should cause no usage excess
>> of
>>> the transient free disk space I have.
>>> Looking on the infostream I can see that the first merges do succeed but
>>> older segments are kept in reference thus cannot be deleted until all the
>>> merging are done.
>>>
>>> Is there anyway of overcoming this?
>>>
>>
--
Walter Underwood
wun...@wunderwood.org
