Re: merge policy & autocommit

On 10/28/2019 7:23 AM, Danilo Tomasoni wrote: We have a solr instance with around 40MLN docs. In the bulk import phase we noticed a high IO and CPU load and it looks like it's related to autocommit because if I disable autocommit the load of the system is very low. I know that disabling

Re: Merge policy

25% overhead is pretty good. It is easy for a merge to need almost double the space of a minimum sized index. It is possible to use 3X the space. Don’t try use the least possible disk space. If there isn’t enough free space on the disk, Solr cannot merge the big indexes. Ever. That may be what

Re: Merge policy

I got some notification from mailer, so not sure if my reply reached you: "If you are using TieredMergePolicy, you can try setting /*reclaimDeletesWeight*/." HTH, Emir On 28.10.2016 09:20, Arkadi Colson wrote: The index size of 1 shard is about 125GB and we are running 11 shards with

Re: Merge policy

It's a default installation using the default settings and parameters. Should I perhaps change the segment size or so? Is it possible to do live without re-indexing? If you need more info, just let me know... Thx! On 27-10-16 19:03, Walter Underwood wrote: That distribution of segment sizes

Re: Merge policy

The index size of 1 shard is about 125GB and we are running 11 shards with replication factor 2 so it's a lot of data. The deletions percentage at the bottom of the segment page is around 25%. So it's quite some space which we could recover. That's why I was looking for an optimize. Do you

Re: Merge policy

That distribution of segment sizes seems odd. Why so many medium-large segments? Are there custom settings for merge policy? I think the default policy would avoid so many segments that are mostly deleted documents. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/

Re: Merge policy

On 10/27/2016 9:50 AM, Yonik Seeley wrote: > On Thu, Oct 27, 2016 at 9:56 AM, Arkadi Colson > wrote: >> Thanks for the answer! Do you know if there is a way to trigger an >> optimize for only 1 shard and not the whole collection at once? > Adding a "distrib=false" parameter

Re: Merge policy

On Thu, Oct 27, 2016 at 9:56 AM, Arkadi Colson wrote: > Thanks for the answer! > Do you know if there is a way to trigger an optimize for only 1 shard and > not the whole collection at once? > Adding a "distrib=false" parameter should work I think. -Yonik

Re: Merge policy

Why do you think you need to get rid of the deleted data? During normal indexing, these will be "merged away". Optimizing has some downsides for continually changing indexes, in particular since the default tieredmergepolicy tries to merge "like size" segments, deletions will accumulate in your

Re: Merge policy

Thanks for the answer! Do you know if there is a way to trigger an optimize for only 1 shard and not the whole collection at once? On 27-10-16 15:30, Pushkar Raste wrote: Try commit with expungeDeletes="true" I am not sure if it will merge old segments that have deleted documents. In the

Re: Merge policy

Try commit with expungeDeletes="true" I am not sure if it will merge old segments that have deleted documents. In the worst case you can 'optimize' your index which should take care of removing deleted document On Oct 27, 2016 4:20 AM, "Arkadi Colson" wrote: > Hi > > As

Re: Merge Policy Recommendation for 3.6.1

Thanks Shawn,that helps a lot .our current OS limit is set to 300,000+, I guess, which is I heard is maximum for the OS .. not sure of the soft and hard limits .Will check this . Regards, Sujatha On Fri, Sep 28, 2012 at 8:14 PM, Shawn Heisey s...@elyograg.org wrote: On 9/28/2012 12:43 AM,

Re: Merge Policy Recommendation for 3.6.1

On 9/28/2012 12:43 AM, Sujatha Arun wrote: Hello, In the case where there are over 200+ cores on a single node , is it recommended to go with Tiered MP with segment size of 4 ? Our Index size vary from a few MB to 4 GB . Will there be any issue with Too many open files and the number of

Re: Merge Policy

: SolrIndexConfig accepts a mergePolicy class name, however how does one : inject properties into it? At the moment you can't. If you look at the history of MergePolicy, users have never been encouraged to implement their own (the API actively discourages it, without going so far as to make

Re: Merge Policy

I am referring to setting properties on the *existing* policy available in Lucene such as LogByteSizeMergePolicy.setMaxMergeMB On Tue, Jul 21, 2009 at 5:11 PM, Chris Hostetterhossman_luc...@fucit.org wrote: : SolrIndexConfig accepts a mergePolicy class name, however how does one : inject