gger and
> >> tend to use time on the theory that it’s easier to explain,
> >> whereas when commits happen when using maxDocs
> >> varies depending on the throughput rate.
> >>
> >> Best,
> >> Erick
> >>
> >>> On Apr 15, 2020,
that it’s easier to explain,
>> whereas when commits happen when using maxDocs
>> varies depending on the throughput rate.
>>
>> Best,
>> Erick
>>
>>> On Apr 15, 2020, at 1:28 PM, Kayak28 wrote:
>>>
>>> Hello, Solr Community:
>>&
> > On Apr 15, 2020, at 1:28 PM, Kayak28 wrote:
> >
> > Hello, Solr Community:
> >
> > I would like to ask about Default's Merge Policy for Solr 8.3.0.
> > My client (SolrJ) makes a commit every 10'000 doc.
> > I have not explicitly configured Merge
rate.
Best,
Erick
> On Apr 15, 2020, at 1:28 PM, Kayak28 wrote:
>
> Hello, Solr Community:
>
> I would like to ask about Default's Merge Policy for Solr 8.3.0.
> My client (SolrJ) makes a commit every 10'000 doc.
> I have not explicitly configured Merge Polic
Hello, Solr Community:
I would like to ask about Default's Merge Policy for Solr 8.3.0.
My client (SolrJ) makes a commit every 10'000 doc.
I have not explicitly configured Merge Policy via solrconfig.xml
For each indexing time, some documents are updated or deleted.
I think the Def
;
> Few questions before I tackled an upgrade here. Looking to go from 7.4 to
7.7.2 to take advantage of the improved Tiered Merge Policy and segment cleanup
– we are dealing with some high (45%) deleted doc counts in a few cores. Would
simply upgrading Solr and setting the cores to u
, at 5:42 PM, Zimmermann, Thomas
> wrote:
>
> Hi Folks –
>
> Few questions before I tackled an upgrade here. Looking to go from 7.4 to
> 7.7.2 to take advantage of the improved Tiered Merge Policy and segment
> cleanup – we are dealing with some high (45%) deleted doc count
Hi Folks –
Few questions before I tackled an upgrade here. Looking to go from 7.4 to 7.7.2
to take advantage of the improved Tiered Merge Policy and segment cleanup – we
are dealing with some high (45%) deleted doc counts in a few cores. Would
simply upgrading Solr and setting the cores to use
g a
correctly configured autoCommit would substantially affect indexing speeds.
The same can be true also for the merge policy? how the IO speed can
affect the merge policy parameters?
I kept the default merge policy configuration but it looks like it never
merges segments. How can I know if a mer
y our solr instance to the point of
making it not responsive?
The same can be true also for the merge policy? how the IO speed can
affect the merge policy parameters?
I kept the default merge policy configuration but it looks like it never
merges segments. How can I know if a merge is happening
what Erick said. Erick's info is completely valid.
For the version you are on, specifying a mergeFactor of 25 with no other
merge-policy related config effectively results in this config:
25
25
30
I would recommend replacing mergeFactor with an explicit merge policy
config. Sinc
bq. However we have not specified it in the following way
Is that a typo and you mean "have now specified"?
There's code in SolrIndexConfig:
if (policy instanceof TieredMergePolicy) {
if (mergeFactor != -1) {
tieredMergePolicy.setMaxMergeAtOnce(mergeFactor);
tieredMergePolicy.setSegmen
Hi all,
I am little bit confused.
We are on solr 6. and as per the documentation i think solr 6 uses
TieredMergePolicyFactory.
However we have not specified it in the following way
10
10
We still use 25. which i understand is not used
by TieredMergePolicyFactory.
So my confusion is that w
Thanks Erick, good point on maxMergedSegmentMB, many of my segments really
are max out.
My index isn't 800G, but it's not far from it - it's about 250G per server.
I have high confidence in Solr and my EC2 i3-2xl instances, so far I got
pretty good results.
--
Sent from: http://lucene.472066.n3
So I'm guessing you have something on the order of an 800G index?
The max segment size is roughly 5G (by default) and assuming all your segments
are close to the max size I get 160 * 5G = 800G, but that may be off.
I think you're barking up the wrong tree if these numbers are close to
correct. Th
To be clear - I'm talking about query performance, not indexing performance.
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Thanks for the quick answer Erick,
I'm hoping to improve performance by reducing the number of segments.
Currently I have ~160 segments. Am I wrong thinking it might improve
performance?
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
The merge rate will be limited by the number of merge threads. You'll merge
more often though so the load will change. That said, I wouldn't be
concerned unless you have a very high indexing rate.
Why do you want to change anyway? Unless you've tried the new settings in a
Dev environment, the bigg
Hi,
Is it safe to change the mergePolicyFactory config on production servers?
Specifically maxMergeAtOnce and segmentsPerTier. How will solr reconcile the
current state of the segments with the new config? In case of setting
segmentsPerTier to a lower number - will subsequent merges be particulary
1> merging takes place up until the max segment size is reached (5G in
the default TieredMergePolicy).
2> there are a couple of options, again config changes for TieredMergePolicy
10
might help.
You could also try upping this (the default is 5G).
5000
Best,
Erick
On Mon, Oct 23, 2017 at 10:34
Thanks eric.
(Beginner in solr). Few questions.
1. Does merging take place only when we have deleted docs?
When my segments reach a count of 35+ the search is getting slow.Only on
performing force merge to index the search is efficient.
2. Is there any way we can reduce the number of segments
Amrit,
Thanks for your reply. I have removed that
1000
1
15
false
1024
2
2
hdfs
1
0
--
Sent from: http://lucene.472066.n3.nabble.com
And please define what you mean by "merging is not working". One
parameter is max segments size, which defaults to 5G. Segments
at or near that size are not eligible for merging unless they have
around 50% deleted docs.
Best,
Erick
On Mon, Oct 23, 2017 at 3:11 AM, Amrit Sarkar wrote:
> Chandru,
Chandru,
Didn't try the above config bu whyt have you defined both "mergePolicy" and
"mergePolicyFactory"? and pass different values for same parameters?
> 10
> 1
>
>
> 10
> 10
>
>
Amrit Sarkar
Search Engineer
Lucidworks, Inc.
415-589-9269
www.lucidworks.com
Twitter http://
The following is my solrconfig.xml
1000
1
15
false
1024
10
1
10
10
hdfs
1
0
Please let me know if should I tweak something above
--
Thanks,
Chandru.S
er. That's why I was looking for an optimize.
>
> Do you have any idea why the merge policy does not merge away the deletions?
> Should I tweak some parameters somehow? It's a default installation using the
> default settings and parameters. If you need more info, just let
ards
with replication factor 2 so it's a lot of data. The deletions
percentage at the bottom of the segment page is around 25%. So it's
quite some space which we could recover. That's why I was looking for
an optimize.
Do you have any idea why the merge policy does not merge away
sizes seems odd. Why so many medium-large segments?
Are there custom settings for merge policy? I think the default policy would
avoid so many segments that are mostly deleted documents.
wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/ (my blog)
On Oct 27, 2016,
imize.
Do you have any idea why the merge policy does not merge away the
deletions? Should I tweak some parameters somehow? It's a default
installation using the default settings and parameters. If you need more
info, just let me know...
Thx!
On 27-10-16 17:40, Erick Erickson wrote:
That distribution of segment sizes seems odd. Why so many medium-large segments?
Are there custom settings for merge policy? I think the default policy would
avoid so many segments that are mostly deleted documents.
wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org
On 10/27/2016 9:50 AM, Yonik Seeley wrote:
> On Thu, Oct 27, 2016 at 9:56 AM, Arkadi Colson
> wrote:
>> Thanks for the answer! Do you know if there is a way to trigger an
>> optimize for only 1 shard and not the whole collection at once?
> Adding a "distrib=false" parameter should work I think.
On Thu, Oct 27, 2016 at 9:56 AM, Arkadi Colson wrote:
> Thanks for the answer!
> Do you know if there is a way to trigger an optimize for only 1 shard and
> not the whole collection at once?
>
Adding a "distrib=false" parameter should work I think.
-Yonik
, "Arkadi Colson" wrote:
>
>> Hi
>>
>> As you can see in the screenshot above in the oldest segments there are a
>> lot of deletions. In total the shard has about 26% deletions. How can I get
>> rid of them so the index will be smaller again?
>> Can this o
re are a lot of deletions. In total the shard has about 26%
deletions. How can I get rid of them so the index will be smaller
again?
Can this only be done with an optimize or does it also depend on
the merge policy? If it also depends also on the merge policy
which one sho
> Hi
>
> As you can see in the screenshot above in the oldest segments there are a
> lot of deletions. In total the shard has about 26% deletions. How can I get
> rid of them so the index will be smaller again?
> Can this only be done with an optimize or does it also depend on the merg
Hi
As you can see in the screenshot above in the oldest segments there are
a lot of deletions. In total the shard has about 26% deletions. How can
I get rid of them so the index will be smaller again?
Can this only be done with an optimize or does it also depend on the
merge policy? If it
nd just to be really clear, you _only_ seeing more segments being
> >> added, right? If you're only counting files in the index directory, it's
> >> _possible_ that merging is happening, you're just seeing new files take
> >> the place of old ones.
> >>
> >> Best,
> >> Erick
> >>
> >> On Wed, Mar 4, 2015 at 7:12 PM, Shawn Heisey
> wrote:
> >>> On 3/4/2015 4:12 PM, Erick Erickson wrote:
> >>>> I _think_, but don't know for sure, that the merging stuff doesn't get
> >>>> triggered until you commit, it doesn't "just happen".
> >>>>
> >>>> Shot in the dark...
> >>>
> >>> I believe that new segments are created when the indexing buffer
> >>> (ramBufferSizeMB) fills up, even without commits. I'm pretty sure that
> >>> anytime a new segment is created, the merge policy is checked to see
> >>> whether a merge is needed.
> >>>
> >>> Thanks,
> >>> Shawn
> >>>
> >
>
>
--
Dmitry Kan
Luke Toolbox: http://github.com/DmitryKey/luke
Blog: http://dmitrykan.blogspot.com
Twitter: http://twitter.com/dmitrykan
SemanticAnalyzer: www.semanticanalyzer.info
t merging is happening, you're just seeing new files take
>> the place of old ones.
>>
>> Best,
>> Erick
>>
>> On Wed, Mar 4, 2015 at 7:12 PM, Shawn Heisey wrote:
>>> On 3/4/2015 4:12 PM, Erick Erickson wrote:
>>>> I _think_, but don't know for sure, that the merging stuff doesn't get
>>>> triggered until you commit, it doesn't "just happen".
>>>>
>>>> Shot in the dark...
>>>
>>> I believe that new segments are created when the indexing buffer
>>> (ramBufferSizeMB) fills up, even without commits. I'm pretty sure that
>>> anytime a new segment is created, the merge policy is checked to see
>>> whether a merge is needed.
>>>
>>> Thanks,
>>> Shawn
>>>
>
7;t get
>>> triggered until you commit, it doesn't "just happen".
>>>
>>> Shot in the dark...
>>
>> I believe that new segments are created when the indexing buffer
>> (ramBufferSizeMB) fills up, even without commits. I'm pretty sure that
>> anytime a new segment is created, the merge policy is checked to see
>> whether a merge is needed.
>>
>> Thanks,
>> Shawn
>>
sure, that the merging stuff doesn't get
>> triggered until you commit, it doesn't "just happen".
>>
>> Shot in the dark...
>
> I believe that new segments are created when the indexing buffer
> (ramBufferSizeMB) fills up, even without commits. I&
ffer
(ramBufferSizeMB) fills up, even without commits. I'm pretty sure that
anytime a new segment is created, the merge policy is checked to see
whether a merge is needed.
Thanks,
Shawn
actually after every commit a new segment gets created. I don't see them
merging down.
what all could i do to debug this better. Hasn't anyone else tried to merge
their segments down to a specific range :) ?
On Wed, Mar 4, 2015 at 3:12 PM, Erick Erickson
wrote:
> I _think_, but don't know for s
I _think_, but don't know for sure, that the merging stuff doesn't get
triggered until you commit, it doesn't "just happen".
Shot in the dark...
Erick
On Wed, Mar 4, 2015 at 1:15 PM, Summer Shire wrote:
> Hi All,
>
> I am using solr 4.7.2 is there a bug wrt merging the segments down ?
>
> I rec
Hi All,
I am using solr 4.7.2 is there a bug wrt merging the segments down ?
I recently added the following to my solrConfig.xml
false
100
1000
5
But I do not see any merging of the segments happening. I saw some other
people have
the same issue but there wasn’t much info
; In earlier lucene version it merges segements periodically
> according to merge policy, when it reached merge time, indexing
> request may take longer time to finish (in my test it may delay
> 10-30 seconds, depending on indexed data size).
>
> I read solr 3.6 - 4.1 doc and w
Hi,
In earlier lucene version it merges segements periodically
according to merge policy, when it reached merge time, indexing
request may take longer time to finish (in my test it may delay
10-30 seconds, depending on indexed data size).
I read solr 3.6 - 4.1 doc and we have entries in
r 14 segments, and possibly more.
>
> Assuming some things, which lead to using the 13 segment figure:
> simultaneous indexing to multiple cores at once, with termvectors turned
> on. With these assumptions, a 200 core Solr installation using 4 segments
> might potentially have nearly
ts might potentially have nearly 37000 files open, but is more
likely to have significantly less. If you increase your merge policy
segment limit, the numbers will go up from there.
I have configured my Linux servers with a soft file limit of 49152 and a
hard limit of 65536. My segment limit is
Hello,
In the case where there are over 200+ cores on a single node , is it
recommended to go with Tiered MP with segment size of 4 ? Our Index size
vary from a few MB to 4 GB .
Will there be any issue with "Too many open files " and the number of
indexes with respect to MP ? At the moment we ar
I am referring to setting properties on the *existing* policy
available in Lucene such as LogByteSizeMergePolicy.setMaxMergeMB
On Tue, Jul 21, 2009 at 5:11 PM, Chris
Hostetter wrote:
>
> : SolrIndexConfig accepts a mergePolicy class name, however how does one
> : inject properties into it?
>
> At
: SolrIndexConfig accepts a mergePolicy class name, however how does one
: inject properties into it?
At the moment you can't.
If you look at the history of MergePolicy, users have never been
encouraged to implement their own (the API actively discourages it,
without going so far as to make
SolrIndexConfig accepts a mergePolicy class name, however how does one
inject properties into it?
52 matches
Mail list logo
