Re: Memory Usage increases by a lot during and after optimization .

Hi Shawn, I am using Java Version 8 Update 45 (build 1.8.0_45-b15). It is a 64-bit Java. Thank you. Regards, Edwin On 8 January 2016 at 00:15, Shawn Heisey wrote: > On 1/7/2016 12:53 AM, Zheng Lin Edwin Yeo wrote: > >> Subtracting SHR from RES (or in your case,

Re: Memory Usage increases by a lot during and after optimization .

Hi Shawn, Thank you for your explanation. Yes, both of the top two processes are Solr. I have two Solr processes on one machine now, as the second one is a replica of the first one. In the future, the plan is to have them on separate machine. >Subtracting SHR from RES (or in your case,

Re: Memory Usage increases by a lot during and after optimization .

On 1/5/2016 11:50 PM, Zheng Lin Edwin Yeo wrote: > Here is the new screenshot of the Memory tab of the Resource Monitor. > https://www.dropbox.com/s/w4bnrb66r16lpx1/Resource%20Monitor.png?dl=0 > > Yes, I found that the value under the "Working Set" column is much higher > than the others. Also,

Re: Memory Usage increases by a lot during and after optimization .

Hi Shawn, Thanks for your reply. I have uploaded the screenshot here https://www.dropbox.com/s/l5itfbaus1c9793/Memmory%20Usage.png?dl=0 Basically, Java(TM) Platform SE Library, which Solr is running on, is only using about 22GB currently. However, the memory usage at the top says it is using

Re: Memory Usage increases by a lot during and after optimization .

Hi Toke, I read the server's memory usage from the Task manager under Windows, Regards, Edwin On 4 January 2016 at 17:17, Toke Eskildsen wrote: > On Mon, 2016-01-04 at 10:05 +0800, Zheng Lin Edwin Yeo wrote: > > A) Before I start the optimization, the server's

Re: Memory Usage increases by a lot during and after optimization .

On 1/5/2016 9:59 AM, Zheng Lin Edwin Yeo wrote: > I have uploaded the screenshot here > https://www.dropbox.com/s/l5itfbaus1c9793/Memmory%20Usage.png?dl=0 > > Basically, Java(TM) Platform SE Library, which Solr is running on, is only > using about 22GB currently. However, the memory usage at the

Re: Memory Usage increases by a lot during and after optimization .

Hi Shawn, Here is the new screenshot of the Memory tab of the Resource Monitor. https://www.dropbox.com/s/w4bnrb66r16lpx1/Resource%20Monitor.png?dl=0 Yes, I found that the value under the "Working Set" column is much higher than the others. Also, the value which I was previously looking at under

Re: Memory Usage increases by a lot during and after optimization .

On Mon, 2016-01-04 at 10:05 +0800, Zheng Lin Edwin Yeo wrote: > A) Before I start the optimization, the server's memory usage > is consistent at around 16GB, when Solr startsup and we did some searching. How do you read this number? > However, when I click on the optimization button, the memory

Re: Memory Usage increases by a lot during and after optimization .

On 1/3/2016 7:05 PM, Zheng Lin Edwin Yeo wrote: > A) Before I start the optimization, the server's memory usage > is consistent at around 16GB, when Solr startsup and we did some searching. > However, when I click on the optimization button, the memory usage > increases gradually, until it reaches

Re: Memory Usage increases by a lot during and after optimization .

Thanks for the reply Shawn and Erick. What *exactly* are you looking at that says Solr is using all your memory? You must be extremely specific when answering this question. This will determine whether we should be looking for a bug or not. A) Before I start the optimization, the server's

Re: Memory Usage increases by a lot during and after optimization .

If you happen to be looking at "top" or the like, you might be seeing virtual memory, see Uwe's excellent blog here: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html Best, Erick On Fri, Jan 1, 2016 at 11:46 PM, Shawn Heisey wrote: > On 12/31/2015 8:03

Re: Memory Usage increases by a lot during and after optimization .

On 12/31/2015 8:03 PM, Zheng Lin Edwin Yeo wrote: > But the problem I'm facing now is that during optimizing, the memory usage > of the server hit the maximum of 64GB, and I believe the optimization could > not be completed fully as there is not enough memory, so when I check the > index again, it

Re: Memory Usage increases by a lot during and after optimization .

Wouldn't collection swapping be a better strategy in that case? Load and optimise in a separate server, then swap it in. On 30 Dec 2015 10:08 am, "Walter Underwood" wrote: > The only time that a force merge might be useful is when you reindex all > content every night or

Re: Memory Usage increases by a lot during and after optimization .

Hi Yonik, Yes, the plan is to do the optimizing at night after indexing, when there are lesser user who will use the system. But the problem I'm facing now is that during optimizing, the memory usage of the server hit the maximum of 64GB, and I believe the optimization could not be completed

Memory Usage increases by a lot during and after optimization .

Hi, I am facing a situation, when I do an optimization by clicking on the "Optimized" button on the Solr Admin Overview UI, the memory usage of the server increases gradually, until it reaches near the maximum memory available. There is 64GB of memory available in the server. Even after the

Re: Memory Usage increases by a lot during and after optimization .

Do not “optimize". It is a forced merge, not an optimization. It was a mistake to ever name it “optimize”. Solr automatically merges as needed. There are a few situations where a force merge might make a small difference. Maybe 10% or 20%, no one had bothered to measure it. If your index is

Re: Memory Usage increases by a lot during and after optimization .

Hi Walter, Thanks for your reply. Then how about optimization after indexing? Normally the index size is much larger after indexing, then after optimization, the index size reduces. Do we still need to do that? Regards, Edwin On 30 December 2015 at 10:45, Walter Underwood

Re: Memory Usage increases by a lot during and after optimization .

The only time that a force merge might be useful is when you reindex all content every night or every week, then do not make any changes until the next reindex. But even then, it probably does not matter. Just let Solr do its thing. Solr is pretty smart. A long time ago (1996-2006), I worked

Re: Memory Usage increases by a lot during and after optimization .

Some people also want to control when major segment merges happen, and optimizing at a known time helps prevent a major merge at an unknown time (which can be equivalent to an optimize/forceMerge). The benefits of optimizing (and having fewer segments to search across) will vary depending on the

Re: Memory Usage increases by a lot during and after optimization .

Thanks for the information. Another thing I like to confirm is, will the Java Heap size setting affect the optimization process or the memory usage? Is the any recommended setting that we can use, for an index size of 200GB? Regards, Edwin On 30 December 2015 at 11:07, Walter Underwood

Re: Memory Usage increases by a lot during and after optimization .

Question: does anyone have example good merge settings for solrconfig ? To keep the number of segments small like 6? On Tue, Dec 29, 2015 at 8:49 PM, Yonik Seeley wrote: > Some people also want to control when major segment merges happen, and > optimizing at a known time