Hi Shawn, Thanks for your reply.
I have uploaded the screenshot here https://www.dropbox.com/s/l5itfbaus1c9793/Memmory%20Usage.png?dl=0 Basically, Java(TM) Platform SE Library, which Solr is running on, is only using about 22GB currently. However, the memory usage at the top says it is using 73% now (which I think is already higher than the figures, given that I have 64GB of RAM), and it could potentially go up to 100%, even though the memory usage of Java(TM) Platform SE Library remains around 22GB, and there is no other new task which uses alot of memory are running. The figure is sorted according to the memory usage already. Regards, Edwin On 5 January 2016 at 08:16, Shawn Heisey <apa...@elyograg.org> wrote: > On 1/3/2016 7:05 PM, Zheng Lin Edwin Yeo wrote: > > A) Before I start the optimization, the server's memory usage > > is consistent at around 16GB, when Solr startsup and we did some > searching. > > However, when I click on the optimization button, the memory usage > > increases gradually, until it reaches the maximum of 64GB which the > server > > has. But this only happens to the collection with index of 200GB, and not > > other collections which has smaller index size (they are at most 1GB at > the > > moment). > > <snip> > > > A) I am quite curious at this also, because in the Task Manager of the > > server, the amount of memory usage stated does not tally with the > > percentage of memory usage. When I start optimizatoin, the memory usage > > states the JVM is only using 14GB, but the percentage of memory usage is > > almost 100%, when I have 64GB RAM. I have check the other processes > running > > in the server, and did not found any other processes that takes up a > large > > amount of memory, and the total amount of memory usage for the whole > sever > > is only around 16GB. > > Toke's reply is spot on. > > In your first answer above, you didn't really answer my question, which > was "What *exactly* are you looking at that says Solr is using all your > memory?" You've said "the server's memory usage" but haven't described > how you got that number. > > Here's a screenshot of "top" on one of my Solr servers, with the list > sorted by memory usage: > > https://www.dropbox.com/s/i49s2uyfetwo3xq/solr-mem-prod-8g-heap.png?dl=0 > > This machine has 165GB (base 2 number) of index data on it, and 64GB of > memory. Solr has been assigned an 8GB heap. Here's more specific info > about the size of the index data: > > root@idxb3:/index/solr5/data# du -hs data > 165G data > root@idxb3:/index/solr5/data# du -s data > 172926520 data > > You can see that the VIRT memory size of the Solr process is > approximately the same as the total index size (165GB) plus the max heap > (8GB), which adds up to 173GB. The RES memory size of the java process > is 8.3GB -- just a little bit larger than the max heap. > > At the OS level, my server shows 46GB used out of 64GB total ... which > probably seems excessive, until you consider the 36 million kilobytes in > the "cached" statistic. This is the amount of memory being used for the > page cache. If you subtract that memory, then you can see that this > server has only allocated about 10GB of RAM total -- exactly what I > would expect for a Linux machine dedicated to Solr with the max heap at > 8GB. > > Although my server is indicating about 18GB of memory free, I have seen > perfectly functioning servers with that number very close to zero. It > is completely normal for the "free" memory statistic on Linux and > Windows to show a few megabytes or less, especially when you optimize a > Solr index, which reads (and writes) all of the index data, and will > fill up the page cache. > > So, I will ask something very similar to my initial question. Where > *exactly* are you looking to see the memory usage that you believe is a > problem? A screenshot would be very helpful. > > Here's a screenshot from my Windows client. This machine is NOT running > Solr, but the situation with free and cached memory is similar. > > https://www.dropbox.com/s/wex1gbj7e45g8ed/windows7-mem-usage.png?dl=0 > > I am not doing anything particularly unusual with this machine, but it > says there is *zero* free memory, out of 16GB total. There is 9GB of > memory in the page cache, though -- memory that the OS will instantly > give up if any program requests it, which you can see because the > "available" stat is also about 9GB. This Windows machine is doing > perfectly fine as far as memory. > > Thanks, > Shawn > >
