Otis, Thank you for your response. I know there are a few variables here but the difference in memory utilization with and without shards somehow leads me to believe that the leak could be within Solr.
I tried using a profiling tool - Yourkit. The trial version was free for 15 days. But I couldn't find anything of significance. Regards Rahul On Tue, Aug 4, 2009 at 7:35 PM, Otis Gospodnetic <otis_gospodne...@yahoo.com > wrote: > Hi Rahul, > > A) There are no known (to me) memory leaks. > I think there are too many variables for a person to tell you what exactly > is happening, plus you are dealing with the JVM here. :) > > Try jmap -histo:live PID-HERE | less and see what's using your memory. > > Otis > -- > Sematext is hiring -- http://sematext.com/about/jobs.html?mls > Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR > > > > ----- Original Message ---- > > From: Rahul R <rahul.s...@gmail.com> > > To: solr-user@lucene.apache.org > > Sent: Tuesday, August 4, 2009 1:09:06 AM > > Subject: JVM Heap utilization & Memory leaks with Solr > > > > I am trying to track memory utilization with my Application that uses > Solr. > > Details of the setup : > > -3rd party Software : Solaris 10, Weblogic 10, jdk_150_14, Solr 1.3.0 > > - Hardware : 12 CPU, 24 GB RAM > > > > For testing during PSR I am using a smaller subset of the actual data > that I > > want to work with. Details of this smaller sub-set : > > - 5 million records, 4.5 GB index size > > > > Observations during PSR: > > A) I have allocated 3.2 GB for the JVM(s) that I used. After all users > > logout and doing a force GC, only 60 % of the heap is reclaimed. As part > of > > the logout process I am invalidating the HttpSession and doing a close() > on > > CoreContainer. From my application's side, I don't believe I am holding > on > > to any resource. I wanted to know if there are known issues surrounding > > memory leaks with Solr ? > > B) To further test this, I tried deploying with shards. 3.2 GB was > allocated > > to each JVM. All JVMs had 96 % free heap space after start up. I got > varying > > results with this. > > Case 1 : Used 6 weblogic domains. My application was deployed one 1 > domain. > > I split the 5 million index into 5 parts of 1 million each and used them > as > > shards. After multiple users used the system and doing a force GC, around > 94 > > - 96 % of heap was reclaimed in all the JVMs. > > Case 2: Used 2 weblogic domains. My application was deployed on 1 domain. > On > > the other, I deployed the entire 5 million part index as one shard. After > > multiple users used the system and doing a gorce GC, around 76 % of the > heap > > was reclaimed in the shard JVM. And 96 % was reclaimed in the JVM where > my > > application was running. This result further convinces me that my > > application can be absolved of holding on to memory resources. > > > > I am not sure how to interpret these results ? For searching, I am using > > Without Shards : EmbeddedSolrServer > > With Shards :CommonsHttpSolrServer > > In terms of Solr objects this is what differs in my code between normal > > search and shards search (distributed search) > > > > After looking at Case 1, I thought that the CommonsHttpSolrServer was > more > > memory efficient but Case 2 proved me wrong. Or could there still be > memory > > leaks in my application ? Any thoughts, suggestions would be welcome. > > > > Regards > > Rahul > >
