Re: Interesting OutOfMemoryError on a 170M index
On Thu, Jan 14, 2010 at 4:04 AM, Minutello, Nick nick.minute...@credit-suisse.com wrote: Agreed, commit every second. Assuming I understand what you're saying correctly: There shouldn't be any index readers - as at this point, just writing to the index. Did I understand correctly what you meant? Solr opens a new IndexSearcher after a commit whether or not you are querying it. So if you are committing every second, you are going to have a number of IndexSearchers trying to warm themselves. That can cause an OutOfMemoryException. Just indexing documents with a reasonable heap size will not cause the JVM to go out of memory. -- Regards, Shalin Shekhar Mangar.
Interesting OutOfMemoryError on a 170M index
Hi, I have a bit of an interesting OutOfMemoryError that I'm trying to figure out. My client Solr server are running in the same JVM (for deployment simplicity). FWIW, I'm using Jetty to host Solr. I'm using the supplied code for the http-based client interface. Solr 1.3.0. My app is adding about 20,000 documents per minute to the index - one at a time (it is listening to an event stream and for every event, it adds a new document to the index). The size of the documents, however, is tiny - the total index growth is only about 170M (after about 1 hr and the OutOfMemoryError) At this point, there is zero querying happening - just updates to the index (only adding documents, no updates or deletes) After about an hour or so, my JVM runs out of heap space - and if I look at the memory utilisation over time, it looks like a classic memory leak. It slowly ramps up until we end up with constant FULL GC's and eventual OOME. Max heap space is 512M. In Solr, I'm using autocommit (to buffer the updates) autoCommit maxDocs1/maxDocs maxTime1000/maxTime /autoCommit (Aside: Now, I'm not sure if I am meant to call commit or not on the client SolrServer class if I am using autocommit - but as it turns out, I get OOME whether I do that or not) Any suggestions/advice of quick things to check before I dust off the profiler? Thanks in advance. Cheers, Nick === Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ===
Re: Interesting OutOfMemoryError on a 170M index
The time in autocommit is in milliseconds. You are committing every second while indexing. This then causes a build-up of sucessive index readers that absorb each commit, which is probably the out-of-memory. On Wed, Jan 13, 2010 at 10:36 AM, Minutello, Nick nick.minute...@credit-suisse.com wrote: Hi, I have a bit of an interesting OutOfMemoryError that I'm trying to figure out. My client Solr server are running in the same JVM (for deployment simplicity). FWIW, I'm using Jetty to host Solr. I'm using the supplied code for the http-based client interface. Solr 1.3.0. My app is adding about 20,000 documents per minute to the index - one at a time (it is listening to an event stream and for every event, it adds a new document to the index). The size of the documents, however, is tiny - the total index growth is only about 170M (after about 1 hr and the OutOfMemoryError) At this point, there is zero querying happening - just updates to the index (only adding documents, no updates or deletes) After about an hour or so, my JVM runs out of heap space - and if I look at the memory utilisation over time, it looks like a classic memory leak. It slowly ramps up until we end up with constant FULL GC's and eventual OOME. Max heap space is 512M. In Solr, I'm using autocommit (to buffer the updates) autoCommit maxDocs1/maxDocs maxTime1000/maxTime /autoCommit (Aside: Now, I'm not sure if I am meant to call commit or not on the client SolrServer class if I am using autocommit - but as it turns out, I get OOME whether I do that or not) Any suggestions/advice of quick things to check before I dust off the profiler? Thanks in advance. Cheers, Nick === Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html === -- Lance Norskog goks...@gmail.com
RE: Interesting OutOfMemoryError on a 170M index
Agreed, commit every second. Assuming I understand what you're saying correctly: There shouldn't be any index readers - as at this point, just writing to the index. Did I understand correctly what you meant? -Nick -Original Message- From: Lance Norskog [mailto:goks...@gmail.com] Sent: 13 January 2010 22:28 To: solr-user@lucene.apache.org Subject: Re: Interesting OutOfMemoryError on a 170M index The time in autocommit is in milliseconds. You are committing every second while indexing. This then causes a build-up of sucessive index readers that absorb each commit, which is probably the out-of-memory. On Wed, Jan 13, 2010 at 10:36 AM, Minutello, Nick nick.minute...@credit-suisse.com wrote: Hi, I have a bit of an interesting OutOfMemoryError that I'm trying to figure out. My client Solr server are running in the same JVM (for deployment simplicity). FWIW, I'm using Jetty to host Solr. I'm using the supplied code for the http-based client interface. Solr 1.3.0. My app is adding about 20,000 documents per minute to the index - one at a time (it is listening to an event stream and for every event, it adds a new document to the index). The size of the documents, however, is tiny - the total index growth is only about 170M (after about 1 hr and the OutOfMemoryError) At this point, there is zero querying happening - just updates to the index (only adding documents, no updates or deletes) After about an hour or so, my JVM runs out of heap space - and if I look at the memory utilisation over time, it looks like a classic memory leak. It slowly ramps up until we end up with constant FULL GC's and eventual OOME. Max heap space is 512M. In Solr, I'm using autocommit (to buffer the updates) autoCommit maxDocs1/maxDocs maxTime1000/maxTime /autoCommit (Aside: Now, I'm not sure if I am meant to call commit or not on the client SolrServer class if I am using autocommit - but as it turns out, I get OOME whether I do that or not) Any suggestions/advice of quick things to check before I dust off the profiler? Thanks in advance. Cheers, Nick == = Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html == = -- Lance Norskog goks...@gmail.com === Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ===
Re: Interesting OutOfMemoryError on a 170M index
On Jan 13, 2010, at 5:34 PM, Minutello, Nick wrote: Agreed, commit every second. Do you need the index to be updated this often? Are you reading from it every second? and need results that are that fresh If not, i imagine increasing the auto-commit time to 1min or even 10 secs would help some. Re, calling commit from the client with auto-commit... if you are using auto-commit, you should not call commit from the client ryan Assuming I understand what you're saying correctly: There shouldn't be any index readers - as at this point, just writing to the index. Did I understand correctly what you meant? -Nick -Original Message- From: Lance Norskog [mailto:goks...@gmail.com] Sent: 13 January 2010 22:28 To: solr-user@lucene.apache.org Subject: Re: Interesting OutOfMemoryError on a 170M index The time in autocommit is in milliseconds. You are committing every second while indexing. This then causes a build-up of sucessive index readers that absorb each commit, which is probably the out-of- memory. On Wed, Jan 13, 2010 at 10:36 AM, Minutello, Nick nick.minute...@credit-suisse.com wrote: Hi, I have a bit of an interesting OutOfMemoryError that I'm trying to figure out. My client Solr server are running in the same JVM (for deployment simplicity). FWIW, I'm using Jetty to host Solr. I'm using the supplied code for the http-based client interface. Solr 1.3.0. My app is adding about 20,000 documents per minute to the index - one at a time (it is listening to an event stream and for every event, it adds a new document to the index). The size of the documents, however, is tiny - the total index growth is only about 170M (after about 1 hr and the OutOfMemoryError) At this point, there is zero querying happening - just updates to the index (only adding documents, no updates or deletes) After about an hour or so, my JVM runs out of heap space - and if I look at the memory utilisation over time, it looks like a classic memory leak. It slowly ramps up until we end up with constant FULL GC's and eventual OOME. Max heap space is 512M. In Solr, I'm using autocommit (to buffer the updates) autoCommit maxDocs1/maxDocs maxTime1000/maxTime /autoCommit (Aside: Now, I'm not sure if I am meant to call commit or not on the client SolrServer class if I am using autocommit - but as it turns out, I get OOME whether I do that or not) Any suggestions/advice of quick things to check before I dust off the profiler? Thanks in advance. Cheers, Nick = = = Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html = = = -- Lance Norskog goks...@gmail.com = = = = = = = = = == Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html = = = = = = = = = ==
RE: Interesting OutOfMemoryError on a 170M index
if you are using auto-commit, you should not call commit from the client Cheers, thanks. Do you need the index to be updated this often? Wouldn't increasing the autocommit time make it worse? (ie more documents buffered) I can extend it and see what effect it has -Nick -Original Message- From: Ryan McKinley [mailto:ryan...@gmail.com] Sent: 13 January 2010 23:16 To: solr-user@lucene.apache.org Subject: Re: Interesting OutOfMemoryError on a 170M index On Jan 13, 2010, at 5:34 PM, Minutello, Nick wrote: Agreed, commit every second. Do you need the index to be updated this often? Are you reading from it every second? and need results that are that fresh If not, i imagine increasing the auto-commit time to 1min or even 10 secs would help some. Re, calling commit from the client with auto-commit... if you are using auto-commit, you should not call commit from the client ryan Assuming I understand what you're saying correctly: There shouldn't be any index readers - as at this point, just writing to the index. Did I understand correctly what you meant? -Nick -Original Message- From: Lance Norskog [mailto:goks...@gmail.com] Sent: 13 January 2010 22:28 To: solr-user@lucene.apache.org Subject: Re: Interesting OutOfMemoryError on a 170M index The time in autocommit is in milliseconds. You are committing every second while indexing. This then causes a build-up of sucessive index readers that absorb each commit, which is probably the out-of- memory. On Wed, Jan 13, 2010 at 10:36 AM, Minutello, Nick nick.minute...@credit-suisse.com wrote: Hi, I have a bit of an interesting OutOfMemoryError that I'm trying to figure out. My client Solr server are running in the same JVM (for deployment simplicity). FWIW, I'm using Jetty to host Solr. I'm using the supplied code for the http-based client interface. Solr 1.3.0. My app is adding about 20,000 documents per minute to the index - one at a time (it is listening to an event stream and for every event, it adds a new document to the index). The size of the documents, however, is tiny - the total index growth is only about 170M (after about 1 hr and the OutOfMemoryError) At this point, there is zero querying happening - just updates to the index (only adding documents, no updates or deletes) After about an hour or so, my JVM runs out of heap space - and if I look at the memory utilisation over time, it looks like a classic memory leak. It slowly ramps up until we end up with constant FULL GC's and eventual OOME. Max heap space is 512M. In Solr, I'm using autocommit (to buffer the updates) autoCommit maxDocs1/maxDocs maxTime1000/maxTime /autoCommit (Aside: Now, I'm not sure if I am meant to call commit or not on the client SolrServer class if I am using autocommit - but as it turns out, I get OOME whether I do that or not) Any suggestions/advice of quick things to check before I dust off the profiler? Thanks in advance. Cheers, Nick = = = Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html = = = -- Lance Norskog goks...@gmail.com = = = = = = = = = == Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html = = = = = = = = = == === Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ===
RE: Interesting OutOfMemoryError on a 170M index
Hm, Ryan, you may have inadvertently solved the problem. :) Going flat out in a loop, indexing 1 doc at a time, I can only index about 17,000 per minute - roughly what I was seeing with my app running... which makes me suspicious. The number is too close to be coincidental. It could very well be that I may be getting many more than 17,000 updates per minute - and because I cant index them fast enough, the event queue in the underlying library (that is providing me the events) may be growing without bound... So, looks like I have to increase the throughput with the indexing. (indexing 1 at a time is far from ideal - even with the buffering). I may have to either implement some client-side buffering to make it more efficient - or eliminate the http layer (go embedded). Thanks. -Nick -Original Message- From: Minutello, Nick Sent: 13 January 2010 23:29 To: solr-user@lucene.apache.org Subject: RE: Interesting OutOfMemoryError on a 170M index if you are using auto-commit, you should not call commit from the client Cheers, thanks. Do you need the index to be updated this often? Wouldn't increasing the autocommit time make it worse? (ie more documents buffered) I can extend it and see what effect it has -Nick -Original Message- From: Ryan McKinley [mailto:ryan...@gmail.com] Sent: 13 January 2010 23:16 To: solr-user@lucene.apache.org Subject: Re: Interesting OutOfMemoryError on a 170M index On Jan 13, 2010, at 5:34 PM, Minutello, Nick wrote: Agreed, commit every second. Do you need the index to be updated this often? Are you reading from it every second? and need results that are that fresh If not, i imagine increasing the auto-commit time to 1min or even 10 secs would help some. Re, calling commit from the client with auto-commit... if you are using auto-commit, you should not call commit from the client ryan Assuming I understand what you're saying correctly: There shouldn't be any index readers - as at this point, just writing to the index. Did I understand correctly what you meant? -Nick -Original Message- From: Lance Norskog [mailto:goks...@gmail.com] Sent: 13 January 2010 22:28 To: solr-user@lucene.apache.org Subject: Re: Interesting OutOfMemoryError on a 170M index The time in autocommit is in milliseconds. You are committing every second while indexing. This then causes a build-up of sucessive index readers that absorb each commit, which is probably the out-of- memory. On Wed, Jan 13, 2010 at 10:36 AM, Minutello, Nick nick.minute...@credit-suisse.com wrote: Hi, I have a bit of an interesting OutOfMemoryError that I'm trying to figure out. My client Solr server are running in the same JVM (for deployment simplicity). FWIW, I'm using Jetty to host Solr. I'm using the supplied code for the http-based client interface. Solr 1.3.0. My app is adding about 20,000 documents per minute to the index - one at a time (it is listening to an event stream and for every event, it adds a new document to the index). The size of the documents, however, is tiny - the total index growth is only about 170M (after about 1 hr and the OutOfMemoryError) At this point, there is zero querying happening - just updates to the index (only adding documents, no updates or deletes) After about an hour or so, my JVM runs out of heap space - and if I look at the memory utilisation over time, it looks like a classic memory leak. It slowly ramps up until we end up with constant FULL GC's and eventual OOME. Max heap space is 512M. In Solr, I'm using autocommit (to buffer the updates) autoCommit maxDocs1/maxDocs maxTime1000/maxTime /autoCommit (Aside: Now, I'm not sure if I am meant to call commit or not on the client SolrServer class if I am using autocommit - but as it turns out, I get OOME whether I do that or not) Any suggestions/advice of quick things to check before I dust off the profiler? Thanks in advance. Cheers, Nick = = = Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html = = = -- Lance Norskog goks...@gmail.com = = = = = = = = = == Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html = = = = = = = = = == === Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en
