On Jan 13, 2010, at 5:34 PM, Minutello, Nick wrote:
Agreed, commit every second.
Do you need the index to be updated this often? Are you reading from
it every second? and need results that are that "fresh"
If not, i imagine increasing the auto-commit time to 1min or even 10
secs would help some.
Re, calling commit from the client with auto-commit... if you are
using auto-commit, you should not call commit from the client
ryan
Assuming I understand what you're saying correctly:
There shouldn't be any index readers - as at this point, just
writing to the index.
Did I understand correctly what you meant?
-Nick
-----Original Message-----
From: Lance Norskog [mailto:goks...@gmail.com]
Sent: 13 January 2010 22:28
To: solr-user@lucene.apache.org
Subject: Re: Interesting OutOfMemoryError on a 170M index
The time in autocommit is in milliseconds. You are committing every
second while indexing. This then causes a build-up of sucessive
index readers that absorb each commit, which is probably the out-of-
memory.
On Wed, Jan 13, 2010 at 10:36 AM, Minutello, Nick <nick.minute...@credit-suisse.com
> wrote:
Hi,
I have a bit of an interesting OutOfMemoryError that I'm trying to
figure out.
My client & Solr server are running in the same JVM (for deployment
simplicity). FWIW, I'm using Jetty to host Solr. I'm using the
supplied code for the http-based client interface. Solr 1.3.0.
My app is adding about 20,000 documents per minute to the index - one
at a time (it is listening to an event stream and for every event, it
adds a new document to the index).
The size of the documents, however, is tiny - the total index growth
is only about 170M (after about 1 hr and the OutOfMemoryError) At
this
point, there is zero querying happening - just updates to the index
(only adding documents, no updates or deletes) After about an hour or
so, my JVM runs out of heap space - and if I look at the memory
utilisation over time, it looks like a classic memory leak. It slowly
ramps up until we end up with constant FULL GC's and eventual OOME.
Max heap space is 512M.
In Solr, I'm using autocommit (to buffer the updates)
<autoCommit>
<maxDocs>10000</maxDocs>
<maxTime>1000</maxTime>
</autoCommit>
(Aside: Now, I'm not sure if I am meant to call commit or not on the
client SolrServer class if I am using autocommit - but as it turns
out, I get OOME whether I do that or not)
Any suggestions/advice of quick things to check before I dust off the
profiler?
Thanks in advance.
Cheers,
Nick
=
=====================================================================
=========
Please access the attached hyperlink for an important electronic
communications disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
=
=====================================================================
=========
--
Lance Norskog
goks...@gmail.com
=
=
=
=
=
=
=
=
=
======================================================================
Please access the attached hyperlink for an important electronic
communications disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
=
=
=
=
=
=
=
=
=
======================================================================
