Hi Ian
Thanks for the info. Its difficult to reuse searchers as my users are
performing realtime searches, so I need to open an IndexReader for every
live search query.
I've since tracked the OutOfMemory issue down to sort on date. I am
using too high a precision (down to the second) which is causing the
number of search terms to escalate. Also, I haven't since moved the date
field in the index over to using Numerics instead of a String value for
fear of breaking compatibility with my old index format.
I dont yet quite understand the implications of the precisionStep and
what it all means. If I change my date string to a Numeric integer in
the format yyyyMMddHHmm, what should the precisionStep value be?
Jamie
On 2010/03/17 01:20 PM, Ian Lea wrote:
Hi
Caching searchers at some level should help keep memory usage down -
and will help performance too. Searchers themselves don't generally
consume large amounts of memory, but if you've got loads of them then
obviously things will add up.
Unless you can change the whole design of your app (single index with
a user field that you use as a query filter to restrict users to
"their" data?) you may be stuck with giving the app more memory or
restricting the number of concurrent searchers.
---
Ian.
On Tue, Mar 16, 2010 at 10:00 PM, Jamie<ja...@stimulussoft.com> wrote:
Hi There
I have an index which is 36 GB large. When I perform eight simultaneous
searches (performed by JMeter) on the index, an OutOfMemory error occurs.
Since I need to potentially search across multiple indexes and those indexes
can change from one search query to the next, each user has their own
ParallelMultiSearcher object. Before each search operation, I reconstruct
the ParrallelMultisearcher with the appropriate Searchers to each of the
indexes that need to be included for that particular search query.
The problem is that requiring each user to have their own
ParallelMultisearcher seems to limit the number of users that can use the
system at the same time.
While experimenting, when I make the ParallelMultiSearcher static, the same
object used by all users, the OutOfMemory problem goes away and I am able to
execute 50 simultaneous searches. The problem I have is I cannot make
ParallelMultisearcher static, since the specific indexes used are variable
from one search query to the next. I initially thought one could just cache
the underlying Searchers and all would be okay, but this does not appear to
be the case.
My question: Will ParallelMultisearcher tend to consume a large amount of
memory by itself when used on large indices? If so, do you have any
suggestions on how I might support the above scenario (i.e. when the indexes
used change from one query to the next)
Thanks in advance
Jamie
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
