Thanks for your reply Artem. The operation that we have seen suffer the
performance degradation is what we call a "Word Frequency" query. This involves
iterating through every term in the index
reader = IndexReader.Open(Directory, true);
while (terms.Next())
{
TermDocs docs = reader.TermDocs(term);
while (docs.Next())
{
Document doc = reader.Document(docs.Doc(), fieldSelector);
// do stuff with doc
}
}
Our normal text search queries for individual terms don't seem to have suffered
from the database size increase. I am open to the possibility of there being a
better way to perform such a query, but as it stands, this query is up to 50
times slower than it was with our previous version of the software.
Also it is important to note that I am performing an Optimize on the index
before I try to produce a word frequency query.
Thanks
-----Original Message-----
From: Artem Chereisky [mailto:[email protected]]
Sent: Tuesday, 29 May 2012 9:30 PM
To: [email protected]
Subject: Re: Lucene upgrade causes significant slow down
Scott,
I don't believe the index size of 138MB is your issue. We have multiple indexes
close to 1GB and it's super fast. In fact we didn't notice any performance
degradation as the index grew from 100MB to almost a gig. Our search engine
performs thousands of searches per second on a quad core server with 32GB of
ram...and the server is not even busy.
Do you retrieve field values stored in the index using doc Id? That, I found,
is the most common performance issue.
Thanks,
Art
On Tue, May 29, 2012 at 8:01 PM, Scott Baldwin <[email protected]> wrote:
> Hi all, I'm a bit of a newb when it comes to Lucene, but I am in need
> of some sage counsel.****
>
> ** **
>
> I have an application that uses the Lucene engine to index various items.
> We upgraded to Lucene 2.9.2 and we now find that on a re-index of a
> significantly large project, our index size has doubled (from roughly
> 73MB to 137MB). This causes significant performance issues especially
> as we have written a custom Directory that queries the index from a
> SQL Database instead of a disk file of RAM index.****
>
> ** **
>
> I was just wondering if anyone knows there would be such a huge
> increase in index size, and if there are any options/settings we might
> be able to change to reduce the size of the index?****
>
> ** **
>
> Thanks heaps.****
>
> ** **
>
> Scott Baldwin Technical Architect****
>
> *QSR International Pty Ltd*
> 2nd Floor, 651 Doncaster Road | Doncaster Victoria 3108 Australia
> T +61 3 9840 4934 F +61 3 9840 1500
> [email protected] | www.qsrinternational.com ****
>
> *Please consider the environment before printing this email.*****
>
> [image: Description: cid:[email protected]]****
> ------------------------------
>
> *Disclaimer*
> This transmission may contain information which is confidential and
> privileged and intended only for the addressee. If you are not the
> addressee you may not use, disseminate or copy this information. If
> you have received this information in error please notify the sender
> immediately. Thank you. ****
>
> ** **
>
> ** **
>
