HI Andrew, we're using Lucene.NET version 2.9.2.
We found Lucene excellent at indexing and searching, but found that when
using it like a database that retrieval of data that was stored while
indexing was very slow. So to optimise we stored the minimal amount of data
possible, by storing identifiers for data we wanted to retrieve, and then
used those id's to return back the data by calling a database lookup to get
all the data in a single trip to the DB server.
That worked out very well for us, with both searching and data retrieval/
gathering taking about the same amount of time. It's a SQL2005 database that
we get the data from.
I don't know if that would work in your situation you may be dependant on
Lucene for both storage and searching.
Kind Regards
Noel
-----Original Message-----
From: Andrew W.
Sent: Monday, July 04, 2011 5:07 PM
To: lucene-net-user@lucene.apache.org
Subject: [Lucene.Net] Lucene Document Retrieval
Hi,
I have a Lucene index with 70,000 documents. The size of the index is round
300MB - I have 32 fields, but I only retrieve values for 11 fields when I
display results on the page (I use FieldSelector). The query time is great -
most of the queries execute under 50ms, however when I loop through results
to retrieve documents (I only retrieve 10 documents at once, because this is
my pagination size), the loop takes sometimes even longer than 300ms
I applied all recommendations mentioned on that page
http://wiki.apache.org/lucene-java/ImproveSearchingSpeed to optimize query
time.
I was really surprised when I noticed that the bottleneck is in the loop
below, especially becuase the index is not that big (only 300MB) and the
hits array contains only 10 elements
Is there anything else I can do to optimize document retrieval from lucene
index?
Here is the sample code:
Dim FastFieldSelector As New FastFieldSelector(Fields)
For i As Integer = 0 To hits.Length - 1
Dim DocId As Integer = hits(i).DocId
* Dim doc As Document = reader.Document(DocId, FastFieldSelector) ' this
line is taking long time*
For Each f As Field In Fields
...
Dim values() As Fieldable = doc.GetFieldables(f.Name)
....
Next
Next
Thanks,
Andrew
