Re: update performance

[Chris Kimm]

Unfortunately, I'm not able to batch the updates.  The application needs 
to make some descisions based on what each document looks like before 
and  after the update, so I have to do it one at a time.  I guess this 
is not a common useage scenario for Lucene.  Otherwise, an update() 
might already be built in somewhere.

Is there anything in the locking/sync framework which precludes saving 
the cost of closing the Directory object and deleting the temp lock file 
each time an update is made?

-Chris

Doug Cutting wrote:

It sounds like you're not batching your updates.

The most efficient approch to update 1000 documents would be to:

  1. Open an IndexReader;
  2. Delete all 1000 documents.
  3. Close the reader;
  4. Open an IndexWriter;
  5. Add all 1000 updated documents;
  6. Close the IndexWriter.
Is that what you're doing?

Doug

Chris Kimm wrote:

The standard pattern for updating an index - removing a document then 
re-adding the modified document to the index - is currently a 
significant performance bottleneck in my application.  I sometimes 
need to update ~1000 documents at a time.  The major cost of this 
pattern as far as I can see is IndexWriter.close ().   Average times 
for an update to an FSDirectory look like this:

delete document: 7 ms
create document: 6 ms
add document: 11 ms
IndexWriter.close: 59 ms
Is there a way to synchronize IndexWriter and IndexReader so that a 
call to IndexWriter.close is not required for each update?  I guess I 
mean to ask if there is a *simple* way to do this.  I imagine that one 
could write an IndexUpdater class which manages the synchronization of 
Locks, temp files, etc.

Thanks,

Chris



---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]