Re: IndexWriter.updateDocument performance improvement

2009-11-20 Thread Michael McCandless
Opened LUCENE-2086. Mike On Fri, Nov 20, 2009 at 9:43 AM, Michael McCandless wrote: > +1 > > I'll open an issue. > > Mike > > On Fri, Nov 20, 2009 at 8:11 AM, Yonik Seeley > wrote: >> Thanks Bogdan, I've been meaning to bring this up. >> Solr used a TreeMap in the past (when it handled it's own

Re: IndexWriter.updateDocument performance improvement

2009-11-20 Thread Michael McCandless
+1 I'll open an issue. Mike On Fri, Nov 20, 2009 at 8:11 AM, Yonik Seeley wrote: > Thanks Bogdan, I've been meaning to bring this up. > Solr used a TreeMap in the past (when it handled it's own deletes) for > the same exact reason.  In my profiling, I've also seen applyDeletes() > taking the bu

Re: IndexWriter.updateDocument performance improvement

2009-11-20 Thread Yonik Seeley
Thanks Bogdan, I've been meaning to bring this up. Solr used a TreeMap in the past (when it handled it's own deletes) for the same exact reason. In my profiling, I've also seen applyDeletes() taking the bulk of the time with small/simple document indexing. So we should definitely go in sorted ord

IndexWriter.updateDocument performance improvement

2009-11-20 Thread Bogdan Ghidireac
Hi, One of the use case of my application involves updating the index with 10 to 10k docs every few minutes. Because we maintain a PK for each doc we have to use IndexWriter.updateDocument to be consistent. The average time for an update when we commit every 10k docs is around 17ms (the IndexWrit