Re: performance feedback

2008-07-10 Thread Yonik Seeley
On Thu, Jul 10, 2008 at 11:13 AM, Beard, Brian [EMAIL PROTECTED] wrote: Question: If autoCommit is false, does this apply to optimization also, so that during an hour long optimization that gets killed in the middle, will the index be in the left in the initial state before optimization

performance feedback

2008-07-09 Thread Beard, Brian
I just did an update from lucene 2.2.0 to 2.3.2 and thought I'd give some kudos for the indexing performance enhancements. The lucene indexing portion is about 6-8 times faster. Previously we were doing ~60-120 documents per second, now we're between 400-1000, depending on the type of document,

performance feedback

2008-07-09 Thread Beard, Brian
I just did an update from lucene 2.2.0 to 2.3.2 and thought I'd give some kudos for the indexing performance enhancements. The lucene indexing portion is about 6-8 times faster. Previously we were doing ~60-120 documents per second, now we're between 400-1000, depending on the type of document,

Re: performance feedback

2008-07-09 Thread Michael McCandless
This is great to hear! If you tweak things a bit (increase RAM buffer size, use autoCommit=false, use threads, etc) you should be able to eke out some more gains... Are you storing fields using term vectors on any of your fields? Mike Beard, Brian wrote: I just did an update from

RE: performance feedback

2008-07-09 Thread Beard, Brian
save a lot on index size, which would speed up copy time during swapping between search and update indexes). -Original Message- From: Michael McCandless [mailto:[EMAIL PROTECTED] Sent: Wednesday, July 09, 2008 10:38 AM To: java-user@lucene.apache.org Subject: Re: performance feedback

Re: performance feedback

2008-07-09 Thread Yonik Seeley
On Wed, Jul 9, 2008 at 11:35 AM, Beard, Brian [EMAIL PROTECTED] wrote: I will try tweaking RAM, and check about autoCommit=false. It's on the future agenda to multi-thread through the index writer. The indexing time I quoted includes the document creation time which would definitely improve

Re: performance feedback

2008-07-09 Thread Yonik Seeley
On Wed, Jul 9, 2008 at 11:35 AM, Beard, Brian [EMAIL PROTECTED] wrote: I will try tweaking RAM, and check about autoCommit=false. It's on the future agenda to multi-thread through the index writer. The indexing time I quoted includes the document creation time which would definitely improve