Well... since VS2010 will be out by then (April 2010) you might do the big leap :)
On Wed, Jan 6, 2010 at 10:24 PM, Michael Garski <[email protected]>wrote: > That will be with version 3.0. > > Michael > > -----Original Message----- > From: Simone Chiaretta [mailto:[email protected]] > Sent: Wednesday, January 06, 2010 1:23 PM > To: [email protected] > Subject: Re: Possible memory leak in Lucene.NET 2.4? > > One last thing: > any idea when you'll update the solution and project files to VS2008? > Simone > > On Wed, Jan 6, 2010 at 8:57 PM, Michael Garski > <[email protected]>wrote: > > > Simone, > > > > There is a trade-off in the use of filters - more memory consumed but > > faster performance. It's a good idea to test both approaches (filter > > vs. Boolean clause) to find what works best for you. > > > > Each filter will consume 1 bit in memory for each document plus the > > overhead of the object itself. With 50K documents, each filter would > > consume approximately 6.5KB of memory, with 1500 consuming > approximately > > 10MB. That's not a whole lot of memory, and will give you a bump in > > search performance. How much of a bump, you'll have to test to > > determine. I don't have much experience with indexes of you size, > > however for the large indexes we work with the gain can be > significant. > > > > Michael > > > > -----Original Message----- > > From: Simone Chiaretta [mailto:[email protected]] > > Sent: Wednesday, January 06, 2010 11:46 AM > > To: [email protected] > > Subject: Re: Possible memory leak in Lucene.NET 2.4? > > > > Michel, > > great suggestions thank you > > > > More below > > > > On Wed, Jan 6, 2010 at 8:29 PM, Michael Garski > > <[email protected]>wrote: > > > > > Simone - > > > > > > Filters will provide for more efficient queries in your case if you > > > filter on the blog id rather than using it as a query clause as the > > > filter can be cached and re-used for future queries. Be sure to use > > the > > > FilterManager to ensure your filters are being cached and not > > re-created > > > for each query. > > > > > > > It makes sense when I've just one blog... but can I have, let's say, > > 1500 > > different filters one per blog? > > What I need is filtering blog posts based on the blog I'm currently > in: > > so I've to search over blog 1 if I'm in blog 1, and in blog 1500 if > I'm > > in > > blog 1500. > > This will mean I'll have to cache 1500 different filters? Or in this > > case a > > simple plain query will be better? > > > > > > > > > Optimizing on startup could delay the app pool being available. I'd > > > suggest that rather than optimizing on a periodic basis that you > > create > > > a custom MergePolicy to control the number of segments in the index > > and > > > when segments are merged. With 2.9 I take this approach and don't > use > > > the Optimize call at all anymore. Provided you don't have thousands > > of > > > segments, a multi-segment index should not pose a performance issue. > > > There are quite a number of other performance improvements that can > be > > > made that have a bigger impact, such as filters. > > > > > > > I'll take a look at that. > > > > > > > > > > > > The 2.9 version I am using is the current trunk version. It's very > > > stable and I have not encountered any issues. > > > > > > > Ok.. great...thx > > > > > > > > > > Retrieving the IndexReader from the IndexWriter will give you the > most > > > recent changes, including those that have not yet been committed. > > > > > > http://lucene.apache.org/java/2_9_1/api/all/org/apache/lucene/index/Inde > > > xWriter.html#getReader%28%29 > > > > > > > > OK... 2.9 seems to be solving most of my problems :) > > > > > > > Unfortunately I don't have the bandwidth at this time to help you > out > > > with a code review, but should you have any questions, continue to > > post > > > them to the list and I'll provide some feedback as time allows. > > > > > > > Ok.. no problem... thank you anyway.. you are being very helpful > > Simo > > > > > > > > > > Michael > > > > > > -----Original Message----- > > > From: Simone Chiaretta [mailto:[email protected]] > > > Sent: Wednesday, January 06, 2010 11:11 AM > > > To: [email protected] > > > Subject: Re: Possible memory leak in Lucene.NET 2.4? > > > > > > Hi Michael, > > > > > > more below > > > > > > On Wed, Jan 6, 2010 at 7:41 PM, Michael Garski > > > <[email protected]>wrote: > > > > > > > Simone, > > > > > > > > Filters work to constrain the query to the subset of documents > that > > > are > > > > contained in the filter, which can improve performance. > > > > > > > > > Ok, from what I see, filtering can help me filter out posts from > other > > > blogs. > > > But can filters change with every query? > > > What's the difference between: > > > query for "xyz" on blog 1 over all index > > > vs. > > > query for "xyz" over the index filtered by blog 1? > > > > > > > > > > The field cache > > > > is used to cache values if you are sorting by something other than > > the > > > > score, such as by date or some other value in the index. > > > > > > > > > > > > > I'm just sorting by score... so probably not needed > > > > > > > > > > > > > > Optimizing after each document incurs an unnecessary overhead as > all > > > > segments are merged into one, which is not necessary even in > > versions > > > > prior to 2.9. > > > > > > > > > > Great, thank you... I can remove this, would help speed up the add > > > document > > > procedure on large indexes... > > > and since in web app the pool recycles anyway every day or so, doing > > an > > > optimize at the creation of the index write will be enough, correct? > > > > > > > > > > If your app has not yet been released, I would suggest using 2.9 > and > > > > ensuring you are not using any methods or properties marked with > the > > > > Obsolete attribute to streamline migration to future versions. > > > > > > > > > Great... thank you again... is 2.9 the trunk, right? I don't see a > tag > > > for > > > it in SVN > > > > > > > > > > Another > > > > change in 2.9 you could take advantage of is retrieving an > > IndexReader > > > > from the IndexWriter through the GetReader method, which will save > > you > > > > from having to have both a writer and a reader in application > scope. > > > > The writer could be held at the application level and the reader > > > > retrieved from it directly. > > > > > > > > > > And that will give the most current reader updated with the latest > new > > > docs? > > > > > > One last thing: > > > Would you be so kind (if you have time, and with the proper credit > > given > > > in > > > the source code and in the release notes) to do a kind of source > code > > > review > > > to the search engine of the blog? > > > Thx > > > > > > Simone > > > > > > > > > > > > > > Michael > > > > > > > > > > > > -----Original Message----- > > > > From: Simone Chiaretta [mailto:[email protected]] > > > > Sent: Wednesday, January 06, 2010 10:28 AM > > > > To: [email protected] > > > > Subject: Re: Possible memory leak in Lucene.NET 2.4? > > > > > > > > I'm just using queries... I'm pretty new to Lucene, so I went for > > the > > > > easier > > > > solution. > > > > Would you recommend using filters and caching instead of queries? > > > > > > > > At the moment I'm on Lucene 2.3.1... would you recommend moving to > > > 2.9? > > > > My app has not been released yet (an open source blogging engine), > > but > > > > will > > > > be shortly. > > > > The number of documents indexed will range from 0 to 50.000 blog > > posts > > > > (our > > > > biggest installation atm). > > > > > > > > Will not optimizing after every new document reduce the > performances > > > of > > > > the > > > > searches on such indexes? > > > > > > > > Simone > > > > > > > > On Wed, Jan 6, 2010 at 7:08 PM, Michael Garski > > > > <[email protected]>wrote: > > > > > > > > > Simone, > > > > > > > > > > Are you using any field caches or filters? > > > > > > > > > > In versions prior to 2.9, reopening the index will completely > > > rebuild > > > > > the field cache and filter bits for all documents in the index, > > > which > > > > > can result in an increase in memory consumption. In 2.9 and > > future > > > > > versions, the field cache and filter bits are cached at a > segment > > > > level, > > > > > which results in significantly faster re-opens as only the new > > > > segments > > > > > are loaded into the caches. > > > > > > > > > > Our applications use very large indexes and 2.9's segment level > > > > caching > > > > > allows us to re-open indexes much faster while utilizing less > > memory > > > > in > > > > > the process. > > > > > > > > > > Michael > > > > > > > > > > -----Original Message----- > > > > > From: Simone Chiaretta [mailto:[email protected]] > > > > > Sent: Wednesday, January 06, 2010 10:01 AM > > > > > To: [email protected] > > > > > Subject: Re: Possible memory leak in Lucene.NET 2.4? > > > > > > > > > > What I am doing is initializing the writer in the App_Start > event > > of > > > > the > > > > > web > > > > > app, and closing everything at the App_End event. > > > > > For the reader, I start it at the first search request, re-open > it > > > > > everytime > > > > > a new document is added, and then closing it in the App_End > > > > > > > > > > If you are interested here is the search engine service I'm > using: > > > > > > > > > > > > > > > http://code.google.com/p/subtext/source/browse/trunk/src/Subtext.Framewo > > > > > > > > > > > > > > > rk/Services/SearchEngine/SearchEngineService.cs<http://code.google.com/p > > > > > > > > > > /subtext/source/browse/trunk/src/Subtext.Framewo%0Ark/Services/SearchEng > > > > > > <http://code.google.com/p%0A/subtext/source/browse/trunk/src/Subtext.Fra > > > mewo%0Ark/Services/SearchEng> > > > > ine/SearchEngineService.cs> > > > > > > > > > > Simone > > > > > > > > > > On Wed, Jan 6, 2010 at 6:31 PM, Matt Honeycutt > > > > > <[email protected]>wrote: > > > > > > > > > > > Won't the various global application events be fired if the > app > > > pool > > > > > is > > > > > > gracefully terminated/recycled? While not ideal, couldn't you > > > > > initialize > > > > > > your Lucene objects during one of the application > > initialization, > > > > then > > > > > > dispose of them in the corresponding shutodwn events? > > > > > > > > > > > > On Wed, Jan 6, 2010 at 11:14 AM, Michael Garski > > > > > <[email protected] > > > > > > >wrote: > > > > > > > > > > > > > If it's not an option to create search functionality in a > > > separate > > > > > > process, > > > > > > > such as in a shared hosting environment, you may be limited > in > > > the > > > > > size > > > > > > of > > > > > > > your index and how you query it. The field cache, and to a > > > lesser > > > > > extent > > > > > > > filters, will consume a fair amount of memory that is > > > proportional > > > > > to the > > > > > > > number of documents in the index. > > > > > > > > > > > > > > As others have mentioned, you will have to ensure that > > resources > > > > are > > > > > > > released when the app pool recycles. > > > > > > > > > > > > > > Michael > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Simone Chiaretta [mailto:[email protected]] > > > > > > > Sent: Wednesday, January 06, 2010 12:45 AM > > > > > > > To: [email protected] > > > > > > > Subject: Re: Possible memory leak in Lucene.NET 2.4? > > > > > > > > > > > > > > Unfortunately not everybody can use another process: I'm > > > building > > > > a > > > > > > > blog engine that must be able to run on shared hosting > > provider. > > > > The > > > > > > > 2nd process is not an option :) > > > > > > > > > > > > > > Simone > > > > > > > > > > > > > > On Tuesday, January 5, 2010, Digy <[email protected]> > wrote: > > > > > > > > As Michael stated, I prefer also not hosting "indexing and > > > > > searching > > > > > > > > sevices" in IIS. > > > > > > > > There are many alternatives such as WCF, Remoting etc. > With > > a > > > > > separate > > > > > > > > service for Lucene, you can control anything you want. > > > > > > > > > > > > > > > > DIGY > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: Michael Garski [mailto:[email protected]] > > > > > > > > Sent: Tuesday, January 05, 2010 11:11 PM > > > > > > > > To: [email protected] > > > > > > > > Subject: RE: Possible memory leak in Lucene.NET 2.4? > > > > > > > > > > > > > > > > Jeff, > > > > > > > > > > > > > > > > Correct - there is no need to optimize the index after > > adding > > > a > > > > > > > > document, and I would recommend against it especially when > > you > > > > > move to > > > > > > > > 2.9 as you will not see any of the benefits of the changes > > to > > > > > composite > > > > > > > > readers such as faster incremental warm-ups to filters and > > > field > > > > > > caches. > > > > > > > > > > > > > > > > I've never run Lucene.Net in the context of a web process > > and > > > > > would > > > > > > > > actually recommend against that approach due to app pool > > > > > recycling, > > > > > > > > opting for a service that exposed search functionality via > > > WCF. > > > > > > > > > > > > > > > > What types of queries are you executing? Are you using > > filters > > > > or > > > > > > > > sorting? How often do you re-open the IndexReader that is > > > used > > > > > for > > > > > > > > searching? Re-opening the reader after each document > > addition > > > > can > > > > > be > > > > > > an > > > > > > > > expensive process, especially if you are using filters > > and/or > > > > > sorts. > > > > > > > > How are you refreshing the IndexReader? > > > > > > > > > > > > > > > > Regarding the IndexReader locking files, this is a feature > > > which > > > > > allows > > > > > > > > you to concurrently index and search on the same index and > > not > > > > > have to > > > > > > > > worry about the IndexWriter deleting a segment file from > > > > > underneath the > > > > > > > > searcher when a segment merge occurs. > > > > > > > > > > > > > > > > The first place to look would be to use a memory profiler > to > > > > > determine > > > > > > > > what is actually consuming the memory. I use the SciTech > > .NET > > > > > Memory > > > > > > > > Profiler for such purposes. > > > > > > > > > > > > > > > > Michael > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: Jeff Pennal [mailto:[email protected]] > > > > > > > > Sent: Tuesday, January 05, 2010 12:42 PM > > > > > > > > To: [email protected] > > > > > > > > Subject: Possible memory leak in Lucene.NET 2.4? > > > > > > > > > > > > > > > > Hello all, > > > > > > > > > > > > > > > > In doing some profiling of our Lucene code, I noticed that > > we > > > > were > > > > > > doing > > > > > > > > > > > > > > > > an optimize code after every update to our index. Though > our > > > > index > > > > > is > > > > > > > > relatively small (~75MB), the optimize task still look way > > to > > > > much > > > > > time > > > > > > > > to run. > > > > > > > > > > > > > > > > I did some research and it seems like it would not be an > > issue > > > > to > > > > > > update > > > > > > > > > > > > > > > > our index without optimizing afterwords, the side effect > > being > > > > > that > > > > > > we'd > > > > > > > > > > > > > > > > have more open file handles. > > > > > > > > > > > > > > > > I made that change and noticed some horrible performance > > side > > > > > effects. > > > > > > > > > > > > > > > > The first thing I noticed was that the CPU for our web > > > > application > > > > > > > > (ASP.NET MVC) that read from the Index never went below > > 60-70% > > > > and > > > > > was > > > > > > > > frequently pegged at 99%. > > > > > > > > > > > > > > > > In addition to the CPU spiking, the memory taken up by the > > > > > w3wp.exe > > > > > > > > process quickly grew to around 800MB, which is about 300MB > > > above > > > > > > normal. > > > > > > > > > > > > > > > > This has all the hallmarks of a memory leak somewhere. > > > > > > > > > > > > > > > > Finally, I noticed that the IndexReader was locking some > of > > > the > > > > > files > > > > > > in > > > > > > > > > > > > > > > > the index folder even though the reader was set to nolock > > > mode. > > > > > This > > > > > > > > seemed to be cause of the increase in the number of files > in > > > the > > > > > index > > > > > > > > folder. > > > > > > > > > > > > > > > > We have the IndexReader set to open once and then be > shared > > > > among > > > > > every > > > > > > > > request to the web application. My understanding is that > > this > > > is > > > > > the > > > > > > > > correct way to do this, and this never caused and issues > > when > > > we > > > > > were > > > > > > > > optimizing the index after every update. > > > > > > > > > > > > > > > > I know this is a pretty vague problem and there could be > any > > > > > number of > > > > > > > > issues involved here. However, if anyone could suggest > areas > > > to > > > > > look > > > > > > > > into for possible solutions, it would be greatly > > appreciated. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Jeff > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Simone Chiaretta > > > > > > > Microsoft MVP ASP.NET - ASPInsider > > > > > > > Blog: http://codeclimber.net.nz > > > > > > > RSS: http://feeds2.feedburner.com/codeclimber > > > > > > > twitter: @simonech > > > > > > > > > > > > > > Any sufficiently advanced technology is indistinguishable > from > > > > magic > > > > > > > "Life is short, play hard" > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Simone Chiaretta > > > > > Microsoft MVP ASP.NET - ASPInsider > > > > > Blog: http://codeclimber.net.nz > > > > > RSS: http://feeds2.feedburner.com/codeclimber > > > > > twitter: @simonech > > > > > > > > > > Any sufficiently advanced technology is indistinguishable from > > magic > > > > > "Life is short, play hard" > > > > > > > > > > > > > > > > > > > > > > -- > > > > Simone Chiaretta > > > > Microsoft MVP ASP.NET - ASPInsider > > > > Blog: http://codeclimber.net.nz > > > > RSS: http://feeds2.feedburner.com/codeclimber > > > > twitter: @simonech > > > > > > > > Any sufficiently advanced technology is indistinguishable from > magic > > > > "Life is short, play hard" > > > > > > > > > > > > > > > > > -- > > > Simone Chiaretta > > > Microsoft MVP ASP.NET - ASPInsider > > > Blog: http://codeclimber.net.nz > > > RSS: http://feeds2.feedburner.com/codeclimber > > > twitter: @simonech > > > > > > Any sufficiently advanced technology is indistinguishable from magic > > > "Life is short, play hard" > > > > > > > > > > > > -- > > Simone Chiaretta > > Microsoft MVP ASP.NET - ASPInsider > > Blog: http://codeclimber.net.nz > > RSS: http://feeds2.feedburner.com/codeclimber > > twitter: @simonech > > > > Any sufficiently advanced technology is indistinguishable from magic > > "Life is short, play hard" > > > > > > > -- > Simone Chiaretta > Microsoft MVP ASP.NET - ASPInsider > Blog: http://codeclimber.net.nz > RSS: http://feeds2.feedburner.com/codeclimber > twitter: @simonech > > Any sufficiently advanced technology is indistinguishable from magic > "Life is short, play hard" > > -- Simone Chiaretta Microsoft MVP ASP.NET - ASPInsider Blog: http://codeclimber.net.nz RSS: http://feeds2.feedburner.com/codeclimber twitter: @simonech Any sufficiently advanced technology is indistinguishable from magic "Life is short, play hard"
