It's hard to debug just by your description. But I think in general, you should close the reader and searcher, after you update the index.
BTW: If you are using database, my software DBSight does this kind of pruning out old data and keep the index up-to-date. No memory leaks etc. It's not related to how you search, but how you release the resources. -- Chris Lu ------------------------- Instant Scalable Full-Text Search On Any Database/Application site: http://www.dbsight.net demo: http://search.dbsight.com Lucene Database Search in 3 minutes: http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes DBSight customer Twenga got 2.6 Million Euro funding! http://www.techcrunch.com/2007/12/07/twenga-gets-e26-million-for-product-search/ On Dec 10, 2007 2:43 PM, Bob Daha <[EMAIL PROTECTED]> wrote: > Thanks for the reply. The backup of the index on disc is around 80 MB. I'm > giving my JVM 4 gigs of heap space... I probably should have mentioned in my > first thread that this is not the first crash; it's done it a few times now > and usually has almost a week of uptime before it crashes even with the > growing index. (last time it crashed the index was about 60MB on disc) It's > also worth noting I am only going to keep around 4 weeks of ticket data and > I'm adding week 5 right now, so the index should be relatively close to this > size "forever", although rather rapidly changing. > > I think *something* is holding onto those terms... just not sure what. =\ > > > > ----- Original Message ---- > From: Grant Ingersoll <[EMAIL PROTECTED]> > To: java-user@lucene.apache.org > Sent: Monday, December 10, 2007 2:24:58 PM > Subject: Re: Out of memory? > > > Those terms could be coming from a lot of diff. places. > > What size JVM heap, etc. are you giving your application? My guess is > > your index is too big to fit into RAM, you probably need to use a > FSDirectory or give it more RAM, but it is hard to say for sure w/o > knowing more about the application. > > See http://wiki.apache.org/lucene-java/BasicsOfPerformance for more on > > performance. > > -Grant > > > On Dec 10, 2007, at 2:56 PM, Bob Daha wrote: > > > Hello, > > > > I'm building a ticketing system for my company and am using Lucene > > for some of the more complicated queries. I'd say my application > > differs from the typical lucene application in that my documents are > > > (re)-indexed more frequently, the query load is actually relatively > > light, and most of the indexed document data is numerical. (ticket > > id, owner id, date ranges, etc are numbers and there's just one > > giant "text" searchable field.) > > > > I just got an out of memory exception on my app. When I fired up > > jhat, I was very surprised at the object allocation: > > > > 7862 instances of class org.apache.lucene.index.Term > > 7787 instances of class org.apache.lucene.index.TermInfo > > 504 instances of class org.apache.lucene.index.FieldInfo > > 224 instances of class org.apache.lucene.index.TermBuffer > > 79 instances of class org.apache.lucene.index.CompoundFileReader > > $CSIndexInput > > 75 instances of class org.apache.lucene.index.SegmentTermEnum > > 70 instances of class org.apache.lucene.store.RAMFile > > 64 instances of class org.apache.lucene.store.RAMInputStream > > 63 instances of class org.apache.lucene.index.FieldInfos > > 20 instances of class > com.facebook.tps_search_server.tps_search_ticket > > 20 instances of class > > com.facebook.tps_search_server.tps_search_ticket$Isset > > 10 instances of class org.apache.lucene.index.IndexReader$FieldOption > > 9 instances of class > org.apache.lucene.index.IndexFileDeleter$RefCount > > 8 instances of class org.apache.lucene.index.CompoundFileReader > > $FileEntry > > 8 instances of class org.apache.lucene.index.CompoundFileWriter > > $FileEntry > > 8 instances of class org.apache.lucene.index.SegmentReader$Norm > > 7 instances of class org.apache.lucene.document.FieldSelectorResult > > 5 instances of class org.apache.lucene.document.Field$TermVector > > 5 instances of class org.apache.lucene.queryParser.Token > > 4 instances of class org.apache.lucene.document.Field$Index > > 4 instances of class org.apache.lucene.search.FieldCacheImpl$Entry > > 3 instances of class com.facebook.thrift.protocol.TBinaryProtocol > > 3 instances of class com.facebook.thrift.transport.TSocket > > 3 instances of class org.apache.lucene.document.Field$Store > > 3 instances of class org.apache.lucene.index.SegmentInfos > > 3 instances of class org.apache.lucene.search.BooleanClause$Occur > > 2 instances of class com.facebook.thrift.protocol.TBinaryProtocol > > $Factory > > 2 instances of class com.facebook.thrift.server.TThreadPoolServer > > $WorkerProcess > > 2 instances of class com.facebook.thrift.transport.TTransportFactory > > 2 instances of class > > org.apache.lucene.analysis.standard.StandardAnalyzer > > 2 instances of class org.apache.lucene.index.SegmentInfo > > 2 instances of class org.apache.lucene.queryParser.QueryParser > > $Operator > > 2 instances of class org.apache.lucene.search.DefaultSimilarity > > 2 instances of class org.apache.lucene.search.FieldSortedHitQueue$2 > > 2 instances of class org.apache.lucene.search.Sort > > 2 instances of class org.apache.lucene.search.SortField > > 2 instances of class org.apache.lucene.store.RAMDirectory > > 2 instances of class > org.apache.lucene.store.SingleInstanceLockFactory > > 2 instances of class [Lorg.apache.lucene.search.SortField; > > ... > > > > Any ideas on why I'd have so many Term and TermInfo objects? To > > give you a little more insight into my application, worker threads > > can be spawned by client requests that do one of 4 things - search, > > index tickets, deindex tickets, or reindex tickets. (reindex a > > ticket is basically deindex then index ticket) I'm using a > > RAMDirectory for all my searches and updates although I have another > > > thread which backs this up to disc periodically. > > > > Thanks, > > B > > > > > > > > > > > > > > ____________________________________________________________________________________ > > Looking for last minute shopping deals? > > Find them fast with Yahoo! Search. > http://tools.search.yahoo.com/newsearch/category.php?category=shopping > > -------------------------- > Grant Ingersoll > http://lucene.grantingersoll.com > > Lucene Helpful Hints: > http://wiki.apache.org/lucene-java/BasicsOfPerformance > http://wiki.apache.org/lucene-java/LuceneFAQ > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > ____________________________________________________________________________________ > Never miss a thing. Make Yahoo your home page. > http://www.yahoo.com/r/hs --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
