HI testn, I used to search now 3 folders of total size 1.5.GB.still it consumes lot of memory for every search.i used to close all the IndexReader once i finish the search.i optimize the FIles using Luke.when i set the IndexSearcher object as application level object,its not possible to see the updated records. could you guide me how to resolve this memory problem.
testn wrote: > > As I mentioned, IndexReader is the one that holds the memory. You should > explicitly close the underlying IndexReader to make sure that the reader > releases the memory. > > > > Sebastin wrote: >> >> Hi testn, >> Every IndexFolder is of size 1.5 GB of size,eventhough when >> i used to Open and close the IndexSearcher it wont release the memory for >> all the searches. >> When i set the IndexSearcher object as the Application >> Scope object its not possile for me to see current day records. >> >> Could you give me an Idea how to trigger out this >> problem. >> >> >> testn wrote: >>> >>> If you know that there are only 15 days of indexes you need to search >>> on, you just need to open only the latest 15 indexes at a time right? >>> You can simply create a wrapper that return MultiReader which you can >>> cache for a while and close the oldest index once the date rolls. >>> >>> >>> Sebastin wrote: >>>> >>>> HI testn, >>>> >>>> it gives performance improvement while optimizing the Index. >>>> >>>> Now i seprate the IndexStore on a daily basis.(ie) >>>> For Every Day it create a new Index store ,sep- 08-2007,sep-09-2007 >>>> like wise it will minimize the size of the IndexStore.could you give me >>>> an idea on how to open every day folders for every search. >>>> >>>> Query I use here is, >>>> >>>> 9840836588 AND dateSc:[070901 TO 070910] >>>> >>>> 07---->year (2007) >>>> 09---->month(september) >>>> 01----->day >>>> >>>> i restrict for 15 days that it is possible to search 15 days record in >>>> my application.at a time 10 users aare going to search every store.is >>>> there any other better way to improve the search performance to avoid >>>> memory problem as well as speed of the search. >>>> >>>> >>>> >>>> >>>> >>>> >>>> testn wrote: >>>>> >>>>> So did you see any improvement in performance? >>>>> >>>>> Sebastin wrote: >>>>>> >>>>>> It works finally .i use Lucene 2.2 in my application.thanks testn >>>>>> and Mike >>>>>> >>>>>> Michael McCandless-2 wrote: >>>>>>> >>>>>>> >>>>>>> It sounds like there may be a Lucene version mismatch? When Luke >>>>>>> was used >>>>>>> it was likely based on Lucene 2.2, but it sounds like an older >>>>>>> version of >>>>>>> Lucene is now being used to open the index? >>>>>>> >>>>>>> Mike >>>>>>> >>>>>>> "testn" <[EMAIL PROTECTED]> wrote: >>>>>>>> >>>>>>>> Should the file be "segments_8" and "segments.gen"? Why is it >>>>>>>> "Segment"? >>>>>>>> The >>>>>>>> case is different. >>>>>>>> >>>>>>>> >>>>>>>> Sebastin wrote: >>>>>>>> > >>>>>>>> > java.io.IoException:File Not Found- Segments is the error >>>>>>>> message >>>>>>>> > >>>>>>>> > testn wrote: >>>>>>>> >> >>>>>>>> >> What is the error message? Probably Mike, Erick or Yonik can >>>>>>>> help you >>>>>>>> >> better on this since I'm no one in index area. >>>>>>>> >> >>>>>>>> >> Sebastin wrote: >>>>>>>> >>> >>>>>>>> >>> HI testn, >>>>>>>> >>> 1.I optimize the Large Indexes of size 10 GB using >>>>>>>> Luke.it >>>>>>>> >>> optimize all the content into a single CFS file and it >>>>>>>> generates >>>>>>>> >>> segments.gen and segments_8 file when i search the item it >>>>>>>> shows an >>>>>>>> >>> error that segments file is not there.could you help me in this >>>>>>>> >>> >>>>>>>> >>> testn wrote: >>>>>>>> >>>> >>>>>>>> >>>> 1. You can close the searcher once you're done. If you want to >>>>>>>> reopen >>>>>>>> >>>> the index, you can close and reopen only the updated 3 readers >>>>>>>> and keep >>>>>>>> >>>> the 2 old indexreaders and reuse it. It should reduce the time >>>>>>>> to >>>>>>>> >>>> reopen it. >>>>>>>> >>>> 2. Make sure that you optimize it every once in a while >>>>>>>> >>>> 3. You might consider separating indices in separated storage >>>>>>>> and use >>>>>>>> >>>> ParallelReader >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>>> Sebastin wrote: >>>>>>>> >>>>> >>>>>>>> >>>>> The problem in my pplication are as follows: >>>>>>>> >>>>> 1.I am not able to see the updated records >>>>>>>> in my >>>>>>>> >>>>> index store because i instantiate >>>>>>>> >>>>> IndexReader and IndexSearcher class once that is in the first >>>>>>>> >>>>> search.further searches use the same IndexReaders(5 >>>>>>>> Directories) and >>>>>>>> >>>>> IndexSearcher with different queries. >>>>>>>> >>>>> >>>>>>>> >>>>> 2.My search is very very slow First 2 >>>>>>>> Directories of >>>>>>>> >>>>> size 10 GB each which are having old index records and no >>>>>>>> update in >>>>>>>> >>>>> that remaining 3 Diretories are updated every second. >>>>>>>> >>>>> >>>>>>>> >>>>> 3.i am Indexing 20 million records per day so >>>>>>>> the >>>>>>>> >>>>> Index store gets growing and it makes search very very >>>>>>>> slower. >>>>>>>> >>>>> >>>>>>>> >>>>> 4.I am using searcherOne class as the global >>>>>>>> >>>>> application helper class ,with the scope as APPLICATION it >>>>>>>> consists of >>>>>>>> >>>>> one IndexReader and IndexSearcher get set method which will >>>>>>>> hold the >>>>>>>> >>>>> IndexReader and IndexSearcher object after the First >>>>>>>> Search.it is used >>>>>>>> >>>>> for all other searches. >>>>>>>> >>>>> >>>>>>>> >>>>> 5.I am using Lucene 2.2.0 version, in a WEB >>>>>>>> Application >>>>>>>> >>>>> which index 15 fields per document and Index 5 Fieds,store 10 >>>>>>>> Fields.i >>>>>>>> >>>>> am not using any sort in my query.for a single query upto the >>>>>>>> maximum >>>>>>>> >>>>> it fetches 600 records from the index store(5 direcories) >>>>>>>> >>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> hossman wrote: >>>>>>>> >>>>>> >>>>>>>> >>>>>> >>>>>>>> >>>>>> : I set IndexSearcher as the application Object after the >>>>>>>> first >>>>>>>> >>>>>> search. >>>>>>>> >>>>>> ... >>>>>>>> >>>>>> : how can i reconstruct the new IndexSearcher for every hour >>>>>>>> to see >>>>>>>> >>>>>> the >>>>>>>> >>>>>> : updated records . >>>>>>>> >>>>>> >>>>>>>> >>>>>> i'm confused ... my understanding based on the comments you >>>>>>>> made >>>>>>>> >>>>>> below >>>>>>>> >>>>>> (in an earlier message) was that you already *were* >>>>>>>> constructing a >>>>>>>> >>>>>> new >>>>>>>> >>>>>> IndexSearcher once an hour -- but every time you do that, >>>>>>>> your memory >>>>>>>> >>>>>> usage grows, and and that sometimes you got OOM Errors. >>>>>>>> >>>>>> >>>>>>>> >>>>>> if that's not what you said, then i think you need to >>>>>>>> explain, in >>>>>>>> >>>>>> detail, >>>>>>>> >>>>>> in one message, exactly what your problem is. And don't >>>>>>>> assume we >>>>>>>> >>>>>> understand anything -- tell us *EVERYTHING* (like, for >>>>>>>> example, what >>>>>>>> >>>>>> the >>>>>>>> >>>>>> word "crore" means, how "searcherOne" is implemented, and >>>>>>>> the answer >>>>>>>> >>>>>> to >>>>>>>> >>>>>> the specfic question i asked in my last message: does your >>>>>>>> >>>>>> application, >>>>>>>> >>>>>> contain anywhere in it, any code that will close anything >>>>>>>> >>>>>> (IndexSearchers >>>>>>>> >>>>>> or IndexReaders) ? >>>>>>>> >>>>>> >>>>>>>> >>>>>> >>>>>>>> >>>>>> : > : I use StandardAnalyzer.the records daily ranges from 5 >>>>>>>> crore to >>>>>>>> >>>>>> 6 crore. >>>>>>>> >>>>>> : > for >>>>>>>> >>>>>> : > : every second i am updating my Index. i instantiate >>>>>>>> >>>>>> IndexSearcher object >>>>>>>> >>>>>> : > one >>>>>>>> >>>>>> : > : time for all the searches. for an hour can i see the >>>>>>>> updated >>>>>>>> >>>>>> records in >>>>>>>> >>>>>> : > the >>>>>>>> >>>>>> : > : indexstore by reinstantiating IndexSearcher object.but >>>>>>>> the >>>>>>>> >>>>>> problem when >>>>>>>> >>>>>> : > i >>>>>>>> >>>>>> : > : reinstantiate IndexSearcher ,my RAM memory gets >>>>>>>> appended.is >>>>>>>> >>>>>> there any >>>>>>>> >>>>>> >>>>>>>> >>>>>> >>>>>>>> >>>>>> : > IndexSearcher are you explicitly closing both the old >>>>>>>> >>>>>> IndexSearcher as >>>>>>>> >>>>>> : > well as all of 4 of those old IndexReaders and the >>>>>>>> MultiReader? >>>>>>>> >>>>>> >>>>>>>> >>>>>> >>>>>>>> >>>>>> >>>>>>>> >>>>>> >>>>>>>> >>>>>> -Hoss >>>>>>>> >>>>>> >>>>>>>> >>>>>> >>>>>>>> >>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> >>>>>> To unsubscribe, e-mail: >>>>>>>> [EMAIL PROTECTED] >>>>>>>> >>>>>> For additional commands, e-mail: >>>>>>>> [EMAIL PROTECTED] >>>>>>>> >>>>>> >>>>>>>> >>>>>> >>>>>>>> >>>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >> >>>>>>>> >> >>>>>>>> > >>>>>>>> > >>>>>>>> >>>>>>>> -- >>>>>>>> View this message in context: >>>>>>>> http://www.nabble.com/Java-Heap-Space--Out-Of-Memory-Error-tf4376803.html#a12655880 >>>>>>>> Sent from the Lucene - Java Users mailing list archive at >>>>>>>> Nabble.com. >>>>>>>> >>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED] >>>>>>>> For additional commands, e-mail: [EMAIL PROTECTED] >>>>>>>> >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED] >>>>>>> For additional commands, e-mail: [EMAIL PROTECTED] >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > > -- View this message in context: http://www.nabble.com/Java-Heap-Space--Out-Of-Memory-Error-tf4376803.html#a12775620 Sent from the Lucene - Java Users mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]