Hi testn,
Every IndexFolder is of size 1.5 GB of size,eventhough when i
used to Open and close the IndexSearcher it wont release the memory for all
the searches.
When i set the IndexSearcher object as the Application Scope
object its not possile for me to see current day records.
Could you give me an Idea how to trigger out this problem.
testn wrote:
>
> If you know that there are only 15 days of indexes you need to search on,
> you just need to open only the latest 15 indexes at a time right? You can
> simply create a wrapper that return MultiReader which you can cache for a
> while and close the oldest index once the date rolls.
>
>
> Sebastin wrote:
>>
>> HI testn,
>>
>> it gives performance improvement while optimizing the Index.
>>
>> Now i seprate the IndexStore on a daily basis.(ie)
>> For Every Day it create a new Index store ,sep- 08-2007,sep-09-2007 like
>> wise it will minimize the size of the IndexStore.could you give me an
>> idea on how to open every day folders for every search.
>>
>> Query I use here is,
>>
>> 9840836588 AND dateSc:[070901 TO 070910]
>>
>> 07---->year (2007)
>> 09---->month(september)
>> 01----->day
>>
>> i restrict for 15 days that it is possible to search 15 days record in my
>> application.at a time 10 users aare going to search every store.is there
>> any other better way to improve the search performance to avoid memory
>> problem as well as speed of the search.
>>
>>
>>
>>
>>
>>
>> testn wrote:
>>>
>>> So did you see any improvement in performance?
>>>
>>> Sebastin wrote:
>>>>
>>>> It works finally .i use Lucene 2.2 in my application.thanks testn and
>>>> Mike
>>>>
>>>> Michael McCandless-2 wrote:
>>>>>
>>>>>
>>>>> It sounds like there may be a Lucene version mismatch? When Luke was
>>>>> used
>>>>> it was likely based on Lucene 2.2, but it sounds like an older version
>>>>> of
>>>>> Lucene is now being used to open the index?
>>>>>
>>>>> Mike
>>>>>
>>>>> "testn" <[EMAIL PROTECTED]> wrote:
>>>>>>
>>>>>> Should the file be "segments_8" and "segments.gen"? Why is it
>>>>>> "Segment"?
>>>>>> The
>>>>>> case is different.
>>>>>>
>>>>>>
>>>>>> Sebastin wrote:
>>>>>> >
>>>>>> > java.io.IoException:File Not Found- Segments is the error message
>>>>>> >
>>>>>> > testn wrote:
>>>>>> >>
>>>>>> >> What is the error message? Probably Mike, Erick or Yonik can help
>>>>>> you
>>>>>> >> better on this since I'm no one in index area.
>>>>>> >>
>>>>>> >> Sebastin wrote:
>>>>>> >>>
>>>>>> >>> HI testn,
>>>>>> >>> 1.I optimize the Large Indexes of size 10 GB using
>>>>>> Luke.it
>>>>>> >>> optimize all the content into a single CFS file and it generates
>>>>>> >>> segments.gen and segments_8 file when i search the item it shows
>>>>>> an
>>>>>> >>> error that segments file is not there.could you help me in this
>>>>>> >>>
>>>>>> >>> testn wrote:
>>>>>> >>>>
>>>>>> >>>> 1. You can close the searcher once you're done. If you want to
>>>>>> reopen
>>>>>> >>>> the index, you can close and reopen only the updated 3 readers
>>>>>> and keep
>>>>>> >>>> the 2 old indexreaders and reuse it. It should reduce the time
>>>>>> to
>>>>>> >>>> reopen it.
>>>>>> >>>> 2. Make sure that you optimize it every once in a while
>>>>>> >>>> 3. You might consider separating indices in separated storage
>>>>>> and use
>>>>>> >>>> ParallelReader
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>> Sebastin wrote:
>>>>>> >>>>>
>>>>>> >>>>> The problem in my pplication are as follows:
>>>>>> >>>>> 1.I am not able to see the updated records in
>>>>>> my
>>>>>> >>>>> index store because i instantiate
>>>>>> >>>>> IndexReader and IndexSearcher class once that is in the first
>>>>>> >>>>> search.further searches use the same IndexReaders(5
>>>>>> Directories) and
>>>>>> >>>>> IndexSearcher with different queries.
>>>>>> >>>>>
>>>>>> >>>>> 2.My search is very very slow First 2
>>>>>> Directories of
>>>>>> >>>>> size 10 GB each which are having old index records and no
>>>>>> update in
>>>>>> >>>>> that remaining 3 Diretories are updated every second.
>>>>>> >>>>>
>>>>>> >>>>> 3.i am Indexing 20 million records per day so
>>>>>> the
>>>>>> >>>>> Index store gets growing and it makes search very very slower.
>>>>>> >>>>>
>>>>>> >>>>> 4.I am using searcherOne class as the global
>>>>>> >>>>> application helper class ,with the scope as APPLICATION it
>>>>>> consists of
>>>>>> >>>>> one IndexReader and IndexSearcher get set method which will
>>>>>> hold the
>>>>>> >>>>> IndexReader and IndexSearcher object after the First Search.it
>>>>>> is used
>>>>>> >>>>> for all other searches.
>>>>>> >>>>>
>>>>>> >>>>> 5.I am using Lucene 2.2.0 version, in a WEB
>>>>>> Application
>>>>>> >>>>> which index 15 fields per document and Index 5 Fieds,store 10
>>>>>> Fields.i
>>>>>> >>>>> am not using any sort in my query.for a single query upto the
>>>>>> maximum
>>>>>> >>>>> it fetches 600 records from the index store(5 direcories)
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>> hossman wrote:
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>> : I set IndexSearcher as the application Object after the
>>>>>> first
>>>>>> >>>>>> search.
>>>>>> >>>>>> ...
>>>>>> >>>>>> : how can i reconstruct the new IndexSearcher for every hour
>>>>>> to see
>>>>>> >>>>>> the
>>>>>> >>>>>> : updated records .
>>>>>> >>>>>>
>>>>>> >>>>>> i'm confused ... my understanding based on the comments you
>>>>>> made
>>>>>> >>>>>> below
>>>>>> >>>>>> (in an earlier message) was that you already *were*
>>>>>> constructing a
>>>>>> >>>>>> new
>>>>>> >>>>>> IndexSearcher once an hour -- but every time you do that, your
>>>>>> memory
>>>>>> >>>>>> usage grows, and and that sometimes you got OOM Errors.
>>>>>> >>>>>>
>>>>>> >>>>>> if that's not what you said, then i think you need to explain,
>>>>>> in
>>>>>> >>>>>> detail,
>>>>>> >>>>>> in one message, exactly what your problem is. And don't
>>>>>> assume we
>>>>>> >>>>>> understand anything -- tell us *EVERYTHING* (like, for
>>>>>> example, what
>>>>>> >>>>>> the
>>>>>> >>>>>> word "crore" means, how "searcherOne" is implemented, and the
>>>>>> answer
>>>>>> >>>>>> to
>>>>>> >>>>>> the specfic question i asked in my last message: does your
>>>>>> >>>>>> application,
>>>>>> >>>>>> contain anywhere in it, any code that will close anything
>>>>>> >>>>>> (IndexSearchers
>>>>>> >>>>>> or IndexReaders) ?
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>> : > : I use StandardAnalyzer.the records daily ranges from 5
>>>>>> crore to
>>>>>> >>>>>> 6 crore.
>>>>>> >>>>>> : > for
>>>>>> >>>>>> : > : every second i am updating my Index. i instantiate
>>>>>> >>>>>> IndexSearcher object
>>>>>> >>>>>> : > one
>>>>>> >>>>>> : > : time for all the searches. for an hour can i see the
>>>>>> updated
>>>>>> >>>>>> records in
>>>>>> >>>>>> : > the
>>>>>> >>>>>> : > : indexstore by reinstantiating IndexSearcher object.but
>>>>>> the
>>>>>> >>>>>> problem when
>>>>>> >>>>>> : > i
>>>>>> >>>>>> : > : reinstantiate IndexSearcher ,my RAM memory gets
>>>>>> appended.is
>>>>>> >>>>>> there any
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>> : > IndexSearcher are you explicitly closing both the old
>>>>>> >>>>>> IndexSearcher as
>>>>>> >>>>>> : > well as all of 4 of those old IndexReaders and the
>>>>>> MultiReader?
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>> -Hoss
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> >>>>>> To unsubscribe, e-mail:
>>>>>> [EMAIL PROTECTED]
>>>>>> >>>>>> For additional commands, e-mail:
>>>>>> [EMAIL PROTECTED]
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>
>>>>>> >>
>>>>>> >
>>>>>> >
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://www.nabble.com/Java-Heap-Space--Out-Of-Memory-Error-tf4376803.html#a12655880
>>>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>>>>>> For additional commands, e-mail: [EMAIL PROTECTED]
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>>>>> For additional commands, e-mail: [EMAIL PROTECTED]
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
--
View this message in context:
http://www.nabble.com/Java-Heap-Space--Out-Of-Memory-Error-tf4376803.html#a12729051
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
