I forgot to mention that the query threads are created only once. Peter
On 8/21/06, Peter Keegan <[EMAIL PROTECTED]> wrote:
Aleksey, My app has precise and predictable control over when the index reader/searcher gets refreshed. When this occurs, the sorting values for all the docs in the new index are generated by the app and cached. Each query thread has it's own ScoreDocComparator which contains a reference to the sorting values and is reused with different 'point' values on each query. So, Lucene only caches one instance of the ScoreDocComparator per query thread. Peter On 8/21/06, Aleksey Serba <[EMAIL PROTECTED]> wrote: > > Peter, > > As i understand correctly, implementing the equals and hashCode > methods let us avoid caching the same sorting values, i.e. if we have > a few points to calc distance from then we can implement equals and > hashCode methods based on point value and we'll have one cached > comparator per different center point value. > > However, if every new search we have different center point value then > we still in enormous memory usage problem. > > Aleksey > > On 8/21/06, Peter Keegan <[EMAIL PROTECTED]> wrote: > > I avoided the cache by implementing the equals and hashCode methods > > appropriately in my custom ScoreDocComparator as described here: > > http://www.lucenebook.com/blog/errata/2006/03/01/memory_leak.html > > > > Peter > > > > On 8/21/06, Aleksey Serba < [EMAIL PROTECTED]> wrote: > > > > > > Hi Chris, > > > > > > On 5/17/06, Peter Keegan <peterlkeegan[at]gmail.com> wrote: > > > > Suppose I have a custom sorting 'DocScoreComparator' for computing > > > > distances > > > > on each search hit from a specified coordinate (similar to the > > > > DistanceComparatorSource example in LIA). Assume that the > 'specified > > > > coordinate' is different for each query. This means a new custom > > > comparator > > > > must be created for each query, which is ok. However, Lucene > caches the > > > > comparator even though it will never be reused. This could result > in > > > heavy > > > > memory usage if many queries are performed before the IndexReader > is > > > > updated. > > > > > > Lets suppose another custom sortings - random, using dynamically > > > changed values from database, etc. Everytime lucene caches _useless_ > > > > custom comparators and application developer has no any control on > it. > > > > > > Also, I have no evidence to close IndexReader if my index is static. > > > > > > Aleksey > > > > > > On 8/18/06, Chris Hostetter <[EMAIL PROTECTED]> wrote: > > > > > > > > : You can reproduce OutOfMemory easily. I've attach test files - > this is > > > > : altered DistanceSortingTest example from LIA book. Also you can > > > > : profile it and see caching of distances arrays. > > > > > > > > An OutOfMemory error is differnet from a memory leak. Sorting > with a > > > > custom Comparator does in fact use a lot of memory -- and if your > Heap > > > > size is not big enough you may get an OutOfMemory -- but that > doesn't > > > mean > > > > you have a memory leak. A Memory leak is a problem where memory > is > > > > allocated for objects, but not freed up once those objects are no > longer > > > > of use -- just becuase you aren't using those objects, doesn't > mean they > > > > aren't "of use" and it doesn't mean they won't be freed up once > some > > > > futher action is taken (ie: closing of an IndexReader) > > > > > > > > Are you sure you've found an actual memory leak? ... ie: do you > have a > > > > demonstratable test case that shows heap usage growing without > bound > > > over > > > > time? If so then seeing *that* code would be helpful to figure > out if > > > > there is in fact a memory leak in Lucene, or if (perhaps) there is > a > > > > mistaken assumption in your code (ie; not closing an IndexReader) > > > > > > > > > > > > > > > > > > > > -Hoss > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > >
