2.1-dev memory leak?

[Otis Gospodnetic]

Hi,

Is anyone running Lucene trunk/HEAD version in a serious production system?  
Anyone noticed any memory leaks?

I'm asking because I recently bravely went from 1.9.1 to 2.1-dev (trunk from 
about a week ago) and all of a sudden my application that was previosly 
consuming about 1.5GB (-Xmx1500m) now consumes 2.2GB, and blows up after it 
exhausts the whole heap and the GC can't make any more room there after running 
for about 3-6 hours and handling several tens of thousands of queries.

I'd love to go back to 2.0.0, or even back to 1.9.1 and run that for a while 
and just double-check that it really is the the Lucene upgrade that is the 
source of the leak, but unfortunately because of LUCENE-701 (lockless commits), 
I can't go back that easily without reindexing...

Moreover, I just looked at CHANGES.txt from 1.9.1 to present, and I think the 
biggest change since then was LUCENE-701.
LUCENE-672 (segment merge policy) was also pretty big, but from what I can 
tell, the memory leak is somewhere in the search part, not indexing part.  
There have been a number of other search-time optimizations since 2.0.0, so 
it's hard to tell what the cause is.  Of course, it could turn out to be a leak 
in my own code, but I'm pretty sure my changes were limited to removal of 
deprecated methods, so I can start using 2.1.

        IndexDescriptor indexDescriptor = getIndexDescriptorFromCache(indexID);

        try {
            // if this is a known index
            if (indexDescriptor != null) {
                cacheHits++;
                // if the index has changed since this Searcher was created, 
make a new Searcher
                long currentVersion = IndexReader.getCurrentVersion(indexID);
                if (currentVersion > indexDescriptor.lastKnownVersion) {
                    hitButChanged++;
                    // modified index detected
                    indexDescriptor.lastKnownVersion = currentVersion;
                    indexDescriptor.searcher = new LuceneSearcher(new 
File(indexID));
                }
                else {
                    // index not modified, reusing searcher
                }
            }
            // if this is a new index
            else {
                cacheMisses++;
                File indexDir = validateIndex(indexID);
                indexDescriptor = new IndexDescriptor();
                indexDescriptor.indexDir = indexDir;
                indexDescriptor.lastKnownVersion = 
IndexReader.getCurrentVersion(indexDir);
                indexDescriptor.searcher = new LuceneSearcher(indexDir);
            }
            return cacheIndexDescriptor(indexDescriptor);
        }
        catch (IOException e) {
            throw new SearcherException("Cannot open index: " + indexID, e);
        }

So this is just caching of "IndexDescriptor" objects, which have 
"LuceneSearcher" objects in them.
The cache is a small LRU cache with max size of 37.  The app actually consists 
of a few tens of thousands of Lucene indices, so this small cache results in 
only 20% cache hit ratio.

And then the LuceneSearcher ctor looks like this:

    LuceneSearcher(File indexDir) throws IOException {
        _indexDir = FSDirectory.getDirectory(indexDir, false);
        _searcher = new IndexSearcher(_indexDir);
    }

This _searcher (IndexSearcher) is then used in various search methods of this 
class.
There are no close() calls anywhere.  In other words, I don't explicitly close 
IndexSearchers, I just let them get GC collected. 
This stuff has been working for well for 1-2 years, and I just started 
exhausting the JVM heap about a week ago when I went from 1.9.1 to 2.1-dev.
Any other overly brave/crazy souls out there who are running the bleeding edge 
version in production environment?
This is running on FedoraCore3 under JDK 1.5_09 (latest 1.5).

Thanks,
Otis




---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]