Have you checked out solr project that provides a service on top of
Lucene + caching / warming up facilities similar to what you need.
The IndexReaders are expensive ( and are the underlying data source for
a given IndexSearcher ) in terms of time and resources , when being
opened / created anew.
So - having 2 live instances of IndexReaders concurrently might be an
issue to begin with.
Lucene (IndexSearchers / IndexReaders) has the notion of cache as well
so you need to check if you really want a 100% replication of
RAMDirectory / FSDirectory as well concurrently in memory. Have you
tested with the FieldCache policies before moving onto the RAMDirectory
based backup solution.
--
Karthik.
Diamond, Greg wrote:
Hi All -
What is the best way to load a RAM Directory from a FS Directory, and
periodically reload the RAM Directory to pick up new documents?
The scenario I have is I create several large directories which I create to a
file system, then load them into ram for faster searching.
They takes several hours to create, so i want to retain a file copy so in the
event of a service/server crash or reboot, they can load again
in a few seconds. Once a day I append or recreate the FS Directories with new
entries, and then reload the RAM Directories to pick up the
new entries.
Problem is, what I am doing seems to cause a memory leak. The directories take
~12 GB of RAM to load, but bloats
to 24 GB (the -Xmx setting) after a few reloads and stays there.
// Map to hold single instance of the IndexSearcher, 1 per RAM
Directory, to be reused across requests.
private static final Map<IndexName, IndexSearcher> searchers = new
HashMap<IndexName, IndexSearcher>();
// Creates the IndexSearcher and stores it in the static map.
public static void load(IndexName index, ...) {
// sync... etc
RAMDirectory dir = new
RAMDirectory(FSDirectory.getDirectory(directoryPath);
IndexReader reader = IndexReader.open(dir, true);
IndexSearcher searcher = new IndexSearcher(reader);
searchers.put(index, searcher);
}
public static IndexSearcher get(IndexName index, ...) {
// sync... etc
return searchers.get(index);
}
// After an indexing service appends the existing FS Directory, this
reloads it.
public static void reload(IndexName index, ...) {
// sync... etc
IndexSearcher searcher = searchers.get(index);
FSDirectory fsDirectory =
FSDirectory.getDirectory(directoryPath);
Directory ramDirectory = searcher.getIndexReader().directory();
Directory.copy(fsDirectory, ramDirectory, false);
}
I've tries some theme and variations on this with the same issue.
TIA!
gd
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
