My indexes are stored on a NetApp filter via NFS. The indexer process updates the indexes over NFS. I have multiple indexes. My search process determines if the nfs indexes have been updated, and if they have, then loads the index into a RAMDirectory. RAMDirectory is of course much faster than searching over NFS. This way, I can also have multiple search servers running easily. The drawback of course is startup time. It takes a few minutes to start each search server because it has to load the data into memory. RAMDirectory also seems to be kind of memory inneficient, using a lot more memory than the data actually consumes on disk.
On Wed, 24 Nov 2004 14:26:40 -0800, Jonathan Hager <[EMAIL PROTECTED]> wrote: > When comparing RAMDirectory and FSDirectory it is important to mention > what OS you are using. When using linux it will cache the most recent > disk access in memory. Here is a good article that describes its > strategy: http://forums.gentoo.org/viewtopic.php?t=175419 > > The 2% difference you are seeing is the memory copy. With other OSes > you may see a speed up when using the RAMDirectory, because not all > OSes contain a disk cache in memory and must access the disk to read > the index. > > Another consideration is there is currently a 2GB limitation with the > size of the RAMDirectory. Indexes over 2GB causes a overflow in the > int used to create the buffer. [see int len = (int) is.length(); in > RamDirectory] > > I ended up using RAM directory for a very different reason. The index > is 1 to 2MB and is rebuilt every few hours. It takes 3 to 4 minutes > to query the database and rebuild the index. But the search should be > available 100% of the time. Since the index is so small I do the > following: > > on server startup: > - look for semaphore, if it is there delete the index > - if there is no index, build it to FSdirectory > - load the index from FSDirectory into RAMDirectory > > on reindex: > - create semaphore > - rebuild index to FSDirectory > - delete semaphore > - load index from FSDirecttory into RAMDirectory > > to search: > - search the RAMDirectory > > RAMDirectory could be replaced by a regular FSDirectory, but it seemed > silly to copy the index from disk to disk, when it ultimately needs to > be in memory. > > FSDirectory could be replaced by a RAMDirectory, but this means that > it would take the server 3 to 4 minutes longer to startup every time. > By persisting the index, this time would only be necessary if indexing > was interrupted. > > Jonathan > > On Mon, 22 Nov 2004 12:39:07 -0800, Kevin A. Burton > > > <[EMAIL PROTECTED]> wrote: > > Otis Gospodnetic wrote: > > > > >For the Lucene book I wrote some test cases that compare FSDirectory > > >and RAMDirectory. What I found was that with certain settings > > >FSDirectory was almost as fast as RAMDirectory. Personally, I would > > >push FSDirectory and hope that the OS and the Filesystem do their share > > >of work and caching for me before looking for ways to optimize my code. > > > > > > > > Yes... I performed the same benchmark and in my situation RAMDirectory > > for searches was about 2% slower. > > > > I'm willing to bet that it has to do with the fact that its a Hashtable > > and not a HashMap (which isn't synchronized). > > > > Also adding a constructor for the term size could make loading a > > RAMDirectory faster since you could prevent rehash. > > > > If you're on a modern machine your filesystme cache will end up > > buffering your disk anyway which I'm sure was happening in my situation. > > > > Kevin > > > > -- > > > > Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an > > invite! Also see irc.freenode.net #rojo if you want to chat. > > > > Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html > > > > If you're interested in RSS, Weblogs, Social Networking, etc... then you > > should work for Rojo! If you recommend someone and we hire them you'll > > get a free iPod! > > > > Kevin A. Burton, Location - San Francisco, CA > > AIM/YIM - sfburtonator, Web - http://peerfear.org/ > > GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
