...Using RAMDirectory really does not help performance... I kind of agree, but in my experience with lucene, there are cases where RAMDirectory helps a lot, with all its drawbacks (huge heap and gc() tuning).
We had very good experience with MMAP on average, but moving to RAMDirectory with properly tuned gc() reduced 95% of "slow performers" in upper range of response times (e.g. slowest 5% queries). On average it made practically no difference. Maybe is this mitigated by better warm up on solr than our hand-tuned warmup, maybe not, I do not really know. In MMAP, you need to have really smart warm up (MMAP) to beat IO quirks, for RAMDir you need to tune gc(), choose your poison :) I argue, in some cases it is very hard to tame IO quirks (e.g. this is shared resource, you never know what going really on in shared app setup!). Then, see only what is happening on major merge and all these efforts with native linux directory to somehow get a grip on that... If you have spare ram, you are probably safer with RAMDirectory. >From the theoretical perspective, in ideal case, RAM ought to be faster than disk (and more expensive). If this is not the case, we did something wrong. I have a feeling that this work Mike is doing with in memory Codecs (fst TermDictionary, pulsing codec & co) in Lucene 4, native directory features ... will make RAMDirectory really obsolete for production setup. Cheers, eks On Wed, Jun 29, 2011 at 6:00 AM, Lance Norskog <goks...@gmail.com> wrote: > Using RAMDirectory really does not help performance. Java garbage > collection has to work around all of the memory taken by the segments. > It works out that Solr works better (for most indexes) without using > the RAMDirectory. > > > > On Sun, Jun 26, 2011 at 2:07 PM, nipunb <ni...@walmartlabs.com> wrote: >> PS: Sorry if this is a repost, I was unable to see my message in the mailing >> list - this may have been due to my outgoing email different from the one I >> used to subscribe to the list with. >> >> Overview – Trying to evaluate if keeping the index in memory using >> RAMDirectoryFactory can help in query performance.I am trying to perform the >> indexing on the master using solr.StandardDirectoryFactory and make those >> indexes accesible to the slave using solr.RAMDirectoryFactory >> >> Details: >> We have set-up Solr in a master/slave enviornment. The index is built on the >> master and then replicated to slaves which are used to serve the query. >> The replication is done using the in-built Java replication in Solr. >> On the master, in the <indexDefaults> of solrconfig.xml we have >> <directoryFactory name="DirectoryFactory" >> class="solr.StandardDirectoryFactory"/> >> >> On the slave, I tried to use the following in the <indexDefaults> >> >> <directoryFactory name="DirectoryFactory" >> class="solr.RAMDirectoryFactory"/> >> >> My slave shows no data for any queries. In solrconfig.xml it is mentioned >> that replication doesn’t work when using RAMDirectoryFactory, however this ( >> https://issues.apache.org/jira/browse/SOLR-1379) mentions that you can use >> it to have the index on disk and then load into memory. >> >> To test the sanity of my set-up, I changed solrconfig.xml in the slave to >> and replicated: >> <directoryFactory name="DirectoryFactory" >> class="solr.StandardDirectoryFactory"/> >> I was able to see the results. >> >> Shouldn’t RAMDirectoryFactory be used for reading index from disk into >> memory? >> >> Any help/pointers in the right direction would be appreciated. >> >> Thanks! >> >> -- >> View this message in context: >> http://lucene.472066.n3.nabble.com/Using-RAMDirectoryFactory-in-Master-Slave-setup-tp3111792p3111792.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> > > > > -- > Lance Norskog > goks...@gmail.com >
