Re: is there any practice to load index into RAM to accelerate solr performance?

2012-02-08 Thread Ted Dunning
This is true with Lucene as it stands. It would be much faster if there were a specialized in-memory index such as is typically used with high performance search engines. On Tue, Feb 7, 2012 at 9:50 PM, Lance Norskog goks...@gmail.com wrote: Experience has shown that it is much faster to run

Re: is there any practice to load index into RAM to accelerate solr performance?

2012-02-08 Thread Patrick Plaatje
A start maybe to use a RAM disk for that. Mount is as a normal disk and have the index files stored there. Have a read here: http://en.wikipedia.org/wiki/RAM_disk Cheers, Patrick 2012/2/8 Ted Dunning ted.dunn...@gmail.com This is true with Lucene as it stands. It would be much faster if

Re: is there any practice to load index into RAM to accelerate solr performance?

2012-02-08 Thread Dmitry Kan
Hi, This talk has some interesting details on setting up an Lucene index in RAM: http://www.lucidimagination.com/devzone/events/conferences/revolution/2011/lucene-yelp Would be great to hear your findings! Dmitry 2012/2/8 James ljatreey...@163.com Is there any practice to load index into

Re: is there any practice to load index into RAM to accelerate solr performance?

2012-02-08 Thread Andrzej Bialecki
On 08/02/2012 09:17, Ted Dunning wrote: This is true with Lucene as it stands. It would be much faster if there were a specialized in-memory index such as is typically used with high performance search engines. This could be implemented in Lucene trunk as a Codec. The challenge though is to

Re: is there any practice to load index into RAM to accelerate solr performance?

2012-02-08 Thread Robert Stewart
I concur with this. As long as index segment files are cached in OS file cache performance is as about good as it gets. Pulling segment files into RAM inside JVM process may actually be slower, given Lucene's existing data structures and algorithms for reading segment file data. If you have

Re: is there any practice to load index into RAM to accelerate solr performance?

2012-02-08 Thread Ted Dunning
Add this as well: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.155.5030 On Wed, Feb 8, 2012 at 1:56 AM, Andrzej Bialecki a...@getopt.org wrote: On 08/02/2012 09:17, Ted Dunning wrote: This is true with Lucene as it stands. It would be much faster if there were a specialized

Re: is there any practice to load index into RAM to accelerate solr performance?

2012-02-07 Thread Lance Norskog
Experience has shown that it is much faster to run Solr with a small amount of memory and let the rest of the ram be used by the operating system disk cache. That is, the OS is very good at keeping the right disk blocks in memory, much better than Solr. How much RAM is in the server and how much