On Thu, Jun 19, 2008 at 6:11 PM, Sébastien Rainville <[EMAIL PROTECTED]> wrote: > I've been using solr for a little without worrying too much about how it > works but now it's becoming a bottleneck in my application. I have a couple > issues with it: > > 1. My index always gets slower and slower when commiting/optimizing for some > obscure reason. It goes from 1 second with a new index to 45 seconds with an > index with the same amount of data but used for a few days. Restarting solr > doesn't fix it. The only way I found to fix that is to delete the whole > index completely by deleting the index folder. Then when I rebuild the index > everything goes back to normal and fast... and then performance slowly > deteriorates again. So, the amount of data is not a factor because > rebuilding the index from scratch fixes the problem and I am sending > "optimize" once in a while... even maybe too often.
This sounds like OS caching to me. A large amount of a "new" index that was just written will be in cache and thus much faster to optimize. If your index is smaller than the amount of RAM, go to the index directory of an "old" index, then try "cat * > /dev/null" and then try optimize to see of that's the case. > 2. I use acts_as_solr and by default they only make "post" requests, even > for /select. With that setup the response time for most queries, simple or > complex ones, were ranging from 150ms to 600ms, with an average of 250ms. I > changed the select request to use "get" requests instead and now the > response time is down to 10ms to 60ms. Did someone seen that before? Why is > it doing it? Are the get requests being cached by the ruby stuff? But even with no caching, I've seen differences with get/post on Linux with the python client when persistent HTTP connections were in use. I tracked it down to the POST being written in two parts, triggering nagle's algorithm in the networking stack. -Yonik
