On 3-May-08, at 10:06 AM, Daniel Andersson wrote:
Our database/index is 3.5 GB and contains 4,352,471 documents. Most
documents are less than 1 kb. When performing a search, the results
vary between 1.5 seconds up to 60 seconds.
I don't have a big problem with 1.5 seconds (even though below 1
would be nice), but 60 seconds it just.. well, scary.
That is too long, and shouldn't be happening.
How do I optimize Solr to better use all the RAM? I'm using java6,
64bit version, and start Solr using:
java -Xmx7500M -Xms4096M -jar start.jar
But according to top it only seems to be using 7.7% of the memory
(around 600 MB).
Don't try to give Solr _all_ the memory on the system. Solr depends
on the index existing in the OS's disk cache (this is "cached" in
top). You should have at least 2 GB memory for a 3.5GB index,
depending on how much of the index is stored (best is of course to
have 3.5GB available so it can be cached completely).
Solr will require a wide distribution of queries to "warm up" (get the
index in the OS disk cache). This is automatically prioritize the
"hot spots" in the index. If you want to load the whole thing 'cd
datadir; cat * > /dev/null' works, but I don't recommend relying on
that.
Most queries are for make_id + model_id or city + state and almost
all of the queries are ordered by datetime_found (newest -> oldest).
How many documents match, typically? How many documents are returned,
typically? How often do you commit() [I suspect frequently, based on
the problems you are having]?
-Mike
