Just to re-emphasize the point - when provisioning Solr, you need to ASSURE
that the system has enough system memory so that the Solr index on that
system fits entirely in the OS file system cache. No ifs, ands, or buts. If
you fail to follow that RULE, all bets are off for performance and don't
even bother complaining about poor performance on this mailing list!! Either
get more memory or shard your index more heavily - again, no ifs, ands, or
buts!!
Any questions on that rule?
Maybe somebody else can phrase this "guidance" more clearly, so that fewer
people will fail to follow it.
Or, maybe we should enhance Solr to check available memory and log a stern
warning if the index size exceeds system memory when Solr is started.
-- Jack Krupansky
-----Original Message-----
From: Shawn Heisey
Sent: Tuesday, May 20, 2014 1:49 PM
To: solr-user@lucene.apache.org
Subject: Re: solr-user Digest of: get.100322
On 5/20/2014 2:01 AM, Jeongseok Son wrote:
Though it uses only small amount of memory I'm worried about memory
usage because I have to store so many documents. (32GB RAM / total 5B
docs, sum of docs. of all cores)
If you've only got 32GB of RAM and there are five billion docs on the
system, Solr performance will be dismal no matter what you do with
docValues. Your index will be FAR larger than the amount of available
RAM for caching.
http://wiki.apache.org/solr/SolrPerformanceProblems#RAM
With that many documents, even if you don't use RAM-hungry features like
sorting and facets, you'll need a significant heap size, which will
further reduce the amount of RAM on the system that the OS can use to
cache the index.
For good performance, Solr *relies* on the operating system caching a
significant portion of the index.
Thanks,
Shawn
