On 3/10/2018 9:44 AM, Erick Erickson wrote:
There are quite a number of reasons you may be seeing this, all having
to do with trying to put too much stuff in too little hardware.
At any rate, there's no a-priori limit to the number of
collections/replicas/whatever that Solr can deal with, the limits are
your hardware.

Big +1 to everything Erick said.  I was working on a response, but I think the things that Erick said are better than what I was writing.  I did want to add a little bit of detail, though.

Let me throw some numbers at you:

With 900 collections that have 25 shards, the SolrCloud cluster is handling 22500 individual indexes -- and that's for only one replica.  If I assume that you've only got one copy of all that data in Solr, then every one of those 49 servers will average over 450 index cores!  If you planned for redundancy and replicationFactor is 2, then each server will have over 900 index cores on it (45000 total indexes).

The index count numbers on each server might look like small numbers to you ... but when you start understanding some of what Solr and Lucene are doing under the covers, you start to realize that these numbers are quite large.

If there's even a hint of significant index/query traffic going to those collections, then those servers are going to be hard-pressed to keep up, unless you've spent a LOT more money for hardware than what I see in a typical server.

Have you ever heard of a phenomenon known as a performance knee? The following link should open up to page 482 of the associated book.  Scroll up to page 481 and reading the two sections named "Hardware Resource Requirements" and "Peak Load Behavior".  Your reading would end partway through page 482.


A common tendency in the performance of computer systems/software is that they show an increase in response time that's more or less linear with load increases, until the system reaches a breaking point where the smallest increase in load causes an incredibly large increase in response times.  That can lead to an outage situation where things take so long that clients give up and go to an error state.

FYI: You're running a "dot zero" release.  6.0 is the first release where MAJOR development changes had been accumulating that weren't in 5.x versions.  The 6.x timeframe was an absolute hotbed of innovation on SolrCloud.

There are typically more bugs in a .0 release than later minor releases.  If there are reasons for you to avoid running 7.x releases, then you should upgrade to the latest 6.x.  Right now this is 6.6.2, but we are in the middle of the 6.6.3 release right now, and that is likely to be announced any day now.

SolrCloud does not handle large numbers of collections very well, especially if the shard and/or replica counts are high for each collection.  Up to a point, throwing a LOT of hardware at the problem helps, but there are scalability issues related to the way SolrCloud uses the ZooKeeper database.  Eventually it just can't handle it.  See this issue for some experiments that I conducted on the problem:



Reply via email to