Impossible to say, see: https://lucidworks.com/post/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/
Lucene uses MMapDirectory, see: https://dzone.com/articles/use-lucene%E2%80%99s-mmapdirectory, so the size of the searchable portions of your index relative to physical memory matters. “nodes going down” What’s in the logs? OOM? Leader Initiated Recovery? Details matter. So here’s what I’d be looking at: - during indexing, do you start seeing increasing I/O times? If so, you need more physical RAM - look at your GC logs. Do you see excessive GC activity? If so, you need more heap Without knowing lots of details, it really looks like you need more hardware or fewer replicas. You have at least 90*8*2 replicas, or 1,440 replicas on 2 servers, which is a lot. You haven’t mentioned how big your aggregate index across all replicas is on disk, but I suspect the aggregate index size _far_ exceeds your available RAM. While it isn’t necessary for all of the index to fit into RAM, at some point you do need to have enough RAM to contain the Java heap and enough left over to allow Lucene to use MMap without swapping. Best, Erick > On Oct 21, 2019, at 4:07 PM, saravanamanoj <jsmanoj...@gmail.com> wrote: > > Hi, > > We have a solrCloud with 2 solr servers with 3 zookeepers. We have around 90 > collections. Each having shard size of 8 and replication factor of 2. Some > collections have shard size of 16 as well. Atleast 10 collections will have > 200M records. Some has 100M records with ~400 fields. We are currently > running with 16 GB JVM. > > We experience slow down or complete node down some times while we update > some large collections (in terms of number of records). > > What can we do to improve the performance while update records using csv > update handler? To avoid nodes going down? > > Can we add more solr nodes? increase or decrease shard counts? > Can someone let us know the below? > 1.optimum number of collections per node (say each collections has around > 50M records) > 2.optimum number of shards per collection(say each collections has around > 50M records) > 3.optimum heap size per node (say each collections has around 50M records) > > Thanks in advance! > > > > -- > Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
