30 Dedicated physical Nodes in the Solr Cloud Cluster, all of identical configuration Server01 RHEL 7.x 256GB RAM 10 2TB Spinning Disk in a RAID 10 Configuration (Leaving us 9.8TB usable per node) 64GB JVM Heap, Tried has high as 100GB, but it appeared that 64GB was faster. If we set a higher heap, do we starve the OS for caching? Huge Pages is off on the system, and thus UseLargePages is off on Solr Startup G1GC, Java 11 (ZGC with Java 15 and HugePages turned on was a disaster. We suspect it was due to the Huge Pages configuration) At one time we discussed putting two instances of Solr on each node, giving us a cloud of 60 instances instead of 30. Load Average is high on these nodes during certain types of queries or updates, but CPU Load is relatively low and should be able to accommodate a second instance, but all the data would still be on the same RAID10 group of disks. Collection 4 is the trouble collection. It has nearly a billion documents, and there are between 200 and 400 million updates every day. How do we get that kind of update performance, and still serve 10 million queries a day? Schemas have been reviewed and re-reviewed to ensure we are only indexing and storing what is absolutely necessary. What are we missing? Do we need to revisit our replica policy? Number of replicas or types of replicas (to ensure some are only used for reading, etc?) [Grabbed from the Admin UI] 755.6Gb Index Size according to Solr Cloud UI Total #docs: 371.8mn Avg size/doc: 2.1Kb 90 Shards, 2 NRT Replicas per Shard, 1,750,612,476 documents, avg size/doc: 1.7Kb, uses nested documents collection-1_s69r317 31.1Gb collection-1_s49r96 30.7Gb collection-1_s78r154 30.2Gb collection-1_s40r259 30.1Gb collection-1_s9r197 29.1Gb collection-1_s18r34 28.9Gb 120 Shards, 2 TLOG Replicas per Shard, 2,230,207,046 documents, avg size/doc: 1.3Kb collection-2_s78r154 22.8Gb collection-2_s49r96 22.8Gb collection-2_s46r331 22.8Gb collection-2_s18r34 22.7Gb collection-2_s109r216 22.7Gb collection-2_s104r447 22.7Gb collection-2_s15r269 22.7Gb collection-2_s73r385 22.7Gb 120 Shards, 2 TLOG Replicas per Shard, 733,588,503 documents, avg size/doc: 1.9Kb collection-3_s19r277 10.6Gb collection-3_s108r214 10.6Gb collection-3_s48r94 10.6Gb collection-3_s109r457 10.6Gb collection-3_s47r333 10.5Gb collection-3_s78r154 10.5Gb collection-3_s18r34 10.5Gb collection-3_s77r393 10.5Gb
120 Shards, 2 TLOG Replicas per Shard, 864,372,654 documents, avg size/doc: 5.6Kb collection-4_s109r216 38.7Gb collection-4_s100r439 38.7Gb collection-4_s49r96 38.7Gb collection-4_s35r309 38.6Gb collection-4_s18r34 38.6Gb collection-4_s78r154 38.6Gb collection-4_s7r253 38.6Gb collection-4_s69r377 38.6Gb
