Re: How to accomadate huge data
On 8/28/2014 11:57 AM, Ethan wrote: Our index size is 110GB and growing, crossed RAM capacity of 96GB, and we are seeing a lot of disk and network IO resulting in huge latencies and instability(one of the server used to shutdown and stay in recovery mode when restarted). Our admin added swap space and that seemed to have mitigated the issue. Adding swap space doesn't seem like it would actually fix anything. If the system is actively swapping, performance will be terrible. Assuming your heap size and query volume are not enormous, 96GB of RAM for an index size of 110GB seems like it would actually be pretty good. Remember that you have to subtract all heap requirements (java and otherwise) from the total RAM in order to determine how much RAM is left for caching the index. The ideal setup has enough extra RAM (beyond what's required for the software itself) to cache the entire index, but that ideal is usually not required. In most cases, getting between half and two thirds of the index into RAM is enough. One thing to note: If you don't have the entire index fitting into RAM, the server will probably not be able to handle an extreme query volume. But what is the usual practice in such scenario? Index size eventually outgrows RAM and is pushed on to disk. Is it advisable to shard(solr forum says no)? Or is there a different mechanism? System config: We have 3 node cluster with RAID1 SSD. Two nodes are running solr and the other is to maintain Quorum. Whether or not to shard depends on several factors, not the least of which is whether or not the features that you are using will work on a distributed index. My index is slightly larger than yours, and it's sharded. I don't run SolrCloud, the sharding is completely manual. Thanks, Shawn
re: How to accomadate huge data
Look into SolrCloud. From: Ethan eh198...@gmail.com Sent: Thursday, August 28, 2014 1:59 PM To: solr-user solr-user@lucene.apache.org Subject: How to accomadate huge data Our index size is 110GB and growing, crossed RAM capacity of 96GB, and we are seeing a lot of disk and network IO resulting in huge latencies and instability(one of the server used to shutdown and stay in recovery mode when restarted). Our admin added swap space and that seemed to have mitigated the issue. But what is the usual practice in such scenario? Index size eventually outgrows RAM and is pushed on to disk. Is it advisable to shard(solr forum says no)? Or is there a different mechanism? System config: We have 3 node cluster with RAID1 SSD. Two nodes are running solr and the other is to maintain Quorum. -E
RE: How to accomadate huge data
kokatnur.vi...@gmail.com [kokatnur.vi...@gmail.com] On Behalf Of Ethan [eh198...@gmail.com] wrote: Our index size is 110GB and growing, crossed RAM capacity of 96GB, and we are seeing a lot of disk and network IO resulting in huge latencies and instability(one of the server used to shutdown and stay in recovery mode when restarted). Our admin added swap space and that seemed to have mitigated the issue. Something is off here. I can understand disk IO going up when the index size increases, but why would it cause more network IO? Are you using networked storage or performing aggressive synchronization? Can you describe how hard you are hitting your indexes, both for updates and queries? What is huge latencies? Have you tried profiling the running Solrs to see if the heap size is large enough? We have 3 node cluster with RAID1 SSD. Two nodes are running solr and the other is to maintain Quorum. Is that on the same physical hardware or on three separate ones? - Toke Eskildsen
Re: How to accomadate huge data
On Thu, Aug 28, 2014 at 11:12 AM, Shawn Heisey s...@elyograg.org wrote: On 8/28/2014 11:57 AM, Ethan wrote: Our index size is 110GB and growing, crossed RAM capacity of 96GB, and we are seeing a lot of disk and network IO resulting in huge latencies and instability(one of the server used to shutdown and stay in recovery mode when restarted). Our admin added swap space and that seemed to have mitigated the issue. Adding swap space doesn't seem like it would actually fix anything. If the system is actively swapping, performance will be terrible. Assuming your heap size and query volume are not enormous, 96GB of RAM for an index size of 110GB seems like it would actually be pretty good. *E Before adding swap space nodes used to shutdown due to OOM or crash after 2-5 minutes of uptime. By bumping swap space the server came up cleanly. ** We have 7GB of heap. I'll need to ask admin more questions to know how it was solved.* Remember that you have to subtract all heap requirements (java and otherwise) from the total RAM in order to determine how much RAM is left for caching the index. The ideal setup has enough extra RAM (beyond what's required for the software itself) to cache the entire index, but that ideal is usually not required. In most cases, getting between half and two thirds of the index into RAM is enough. One thing to note: If you don't have the entire index fitting into RAM, the server will probably not be able to handle an extreme query volume. *E Our query volume is low right now, about 30 **TPS for /select. But** /update is 80 and /get around 100 TPS. In our SolrCloud setup we don't have a separate replication node that handles select traffic. The server currently has 12-40ms TP99 as we don't have any facets or complex queries. * But what is the usual practice in such scenario? Index size eventually outgrows RAM and is pushed on to disk. Is it advisable to shard(solr forum says no)? Or is there a different mechanism? System config: We have 3 node cluster with RAID1 SSD. Two nodes are running solr and the other is to maintain Quorum. Whether or not to shard depends on several factors, not the least of which is whether or not the features that you are using will work on a distributed index. My index is slightly larger than yours, and it's sharded. I don't run SolrCloud, the sharding is completely manual. *E Interesting. Whats your select and update TPS/TP99? We index around 6-8Gb data every month. I think we will need more than one server to handle our index in the long run without degrading performance.* Thanks, Shawn
RE: How to accomadate huge data
kokatnur.vi...@gmail.com [kokatnur.vi...@gmail.com] On Behalf Of Ethan [eh198...@gmail.com] wrote: Before adding swap space nodes used to shutdown due to OOM or crash after 2-5 minutes of uptime. By bumping swap space the server came up cleanly. ** We have 7GB of heap. I'll need to ask admin more questions to know how it was solved.* Yes, please. What you are describing is not solved by adding swap, unless the system has very little free RAM. - Toke Eskildsen
