On Thu, Aug 28, 2014 at 11:12 AM, Shawn Heisey <s...@elyograg.org> wrote:
> On 8/28/2014 11:57 AM, Ethan wrote: > > Our index size is 110GB and growing, crossed RAM capacity of 96GB, and we > > are seeing a lot of disk and network IO resulting in huge latencies and > > instability(one of the server used to shutdown and stay in recovery mode > > when restarted). Our admin added swap space and that seemed to have > > mitigated the issue. > > Adding swap space doesn't seem like it would actually fix anything. If > the system is actively swapping, performance will be terrible. > Assuming your heap size and query volume are not enormous, 96GB of RAM > for an index size of 110GB seems like it would actually be pretty good. > *E >> Before adding swap space nodes used to shutdown due to OOM or crash after 2-5 minutes of uptime. By bumping swap space the server came up cleanly. ** We have 7GB of heap. I'll need to ask admin more questions to know how it was solved.* > Remember that you have to subtract all heap requirements (java and > otherwise) from the total RAM in order to determine how much RAM is left > for caching the index. The ideal setup has enough extra RAM (beyond > what's required for the software itself) to cache the entire index, but > that ideal is usually not required. In most cases, getting between half > and two thirds of the index into RAM is enough. One thing to note: If > you don't have the entire index fitting into RAM, the server will > probably not be able to handle an extreme query volume. > *E >> Our query volume is low right now, about 30 **TPS for /select. But** /update is 80 and /get around 100 TPS. In our SolrCloud setup we don't have a separate replication node that handles select traffic. The server currently has 12-40ms TP99 as we don't have any facets or complex queries. * > > But what is the usual practice in such scenario? Index size eventually > > outgrows RAM and is pushed on to disk. Is it advisable to shard(solr > forum > > says no)? Or is there a different mechanism? > > > > System config: > > We have 3 node cluster with RAID1 SSD. Two nodes are running solr and > the > > other is to maintain Quorum. > > Whether or not to shard depends on several factors, not the least of > which is whether or not the features that you are using will work on a > distributed index. My index is slightly larger than yours, and it's > sharded. I don't run SolrCloud, the sharding is completely manual. > > *E >> Interesting. Whats your select and update TPS/TP99? We index around 6-8Gb data every month. I think we will need more than one server to handle our index in the long run without degrading performance.* Thanks, > Shawn > >
