Shawn: unfortunately the current problems are with facet.method=enum! Erick: We already round our date queries so they're the same for at least an hour so thankfully our fq entries will be reusable. However, I'll take a look at reducing the cache and autowarming counts and see what the effect on hit ratios and performance are.
For SolrCloud our soft commit (openSearcher=false) interval is 15 seconds and our hard commit is 15 minutes. You're right about those sorted fields having a lot of unique values. They can be any number between 0 and 10,000,000 (it's sparsely populated across the documents) and could appear in several variants across multiple documents. This is probably a good area for seeing what we can bend with regard to our requirements for sorting/boosting. I've just looked at two shards and they've each got upwards of 1000 terms showing in the schema browser for one (potentially out of 60) fields. On 21 September 2013 20:07, Erick Erickson <erickerick...@gmail.com> wrote: > About caches. The queryResultCache is only useful when you expect there > to be a number of _identical_ queries. Think of this cache as a map where > the key is the query and the value is just a list of N document IDs > (internal) > where N is your window size. Paging is often the place where this is used. > Take a look at your admin page for this cache, you can see the hit rates. > But, the take-away is that this is a very small cache memory-wise, varying > it is probably not a great predictor of memory usage. > > The filterCache is more intense memory wise, it's another map where the > key is the fq clause and the value is bounded by maxDoc/8. Take a > close look at this in the admin screen and see what the hit ratio is. It > may > be that you can make it much smaller and still get a lot of benefit. > _Especially_ considering it could occupy about 44G of memory. > (43,000,000 / 8) * 8192........ And the autowarm count is excessive in > most cases from what I've seen. Cutting the autowarm down to, say, 16 > may not make a noticeable difference in your response time. And if > you're using NOW in your fq clauses, it's almost totally useless, see: > http://searchhub.org/2012/02/23/date-math-now-and-filter-queries/ > > Also, read Uwe's excellent blog about MMapDirectory here: > http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html > for some problems with over-allocating memory to the JVM. Of course > if you're hitting OOMs, well..... > > bq: order them by one of their fields. > This is one place I'd look first. How many unique values are in each field > that you sort on? This is one of the major memory consumers. You can > get a sense of this by looking at admin/schema-browser and selecting > the fields you sort on. There's a text box with the number of terms > returned, > then a / ### where ### is the total count of unique terms in the field. > NOTE: > in 4.4 this will be -1 for multiValued fields, but you shouldn't be > sorting on > those anyway. How many fields are you sorting on anyway, and of what types? > > For your SolrCloud experiments, what are your soft and hard commit > intervals? > Because something is really screwy here. Your sharding moving the > number of docs down this low per shard should be fast. Back to the point > above, the only good explanation I can come up with from this remove is > that the fields you sort on have a LOT of unique values. It's possible that > the total number of unique values isn't scaling with sharding. That is, > each > shard may have, say, 90% of all unique terms (number from thin air). Worth > checking anyway, but a stretch. > > This is definitely unusual... > > Best, > Erick > > > On Thu, Sep 19, 2013 at 8:20 AM, Neil Prosser <neil.pros...@gmail.com> > wrote: > > Apologies for the giant email. Hopefully it makes sense. > > > > We've been trying out SolrCloud to solve some scalability issues with our > > current setup and have run into problems. I'd like to describe our > current > > setup, our queries and the sort of load we see and am hoping someone > might > > be able to spot the massive flaw in the way I've been trying to set > things > > up. > > > > We currently run Solr 4.0.0 in the old style Master/Slave replication. We > > have five slaves, each running Centos with 96GB of RAM, 24 cores and with > > 48GB assigned to the JVM heap. Disks aren't crazy fast (i.e. not SSDs) > but > > aren't slow either. Our GC parameters aren't particularly exciting, just > > -XX:+UseConcMarkSweepGC. Java version is 1.7.0_11. > > > > Our index size ranges between 144GB and 200GB (when we optimise it back > > down, since we've had bad experiences with large cores). We've got just > > over 37M documents some are smallish but most range between 1000-6000 > > bytes. We regularly update documents so large portions of the index will > be > > touched leading to a maxDocs value of around 43M. > > > > Query load ranges between 400req/s to 800req/s across the five slaves > > throughout the day, increasing and decreasing gradually over a period of > > hours, rather than bursting. > > > > Most of our documents have upwards of twenty fields. We use different > > fields to store territory variant (we have around 30 territories) values > > and also boost based on the values in some of these fields (integer > ones). > > > > So an average query can do a range filter by two of the territory variant > > fields, filter by a non-territory variant field. Facet by a field or two > > (may be territory variant). Bring back the values of 60 fields. Boost > query > > on field values of a non-territory variant field. Boost by values of two > > territory-variant fields. Dismax query on up to 20 fields (with boosts) > and > > phrase boost on those fields too. They're pretty big queries. We don't do > > any index-time boosting. We try to keep things dynamic so we can alter > our > > boosts on-the-fly. > > > > Another common query is to list documents with a given set of IDs and > > select documents with a common reference and order them by one of their > > fields. > > > > Auto-commit every 30 minutes. Replication polls every 30 minutes. > > > > Document cache: > > * initialSize - 32768 > > * size - 32768 > > > > Filter cache: > > * autowarmCount - 128 > > * initialSize - 8192 > > * size - 8192 > > > > Query result cache: > > * autowarmCount - 128 > > * initialSize - 8192 > > * size - 8192 > > > > After a replicated core has finished downloading (probably while it's > > warming) we see requests which usually take around 100ms taking over 5s. > GC > > logs show concurrent mode failure. > > > > I was wondering whether anyone can help with sizing the boxes required to > > split this index down into shards for use with SolrCloud and roughly how > > much memory we should be assigning to the JVM. Everything I've read > > suggests that running with a 48GB heap is way too high but every attempt > > I've made to reduce the cache sizes seems to wind up causing > out-of-memory > > problems. Even dropping all cache sizes by 50% and reducing the heap by > 50% > > caused problems. > > > > I've already tried using SolrCloud 10 shards (around 3.7M documents per > > shard, each with one replica) and kept the cache sizes low: > > > > Document cache: > > * initialSize - 1024 > > * size - 1024 > > > > Filter cache: > > * autowarmCount - 128 > > * initialSize - 512 > > * size - 512 > > > > Query result cache: > > * autowarmCount - 32 > > * initialSize - 128 > > * size - 128 > > > > Even when running on six machines in AWS with SSDs, 24GB heap (out of > 60GB > > memory) and four shards on two boxes and three on the rest I still see > > concurrent mode failure. This looks like it's causing ZooKeeper to mark > the > > node as down and things begin to struggle. > > > > Is concurrent mode failure just something that will inevitably happen or > is > > it avoidable by dropping the CMSInitiatingOccupancyFraction? > > > > If anyone has anything that might shove me in the right direction I'd be > > very grateful. I'm wondering whether our set-up will just never work and > > maybe we're expecting too much. > > > > Many thanks, > > > > Neil >
