On Tue, Oct 20, 2009 at 11:57 PM, Shalin Shekhar Mangar < shalinman...@gmail.com> wrote:
> On Tue, Oct 20, 2009 at 3:56 PM, Mark Miller <markrmil...@gmail.com> > wrote: > > > > > > > On Oct 20, 2009, at 12:12 AM, Shalin Shekhar Mangar < > > shalinman...@gmail.com> wrote: > > > > I don't think the debate is about weak reference vs. soft references. > >> > > > > There appears to be confusion between the two here no matter what the > > debate - soft references are for cachinh, weak references are not so > much. > > Getting it right is important. > > > > I > >> guess the point that Lance is making is that using such a technique will > >> make application performance less predictable. There's also a good > chance > >> that a soft reference based cache will cause cache thrashing and will > hide > >> OOMs caused by inadequate cache sizes. So basically we trade an OOM for > >> more > >> CPU usage (due to re-computation of results). > >> > > > > That's the whole point. Your not hiding anything. I don't follow you. > > > > Using a soft reference based cache can hide the fact that one has > inadequate > memory for the cache size one has configured. Don't get me wrong. I'm not > against the feature. I was merely trying to explain Lance's concerns as I > understood them. > Lance concern is valid. Assuming that we are going to have this feature (non-default) we need a way to know that cache trashing has happened.I mean the statistics should also expose the no:of cache entries which got removed. This should enable the user to decide whether there should be more RAM or he is happy to live w/ the extra cpu cycles for recomputation > > > > > > > > > >> Personally, I think giving an option is fine. What if the user does not > >> have > >> enough RAM and he is willing to pay the price? Right now, there is no > way > >> he > >> can do that at all. However, the most frequent reason behind OOMs is not > >> having enough RAM to create the field caches and not Solr caches, so I'm > >> not > >> sure how important this is. > >> > > > > How important is any feature? You don't have a use for it, so it's not > > important to you - someone else does so it is important to them. Soft > value > > caches can be useful. > > > Don't jump to conclusions :) > > The reason behind this feature request is to have Solr caches which resize > themselves when enough memory is not available. I agree that soft value > caches are useful for this. All I'm saying is that most OOMs that get > reported on the list are due to inadequate free memory for allocating field > caches. Finding a way around that will be the key to make a Lucene/Solr > application practical in a limited memory environment. > > Just for the record, I'm +1 for adding this feature but keeping the current > behavior as the default. > > -- > Regards, > Shalin Shekhar Mangar. > -- ----------------------------------------------------- Noble Paul | Principal Engineer| AOL | http://aol.com
