Re: Understanding fieldCache SUBREADER insanity

2012-10-02 Thread Aaron Daubman
Hi Yonik, I've been attempting to fix the SUBREADER insanity in our custom component, and have made perhaps some progress (or is this worse?) - I've gone from SUBREADER to VALUEMISMATCH insanity: ---snip--- entries_count : 12 entry#0 :

Re: Understanding fieldCache SUBREADER insanity

2012-09-21 Thread Aaron Daubman
Yonik, et al. I believe I found the section of code pushing me into 'insanity' status: ---snip--- int[] collapseIDs = null; float[] hotnessValues = null; String[] artistIDs = null; try { collapseIDs =

Understanding fieldCache SUBREADER insanity

2012-09-19 Thread Aaron Daubman
Hi all, In reviewing a solr instance with somewhat variable performance, I noticed that its fieldCache stats show an insanity_count of 1 with the insanity type SUBREADER: ---snip--- insanity_count : 1 insanity#0 : SUBREADER: Found caches for descendants of ReadOnlyDirectoryReader(segments_k

Re: Understanding fieldCache SUBREADER insanity

2012-09-19 Thread Tomás Fernández Löbbe
Hi Aaron, here there is some information about the insanity count: http://wiki.apache.org/solr/SolrCaching#The_Lucene_FieldCache As for the SUBREADER type, the javadocs say: Indicates an overlap in cache usage on a given field in sub/super readers. This probably means that you are using the same

Re: Understanding fieldCache SUBREADER insanity

2012-09-19 Thread Aaron Daubman
Hi Tomás, This probably means that you are using the same field for faceting and for sorting (tf_normalizedTotalHotttnesss), sorting uses the segment level cache and faceting uses by default the global field cache. This can be a problem because the field is duplicated in cache, and then it

Re: Understanding fieldCache SUBREADER insanity

2012-09-19 Thread Yonik Seeley
The other thing to realize is that it's only insanity if it's unexpected or not-by-design (so the term is rather mis-named). It's more for core developers - if you are just using Solr without custom plugins, don't worry about it. -Yonik http://lucidworks.com On Wed, Sep 19, 2012 at 3:27 PM,

Re: Understanding fieldCache SUBREADER insanity

2012-09-19 Thread Tomás Fernández Löbbe
Some function queries also use the field cache. I *think* those usually use the segment level cache, but I'm not sure. On Wed, Sep 19, 2012 at 4:36 PM, Yonik Seeley yo...@lucidworks.com wrote: The other thing to realize is that it's only insanity if it's unexpected or not-by-design (so the

Re: Understanding fieldCache SUBREADER insanity

2012-09-19 Thread Yonik Seeley
already-optimized, single-segment index That part is interesting... if true, then the type of insanity you saw should be impossible, and either the insanity detection or something else is broken. -Yonik http://lucidworks.com