Yeah, I'm not sure why this would help. There should be nothing in FieldCaches unless you sort or use facets.
Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch ----- Original Message ---- > From: vivek sar <vivex...@gmail.com> > To: solr-user@lucene.apache.org > Sent: Wednesday, May 13, 2009 5:53:45 PM > Subject: Re: Solr memory requirements? > > Just an update on the memory issue - might be useful for others. I > read the following, > > http://wiki.apache.org/solr/SolrCaching?highlight=(SolrCaching) > > and looks like the first and new searcher listeners would populate the > FieldCache. Commenting out these two listener entries seems to do the > trick - at least the heap size is not growing as soon as Solr starts > up. > > I ran some searches and they all came out fine. Index rate is also > pretty good. Would there be any impact of disabling these listeners? > > Thanks, > -vivek > > On Wed, May 13, 2009 at 2:12 PM, vivek sar wrote: > > Otis, > > > > In that case, I'm not sure why Solr is taking up so much memory as > > soon as we start it up. I checked for .tii file and there is only one, > > > > -rw-r--r-- 1 search staff 20306 May 11 21:47 > ./20090510_1/data/index/_3au.tii > > > > I have all the cache disabled - so that shouldn't be a problem too. My > > ramBuffer size is only 64MB. > > > > I read note on sorting, > > http://wiki.apache.org/solr/SchemaDesign?highlight=(sort), and see > > something related to FieldCache. I don't see this as parameter defined > > in either solrconfig.xml or schema.xml. Could this be something that > > can load things in memory at startup? How can we disable it? > > > > I'm trying to find out if there is a way to tell how much memory Solr > > would consume and way to cap it. > > > > Thanks, > > -vivek > > > > > > > > > > On Wed, May 13, 2009 at 1:50 PM, Otis Gospodnetic > > wrote: > >> > >> Hi, > >> > >> Sorting is triggered by the sort parameter in the URL, not a > >> characteristic > of a field. :) > >> > >> Otis > >> -- > >> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > >> > >> > >> > >> ----- Original Message ---- > >>> From: vivek sar > >>> To: solr-user@lucene.apache.org > >>> Sent: Wednesday, May 13, 2009 4:42:16 PM > >>> Subject: Re: Solr memory requirements? > >>> > >>> Thanks Otis. > >>> > >>> Our use case doesn't require any sorting or faceting. I'm wondering if > >>> I've configured anything wrong. > >>> > >>> I got total of 25 fields (15 are indexed and stored, other 10 are just > >>> stored). All my fields are basic data type - which I thought are not > >>> sorted. My id field is unique key. > >>> > >>> Is there any field here that might be getting sorted? > >>> > >>> > >>> required="true" omitNorms="true" compressed="false"/> > >>> > >>> > >>> compressed="false"/> > >>> > >>> omitNorms="true" compressed="false"/> > >>> > >>> omitNorms="true" compressed="false"/> > >>> > >>> omitNorms="true" compressed="false"/> > >>> > >>> default="NOW/HOUR" compressed="false"/> > >>> > >>> omitNorms="true" compressed="false"/> > >>> > >>> omitNorms="true" compressed="false"/> > >>> > >>> compressed="false"/> > >>> > >>> compressed="false"/> > >>> > >>> omitNorms="true" compressed="false"/> > >>> > >>> omitNorms="true" compressed="false"/> > >>> > >>> omitNorms="true" compressed="false"/> > >>> > >>> omitNorms="true" compressed="false"/> > >>> > >>> omitNorms="true" compressed="false"/> > >>> > >>> compressed="false"/> > >>> > >>> compressed="false"/> > >>> > >>> compressed="false"/> > >>> > >>> omitNorms="true" compressed="false"/> > >>> > >>> compressed="false"/> > >>> > >>> default="NOW/HOUR" omitNorms="true"/> > >>> > >>> > >>> > >>> > >>> omitNorms="true" multiValued="true"/> > >>> > >>> Thanks, > >>> -vivek > >>> > >>> On Wed, May 13, 2009 at 1:10 PM, Otis Gospodnetic > >>> wrote: > >>> > > >>> > Hi, > >>> > Some answers: > >>> > 1) .tii files in the Lucene index. When you sort, all distinct values > >>> > for > the > >>> field(s) used for sorting. Similarly for facet fields. Solr caches. > >>> > 2) ramBufferSizeMB dictates, more or less, how much Lucene/Solr will > consume > >>> during indexing. There is no need to commit every 50K docs unless you > >>> want > to > >>> trigger snapshot creation. > >>> > 3) see 1) above > >>> > > >>> > 1.5 billion docs per instance where each doc is cca 1KB? I doubt > >>> > that's > going > >>> to fly. :) > >>> > > >>> > Otis > >>> > -- > >>> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > >>> > > >>> > > >>> > > >>> > ----- Original Message ---- > >>> >> From: vivek sar > >>> >> To: solr-user@lucene.apache.org > >>> >> Sent: Wednesday, May 13, 2009 3:04:46 PM > >>> >> Subject: Solr memory requirements? > >>> >> > >>> >> Hi, > >>> >> > >>> >> I'm pretty sure this has been asked before, but I couldn't find a > >>> >> complete answer in the forum archive. Here are my questions, > >>> >> > >>> >> 1) When solr starts up what does it loads up in the memory? Let's say > >>> >> I've 4 cores with each core 50G in size. When Solr comes up how much > >>> >> of it would be loaded in memory? > >>> >> > >>> >> 2) How much memory is required during index time? If I'm committing > >>> >> 50K records at a time (1 record = 1KB) using solrj, how much memory do > >>> >> I need to give to Solr. > >>> >> > >>> >> 3) Is there a minimum memory requirement by Solr to maintain a certain > >>> >> size index? Is there any benchmark on this? > >>> >> > >>> >> Here are some of my configuration from solrconfig.xml, > >>> >> > >>> >> 1) 64 > >>> >> 2) All the caches (under query tag) are commented out > >>> >> 3) Few others, > >>> >> a) true ==> > >>> >> would this require memory? > >>> >> b) 50 > >>> >> c) 200 > >>> >> d) > >>> >> e) false > >>> >> f) 2 > >>> >> > >>> >> The problem we are having is following, > >>> >> > >>> >> I've given Solr RAM of 6G. As the total index size (all cores > >>> >> combined) start growing the Solr memory consumption goes up. With 800 > >>> >> million documents, I see Solr already taking up all the memory at > >>> >> startup. After that the commits, searches everything become slow. We > >>> >> will be having distributed setup with multiple Solr instances (around > >>> >> 8) on four boxes, but our requirement is to have each Solr instance at > >>> >> least maintain around 1.5 billion documents. > >>> >> > >>> >> We are trying to see if we can somehow reduce the Solr memory > >>> >> footprint. If someone can provide a pointer on what parameters affect > >>> >> memory and what effects it has we can then decide whether we want that > >>> >> parameter or not. I'm not sure if there is any minimum Solr > >>> >> requirement for it to be able maintain large indexes. I've used Lucene > >>> >> before and that didn't require anything by default - it used up memory > >>> >> only during index and search times - not otherwise. > >>> >> > >>> >> Any help is very much appreciated. > >>> >> > >>> >> Thanks, > >>> >> -vivek > >>> > > >>> > > >> > >> > >
