Re: Solr/Lucene keeps eating up memory while idling

2009-10-15 Thread Grant Ingersoll 


On Oct 14, 2009, at 12:26 PM, nonrenewable wrote:



I'm curious why this is occurring and whether i can prevent it. This  
is my

scenario:

Locally I have an idle running solr 1.3 service using lucene 2.4.1  
which has
an index of ~330K documents containing ~10 fields each(total size  
~12GB).


Did I read that right?  330K docs == 12 GB index.

Currently I've turned off all caching, lazy field loading, however i  
do have

facet fields set for some request handlers.

What i'm seeing is heap space usage increasing by ~1.2MB per 2 sec (by
java.lang.String objects). I'm assuming they're being used by lucene  
but i
may be wrong about that, since i have no actual data to confirm it.  
Why

exactly is this happening, considering no requests are being serviced?
Shouldn't the memory usage stabilise with a certain set of  
information and
only be affected on requests? Additionally there is a full GC every  
half
hour, which seems very unreasonable on a machine that isn't actually  
being

used as a service.


Can you share the Solr logs and/or your config?  Is this happening  
around a commit or some warming process?  After startup, with no  
requests hitting it and no warming/commits/indexing, I don't see why  
it would be growing.  Do you have custom code?





I really hope there's just a certain setting that i've overlooked,  
or a
concept i'm not understanding because otherwise this behaviour seems  
very

unreasonable...

Thanks beforehand,
Tony
--
View this message in context: 
http://www.nabble.com/Solr-Lucene-keeps-eating-up-memory-while-idling-tp25894357p25894357.html
Sent from the Solr - User mailing list archive at Nabble.com.



--
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)  
using Solr/Lucene:

http://www.lucidimagination.com/search

Re: Solr/Lucene keeps eating up memory while idling

2009-10-15 Thread nonrenewable 
/lst
 /requestHandler
  
  updateRequestProcessor
factory name=standard class=solr.ChainedUpdateProcessorFactory
default=true
  chain
class=com.pjaol.search.solr.update.LocalUpdateProcessorFactory
str name=latFieldlat/str
str name=lngFieldlng/str
int name=startTier9/int
int name=endTier17/int
  /chain
  chain class=solr.LogUpdateProcessorFactory 
   /chain
  chain class=solr.RunUpdateProcessorFactory /
/factory
  /updateRequestProcessor

  requestHandler name=geo
class=com.pjaol.search.solr.LocalSolrRequestHandler
  str name=latFieldlat/str
 str name=lngFieldlng/str
  /requestHandler
 
  searchComponent name=spellcheck class=solr.SpellCheckComponent

str name=queryAnalyzerFieldTypetextSpell/str

lst name=spellchecker
  str name=namedefault/str
  str name=fieldspell/str
  str name=spellcheckIndexDir./spellchecker1/str

/lst
lst name=spellchecker
  str name=namejarowinkler/str
  str name=fieldspell/str
   str
name=distanceMeasureorg.apache.lucene.search.spell.JaroWinklerDistance/str
  str name=spellcheckIndexDir./spellchecker2/str

/lst
 /searchComponent

   requestHandler name=/spellCheckCompRH class=solr.SearchHandler
lst name=defaults
   str name=spellcheck.onlyMorePopularfalse/str
   str name=spellcheck.extendedResultsfalse/str
   str name=spellcheck.count1/str
/lst
arr name=last-components
  strspellcheck/str
/arr
  /requestHandler
 
  searchComponent name=elevator class=solr.QueryElevationComponent 
 str name=queryFieldTypestring/str
str name=config-fileelevate.xml/str
  /searchComponent
 
   requestHandler name=/elevate class=solr.SearchHandler
startup=lazy
lst name=defaults
  str name=echoParamsexplicit/str
/lst
arr name=last-components
  strelevator/str
/arr
  /requestHandler
  

  requestHandler name=/update class=solr.XmlUpdateRequestHandler /
 requestHandler name=/analysis class=solr.AnalysisRequestHandler /
  

  requestHandler name=/update/csv class=solr.CSVRequestHandler
startup=lazy /


  requestHandler name=/admin/
class=org.apache.solr.handler.admin.AdminHandlers /
  
   requestHandler name=/admin/ping class=PingRequestHandler
lst name=defaults
  str name=qtstandard/str
  str name=qsolrpingquery/str
  str name=echoParamsall/str
/lst
  /requestHandler

  requestHandler name=/debug/dump class=solr.DumpRequestHandler 
lst name=defaults
 str name=echoParamsexplicit/str 
 str name=echoHandlertrue/str
/lst
  /requestHandler
  
  highlighting
   fragmenter name=gap class=org.apache.solr.highlight.GapFragmenter
default=true
lst name=defaults
 int name=hl.fragsize100/int
/lst
   /fragmenter

   fragmenter name=regex
class=org.apache.solr.highlight.RegexFragmenter
lst name=defaults
   int name=hl.fragsize70/int
   float name=hl.regex.slop0.5/float 
   str name=hl.regex.pattern[-\w ,/\n\']{20,200}/str
/lst
   /fragmenter
   
   formatter name=html class=org.apache.solr.highlight.HtmlFormatter
default=true
lst name=defaults
 str name=hl.simple.pre![CDATA[em]]/str
 str name=hl.simple.post![CDATA[/em]]/str
/lst
   /formatter
  /highlighting
  
  queryResponseWriter name=xslt
class=org.apache.solr.request.XSLTResponseWriter
int name=xsltCacheLifetimeSeconds5/int
  /queryResponseWriter 

  admin
defaultQuerysolr/defaultQuery

  /admin

/config
-- 
View this message in context: 
http://www.nabble.com/Solr-Lucene-keeps-eating-up-memory-while-idling-tp25894357p25912727.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr/Lucene keeps eating up memory while idling

2009-10-15 Thread Grant Ingersoll 
=bqincubationdate_dt:[* TO NOW/DAY-1MONTH]^2.2/str
   /lst
   lst name=appends
 str name=fqinStock:true/str
   /lst
/requestHandler

 updateRequestProcessor
   factory name=standard class=solr.ChainedUpdateProcessorFactory
default=true
 chain
class=com.pjaol.search.solr.update.LocalUpdateProcessorFactory
   str name=latFieldlat/str
   str name=lngFieldlng/str
   int name=startTier9/int
   int name=endTier17/int
 /chain
 chain class=solr.LogUpdateProcessorFactory 
  /chain
 chain class=solr.RunUpdateProcessorFactory /
   /factory
 /updateRequestProcessor

 requestHandler name=geo
class=com.pjaol.search.solr.LocalSolrRequestHandler
 str name=latFieldlat/str
str name=lngFieldlng/str
 /requestHandler

 searchComponent name=spellcheck class=solr.SpellCheckComponent

   str name=queryAnalyzerFieldTypetextSpell/str

   lst name=spellchecker
 str name=namedefault/str
 str name=fieldspell/str
 str name=spellcheckIndexDir./spellchecker1/str

   /lst
   lst name=spellchecker
 str name=namejarowinkler/str
 str name=fieldspell/str
  str
name 
= 
distanceMeasure 
org.apache.lucene.search.spell.JaroWinklerDistance/str

 str name=spellcheckIndexDir./spellchecker2/str

   /lst
/searchComponent

  requestHandler name=/spellCheckCompRH class=solr.SearchHandler
   lst name=defaults
  str name=spellcheck.onlyMorePopularfalse/str
  str name=spellcheck.extendedResultsfalse/str
  str name=spellcheck.count1/str
   /lst
   arr name=last-components
 strspellcheck/str
   /arr
 /requestHandler

 searchComponent name=elevator  
class=solr.QueryElevationComponent 

str name=queryFieldTypestring/str
   str name=config-fileelevate.xml/str
 /searchComponent

  requestHandler name=/elevate class=solr.SearchHandler
startup=lazy
   lst name=defaults
 str name=echoParamsexplicit/str
   /lst
   arr name=last-components
 strelevator/str
   /arr
 /requestHandler


 requestHandler name=/update  
class=solr.XmlUpdateRequestHandler /
requestHandler name=/analysis  
class=solr.AnalysisRequestHandler /



 requestHandler name=/update/csv class=solr.CSVRequestHandler
startup=lazy /


 requestHandler name=/admin/
class=org.apache.solr.handler.admin.AdminHandlers /

  requestHandler name=/admin/ping class=PingRequestHandler
   lst name=defaults
 str name=qtstandard/str
 str name=qsolrpingquery/str
 str name=echoParamsall/str
   /lst
 /requestHandler

 requestHandler name=/debug/dump class=solr.DumpRequestHandler 
   lst name=defaults
str name=echoParamsexplicit/str
str name=echoHandlertrue/str
   /lst
 /requestHandler

 highlighting
  fragmenter name=gap  
class=org.apache.solr.highlight.GapFragmenter

default=true
   lst name=defaults
int name=hl.fragsize100/int
   /lst
  /fragmenter

  fragmenter name=regex
class=org.apache.solr.highlight.RegexFragmenter
   lst name=defaults
  int name=hl.fragsize70/int
  float name=hl.regex.slop0.5/float
  str name=hl.regex.pattern[-\w ,/\n\']{20,200}/str
   /lst
  /fragmenter

  formatter name=html  
class=org.apache.solr.highlight.HtmlFormatter

default=true
   lst name=defaults
str name=hl.simple.pre![CDATA[em]]/str
str name=hl.simple.post![CDATA[/em]]/str
   /lst
  /formatter
 /highlighting

 queryResponseWriter name=xslt
class=org.apache.solr.request.XSLTResponseWriter
   int name=xsltCacheLifetimeSeconds5/int
 /queryResponseWriter

 admin
   defaultQuerysolr/defaultQuery

 /admin

/config
--
View this message in context: 
http://www.nabble.com/Solr-Lucene-keeps-eating-up-memory-while-idling-tp25894357p25912727.html
Sent from the Solr - User mailing list archive at Nabble.com.



--
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)  
using Solr/Lucene:

http://www.lucidimagination.com/search

Re: Solr/Lucene keeps eating up memory while idling

2009-10-15 Thread nonrenewable 

Here is exactly half an hour from roughly the beginning of logging. There's
nothing to see really because no requests are sent, you just see the GC
behaviour:
[Full GC 211987K-208493K(432448K), 0.6273480 secs]
[GC 276333K-212269K(438720K), 0.0929710 secs]
[GC 289133K-216269K(439936K), 0.1019780 secs]
[GC 293133K-220205K(436672K), 0.1128410 secs]
[GC 304301K-224429K(441472K), 0.1358250 secs]
[GC 308525K-228685K(431744K), 0.1559950 secs]
[GC 317197K-233069K(437312K), 0.1642160 secs]
[GC 321581K-237613K(432832K), 0.1772830 secs]
[GC 329197K-242093K(435136K), 0.1896270 secs]
[GC 333677K-246701K(436352K), 0.2039880 secs]
[GC 274165K-247917K(437760K), 0.2022640 secs]
[Full GC 247917K-208726K(437760K), 0.7195200 secs]

The heap is set to 1400m so it'll take it awhile to hit the roof. I also
haven't tested to see if it stabilises but i'll leave it running now and see
what happens to it overnight. I assume that when(if) it reaches the heap
limit i'll just do full GCs more often. 


Grant Ingersoll-6 wrote:
 
 Please send a log covering at least the 2.5 minutes you discuss, but  
 upwards of 5 minutes would be good.
 

-- 
View this message in context: 
http://www.nabble.com/Solr-Lucene-keeps-eating-up-memory-while-idling-tp25894357p25916348.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr/Lucene keeps eating up memory while idling

2009-10-15 Thread Yonik Seeley 
I just did some allocation profiling on the stock Solr example... it's
not completely idle when no requests are being made.

There's only one thing allocating memory: org.mortbay.util.Scanner.scanFiles()
That must be Jetty looking to see if any of the files under webapps has changed.

It's really nothing to worry about - there's no memory leaks, and the
activity is extremely minimal, but if you want to shut it off, it
would be a Jetty config option somewhere.

-Yonik
http://www.lucidimagination.com



On Wed, Oct 14, 2009 at 12:26 PM, nonrenewable nonrenewa...@gmail.com wrote:

 I'm curious why this is occurring and whether i can prevent it. This is my
 scenario:

 Locally I have an idle running solr 1.3 service using lucene 2.4.1 which has
 an index of ~330K documents containing ~10 fields each(total size ~12GB).
 Currently I've turned off all caching, lazy field loading, however i do have
 facet fields set for some request handlers.

 What i'm seeing is heap space usage increasing by ~1.2MB per 2 sec (by
 java.lang.String objects). I'm assuming they're being used by lucene but i
 may be wrong about that, since i have no actual data to confirm it. Why
 exactly is this happening, considering no requests are being serviced?
 Shouldn't the memory usage stabilise with a certain set of information and
 only be affected on requests? Additionally there is a full GC every half
 hour, which seems very unreasonable on a machine that isn't actually being
 used as a service.

 I really hope there's just a certain setting that i've overlooked, or a
 concept i'm not understanding because otherwise this behaviour seems very
 unreasonable...

 Thanks beforehand,
 Tony
 --
 View this message in context: 
 http://www.nabble.com/Solr-Lucene-keeps-eating-up-memory-while-idling-tp25894357p25894357.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Solr/Lucene keeps eating up memory while idling

2009-10-14 Thread nonrenewable 

I'm curious why this is occurring and whether i can prevent it. This is my
scenario:

Locally I have an idle running solr 1.3 service using lucene 2.4.1 which has
an index of ~330K documents containing ~10 fields each(total size ~12GB).
Currently I've turned off all caching, lazy field loading, however i do have
facet fields set for some request handlers. 

What i'm seeing is heap space usage increasing by ~1.2MB per 2 sec (by
java.lang.String objects). I'm assuming they're being used by lucene but i
may be wrong about that, since i have no actual data to confirm it. Why
exactly is this happening, considering no requests are being serviced?
Shouldn't the memory usage stabilise with a certain set of information and
only be affected on requests? Additionally there is a full GC every half
hour, which seems very unreasonable on a machine that isn't actually being
used as a service. 

I really hope there's just a certain setting that i've overlooked, or a
concept i'm not understanding because otherwise this behaviour seems very
unreasonable...

Thanks beforehand,
Tony
-- 
View this message in context: 
http://www.nabble.com/Solr-Lucene-keeps-eating-up-memory-while-idling-tp25894357p25894357.html
Sent from the Solr - User mailing list archive at Nabble.com.