RE: Memory problems when highlight with not very big index
Hi Yonik, I've tried to change de documentCache to 60 as you told me but the problem persist. If I set hl=off the memory never pass 65.000KB. But if I set it to on in my first search, using a common word as a, the memory increase until 763000KB. If after this I search for other common word as web the memory grows up until 120KB..and continue increasing with new searchs. I can't understand very well how solr uses the RAM. Should I read how Lucene use the memory or it's not the same management. My solr configuration is: ### requestHandler name=standard class=solr.DisMaxRequestHandler lst name=defaults str name=echoParamsexplicit/str float name=tie0.01/float str name=qfcontent^0.5 title^0.7/str int name=rows10/int str name=flpath,date,format,score,author,title/str str name=version2.1/str str name=hloff/str str name=hl.flcontent/str int name=hl.fragsize50/int int name=hl.snippets2/int - str name=hl.simple.pre ![CDATA[ span class=Highlight ]] /str - str name=hl.simple.post ![CDATA[ /span ]] /str /lst /requestHandler ... filterCache class=solr.LRUCache size=1000 initialSize=500 autowarmCount=0/ queryResultCache class=solr.LRUCache size=640 initialSize=320 autowarmCount=0/ documentCache class=solr.LRUCache size=60 initialSize=60 autowarmCount=0/ ### Any help will be very usefull for me. Thanks for your attention. -Rober -Mensaje original- De: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] En nombre de Yonik Seeley Enviado el: viernes, 13 de junio de 2008 21:48 Para: solr-user@lucene.apache.org Asunto: Re: Memory problems when highlight with not very big index On Fri, Jun 13, 2008 at 3:30 PM, Roberto Nieto [EMAIL PROTECTED] wrote: The part that i can't understand very well is why if i desactivate highlighting the memory doesnt grows. It only uses doc cache if highlighting is used or if content retrieve is activated? Perhaps you are highlighting some fields that you normally don't return? What is fl vs hl.fl? -Yonik
Re: Memory problems when highlight with not very big index
Hi Yonik, I think your are right, it must be that. If i activate the highlighting of a field that i´m not specifing in fl, it will have the same use of RAM as if i return it? Internally it will be as if I add it to fl? 2008/6/13 Yonik Seeley [EMAIL PROTECTED]: On Fri, Jun 13, 2008 at 3:30 PM, Roberto Nieto [EMAIL PROTECTED] wrote: The part that i can't understand very well is why if i desactivate highlighting the memory doesnt grows. It only uses doc cache if highlighting is used or if content retrieve is activated? Perhaps you are highlighting some fields that you normally don't return? What is fl vs hl.fl? -Yonik
Memory problems when highlight with not very big index
Hi users/developers, I´m new with solr and i have been reading the list for a few hours but i didn´t found anything to solve my doubt. I´m using 5GB index in a 2GB RAM maquine, and i´m trying to optimize the solr configuration for searching. I´ve have good searching times but when i activate highlighting the RAM memory grows a lot, it grows the same as if a want to retrieve the content of the files found. I´m not sure if for highlighting solr needs to allocate all the content of the resulting documents to be able to highlight them. How it works? It´s possible to only allocate the 10 first results to make the snippet of only those results and use less memory? Thanks in advance. Rober.
Re: Memory problems when highlight with not very big index
On Fri, Jun 13, 2008 at 1:07 PM, Roberto Nieto [EMAIL PROTECTED] wrote: It´s possible to only allocate the 10 first results to make the snippet of only those results and use less memory? That's how it currently works. But there is a Document cache to make things more efficient. If you have large documents, you might want to decrease this from it's default size (see solrconfig.xml) which is currently 512. Perhaps move it down to 60 (which would allow for 6 concurrent requests of 10 docs each w/o re-fetching the doc between highlighting and response writing). -Yonik
Re: Memory problems when highlight with not very big index
Thanks for your fast answer, I think i tried to put default size to 0 and the problems persist but i will probe it on Monday again. The part that i can't understand very well is why if i desactivate highlighting the memory doesnt grows. It only uses doc cache if highlighting is used or if content retrieve is activated? Thnx Rober. 2008/6/13 Yonik Seeley [EMAIL PROTECTED]: On Fri, Jun 13, 2008 at 1:07 PM, Roberto Nieto [EMAIL PROTECTED] wrote: It´s possible to only allocate the 10 first results to make the snippet of only those results and use less memory? That's how it currently works. But there is a Document cache to make things more efficient. If you have large documents, you might want to decrease this from it's default size (see solrconfig.xml) which is currently 512. Perhaps move it down to 60 (which would allow for 6 concurrent requests of 10 docs each w/o re-fetching the doc between highlighting and response writing). -Yonik
Re: Memory problems when highlight with not very big index
On Fri, Jun 13, 2008 at 3:30 PM, Roberto Nieto [EMAIL PROTECTED] wrote: The part that i can't understand very well is why if i desactivate highlighting the memory doesnt grows. It only uses doc cache if highlighting is used or if content retrieve is activated? Perhaps you are highlighting some fields that you normally don't return? What is fl vs hl.fl? -Yonik
