Re: Solr 3.6 optimize and field cache question
Not a solution for the short term but sounds like a good use case to migrate to Solr 4.X and use DocValues instead of FieldCache for faceting. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-3-6-optimize-and-field-cache-question-tp4076398p4076822.html Sent from the Solr - User mailing list archive at Nabble.com.
Solr 3.6 optimize and field cache question
Hi, We have Solr 3.6 set up with master and two slaves, each one with 70GB JVM. We run into java.lang.OutOfMemoryError when we cross 250 million documents. Every time this happens we purge documents, bring it below 200 million and bounce both slaves. We have facets on 14 fields. We usually don't run optimize after the purge. Will the deleted documents be part of field cache if we don't run optimize after purge? Will I see difference in java heap memory utilization before and after running optimize? I thought optimize only affects the disk space. Thanks!
RE: Solr 3.6 optimize and field cache question
I'm 99% sure that the deleted docs will indeed use up space in the field cache, at least until the segments that those documents are in are merged - that is what an optimize will do. Of course, these segments will automatically be merged eventually, but it might take days for this to happen, depending on how often your index is updated. -Michael -Original Message- From: Joshi, Shital [mailto:shital.jo...@gs.com] Sent: Monday, July 08, 2013 6:12 PM To: 'solr-user@lucene.apache.org' Subject: Solr 3.6 optimize and field cache question Hi, We have Solr 3.6 set up with master and two slaves, each one with 70GB JVM. We run into java.lang.OutOfMemoryError when we cross 250 million documents. Every time this happens we purge documents, bring it below 200 million and bounce both slaves. We have facets on 14 fields. We usually don't run optimize after the purge. Will the deleted documents be part of field cache if we don't run optimize after purge? Will I see difference in java heap memory utilization before and after running optimize? I thought optimize only affects the disk space. Thanks!
Re: Solr 3.6 optimize and field cache question
Hi, 70 GB heap and still OOMing? H sure, 14 fields for faceting, but still - 70 GB heap! Don't have source handy, but I quickly looked at FC src here - http://search-lucene.com/c/Lucene:core/src/java/org/apache/lucene/search/FieldCache.java - I see mentions of delete there, so I would guess FC is delete-aware. Have you tried just committing after your purge? (purge == delete (by query), I assume) Try that and lets see your heap before/after. Otis -- Solr ElasticSearch Support -- http://sematext.com/ Performance Monitoring -- http://sematext.com/spm On Mon, Jul 8, 2013 at 6:11 PM, Joshi, Shital shital.jo...@gs.com wrote: Hi, We have Solr 3.6 set up with master and two slaves, each one with 70GB JVM. We run into java.lang.OutOfMemoryError when we cross 250 million documents. Every time this happens we purge documents, bring it below 200 million and bounce both slaves. We have facets on 14 fields. We usually don't run optimize after the purge. Will the deleted documents be part of field cache if we don't run optimize after purge? Will I see difference in java heap memory utilization before and after running optimize? I thought optimize only affects the disk space. Thanks!
