Re: solr indexing slows down after few minutes
Commit is not too often, it's a batch of 100 records, takes 40 to 60 secs before another commit. No I am not indexing with multi threads. It uses a single thread executor. I have seen steady performance for now after increasing the merge factor from 10 to 25. Will have to wait and watch if that reduces the search speed, but so far so good. Thanks Amit On Thu, Aug 30, 2012 at 10:53 PM, pravesh [via Lucene] < ml-node+s472066n4004421...@n3.nabble.com> wrote: > Did you checked wiki: > http://wiki.apache.org/lucene-java/ImproveIndexingSpeed > > Do you commit often? Do you index with multiple threads? Also try > experimenting with various available MergePolicies introduced from SOLR 3.4 > onwards > > Thanx > Pravesh > > -- > If you reply to this email, your message will be added to the discussion > below: > > http://lucene.472066.n3.nabble.com/solr-indexing-slows-down-after-few-minutes-tp4004337p4004421.html > To unsubscribe from solr indexing slows down after few minutes, click > here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4004337&code=YW1pdC5tYWxsaWtAZ21haWwuY29tfDQwMDQzMzd8LTk5Njc5OTA3NA==> > . > NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://lucene.472066.n3.nabble.com/solr-indexing-slows-down-after-few-minutes-tp4004337p4005864.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr indexing slows down after few minutes
Did you checked wiki: http://wiki.apache.org/lucene-java/ImproveIndexingSpeed http://wiki.apache.org/lucene-java/ImproveIndexingSpeed Do you commit often? Do you index with multiple threads? Also try experimenting with various available MergePolicies introduced from SOLR 3.4 onwards Thanx Pravesh -- View this message in context: http://lucene.472066.n3.nabble.com/solr-indexing-slows-down-after-few-minutes-tp4004337p4004421.html Sent from the Solr - User mailing list archive at Nabble.com.
solr indexing slows down after few minutes
Hi I am indexing our website using solr. After crawling the website I am adding the contents to the solr server and committing. My slor server is on 3.6 version. I observed that initially the indexing is very fast. It added around 33 docs/sec; making 2000 / minute. But then after the 30 mins or so the performance reduces drastically. Within an hour of indexing it is adding 200 docs/min which is 1/10 times what it used to add. Here is the details the time and the total docs in slor. TimeDocs in solr (min) 3.480 3.5410,000 3.5816,000 4.0324,000 4:0830,000 4:1538,000 4:1841,000 4:2542,230 4:3143,140 4:3844,410 This is after making the name field stored="false". Otherwise the same behavior can be duplicated much before. My doc has 3 fields and the name field is the extracted text from the webpages. Others are too small Not sure what is wrong. I am using a win 7 machine dual core and 4 GB ram. Thanks Amit -- View this message in context: http://lucene.472066.n3.nabble.com/solr-indexing-slows-down-after-few-minutes-tp4004337.html Sent from the Solr - User mailing list archive at Nabble.com.