Solrcloud (4.10) reports the end of soft commit before all shard replicas finished committing

2015-10-22 Thread vsolakhian
We have a strange behavior of our Sorlcloud related code after upgrading from from Solr 4.4 to Solr 4.10 (as part of upgrading from Cloudera CDH 4.6 to Cloudera CDH 5.4.5). We have a Solrcloud collection with three replicas of one shard. Our code does batch indexing, then submits a soft commit

Re: Very Slow Commits After Solr Index Optimization

2016-09-22 Thread vsolakhian
Thanks again, Shawn. You are completely right about the use of disk cache and the special note regarding the optimize operation in Solr wiki. This is not the cause of the problem though. The disk cache is important for queries and overall performance during optimization, but once it is done,

Re: Very Slow Commits After Solr Index Optimization

2016-09-22 Thread vsolakhian
Hi Shawn, Thank you for response. Everything you said is correct in general. Our index is in HDFS, but we did not change any configuration after we deleted 35% of records and optimized. The relatively slow commit (soft commit and warming up took 1.5 minutes) is OK for our use case (adding

Very Slow Commits After Solr Index Optimization

2016-09-20 Thread vsolakhian
We are using Solr Cloud 4.10.3-cdh5.4.5 that is part of CLoudera CDH 5.4.5. Our collection (one shard with three replicas) became really big and we decided to delete some old records to improve performance (tests in staging environment have shown that after reaching 500 million records the index