Re: Lucene 6.6: "Too many open files"

Hi Varun, Thanks, I will try that and open the jira. Btw, is it possible (and reasonable?) to create large segments during bulk-indexing (without interleaved commits) and then merge them into fewer (e.g. 40) segments when committing at the end? So far my feeling (without logging enabled) is that

Re: Lucene 6.6: "Too many open files"

HI Nawab, I don't see a way to turn it of directly from Solr. Can you please create a Jira for this? In your case do you see evidence why you need to turn it off? I'd suggest turning on Lucene level info stream logging by enabling then in the resources/log4j.properties file . Index the documents

Re: Lucene 6.6: "Too many open files"

varun: How do you set auto-IOThrottle? Is there a way to turn it off? I don't see a setting for it. Nawab On Fri, Aug 4, 2017 at 9:49 AM, Varun Thacker wrote: > I just ran into this issue yesterday and a couple of times last week. > > The team I was working with was

Re: Lucene 6.6: "Too many open files"

I just ran into this issue yesterday and a couple of times last week. The team I was working with was running Solr 5.4.1 so the auto-throttle merge policy applies. We were using the default merge policy. When infoStream was turned on for a dummy collection on the same box the output for the

Re: Lucene 6.6: "Too many open files"

Hi, It's documented in the Javadocs of ConcurrentMergeScheduler. It depends on the number of CPUs (with some upper bound) and if the index is on SSD. Without SSD it uses only one thread for merging. Uwe Am 2. August 2017 22:01:51 MESZ schrieb Nawab Zada Asad Iqbal : >Thanks

Re: Lucene 6.6: "Too many open files"

Thanks Uwe, That worked actually. After running for 3 hours, I observed about 88% of indexing rate as compared to 4.5.0 without any file descriptor issues. It seems that I can probably do some tweaking to get same throughput as before. I looked at the code and the default values for

Re: Lucene 6.6: "Too many open files"

Hi, You should reset those settings back to defaults (remove the inner settings in the factory). 30 merge threads will eat up all your file handles. In earlier versions of Lucene, internal limitations in IndexWriter make it unlikely, that you spawn too many threads, so 30 had no effect. In

Re: Lucene 6.6: "Too many open files"

Thanks Jigar I haven't tweaked ConcurrentMergeStrategy between 4.5.0 and 6.6. Here is what I have: 30 30 On Mon, Jul 31, 2017 at 8:56 PM, Jigar Shah wrote: > I faced such problem when I used nomergepolicy, and did some code to manual > merging segments which had

Re: Lucene 6.6: "Too many open files"

Hi Erick, I am committing once after all the indexing finishes. This is a batch indexing scenario to prepare new servers or replicas, and we don't use this config for everyday operations. While batch indexing, we also modify code to skip IndexWriter.maybeApplyDeletes to optimize the commit. I

Re: Lucene 6.6: "Too many open files"

I faced such problem when I used nomergepolicy, and did some code to manual merging segments which had bug and I had same issue. Make sure you have default AFAIR ConcurrentMergeStrategy enabled. And its is configured with appropriate settings. On Jul 31, 2017 11:21 PM, "Erick Erickson"

Re: Lucene 6.6: "Too many open files"

No, nothing's changed fundamentally. But you say: "We have some batch indexing scripts, which flood the solr servers with indexing requests (while keeping open-searcher false)" What is your commit interval? Regardless of whether openSearcher is false or not, background merging continues apace

Lucene 6.6: "Too many open files"

Hi, I am upgrading from solr4.5 to solr6.6 and hitting this issue during complete reindexing scenario. We have some batch indexing scripts, which flood the solr servers with indexing requests (while keeping open-searcher false) for many hours and then perform one commit. This used to work fine