Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
I just realized I failed my own reading comprehension :) You have maxDocs, not maxTime for hard commit. Please disregard. On May 30, 2014, at 1:46 PM, Jason Hellman wrote: > I’m also not sure I understand the practical purpose of your hard/soft auto > commit settings. You are stating the following: > > Every 10 seconds I want data written to disk, but not be searchable. > Every 15 seconds I want data to be written into memory and searchable. > > I would consider whether your soft commit window is too long, or if you can > lengthen your hard commit period. It’s typical to see hard commits occur > *less* frequently than soft commits. > > > On May 30, 2014, at 11:04 AM, Shawn Heisey wrote: > >> On 5/29/2014 9:55 PM, M, Arjun (NSN - IN/Bangalore) wrote: >>> Thanks a lot for your nice explanation.. Now I understood the >>> difference between autoCommit and autoSoftCommit.. Now my config looks like >>> below. >>> >>> >>> 1 >>> false >>> >>> >>> >>> 15000 >>> >>> >>> >>> With this now I am getting some other error like this. >>> >>> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: >>> version conflict for 140142167803912812800030383128128 >>> expected=1469497192978841608 actual=1469497212082847746 >> >> This sounds like you are including the _version_ field in your document >> when you index. You probably shouldn't be doing that. Here's what that >> field is for, and how it works: >> >> http://heliosearch.org/solr/optimistic-concurrency/ >> >> Thanks, >> Shawn >> >
Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
I’m also not sure I understand the practical purpose of your hard/soft auto commit settings. You are stating the following: Every 10 seconds I want data written to disk, but not be searchable. Every 15 seconds I want data to be written into memory and searchable. I would consider whether your soft commit window is too long, or if you can lengthen your hard commit period. It’s typical to see hard commits occur *less* frequently than soft commits. On May 30, 2014, at 11:04 AM, Shawn Heisey wrote: > On 5/29/2014 9:55 PM, M, Arjun (NSN - IN/Bangalore) wrote: >> Thanks a lot for your nice explanation.. Now I understood the >> difference between autoCommit and autoSoftCommit.. Now my config looks like >> below. >> >> >> 1 >> false >> >> >> >> 15000 >> >> >> >> With this now I am getting some other error like this. >> >> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: >> version conflict for 140142167803912812800030383128128 >> expected=1469497192978841608 actual=1469497212082847746 > > This sounds like you are including the _version_ field in your document > when you index. You probably shouldn't be doing that. Here's what that > field is for, and how it works: > > http://heliosearch.org/solr/optimistic-concurrency/ > > Thanks, > Shawn >
Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
On 5/29/2014 9:55 PM, M, Arjun (NSN - IN/Bangalore) wrote: > Thanks a lot for your nice explanation.. Now I understood the > difference between autoCommit and autoSoftCommit.. Now my config looks like > below. > > >1 >false > > > >15000 > > > > With this now I am getting some other error like this. > > org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: version > conflict for 140142167803912812800030383128128 > expected=1469497192978841608 actual=1469497212082847746 This sounds like you are including the _version_ field in your document when you index. You probably shouldn't be doing that. Here's what that field is for, and how it works: http://heliosearch.org/solr/optimistic-concurrency/ Thanks, Shawn
RE: Error enquiry- exceeded limit of maxWarmingSearchers=2
Hi Shawn, Thanks a lot for your nice explanation.. Now I understood the difference between autoCommit and autoSoftCommit.. Now my config looks like below. 1 false 15000 With this now I am getting some other error like this. org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: version conflict for 140142167803912812800030383128128 expected=1469497192978841608 actual=1469497212082847746 What could be the reason? Thanks & Regards, Arjun M -Original Message- From: ext Shawn Heisey [mailto:s...@elyograg.org] Sent: Thursday, May 29, 2014 10:14 PM To: solr-user@lucene.apache.org Subject: Re: Error enquiry- exceeded limit of maxWarmingSearchers=2 On 5/29/2014 7:52 AM, M, Arjun (NSN - IN/Bangalore) wrote: > Thanks Shawn... Just one more question.. > > Can both autoCommit and autoSoftCommit be enabled? If both are enabled, which > one takes precedence? Yes, and it's a very common configuration. If you do enable both, you want openSearcher to be false on autoCommit, so that your hard commits are not making documents visible. That is a job for autoSoftCommit. If you use openSearcher=false on autoCommit, then the question of which one takes precendence actually has no meaning, because the two kinds of commits will be doing different things. Read this until you completely understand it: http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ Thanks, Shawn
Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
On 5/29/2014 7:52 AM, M, Arjun (NSN - IN/Bangalore) wrote: > Thanks Shawn... Just one more question.. > > Can both autoCommit and autoSoftCommit be enabled? If both are enabled, which > one takes precedence? Yes, and it's a very common configuration. If you do enable both, you want openSearcher to be false on autoCommit, so that your hard commits are not making documents visible. That is a job for autoSoftCommit. If you use openSearcher=false on autoCommit, then the question of which one takes precendence actually has no meaning, because the two kinds of commits will be doing different things. Read this until you completely understand it: http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ Thanks, Shawn
RE: Error enquiry- exceeded limit of maxWarmingSearchers=2
Thanks Shawn... Just one more question.. Can both autoCommit and autoSoftCommit be enabled? If both are enabled, which one takes precedence? Thanks & Regards, Arjun M -Original Message- From: ext Shawn Heisey [mailto:s...@elyograg.org] Sent: Thursday, May 29, 2014 7:02 PM To: solr-user@lucene.apache.org Subject: Re: Error enquiry- exceeded limit of maxWarmingSearchers=2 On 5/29/2014 4:18 AM, M, Arjun (NSN - IN/Bangalore) wrote: > Thanks for your valuable inputs... Find below my code and config in > solrconfig.xml. Index update is successful but I am not able to see any data > from solr admin console. What could be the issue? Any help here is highly > appreciated. > > I can see the data in the solr admin gui after tomcat restart(solr is > running in tomcat in my case) > > private void addToSolr(List c) throws SolrServerException, > IOException { > if (!c.isEmpty()) { > try { > > > solr.add(c); > logger.info("Commit size after Add=" + c.size()); > > } finally { > //renew lock > } > } > } > > autoCommit config in solrconfig.xml > = > > >${solr.autoCommit.maxTime:15000} > 1 >false > > > >${solr.autoSoftCommit.maxTime:-1} > The code snippet does not include a commit. I am not really clear on what using a value of -1 would do on maxTime here. I suspect that it efectively disables autoSoftCommit. If that's the case, then there is nothing at all in your code or your config that will open a new searcher -- that option is set to false in your autoCommit. If you want Solr to automatically do commits to make documents visible, I think you should configure a maxTime value for autoSoftCommit, and make it as long as you can possibly stand to not have new documents available. Then you won't have to worry about commits in your code at all. > Few more questions.. > > 2) If I use solrServer.add(,), should I do > solrServer.commit() also No. The commitWithin would do a soft commit for you once that much time has elapsed since indexing started (or the last commit with openSearcher=true), so you would not need to do a commit(). My opinion is that you should not combine manual commits with autoSoftCommit. Depending on exactly what your needs are, you might want to use commitWithin, and have autoSoftCommit as a last guarantee against errors in your indexing process. Thanks, Shawn
Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
On 5/29/2014 4:18 AM, M, Arjun (NSN - IN/Bangalore) wrote: > Thanks for your valuable inputs... Find below my code and config in > solrconfig.xml. Index update is successful but I am not able to see any data > from solr admin console. What could be the issue? Any help here is highly > appreciated. > > I can see the data in the solr admin gui after tomcat restart(solr is > running in tomcat in my case) > > private void addToSolr(List c) throws SolrServerException, > IOException { > if (!c.isEmpty()) { > try { > > > solr.add(c); > logger.info("Commit size after Add=" + c.size()); > > } finally { > //renew lock > } > } > } > > autoCommit config in solrconfig.xml > = > > >${solr.autoCommit.maxTime:15000} > 1 >false > > > >${solr.autoSoftCommit.maxTime:-1} > The code snippet does not include a commit. I am not really clear on what using a value of -1 would do on maxTime here. I suspect that it efectively disables autoSoftCommit. If that's the case, then there is nothing at all in your code or your config that will open a new searcher -- that option is set to false in your autoCommit. If you want Solr to automatically do commits to make documents visible, I think you should configure a maxTime value for autoSoftCommit, and make it as long as you can possibly stand to not have new documents available. Then you won't have to worry about commits in your code at all. > Few more questions.. > > 2) If I use solrServer.add(,), should I do > solrServer.commit() also No. The commitWithin would do a soft commit for you once that much time has elapsed since indexing started (or the last commit with openSearcher=true), so you would not need to do a commit(). My opinion is that you should not combine manual commits with autoSoftCommit. Depending on exactly what your needs are, you might want to use commitWithin, and have autoSoftCommit as a last guarantee against errors in your indexing process. Thanks, Shawn
RE: Error enquiry- exceeded limit of maxWarmingSearchers=2
Hi, Thanks for your valuable inputs... Find below my code and config in solrconfig.xml. Index update is successful but I am not able to see any data from solr admin console. What could be the issue? Any help here is highly appreciated. I can see the data in the solr admin gui after tomcat restart(solr is running in tomcat in my case) private void addToSolr(List c) throws SolrServerException, IOException { if (!c.isEmpty()) { try { solr.add(c); logger.info("Commit size after Add=" + c.size()); } finally { //renew lock } } } autoCommit config in solrconfig.xml = ${solr.autoCommit.maxTime:15000} 1 false ${solr.autoSoftCommit.maxTime:-1} Few more questions.. 2) If I use solrServer.add(,), should I do solrServer.commit() also Thanks & Regards, Arjun M -Original Message- From: ext Shawn Heisey [mailto:s...@elyograg.org] Sent: Wednesday, May 28, 2014 6:36 PM To: solr-user@lucene.apache.org Subject: Re: Error enquiry- exceeded limit of maxWarmingSearchers=2 On 5/28/2014 3:45 AM, M, Arjun (NSN - IN/Bangalore) wrote: > Also is there a way to check if autowarming completed (or) how to make > the next commit wait till previous commit finishes? With Solr, probably not. There might be a statistic available from an admin handler that I don't know about, but as far as I know, your code must be aware of approximately how long a commit is likely to take, and not send another commit until you can be sure that the previous commit is done. This includes the commitWithin parameter on an update request. Now that I've just said that, you *can* do an all documents query with rows=0 and look for a change in numFound. An update might actually result in no change to numFound, so you would need to build in a time-based exit to the loop that looks for numFound changes. In the case of commits done automatically by the configuration (autoCommit and/or autoSoftCommit), there is definitely no way to detect when a previous commit is done. The general recommendation with Solr 4.x is to have autoCommit enabled with openSearcher=false, with a relatively short maxTime -- from 5 minutes down to 15 seconds, depending on indexing rate. These commits will not open a new searcher, and they will not make new documents visible. For commits that affect which documents are visible, you need to determine how long you can possibly stand to go without seeing new data that has been indexed. Once you know that time interval, you can use it to do a manual commit, or you can set up autoSoftCommit with that interval. It is not at all unusual to have an autoCommit time interval that's shorter than autoSoftCommit. This blog post mentions SolrCloud, but it is also applicable to Solr 4.x when NOT running in cloud mode: http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ Thanks, Shawn
Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
On 5/28/2014 3:45 AM, M, Arjun (NSN - IN/Bangalore) wrote: > Also is there a way to check if autowarming completed (or) how to make > the next commit wait till previous commit finishes? With Solr, probably not. There might be a statistic available from an admin handler that I don't know about, but as far as I know, your code must be aware of approximately how long a commit is likely to take, and not send another commit until you can be sure that the previous commit is done. This includes the commitWithin parameter on an update request. Now that I've just said that, you *can* do an all documents query with rows=0 and look for a change in numFound. An update might actually result in no change to numFound, so you would need to build in a time-based exit to the loop that looks for numFound changes. In the case of commits done automatically by the configuration (autoCommit and/or autoSoftCommit), there is definitely no way to detect when a previous commit is done. The general recommendation with Solr 4.x is to have autoCommit enabled with openSearcher=false, with a relatively short maxTime -- from 5 minutes down to 15 seconds, depending on indexing rate. These commits will not open a new searcher, and they will not make new documents visible. For commits that affect which documents are visible, you need to determine how long you can possibly stand to go without seeing new data that has been indexed. Once you know that time interval, you can use it to do a manual commit, or you can set up autoSoftCommit with that interval. It is not at all unusual to have an autoCommit time interval that's shorter than autoSoftCommit. This blog post mentions SolrCloud, but it is also applicable to Solr 4.x when NOT running in cloud mode: http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ Thanks, Shawn
RE: Error enquiry- exceeded limit of maxWarmingSearchers=2
Hi, Also is there a way to check if autowarming completed (or) how to make the next commit wait till previous commit finishes? Thanks & Regards, Arjun M -Original Message- From: ext Shawn Heisey [mailto:s...@elyograg.org] Sent: Wednesday, May 28, 2014 10:31 AM To: solr-user@lucene.apache.org Subject: Re: Error enquiry- exceeded limit of maxWarmingSearchers=2 > Hi, > > I am getting the below error. > > org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: > Error opening new searcher. exceeded limit of > maxWarmingSearchers=2, try again later. This error is usually a symptom of a problem, not the actual problem. Either you are running into performance issues that are making your commits slow, or you are committing too frequently. Either way, you've got a situation where one commit (with opensearcher=true) is not able to finish before the next commit starts. Solr puts a limit on the number of searcher objects that can be starting up (warming) at the same time. You've exceeded that limit. Here's a wiki page about slow commits: http://wiki.apache.org/solr/SolrPerformanceProblems#Slow_commits The rest of that wiki page discusses other things that can cause Solr performance issues. Thanks, Shawn
Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
> Hi, > > I am getting the below error. > > org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: > Error opening new searcher. exceeded limit of > maxWarmingSearchers=2, try again later. This error is usually a symptom of a problem, not the actual problem. Either you are running into performance issues that are making your commits slow, or you are committing too frequently. Either way, you've got a situation where one commit (with opensearcher=true) is not able to finish before the next commit starts. Solr puts a limit on the number of searcher objects that can be starting up (warming) at the same time. You've exceeded that limit. Here's a wiki page about slow commits: http://wiki.apache.org/solr/SolrPerformanceProblems#Slow_commits The rest of that wiki page discusses other things that can cause Solr performance issues. Thanks, Shawn
Error enquiry- exceeded limit of maxWarmingSearchers=2
Hi, I am getting the below error. org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Can you please help?