Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
On 5/29/2014 9:55 PM, M, Arjun (NSN - IN/Bangalore) wrote: Thanks a lot for your nice explanation.. Now I understood the difference between autoCommit and autoSoftCommit.. Now my config looks like below. autoCommit maxDocs1/maxDocs openSearcherfalse/openSearcher /autoCommit autoSoftCommit maxTime15000/maxTime /autoSoftCommit With this now I am getting some other error like this. org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: version conflict for 140142167803912812800030383128128 expected=1469497192978841608 actual=1469497212082847746 This sounds like you are including the _version_ field in your document when you index. You probably shouldn't be doing that. Here's what that field is for, and how it works: http://heliosearch.org/solr/optimistic-concurrency/ Thanks, Shawn
Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
I’m also not sure I understand the practical purpose of your hard/soft auto commit settings. You are stating the following: Every 10 seconds I want data written to disk, but not be searchable. Every 15 seconds I want data to be written into memory and searchable. I would consider whether your soft commit window is too long, or if you can lengthen your hard commit period. It’s typical to see hard commits occur *less* frequently than soft commits. On May 30, 2014, at 11:04 AM, Shawn Heisey s...@elyograg.org wrote: On 5/29/2014 9:55 PM, M, Arjun (NSN - IN/Bangalore) wrote: Thanks a lot for your nice explanation.. Now I understood the difference between autoCommit and autoSoftCommit.. Now my config looks like below. autoCommit maxDocs1/maxDocs openSearcherfalse/openSearcher /autoCommit autoSoftCommit maxTime15000/maxTime /autoSoftCommit With this now I am getting some other error like this. org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: version conflict for 140142167803912812800030383128128 expected=1469497192978841608 actual=1469497212082847746 This sounds like you are including the _version_ field in your document when you index. You probably shouldn't be doing that. Here's what that field is for, and how it works: http://heliosearch.org/solr/optimistic-concurrency/ Thanks, Shawn
Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
I just realized I failed my own reading comprehension :) You have maxDocs, not maxTime for hard commit. Please disregard. On May 30, 2014, at 1:46 PM, Jason Hellman jhell...@innoventsolutions.com wrote: I’m also not sure I understand the practical purpose of your hard/soft auto commit settings. You are stating the following: Every 10 seconds I want data written to disk, but not be searchable. Every 15 seconds I want data to be written into memory and searchable. I would consider whether your soft commit window is too long, or if you can lengthen your hard commit period. It’s typical to see hard commits occur *less* frequently than soft commits. On May 30, 2014, at 11:04 AM, Shawn Heisey s...@elyograg.org wrote: On 5/29/2014 9:55 PM, M, Arjun (NSN - IN/Bangalore) wrote: Thanks a lot for your nice explanation.. Now I understood the difference between autoCommit and autoSoftCommit.. Now my config looks like below. autoCommit maxDocs1/maxDocs openSearcherfalse/openSearcher /autoCommit autoSoftCommit maxTime15000/maxTime /autoSoftCommit With this now I am getting some other error like this. org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: version conflict for 140142167803912812800030383128128 expected=1469497192978841608 actual=1469497212082847746 This sounds like you are including the _version_ field in your document when you index. You probably shouldn't be doing that. Here's what that field is for, and how it works: http://heliosearch.org/solr/optimistic-concurrency/ Thanks, Shawn
RE: Error enquiry- exceeded limit of maxWarmingSearchers=2
Hi,
Thanks for your valuable inputs... Find below my code and config in
solrconfig.xml. Index update is successful but I am not able to see any data
from solr admin console. What could be the issue? Any help here is highly
appreciated.
I can see the data in the solr admin gui after tomcat restart(solr is
running in tomcat in my case)
private void addToSolr(ListSolrInputDocument c) throws SolrServerException,
IOException {
if (!c.isEmpty()) {
try {
solr.add(c);
logger.info(Commit size after Add= + c.size());
} finally {
//renew lock
}
}
}
autoCommit config in solrconfig.xml
=
autoCommit
maxTime${solr.autoCommit.maxTime:15000}/maxTime
maxDocs1/maxDocs
openSearcherfalse/openSearcher
/autoCommit
autoSoftCommit
maxTime${solr.autoSoftCommit.maxTime:-1}/maxTime
/autoSoftCommit
Few more questions..
2) If I use solrServer.add(Doc list,commitwithin), should I do
solrServer.commit() also
Thanks Regards,
Arjun M
-Original Message-
From: ext Shawn Heisey [mailto:s...@elyograg.org]
Sent: Wednesday, May 28, 2014 6:36 PM
To: solr-user@lucene.apache.org
Subject: Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
On 5/28/2014 3:45 AM, M, Arjun (NSN - IN/Bangalore) wrote:
Also is there a way to check if autowarming completed (or) how to make
the next commit wait till previous commit finishes?
With Solr, probably not. There might be a statistic available from an
admin handler that I don't know about, but as far as I know, your code
must be aware of approximately how long a commit is likely to take, and
not send another commit until you can be sure that the previous commit
is done. This includes the commitWithin parameter on an update request.
Now that I've just said that, you *can* do an all documents query with
rows=0 and look for a change in numFound. An update might actually
result in no change to numFound, so you would need to build in a
time-based exit to the loop that looks for numFound changes.
In the case of commits done automatically by the configuration
(autoCommit and/or autoSoftCommit), there is definitely no way to detect
when a previous commit is done.
The general recommendation with Solr 4.x is to have autoCommit enabled
with openSearcher=false, with a relatively short maxTime -- from 5
minutes down to 15 seconds, depending on indexing rate. These commits
will not open a new searcher, and they will not make new documents visible.
For commits that affect which documents are visible, you need to
determine how long you can possibly stand to go without seeing new data
that has been indexed. Once you know that time interval, you can use it
to do a manual commit, or you can set up autoSoftCommit with that
interval. It is not at all unusual to have an autoCommit time interval
that's shorter than autoSoftCommit.
This blog post mentions SolrCloud, but it is also applicable to Solr 4.x
when NOT running in cloud mode:
http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
Thanks,
Shawn
Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
On 5/29/2014 4:18 AM, M, Arjun (NSN - IN/Bangalore) wrote:
Thanks for your valuable inputs... Find below my code and config in
solrconfig.xml. Index update is successful but I am not able to see any data
from solr admin console. What could be the issue? Any help here is highly
appreciated.
I can see the data in the solr admin gui after tomcat restart(solr is
running in tomcat in my case)
private void addToSolr(ListSolrInputDocument c) throws SolrServerException,
IOException {
if (!c.isEmpty()) {
try {
solr.add(c);
logger.info(Commit size after Add= + c.size());
} finally {
//renew lock
}
}
}
autoCommit config in solrconfig.xml
=
autoCommit
maxTime${solr.autoCommit.maxTime:15000}/maxTime
maxDocs1/maxDocs
openSearcherfalse/openSearcher
/autoCommit
autoSoftCommit
maxTime${solr.autoSoftCommit.maxTime:-1}/maxTime
/autoSoftCommit
The code snippet does not include a commit.
I am not really clear on what using a value of -1 would do on maxTime
here. I suspect that it efectively disables autoSoftCommit. If that's
the case, then there is nothing at all in your code or your config that
will open a new searcher -- that option is set to false in your autoCommit.
If you want Solr to automatically do commits to make documents visible,
I think you should configure a maxTime value for autoSoftCommit, and
make it as long as you can possibly stand to not have new documents
available. Then you won't have to worry about commits in your code at all.
Few more questions..
2) If I use solrServer.add(Doc list,commitwithin), should I do
solrServer.commit() also
No. The commitWithin would do a soft commit for you once that much time
has elapsed since indexing started (or the last commit with
openSearcher=true), so you would not need to do a commit().
My opinion is that you should not combine manual commits with
autoSoftCommit. Depending on exactly what your needs are, you might
want to use commitWithin, and have autoSoftCommit as a last guarantee
against errors in your indexing process.
Thanks,
Shawn
RE: Error enquiry- exceeded limit of maxWarmingSearchers=2
Thanks Shawn... Just one more question..
Can both autoCommit and autoSoftCommit be enabled? If both are enabled, which
one takes precedence?
Thanks Regards,
Arjun M
-Original Message-
From: ext Shawn Heisey [mailto:s...@elyograg.org]
Sent: Thursday, May 29, 2014 7:02 PM
To: solr-user@lucene.apache.org
Subject: Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
On 5/29/2014 4:18 AM, M, Arjun (NSN - IN/Bangalore) wrote:
Thanks for your valuable inputs... Find below my code and config in
solrconfig.xml. Index update is successful but I am not able to see any data
from solr admin console. What could be the issue? Any help here is highly
appreciated.
I can see the data in the solr admin gui after tomcat restart(solr is
running in tomcat in my case)
private void addToSolr(ListSolrInputDocument c) throws SolrServerException,
IOException {
if (!c.isEmpty()) {
try {
solr.add(c);
logger.info(Commit size after Add= + c.size());
} finally {
//renew lock
}
}
}
autoCommit config in solrconfig.xml
=
autoCommit
maxTime${solr.autoCommit.maxTime:15000}/maxTime
maxDocs1/maxDocs
openSearcherfalse/openSearcher
/autoCommit
autoSoftCommit
maxTime${solr.autoSoftCommit.maxTime:-1}/maxTime
/autoSoftCommit
The code snippet does not include a commit.
I am not really clear on what using a value of -1 would do on maxTime
here. I suspect that it efectively disables autoSoftCommit. If that's
the case, then there is nothing at all in your code or your config that
will open a new searcher -- that option is set to false in your autoCommit.
If you want Solr to automatically do commits to make documents visible,
I think you should configure a maxTime value for autoSoftCommit, and
make it as long as you can possibly stand to not have new documents
available. Then you won't have to worry about commits in your code at all.
Few more questions..
2) If I use solrServer.add(Doc list,commitwithin), should I do
solrServer.commit() also
No. The commitWithin would do a soft commit for you once that much time
has elapsed since indexing started (or the last commit with
openSearcher=true), so you would not need to do a commit().
My opinion is that you should not combine manual commits with
autoSoftCommit. Depending on exactly what your needs are, you might
want to use commitWithin, and have autoSoftCommit as a last guarantee
against errors in your indexing process.
Thanks,
Shawn
Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
On 5/29/2014 7:52 AM, M, Arjun (NSN - IN/Bangalore) wrote: Thanks Shawn... Just one more question.. Can both autoCommit and autoSoftCommit be enabled? If both are enabled, which one takes precedence? Yes, and it's a very common configuration. If you do enable both, you want openSearcher to be false on autoCommit, so that your hard commits are not making documents visible. That is a job for autoSoftCommit. If you use openSearcher=false on autoCommit, then the question of which one takes precendence actually has no meaning, because the two kinds of commits will be doing different things. Read this until you completely understand it: http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ Thanks, Shawn
RE: Error enquiry- exceeded limit of maxWarmingSearchers=2
Hi Shawn, Thanks a lot for your nice explanation.. Now I understood the difference between autoCommit and autoSoftCommit.. Now my config looks like below. autoCommit maxDocs1/maxDocs openSearcherfalse/openSearcher /autoCommit autoSoftCommit maxTime15000/maxTime /autoSoftCommit With this now I am getting some other error like this. org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: version conflict for 140142167803912812800030383128128 expected=1469497192978841608 actual=1469497212082847746 What could be the reason? Thanks Regards, Arjun M -Original Message- From: ext Shawn Heisey [mailto:s...@elyograg.org] Sent: Thursday, May 29, 2014 10:14 PM To: solr-user@lucene.apache.org Subject: Re: Error enquiry- exceeded limit of maxWarmingSearchers=2 On 5/29/2014 7:52 AM, M, Arjun (NSN - IN/Bangalore) wrote: Thanks Shawn... Just one more question.. Can both autoCommit and autoSoftCommit be enabled? If both are enabled, which one takes precedence? Yes, and it's a very common configuration. If you do enable both, you want openSearcher to be false on autoCommit, so that your hard commits are not making documents visible. That is a job for autoSoftCommit. If you use openSearcher=false on autoCommit, then the question of which one takes precendence actually has no meaning, because the two kinds of commits will be doing different things. Read this until you completely understand it: http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ Thanks, Shawn
RE: Error enquiry- exceeded limit of maxWarmingSearchers=2
Hi, Also is there a way to check if autowarming completed (or) how to make the next commit wait till previous commit finishes? Thanks Regards, Arjun M -Original Message- From: ext Shawn Heisey [mailto:s...@elyograg.org] Sent: Wednesday, May 28, 2014 10:31 AM To: solr-user@lucene.apache.org Subject: Re: Error enquiry- exceeded limit of maxWarmingSearchers=2 Hi, I am getting the below error. org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. This error is usually a symptom of a problem, not the actual problem. Either you are running into performance issues that are making your commits slow, or you are committing too frequently. Either way, you've got a situation where one commit (with opensearcher=true) is not able to finish before the next commit starts. Solr puts a limit on the number of searcher objects that can be starting up (warming) at the same time. You've exceeded that limit. Here's a wiki page about slow commits: http://wiki.apache.org/solr/SolrPerformanceProblems#Slow_commits The rest of that wiki page discusses other things that can cause Solr performance issues. Thanks, Shawn
Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
On 5/28/2014 3:45 AM, M, Arjun (NSN - IN/Bangalore) wrote: Also is there a way to check if autowarming completed (or) how to make the next commit wait till previous commit finishes? With Solr, probably not. There might be a statistic available from an admin handler that I don't know about, but as far as I know, your code must be aware of approximately how long a commit is likely to take, and not send another commit until you can be sure that the previous commit is done. This includes the commitWithin parameter on an update request. Now that I've just said that, you *can* do an all documents query with rows=0 and look for a change in numFound. An update might actually result in no change to numFound, so you would need to build in a time-based exit to the loop that looks for numFound changes. In the case of commits done automatically by the configuration (autoCommit and/or autoSoftCommit), there is definitely no way to detect when a previous commit is done. The general recommendation with Solr 4.x is to have autoCommit enabled with openSearcher=false, with a relatively short maxTime -- from 5 minutes down to 15 seconds, depending on indexing rate. These commits will not open a new searcher, and they will not make new documents visible. For commits that affect which documents are visible, you need to determine how long you can possibly stand to go without seeing new data that has been indexed. Once you know that time interval, you can use it to do a manual commit, or you can set up autoSoftCommit with that interval. It is not at all unusual to have an autoCommit time interval that's shorter than autoSoftCommit. This blog post mentions SolrCloud, but it is also applicable to Solr 4.x when NOT running in cloud mode: http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ Thanks, Shawn
Error enquiry- exceeded limit of maxWarmingSearchers=2
Hi, I am getting the below error. org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Can you please help?
Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
Hi, I am getting the below error. org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. This error is usually a symptom of a problem, not the actual problem. Either you are running into performance issues that are making your commits slow, or you are committing too frequently. Either way, you've got a situation where one commit (with opensearcher=true) is not able to finish before the next commit starts. Solr puts a limit on the number of searcher objects that can be starting up (warming) at the same time. You've exceeded that limit. Here's a wiki page about slow commits: http://wiki.apache.org/solr/SolrPerformanceProblems#Slow_commits The rest of that wiki page discusses other things that can cause Solr performance issues. Thanks, Shawn
