Re: Performance warning: Overlapping onDeskSearchers=2 solr
On 5/17/2017 9:15 AM, Jason Gerlowski wrote: > A strawman new message could be: "Performance warning: Overlapping > onDeskSearchers=2; consider reducing commit frequency if performance > problems encountered" > > Happy to create a JIRA/patch for this; just wanted to get some > feedback first in case there's an obvious reason the messages don't > get explicit about the cause. This sounds like a good idea to me. Bikeshedding on the message: "Performance warning: Overlapping onDeskSearchers=2; consider reducing commit frequency and/or cache warming" Thanks, Shawn
RE: Performance warning: Overlapping onDeskSearchers=2 solr
Hi, We have not set the autosoftcommit in solrcofig.xml. The only commit we are doing is through DIH(assuming it commits after the import). Also we have written timely schedulers to check if any records/documents is updated in database and to trigger the re-index of solr on those updated documents. Below are some more config details in solrconfig.xml 20 200 false 2 Thanks and Regards, Srinivas Kashyap -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: 17 May 2017 08:51 PM To: solr-user <solr-user@lucene.apache.org<mailto:solr-user@lucene.apache.org>> Subject: Re: Performance warning: Overlapping onDeskSearchers=2 solr Also, what is your autoSoftCommit setting? That also opens up a new searcher. On Wed, May 17, 2017 at 8:15 AM, Jason Gerlowski <gerlowsk...@gmail.com<mailto:gerlowsk...@gmail.com>> wrote: > Hey Shawn, others. > > This is a pitfall that Solr users seem to run into with some > frequency. (Anecdotally, I've bookmarked the Lucidworks article you > referenced because I end up referring people to it often enough.) > > The immediate first advice when someone encounters these > onDeckSearcher error messages is to examine their commit settings. Is > there any other possible cause for those messages? If not, can we > consider changing the log/exception error message to be more explicit > about the cause? > > A strawman new message could be: "Performance warning: Overlapping > onDeskSearchers=2; consider reducing commit frequency if performance > problems encountered" > > Happy to create a JIRA/patch for this; just wanted to get some > feedback first in case there's an obvious reason the messages don't > get explicit about the cause. > > Jason > > On Wed, May 17, 2017 at 8:49 AM, Shawn Heisey > <apa...@elyograg.org<mailto:apa...@elyograg.org>> wrote: >> On 5/17/2017 5:57 AM, Srinivas Kashyap wrote: >>> We are using Solr 5.2.1 version and are currently experiencing below >>> Warning in Solr Logging Console: >>> >>> Performance warning: Overlapping onDeskSearchers=2 >>> >>> Also we encounter, >>> >>> org.apache.solr.common.SolrException: Error opening new searcher. exceeded >>> limit of maxWarmingSearchers=2, try again later. >>> >>> >>> The reason being, we are doing mass update on our application and solr >>> experiencing the higher loads at times. Data is being indexed using DIH(sql >>> queries). >>> >>> In solrconfig.xml below is the code. >>> >>> >>> >>> Should we be uncommenting the above lines and try to avoid this error? >>> Please help me. >> >> This warning means that you are committing so frequently that there >> are already two searchers warming when you start another commit. >> >> DIH does a commit exactly once -- at the end of the import. One import will >> not cause the warning message you're seeing, so if there is one import >> happening at a time, either you are sending explicit commit requests during >> the import, or you have autoSoftCommit enabled with values that are far too >> small. >> >> You should definitely have autoCommit configured, but I would remove >> maxDocs and set maxTime to something like 6 -- one minute. The >> autoCommit should also set openSearcher to false. This kind of >> commit will not make new changes visible, but it will start a new >> transaction log frequently. >> >> >> 6 >> false >> >> >> An automatic commit (soft or hard) with a one second interval is going to >> cause that warning you're seeing. >> >> https://lucidworks.com/understanding-transaction-logs-softcommit-and- >> commit-in-sorlcloud/ >> >> Thanks, >> Shawn >> DISCLAIMER: E-mails and attachments from Bamboo Rose, LLC are confidential. If you are not the intended recipient, please notify the sender immediately by replying to the e-mail, and then delete it without making copies or using it in any way. No representation is made that this email or any attachments are free of viruses. Virus scanning is recommended and is the responsibility of the recipient. Thanks and Regards, Srinivas Kashyap Senior Software Engineer "GURUDAS HERITAGE" 100 Feet Ring Road, Kadirenahalli, Banashankari 2nd Stage, Bangalore-560070 P: 973-986-6105 Bamboo Rose The only B2B marketplace powered by proven trade engines. www.BambooRose.com<http://www.bamboorose.com/> Make Retail. Fun. Connected. Easier. Smarter. Together. Better. DISCLAIMER: E-mails and attachments from Bamboo Rose, LLC are confidential. If you are not the intended recipient, please notify the sender immediately by replying to the e-mail, and then delete it without making copies or using it in any way. No representation is made that this email or any attachments are free of viruses. Virus scanning is recommended and is the responsibility of the recipient.
RE: Performance warning: Overlapping onDeskSearchers=2 solr
Hi, We have not set the autosoftcommit in solrcofig.xml. The only commit we are doing is through DIH(assuming it commits after the import). Also we have written timely schedulers to check if any records/documents is updated in database and to trigger the re-index of solr on those updated documents. Below are some more config details in solrconfig.xml 20 200 false 2 Thanks and Regards, Srinivas Kashyap -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: 17 May 2017 08:51 PM To: solr-user <solr-user@lucene.apache.org> Subject: Re: Performance warning: Overlapping onDeskSearchers=2 solr Also, what is your autoSoftCommit setting? That also opens up a new searcher. On Wed, May 17, 2017 at 8:15 AM, Jason Gerlowski <gerlowsk...@gmail.com<mailto:gerlowsk...@gmail.com>> wrote: > Hey Shawn, others. > > This is a pitfall that Solr users seem to run into with some > frequency. (Anecdotally, I've bookmarked the Lucidworks article you > referenced because I end up referring people to it often enough.) > > The immediate first advice when someone encounters these > onDeckSearcher error messages is to examine their commit settings. Is > there any other possible cause for those messages? If not, can we > consider changing the log/exception error message to be more explicit > about the cause? > > A strawman new message could be: "Performance warning: Overlapping > onDeskSearchers=2; consider reducing commit frequency if performance > problems encountered" > > Happy to create a JIRA/patch for this; just wanted to get some > feedback first in case there's an obvious reason the messages don't > get explicit about the cause. > > Jason > > On Wed, May 17, 2017 at 8:49 AM, Shawn Heisey > <apa...@elyograg.org<mailto:apa...@elyograg.org>> wrote: >> On 5/17/2017 5:57 AM, Srinivas Kashyap wrote: >>> We are using Solr 5.2.1 version and are currently experiencing below >>> Warning in Solr Logging Console: >>> >>> Performance warning: Overlapping onDeskSearchers=2 >>> >>> Also we encounter, >>> >>> org.apache.solr.common.SolrException: Error opening new searcher. exceeded >>> limit of maxWarmingSearchers=2, try again later. >>> >>> >>> The reason being, we are doing mass update on our application and solr >>> experiencing the higher loads at times. Data is being indexed using DIH(sql >>> queries). >>> >>> In solrconfig.xml below is the code. >>> >>> >>> >>> Should we be uncommenting the above lines and try to avoid this error? >>> Please help me. >> >> This warning means that you are committing so frequently that there >> are already two searchers warming when you start another commit. >> >> DIH does a commit exactly once -- at the end of the import. One import will >> not cause the warning message you're seeing, so if there is one import >> happening at a time, either you are sending explicit commit requests during >> the import, or you have autoSoftCommit enabled with values that are far too >> small. >> >> You should definitely have autoCommit configured, but I would remove >> maxDocs and set maxTime to something like 6 -- one minute. The >> autoCommit should also set openSearcher to false. This kind of >> commit will not make new changes visible, but it will start a new >> transaction log frequently. >> >> >> 6 >> false >> >> >> An automatic commit (soft or hard) with a one second interval is going to >> cause that warning you're seeing. >> >> https://lucidworks.com/understanding-transaction-logs-softcommit-and- >> commit-in-sorlcloud/ >> >> Thanks, >> Shawn >> DISCLAIMER: E-mails and attachments from Bamboo Rose, LLC are confidential. If you are not the intended recipient, please notify the sender immediately by replying to the e-mail, and then delete it without making copies or using it in any way. No representation is made that this email or any attachments are free of viruses. Virus scanning is recommended and is the responsibility of the recipient. DISCLAIMER: E-mails and attachments from Bamboo Rose, LLC are confidential. If you are not the intended recipient, please notify the sender immediately by replying to the e-mail, and then delete it without making copies or using it in any way. No representation is made that this email or any attachments are free of viruses. Virus scanning is recommended and is the responsibility of the recipient.
Re: Performance warning: Overlapping onDeskSearchers=2 solr
+1 to change to new message A strawman new message could be: "Performance warning: Overlapping onDeskSearchers=2; consider reducing commit frequency if performance problems encountered" On Wed, May 17, 2017 at 1:15 PM, Mike Drobwrote: > You're committing too frequently, so you have new searchers getting queued > up before the previous ones have been processed. > > You have several options on how to deal with this. Can increase commit > interval, add hardware, or reduce query warming. > > I don't know if uncommenting that section will help because I don't know > what your current settings are. Or if you are using manual commits. > > Mike > > On Wed, May 17, 2017, 4:58 AM Srinivas Kashyap > wrote: > > > Hi All, > > > > We are using Solr 5.2.1 version and are currently experiencing below > > Warning in Solr Logging Console: > > > > Performance warning: Overlapping onDeskSearchers=2 > > > > Also we encounter, > > > > org.apache.solr.common.SolrException: Error opening new searcher. > exceeded > > limit of maxWarmingSearchers=2, try again later. > > > > > > The reason being, we are doing mass update on our application and solr > > experiencing the higher loads at times. Data is being indexed using > DIH(sql > > queries). > > > > In solrconfig.xml below is the code. > > > > > > > > Should we be uncommenting the above lines and try to avoid this error? > > Please help me. > > > > Thanks and Regards, > > Srinivas Kashyap > > > > > > > > DISCLAIMER: E-mails and attachments from Bamboo Rose, LLC are > > confidential. If you are not the intended recipient, please notify the > > sender immediately by replying to the e-mail, and then delete it without > > making copies or using it in any way. No representation is made that this > > email or any attachments are free of viruses. Virus scanning is > recommended > > and is the responsibility of the recipient. > > >
Re: Performance warning: Overlapping onDeskSearchers=2 solr
You're committing too frequently, so you have new searchers getting queued up before the previous ones have been processed. You have several options on how to deal with this. Can increase commit interval, add hardware, or reduce query warming. I don't know if uncommenting that section will help because I don't know what your current settings are. Or if you are using manual commits. Mike On Wed, May 17, 2017, 4:58 AM Srinivas Kashyapwrote: > Hi All, > > We are using Solr 5.2.1 version and are currently experiencing below > Warning in Solr Logging Console: > > Performance warning: Overlapping onDeskSearchers=2 > > Also we encounter, > > org.apache.solr.common.SolrException: Error opening new searcher. exceeded > limit of maxWarmingSearchers=2, try again later. > > > The reason being, we are doing mass update on our application and solr > experiencing the higher loads at times. Data is being indexed using DIH(sql > queries). > > In solrconfig.xml below is the code. > > > > Should we be uncommenting the above lines and try to avoid this error? > Please help me. > > Thanks and Regards, > Srinivas Kashyap > > > > DISCLAIMER: E-mails and attachments from Bamboo Rose, LLC are > confidential. If you are not the intended recipient, please notify the > sender immediately by replying to the e-mail, and then delete it without > making copies or using it in any way. No representation is made that this > email or any attachments are free of viruses. Virus scanning is recommended > and is the responsibility of the recipient. >
Re: Performance warning: Overlapping onDeskSearchers=2 solr
This has been changed already in 6.4.0. From the CHANGES.txt entry: SOLR-9712: maxWarmingSearchers now defaults to 1, and more importantly commits will now block if this limit is exceeded instead of throwing an exception (a good thing). Consequently there is no longer a risk in overlapping commits. Nonetheless users should continue to avoid excessive committing. Users are advised to remove any pre-existing maxWarmingSearchers entries from their solrconfig.xml files. On Wed, May 17, 2017 at 8:45 PM, Jason Gerlowskiwrote: > Hey Shawn, others. > > This is a pitfall that Solr users seem to run into with some > frequency. (Anecdotally, I've bookmarked the Lucidworks article you > referenced because I end up referring people to it often enough.) > > The immediate first advice when someone encounters these > onDeckSearcher error messages is to examine their commit settings. Is > there any other possible cause for those messages? If not, can we > consider changing the log/exception error message to be more explicit > about the cause? > > A strawman new message could be: "Performance warning: Overlapping > onDeskSearchers=2; consider reducing commit frequency if performance > problems encountered" > > Happy to create a JIRA/patch for this; just wanted to get some > feedback first in case there's an obvious reason the messages don't > get explicit about the cause. > > Jason > > On Wed, May 17, 2017 at 8:49 AM, Shawn Heisey wrote: >> On 5/17/2017 5:57 AM, Srinivas Kashyap wrote: >>> We are using Solr 5.2.1 version and are currently experiencing below >>> Warning in Solr Logging Console: >>> >>> Performance warning: Overlapping onDeskSearchers=2 >>> >>> Also we encounter, >>> >>> org.apache.solr.common.SolrException: Error opening new searcher. exceeded >>> limit of maxWarmingSearchers=2, try again later. >>> >>> >>> The reason being, we are doing mass update on our application and solr >>> experiencing the higher loads at times. Data is being indexed using DIH(sql >>> queries). >>> >>> In solrconfig.xml below is the code. >>> >>> >>> >>> Should we be uncommenting the above lines and try to avoid this error? >>> Please help me. >> >> This warning means that you are committing so frequently that there are >> already two searchers warming when you start another commit. >> >> DIH does a commit exactly once -- at the end of the import. One import will >> not cause the warning message you're seeing, so if there is one import >> happening at a time, either you are sending explicit commit requests during >> the import, or you have autoSoftCommit enabled with values that are far too >> small. >> >> You should definitely have autoCommit configured, but I would remove >> maxDocs and set maxTime to something like 6 -- one minute. The >> autoCommit should also set openSearcher to false. This kind of commit >> will not make new changes visible, but it will start a new transaction >> log frequently. >> >> >> 6 >> false >> >> >> An automatic commit (soft or hard) with a one second interval is going to >> cause that warning you're seeing. >> >> https://lucidworks.com/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ >> >> Thanks, >> Shawn >> -- Regards, Shalin Shekhar Mangar.
Re: Performance warning: Overlapping onDeskSearchers=2 solr
Also, what is your autoSoftCommit setting? That also opens up a new searcher. On Wed, May 17, 2017 at 8:15 AM, Jason Gerlowskiwrote: > Hey Shawn, others. > > This is a pitfall that Solr users seem to run into with some > frequency. (Anecdotally, I've bookmarked the Lucidworks article you > referenced because I end up referring people to it often enough.) > > The immediate first advice when someone encounters these > onDeckSearcher error messages is to examine their commit settings. Is > there any other possible cause for those messages? If not, can we > consider changing the log/exception error message to be more explicit > about the cause? > > A strawman new message could be: "Performance warning: Overlapping > onDeskSearchers=2; consider reducing commit frequency if performance > problems encountered" > > Happy to create a JIRA/patch for this; just wanted to get some > feedback first in case there's an obvious reason the messages don't > get explicit about the cause. > > Jason > > On Wed, May 17, 2017 at 8:49 AM, Shawn Heisey wrote: >> On 5/17/2017 5:57 AM, Srinivas Kashyap wrote: >>> We are using Solr 5.2.1 version and are currently experiencing below >>> Warning in Solr Logging Console: >>> >>> Performance warning: Overlapping onDeskSearchers=2 >>> >>> Also we encounter, >>> >>> org.apache.solr.common.SolrException: Error opening new searcher. exceeded >>> limit of maxWarmingSearchers=2, try again later. >>> >>> >>> The reason being, we are doing mass update on our application and solr >>> experiencing the higher loads at times. Data is being indexed using DIH(sql >>> queries). >>> >>> In solrconfig.xml below is the code. >>> >>> >>> >>> Should we be uncommenting the above lines and try to avoid this error? >>> Please help me. >> >> This warning means that you are committing so frequently that there are >> already two searchers warming when you start another commit. >> >> DIH does a commit exactly once -- at the end of the import. One import will >> not cause the warning message you're seeing, so if there is one import >> happening at a time, either you are sending explicit commit requests during >> the import, or you have autoSoftCommit enabled with values that are far too >> small. >> >> You should definitely have autoCommit configured, but I would remove >> maxDocs and set maxTime to something like 6 -- one minute. The >> autoCommit should also set openSearcher to false. This kind of commit >> will not make new changes visible, but it will start a new transaction >> log frequently. >> >> >> 6 >> false >> >> >> An automatic commit (soft or hard) with a one second interval is going to >> cause that warning you're seeing. >> >> https://lucidworks.com/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ >> >> Thanks, >> Shawn >>
Re: Performance warning: Overlapping onDeskSearchers=2 solr
Hey Shawn, others. This is a pitfall that Solr users seem to run into with some frequency. (Anecdotally, I've bookmarked the Lucidworks article you referenced because I end up referring people to it often enough.) The immediate first advice when someone encounters these onDeckSearcher error messages is to examine their commit settings. Is there any other possible cause for those messages? If not, can we consider changing the log/exception error message to be more explicit about the cause? A strawman new message could be: "Performance warning: Overlapping onDeskSearchers=2; consider reducing commit frequency if performance problems encountered" Happy to create a JIRA/patch for this; just wanted to get some feedback first in case there's an obvious reason the messages don't get explicit about the cause. Jason On Wed, May 17, 2017 at 8:49 AM, Shawn Heiseywrote: > On 5/17/2017 5:57 AM, Srinivas Kashyap wrote: >> We are using Solr 5.2.1 version and are currently experiencing below Warning >> in Solr Logging Console: >> >> Performance warning: Overlapping onDeskSearchers=2 >> >> Also we encounter, >> >> org.apache.solr.common.SolrException: Error opening new searcher. exceeded >> limit of maxWarmingSearchers=2, try again later. >> >> >> The reason being, we are doing mass update on our application and solr >> experiencing the higher loads at times. Data is being indexed using DIH(sql >> queries). >> >> In solrconfig.xml below is the code. >> >> >> >> Should we be uncommenting the above lines and try to avoid this error? >> Please help me. > > This warning means that you are committing so frequently that there are > already two searchers warming when you start another commit. > > DIH does a commit exactly once -- at the end of the import. One import will > not cause the warning message you're seeing, so if there is one import > happening at a time, either you are sending explicit commit requests during > the import, or you have autoSoftCommit enabled with values that are far too > small. > > You should definitely have autoCommit configured, but I would remove > maxDocs and set maxTime to something like 6 -- one minute. The > autoCommit should also set openSearcher to false. This kind of commit > will not make new changes visible, but it will start a new transaction > log frequently. > > > 6 > false > > > An automatic commit (soft or hard) with a one second interval is going to > cause that warning you're seeing. > > https://lucidworks.com/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ > > Thanks, > Shawn >
Re: Performance warning: Overlapping onDeskSearchers=2 solr
On 5/17/2017 5:57 AM, Srinivas Kashyap wrote: > We are using Solr 5.2.1 version and are currently experiencing below Warning > in Solr Logging Console: > > Performance warning: Overlapping onDeskSearchers=2 > > Also we encounter, > > org.apache.solr.common.SolrException: Error opening new searcher. exceeded > limit of maxWarmingSearchers=2, try again later. > > > The reason being, we are doing mass update on our application and solr > experiencing the higher loads at times. Data is being indexed using DIH(sql > queries). > > In solrconfig.xml below is the code. > > > > Should we be uncommenting the above lines and try to avoid this error? Please > help me. This warning means that you are committing so frequently that there are already two searchers warming when you start another commit. DIH does a commit exactly once -- at the end of the import. One import will not cause the warning message you're seeing, so if there is one import happening at a time, either you are sending explicit commit requests during the import, or you have autoSoftCommit enabled with values that are far too small. You should definitely have autoCommit configured, but I would remove maxDocs and set maxTime to something like 6 -- one minute. The autoCommit should also set openSearcher to false. This kind of commit will not make new changes visible, but it will start a new transaction log frequently. 6 false An automatic commit (soft or hard) with a one second interval is going to cause that warning you're seeing. https://lucidworks.com/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ Thanks, Shawn
Performance warning: Overlapping onDeskSearchers=2 solr
Hi All, We are using Solr 5.2.1 version and are currently experiencing below Warning in Solr Logging Console: Performance warning: Overlapping onDeskSearchers=2 Also we encounter, org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. The reason being, we are doing mass update on our application and solr experiencing the higher loads at times. Data is being indexed using DIH(sql queries). In solrconfig.xml below is the code. Should we be uncommenting the above lines and try to avoid this error? Please help me. Thanks and Regards, Srinivas Kashyap DISCLAIMER: E-mails and attachments from Bamboo Rose, LLC are confidential. If you are not the intended recipient, please notify the sender immediately by replying to the e-mail, and then delete it without making copies or using it in any way. No representation is made that this email or any attachments are free of viruses. Virus scanning is recommended and is the responsibility of the recipient.