Re: solr startup
It totally depends on how often you open a new searcher. If you have an index that updates once a day, running extensive autowarming queries can help. If you index frequently, it’s a balance between how much time you spend autowarming and how much time a user might have to wait. In extreme cases, i.e. autoSoftCommits every second or hard-commit-with-opensearcher=true, it can be best to disable autowarming entirely. And pretty often, I find just autowarming the queryResultCache and filterCache with relatively low numbers (< 20) works quite well. I’ll also say that in the bad old days before docValues, autowarming had more of an impact if you faceted or sorted, as uninverting the structures on the heap could get very expensive. DocValues fields are much quicker to load, although they can still take a noticeable amount if time especially if there are lots… FWIW, Erick > On Aug 8, 2020, at 9:48 AM, Dave wrote: > > Ah. Glad you found it. Yeah warming queries are much better substituted with > home made scripts if you need them. I like to use the previous days logs and > run the last couple hundred or so on a cron in the morning. > >> On Aug 8, 2020, at 9:39 AM, Schwartz, Tony wrote: >> >> I did not have a suggester set up. I disabled the spell checker component, >> but that wasn't the problem. I found my issue... it was related to a >> warming query i was running for each newly opened searcher. Early on I >> enabled that, but I completely forgot about it. And i don't believe it's >> needed. I was hoping it would help with performance related to time >> filtering and sorting. But, now it seems to be performing quite well >> without it. >> >> Tony >> >> >> >> From: Schwartz, Tony >> Sent: Friday, August 7, 2020 6:27 PM >> To: solr-user@lucene.apache.org >> Subject: RE: solr startup >> >> suggester? what do i need to look for in the configs? >> >> Tony >> >> >> >> Sent from my Verizon, Samsung Galaxy smartphone >> >> >> >> Original message >> From: Dave >> mailto:hastings.recurs...@gmail.com>> >> Date: 8/7/20 18:23 (GMT-05:00) >> To: solr-user@lucene.apache.org<mailto:solr-user@lucene.apache.org> >> Subject: Re: solr startup >> >> It sounds like you have suggester indexes being built on startup. Without >> them they just come up in a second or so >> >>> On Aug 7, 2020, at 6:03 PM, Schwartz, Tony >>> mailto:tony.schwa...@cinbell.com>> wrote: >>> >>> I have many collections. When I start solr, it takes 30 - 45 minutes to >>> start up and load all the collections. My collections are named per day. >>> During startup, solr loads the collections in alpha-numeric name order. I >>> would like solr to load the collections in the descending order. So the >>> most recent collections are loaded first and are available for searching >>> while the older collections are not as important. Is this possible? >>> >>>
Re: solr startup
Ah. Glad you found it. Yeah warming queries are much better substituted with home made scripts if you need them. I like to use the previous days logs and run the last couple hundred or so on a cron in the morning. > On Aug 8, 2020, at 9:39 AM, Schwartz, Tony wrote: > > I did not have a suggester set up. I disabled the spell checker component, > but that wasn't the problem. I found my issue... it was related to a warming > query i was running for each newly opened searcher. Early on I enabled that, > but I completely forgot about it. And i don't believe it's needed. I was > hoping it would help with performance related to time filtering and sorting. > But, now it seems to be performing quite well without it. > > Tony > > > > From: Schwartz, Tony > Sent: Friday, August 7, 2020 6:27 PM > To: solr-user@lucene.apache.org > Subject: RE: solr startup > > suggester? what do i need to look for in the configs? > > Tony > > > > Sent from my Verizon, Samsung Galaxy smartphone > > > > Original message > From: Dave mailto:hastings.recurs...@gmail.com>> > Date: 8/7/20 18:23 (GMT-05:00) > To: solr-user@lucene.apache.org<mailto:solr-user@lucene.apache.org> > Subject: Re: solr startup > > It sounds like you have suggester indexes being built on startup. Without > them they just come up in a second or so > >> On Aug 7, 2020, at 6:03 PM, Schwartz, Tony >> mailto:tony.schwa...@cinbell.com>> wrote: >> >> I have many collections. When I start solr, it takes 30 - 45 minutes to >> start up and load all the collections. My collections are named per day. >> During startup, solr loads the collections in alpha-numeric name order. I >> would like solr to load the collections in the descending order. So the >> most recent collections are loaded first and are available for searching >> while the older collections are not as important. Is this possible? >> >>
RE: solr startup
I did not have a suggester set up. I disabled the spell checker component, but that wasn't the problem. I found my issue... it was related to a warming query i was running for each newly opened searcher. Early on I enabled that, but I completely forgot about it. And i don't believe it's needed. I was hoping it would help with performance related to time filtering and sorting. But, now it seems to be performing quite well without it. Tony From: Schwartz, Tony Sent: Friday, August 7, 2020 6:27 PM To: solr-user@lucene.apache.org Subject: RE: solr startup suggester? what do i need to look for in the configs? Tony Sent from my Verizon, Samsung Galaxy smartphone Original message From: Dave mailto:hastings.recurs...@gmail.com>> Date: 8/7/20 18:23 (GMT-05:00) To: solr-user@lucene.apache.org<mailto:solr-user@lucene.apache.org> Subject: Re: solr startup It sounds like you have suggester indexes being built on startup. Without them they just come up in a second or so > On Aug 7, 2020, at 6:03 PM, Schwartz, Tony > mailto:tony.schwa...@cinbell.com>> wrote: > > I have many collections. When I start solr, it takes 30 - 45 minutes to > start up and load all the collections. My collections are named per day. > During startup, solr loads the collections in alpha-numeric name order. I > would like solr to load the collections in the descending order. So the most > recent collections are loaded first and are available for searching while the > older collections are not as important. Is this possible? > >
RE: solr startup
suggester? what do i need to look for in the configs? Tony Sent from my Verizon, Samsung Galaxy smartphone Original message From: Dave Date: 8/7/20 18:23 (GMT-05:00) To: solr-user@lucene.apache.org Subject: Re: solr startup It sounds like you have suggester indexes being built on startup. Without them they just come up in a second or so > On Aug 7, 2020, at 6:03 PM, Schwartz, Tony wrote: > > I have many collections. When I start solr, it takes 30 - 45 minutes to > start up and load all the collections. My collections are named per day. > During startup, solr loads the collections in alpha-numeric name order. I > would like solr to load the collections in the descending order. So the most > recent collections are loaded first and are available for searching while the > older collections are not as important. Is this possible? > >
Re: solr startup
It sounds like you have suggester indexes being built on startup. Without them they just come up in a second or so > On Aug 7, 2020, at 6:03 PM, Schwartz, Tony wrote: > > I have many collections. When I start solr, it takes 30 - 45 minutes to > start up and load all the collections. My collections are named per day. > During startup, solr loads the collections in alpha-numeric name order. I > would like solr to load the collections in the descending order. So the most > recent collections are loaded first and are available for searching while the > older collections are not as important. Is this possible? > >
solr startup
I have many collections. When I start solr, it takes 30 - 45 minutes to start up and load all the collections. My collections are named per day. During startup, solr loads the collections in alpha-numeric name order. I would like solr to load the collections in the descending order. So the most recent collections are loaded first and are available for searching while the older collections are not as important. Is this possible?
Re: Can a Solr Plugin be pre-loaded at Solr Startup
: I have a few classes that are Analyzers, Readers, and TokenFilters. These : classes use a large hashmap to map tokens to another value. The code is : working great. I go to the Analysis page on the Solr dashboard and everything : works as I would like. The problem is that the first time each one of these : are used it takes them a long time to load the classes. https://issues.apache.org/jira/browse/SOLR-8349 -Hoss http://www.lucidworks.com/
Can a Solr Plugin be pre-loaded at Solr Startup
I have a few classes that are Analyzers, Readers, and TokenFilters. These classes use a large hashmap to map tokens to another value. The code is working great. I go to the Analysis page on the Solr dashboard and everything works as I would like. The problem is that the first time each one of these are used it takes them a long time to load the classes. Is there a way that I can cause these classes to be loaded when Solr starts up? It also seems that at some time the classes are being reloaded because occasionally I see the analysis take a long time, similar to the first time the field analysis was used from the Solr dashboard. Maybe that is not the case, but it seems to be. Are plugins loaded once they are invoked for the entire Solr process or are plugins loaded for each thread? Thank you.
Re: Solr startup script in version 4.10.3
Thank you for your responses. However, according to my tests, solr 4.10.3 doesn’t use server by default anymore due to the removal of these lines in the bin/solr script. # TODO: see SOLR-3619, need to support server or example # depending on the version of Solr if [ -e $SOLR_TIP/server/start.jar ]; then DEFAULT_SERVER_DIR=$SOLR_TIP/server else DEFAULT_SERVER_DIR=$SOLR_TIP/example fi Solr 5.0.0 does in both standalone and solrcloud modes ! This is great ! Dominique http://www.eolya.fr/ Le jeudi 8 janvier 2015, Anshum Gupta ans...@anshumgupta.net a écrit : Things have changed reasonably for the 5.0 release. In case of a standalone mode, it still defaults to the server directory. So you'd find your logs in server/logs. In case of solrcloud mode e.g. if you ran bin/solr -e cloud -noprompt this would default to stuff being copied into example directory (leaving server directory untouched) and everything would run from there. You will also have the option of just creating a new SOLR home and using that instead. See the following: https://cwiki.apache.org/confluence/display/solr/Getting+Started+with+SolrCloud The link above is for the upcoming Solr 5.0 and is still work in progress but should give you more information. Hope that helps. On Tue, Jan 6, 2015 at 1:29 AM, Dominique Bejean dominique.bej...@eolya.fr javascript:; wrote: Hi, In release 4.10.3, the following lines were removed from solr starting script (bin/solr) # TODO: see SOLR-3619, need to support server or example # depending on the version of Solr if [ -e $SOLR_TIP/server/start.jar ]; then DEFAULT_SERVER_DIR=$SOLR_TIP/server else DEFAULT_SERVER_DIR=$SOLR_TIP/example fi However, the usage message always say -d dir Specify the Solr server directory; defaults to server Either the usage have to be fixed or the removed lines put back to the script. Personally, I like the default to server directory. My installation process in order to have a clean empty solr instance is to copy examples into server and remove directories like example-DIH, example-shemaless, multicore and solr/collection1 Solr server (or node) can be started without the -d parameter. If this makes sense, a Jira issue could be open. Dominique http://www.eolya.fr/ -- Anshum Gupta http://about.me/anshumgupta
Re: Solr startup script in version 4.10.3
Versions 4.10.3 and beyond already use server rather than example, which still finds a reference in the script purely for back compat. A major release 5.0 is coming soon, perhaps the back compat can be removed for that. On 6 Jan 2015 09:30, Dominique Bejean dominique.bej...@eolya.fr wrote: Hi, In release 4.10.3, the following lines were removed from solr starting script (bin/solr) # TODO: see SOLR-3619, need to support server or example # depending on the version of Solr if [ -e $SOLR_TIP/server/start.jar ]; then DEFAULT_SERVER_DIR=$SOLR_TIP/server else DEFAULT_SERVER_DIR=$SOLR_TIP/example fi However, the usage message always say -d dir Specify the Solr server directory; defaults to server Either the usage have to be fixed or the removed lines put back to the script. Personally, I like the default to server directory. My installation process in order to have a clean empty solr instance is to copy examples into server and remove directories like example-DIH, example-shemaless, multicore and solr/collection1 Solr server (or node) can be started without the -d parameter. If this makes sense, a Jira issue could be open. Dominique http://www.eolya.fr/
Re: Solr startup script in version 4.10.3
Things have changed reasonably for the 5.0 release. In case of a standalone mode, it still defaults to the server directory. So you'd find your logs in server/logs. In case of solrcloud mode e.g. if you ran bin/solr -e cloud -noprompt this would default to stuff being copied into example directory (leaving server directory untouched) and everything would run from there. You will also have the option of just creating a new SOLR home and using that instead. See the following: https://cwiki.apache.org/confluence/display/solr/Getting+Started+with+SolrCloud The link above is for the upcoming Solr 5.0 and is still work in progress but should give you more information. Hope that helps. On Tue, Jan 6, 2015 at 1:29 AM, Dominique Bejean dominique.bej...@eolya.fr wrote: Hi, In release 4.10.3, the following lines were removed from solr starting script (bin/solr) # TODO: see SOLR-3619, need to support server or example # depending on the version of Solr if [ -e $SOLR_TIP/server/start.jar ]; then DEFAULT_SERVER_DIR=$SOLR_TIP/server else DEFAULT_SERVER_DIR=$SOLR_TIP/example fi However, the usage message always say -d dir Specify the Solr server directory; defaults to server Either the usage have to be fixed or the removed lines put back to the script. Personally, I like the default to server directory. My installation process in order to have a clean empty solr instance is to copy examples into server and remove directories like example-DIH, example-shemaless, multicore and solr/collection1 Solr server (or node) can be started without the -d parameter. If this makes sense, a Jira issue could be open. Dominique http://www.eolya.fr/ -- Anshum Gupta http://about.me/anshumgupta
Solr startup script in version 4.10.3
Hi, In release 4.10.3, the following lines were removed from solr starting script (bin/solr) # TODO: see SOLR-3619, need to support server or example # depending on the version of Solr if [ -e $SOLR_TIP/server/start.jar ]; then DEFAULT_SERVER_DIR=$SOLR_TIP/server else DEFAULT_SERVER_DIR=$SOLR_TIP/example fi However, the usage message always say -d dir Specify the Solr server directory; defaults to server Either the usage have to be fixed or the removed lines put back to the script. Personally, I like the default to server directory. My installation process in order to have a clean empty solr instance is to copy examples into server and remove directories like example-DIH, example-shemaless, multicore and solr/collection1 Solr server (or node) can be started without the -d parameter. If this makes sense, a Jira issue could be open. Dominique http://www.eolya.fr/
Which time consuming processes are executed during Solr startup?
For example we know that cache warming is executed during startup. Are any other processes executed during Solr startup? Thank you, Ivan
Re: SOLR startup problem
it seems jar files related to DataImport Handler are not being loaded in memory.Try to copy Data Import Handler related jars to solr_home/lib directory and restart the servlet container. -Thanx: Grijesh www.gettinhahead.co.in -- View this message in context: http://lucene.472066.n3.nabble.com/SOLR-startup-problem-tp2879789p2880022.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Startup CPU Spike
Yonik, I have provided an image below gives details on what is causing the blocked http thread. Is there any way to resolve this issue. Thanks, John -- John Williams System Administrator 37signals inline: Screen shot 2010-03-09 at 11.22.20 AM.png On Mar 9, 2010, at 10:41 AM, John Williams wrote: Yonik, I got yourkit setup to profile the Tomcat instance and as you will see in the graph below all of the http threads are blocked (red) until around 4:40. This is the point where the instance becomes responsive and CPU usage drops. I have also ruled out GC being the issue by using the GC monitoring in yourkit. Let me know your thoughts and if you have any questions. Thanks for your assistance. Thanks, John -- John Williams System Administrator 37signals Screen shot 2010-03-09 at 10.35.15 AM.png On Mar 8, 2010, at 5:28 PM, Yonik Seeley wrote: On Mon, Mar 8, 2010 at 6:07 PM, John Williams j...@37signals.com wrote: Yonik, In all cases our autowarmCount is set to 0. Also, here is a link to our config. http://pastebin.com/iUgruqPd Weird... on a quick glance, I don't see anything in your config that would cause work to be done on a commit. I expected something like autowarming, or rebuilding a spellcheck index, etc. I assume this is happening even w/o any requests hitting the server? Could it be GC? You could use -verbose:gc or jconsole to check if this corresponds to a big GC (which could naturally hit on an index change). 5 minutes is really excessive though, and I wouldn't expect it on startup. If it's not GC, perhaps the next step is to get some stack traces during the spike (or use a profiler) to figure out where the time is being spent. And verify that the solrconfig.xml shown actually still matches the one you provided. -Yonik http://www.lucidimagination.com Thanks, John -- John Williams System Administrator 37signals On Mar 8, 2010, at 4:44 PM, Yonik Seeley wrote: Is this just autowarming? Check your autowarmCount parameters in solrconfig.xml -Yonik http://www.lucidimagination.com On Mon, Mar 8, 2010 at 5:37 PM, John Williams j...@37signals.com wrote: Good afternoon. We have been experiencing an odd issue with one of our Solr nodes. Upon startup or when bringing in a new index we get a CPU spike for 5 minutes or so. I have attached a graph of this spike. During this time simple queries return without a problem but more complex queries do not return. Here are some more details about the instance: Index Size: ~16G Max Heap: 6144M GC Option: -XX:+UseConcMarkSweepGC System Memory: 16G We have a very similar instance to this one but with a much larger index that we are not seeing this sort of issue. Your help is greatly appreciated. Let me know if you need any additional information. Thanks, John -- John Williams System Administrator 37signals smime.p7s Description: S/MIME cryptographic signature
Re: Solr Startup CPU Spike
Ah - loading the fieldcache - do you have a *lot* of unique terms in the fields you are sorting/faceting on? localhost:8983/solr/admin/luke is helpful for checking this. -- - Mark http://www.lucidimagination.com On 03/09/2010 12:33 PM, John Williams wrote: Yonik, I have provided an image below gives details on what is causing the blocked http thread. Is there any way to resolve this issue. Thanks, John -- John Williams System Administrator 37signals On Mar 9, 2010, at 10:41 AM, John Williams wrote: Yonik, I got yourkit setup to profile the Tomcat instance and as you will see in the graph below all of the http threads are blocked (red) until around 4:40. This is the point where the instance becomes responsive and CPU usage drops. I have also ruled out GC being the issue by using the GC monitoring in yourkit. Let me know your thoughts and if you have any questions. Thanks for your assistance. Thanks, John -- John Williams System Administrator 37signals Screen shot 2010-03-09 at 10.35.15 AM.png On Mar 8, 2010, at 5:28 PM, Yonik Seeley wrote: On Mon, Mar 8, 2010 at 6:07 PM, John Williamsj...@37signals.com wrote: Yonik, In all cases our autowarmCount is set to 0. Also, here is a link to our config. http://pastebin.com/iUgruqPd Weird... on a quick glance, I don't see anything in your config that would cause work to be done on a commit. I expected something like autowarming, or rebuilding a spellcheck index, etc. I assume this is happening even w/o any requests hitting the server? Could it be GC? You could use -verbose:gc or jconsole to check if this corresponds to a big GC (which could naturally hit on an index change). 5 minutes is really excessive though, and I wouldn't expect it on startup. If it's not GC, perhaps the next step is to get some stack traces during the spike (or use a profiler) to figure out where the time is being spent. And verify that the solrconfig.xml shown actually still matches the one you provided. -Yonik http://www.lucidimagination.com Thanks, John -- John Williams System Administrator 37signals On Mar 8, 2010, at 4:44 PM, Yonik Seeley wrote: Is this just autowarming? Check your autowarmCount parameters in solrconfig.xml -Yonik http://www.lucidimagination.com On Mon, Mar 8, 2010 at 5:37 PM, John Williamsj...@37signals.com wrote: Good afternoon. We have been experiencing an odd issue with one of our Solr nodes. Upon startup or when bringing in a new index we get a CPU spike for 5 minutes or so. I have attached a graph of this spike. During this time simple queries return without a problem but more complex queries do not return. Here are some more details about the instance: Index Size: ~16G Max Heap: 6144M GC Option: -XX:+UseConcMarkSweepGC System Memory: 16G We have a very similar instance to this one but with a much larger index that we are not seeing this sort of issue. Your help is greatly appreciated. Let me know if you need any additional information. Thanks, John -- John Williams System Administrator 37signals
Re: Solr Startup CPU Spike
Ahhh, FieldCache loading... what version of Solr are you using? It's interesting it would take that long to load too (and maxing out one CPU - doesn't look particularly IO bound). How many documents are in this index? -Yonik On Tue, Mar 9, 2010 at 12:33 PM, John Williams j...@37signals.com wrote: Yonik, I have provided an image below gives details on what is causing the blocked http thread. Is there any way to resolve this issue. Thanks, John -- John Williams System Administrator 37signals On Mar 9, 2010, at 10:41 AM, John Williams wrote: Yonik, I got yourkit setup to profile the Tomcat instance and as you will see in the graph below all of the http threads are blocked (red) until around 4:40. This is the point where the instance becomes responsive and CPU usage drops. I have also ruled out GC being the issue by using the GC monitoring in yourkit. Let me know your thoughts and if you have any questions. Thanks for your assistance. Thanks, John -- John Williams System Administrator 37signals Screen shot 2010-03-09 at 10.35.15 AM.png On Mar 8, 2010, at 5:28 PM, Yonik Seeley wrote: On Mon, Mar 8, 2010 at 6:07 PM, John Williams j...@37signals.com wrote: Yonik, In all cases our autowarmCount is set to 0. Also, here is a link to our config. http://pastebin.com/iUgruqPd Weird... on a quick glance, I don't see anything in your config that would cause work to be done on a commit. I expected something like autowarming, or rebuilding a spellcheck index, etc. I assume this is happening even w/o any requests hitting the server? Could it be GC? You could use -verbose:gc or jconsole to check if this corresponds to a big GC (which could naturally hit on an index change). 5 minutes is really excessive though, and I wouldn't expect it on startup. If it's not GC, perhaps the next step is to get some stack traces during the spike (or use a profiler) to figure out where the time is being spent. And verify that the solrconfig.xml shown actually still matches the one you provided. -Yonik http://www.lucidimagination.com Thanks, John -- John Williams System Administrator 37signals On Mar 8, 2010, at 4:44 PM, Yonik Seeley wrote: Is this just autowarming? Check your autowarmCount parameters in solrconfig.xml -Yonik http://www.lucidimagination.com On Mon, Mar 8, 2010 at 5:37 PM, John Williams j...@37signals.com wrote: Good afternoon. We have been experiencing an odd issue with one of our Solr nodes. Upon startup or when bringing in a new index we get a CPU spike for 5 minutes or so. I have attached a graph of this spike. During this time simple queries return without a problem but more complex queries do not return. Here are some more details about the instance: Index Size: ~16G Max Heap: 6144M GC Option: -XX:+UseConcMarkSweepGC System Memory: 16G We have a very similar instance to this one but with a much larger index that we are not seeing this sort of issue. Your help is greatly appreciated. Let me know if you need any additional information. Thanks, John -- John Williams System Administrator 37signals
Re: Solr Startup CPU Spike
Mark, I am trying to load that url but its taking quite a while. I will let you know if/when it loads. -John -- John Williams System Administrator 37signals On Mar 9, 2010, at 11:38 AM, Mark Miller wrote: Ah - loading the fieldcache - do you have a *lot* of unique terms in the fields you are sorting/faceting on? localhost:8983/solr/admin/luke is helpful for checking this. -- - Mark http://www.lucidimagination.com On 03/09/2010 12:33 PM, John Williams wrote: Yonik, I have provided an image below gives details on what is causing the blocked http thread. Is there any way to resolve this issue. Thanks, John -- John Williams System Administrator 37signals On Mar 9, 2010, at 10:41 AM, John Williams wrote: Yonik, I got yourkit setup to profile the Tomcat instance and as you will see in the graph below all of the http threads are blocked (red) until around 4:40. This is the point where the instance becomes responsive and CPU usage drops. I have also ruled out GC being the issue by using the GC monitoring in yourkit. Let me know your thoughts and if you have any questions. Thanks for your assistance. Thanks, John -- John Williams System Administrator 37signals Screen shot 2010-03-09 at 10.35.15 AM.png On Mar 8, 2010, at 5:28 PM, Yonik Seeley wrote: On Mon, Mar 8, 2010 at 6:07 PM, John Williamsj...@37signals.com wrote: Yonik, In all cases our autowarmCount is set to 0. Also, here is a link to our config. http://pastebin.com/iUgruqPd Weird... on a quick glance, I don't see anything in your config that would cause work to be done on a commit. I expected something like autowarming, or rebuilding a spellcheck index, etc. I assume this is happening even w/o any requests hitting the server? Could it be GC? You could use -verbose:gc or jconsole to check if this corresponds to a big GC (which could naturally hit on an index change). 5 minutes is really excessive though, and I wouldn't expect it on startup. If it's not GC, perhaps the next step is to get some stack traces during the spike (or use a profiler) to figure out where the time is being spent. And verify that the solrconfig.xml shown actually still matches the one you provided. -Yonik http://www.lucidimagination.com Thanks, John -- John Williams System Administrator 37signals On Mar 8, 2010, at 4:44 PM, Yonik Seeley wrote: Is this just autowarming? Check your autowarmCount parameters in solrconfig.xml -Yonik http://www.lucidimagination.com On Mon, Mar 8, 2010 at 5:37 PM, John Williamsj...@37signals.com wrote: Good afternoon. We have been experiencing an odd issue with one of our Solr nodes. Upon startup or when bringing in a new index we get a CPU spike for 5 minutes or so. I have attached a graph of this spike. During this time simple queries return without a problem but more complex queries do not return. Here are some more details about the instance: Index Size: ~16G Max Heap: 6144M GC Option: -XX:+UseConcMarkSweepGC System Memory: 16G We have a very similar instance to this one but with a much larger index that we are not seeing this sort of issue. Your help is greatly appreciated. Let me know if you need any additional information. Thanks, John -- John Williams System Administrator 37signals smime.p7s Description: S/MIME cryptographic signature
Re: Solr Startup CPU Spike
Yonik, We are on Solr 1.3. The total number of documents is 54173459. Let me know if need any additional info. Thanks, John -- John Williams System Administrator 37signals On Mar 9, 2010, at 11:39 AM, Yonik Seeley wrote: Ahhh, FieldCache loading... what version of Solr are you using? It's interesting it would take that long to load too (and maxing out one CPU - doesn't look particularly IO bound). How many documents are in this index? -Yonik On Tue, Mar 9, 2010 at 12:33 PM, John Williams j...@37signals.com wrote: Yonik, I have provided an image below gives details on what is causing the blocked http thread. Is there any way to resolve this issue. Thanks, John -- John Williams System Administrator 37signals On Mar 9, 2010, at 10:41 AM, John Williams wrote: Yonik, I got yourkit setup to profile the Tomcat instance and as you will see in the graph below all of the http threads are blocked (red) until around 4:40. This is the point where the instance becomes responsive and CPU usage drops. I have also ruled out GC being the issue by using the GC monitoring in yourkit. Let me know your thoughts and if you have any questions. Thanks for your assistance. Thanks, John -- John Williams System Administrator 37signals Screen shot 2010-03-09 at 10.35.15 AM.png On Mar 8, 2010, at 5:28 PM, Yonik Seeley wrote: On Mon, Mar 8, 2010 at 6:07 PM, John Williams j...@37signals.com wrote: Yonik, In all cases our autowarmCount is set to 0. Also, here is a link to our config. http://pastebin.com/iUgruqPd Weird... on a quick glance, I don't see anything in your config that would cause work to be done on a commit. I expected something like autowarming, or rebuilding a spellcheck index, etc. I assume this is happening even w/o any requests hitting the server? Could it be GC? You could use -verbose:gc or jconsole to check if this corresponds to a big GC (which could naturally hit on an index change). 5 minutes is really excessive though, and I wouldn't expect it on startup. If it's not GC, perhaps the next step is to get some stack traces during the spike (or use a profiler) to figure out where the time is being spent. And verify that the solrconfig.xml shown actually still matches the one you provided. -Yonik http://www.lucidimagination.com Thanks, John -- John Williams System Administrator 37signals On Mar 8, 2010, at 4:44 PM, Yonik Seeley wrote: Is this just autowarming? Check your autowarmCount parameters in solrconfig.xml -Yonik http://www.lucidimagination.com On Mon, Mar 8, 2010 at 5:37 PM, John Williams j...@37signals.com wrote: Good afternoon. We have been experiencing an odd issue with one of our Solr nodes. Upon startup or when bringing in a new index we get a CPU spike for 5 minutes or so. I have attached a graph of this spike. During this time simple queries return without a problem but more complex queries do not return. Here are some more details about the instance: Index Size: ~16G Max Heap: 6144M GC Option: -XX:+UseConcMarkSweepGC System Memory: 16G We have a very similar instance to this one but with a much larger index that we are not seeing this sort of issue. Your help is greatly appreciated. Let me know if you need any additional information. Thanks, John -- John Williams System Administrator 37signals smime.p7s Description: S/MIME cryptographic signature
Solr Startup CPU Spike
Good afternoon. We have been experiencing an odd issue with one of our Solr nodes. Upon startup or when bringing in a new index we get a CPU spike for 5 minutes or so. I have attached a graph of this spike. During this time simple queries return without a problem but more complex queries do not return. Here are some more details about the instance: Index Size: ~16G Max Heap: 6144M GC Option: -XX:+UseConcMarkSweepGC System Memory: 16G We have a very similar instance to this one but with a much larger index that we are not seeing this sort of issue. Your help is greatly appreciated. Let me know if you need any additional information. Thanks, John -- John Williams System Administrator 37signals inline: cpu_spike.png smime.p7s Description: S/MIME cryptographic signature
Re: Solr Startup CPU Spike
Is this just autowarming? Check your autowarmCount parameters in solrconfig.xml -Yonik http://www.lucidimagination.com On Mon, Mar 8, 2010 at 5:37 PM, John Williams j...@37signals.com wrote: Good afternoon. We have been experiencing an odd issue with one of our Solr nodes. Upon startup or when bringing in a new index we get a CPU spike for 5 minutes or so. I have attached a graph of this spike. During this time simple queries return without a problem but more complex queries do not return. Here are some more details about the instance: Index Size: ~16G Max Heap: 6144M GC Option: -XX:+UseConcMarkSweepGC System Memory: 16G We have a very similar instance to this one but with a much larger index that we are not seeing this sort of issue. Your help is greatly appreciated. Let me know if you need any additional information. Thanks, John -- John Williams System Administrator 37signals
Re: Solr Startup CPU Spike
Yonik, In all cases our autowarmCount is set to 0. Also, here is a link to our config. http://pastebin.com/iUgruqPd Thanks, John -- John Williams System Administrator 37signals On Mar 8, 2010, at 4:44 PM, Yonik Seeley wrote: Is this just autowarming? Check your autowarmCount parameters in solrconfig.xml -Yonik http://www.lucidimagination.com On Mon, Mar 8, 2010 at 5:37 PM, John Williams j...@37signals.com wrote: Good afternoon. We have been experiencing an odd issue with one of our Solr nodes. Upon startup or when bringing in a new index we get a CPU spike for 5 minutes or so. I have attached a graph of this spike. During this time simple queries return without a problem but more complex queries do not return. Here are some more details about the instance: Index Size: ~16G Max Heap: 6144M GC Option: -XX:+UseConcMarkSweepGC System Memory: 16G We have a very similar instance to this one but with a much larger index that we are not seeing this sort of issue. Your help is greatly appreciated. Let me know if you need any additional information. Thanks, John -- John Williams System Administrator 37signals smime.p7s Description: S/MIME cryptographic signature
Re: Solr Startup CPU Spike
On Mon, Mar 8, 2010 at 6:07 PM, John Williams j...@37signals.com wrote: Yonik, In all cases our autowarmCount is set to 0. Also, here is a link to our config. http://pastebin.com/iUgruqPd Weird... on a quick glance, I don't see anything in your config that would cause work to be done on a commit. I expected something like autowarming, or rebuilding a spellcheck index, etc. I assume this is happening even w/o any requests hitting the server? Could it be GC? You could use -verbose:gc or jconsole to check if this corresponds to a big GC (which could naturally hit on an index change). 5 minutes is really excessive though, and I wouldn't expect it on startup. If it's not GC, perhaps the next step is to get some stack traces during the spike (or use a profiler) to figure out where the time is being spent. And verify that the solrconfig.xml shown actually still matches the one you provided. -Yonik http://www.lucidimagination.com Thanks, John -- John Williams System Administrator 37signals On Mar 8, 2010, at 4:44 PM, Yonik Seeley wrote: Is this just autowarming? Check your autowarmCount parameters in solrconfig.xml -Yonik http://www.lucidimagination.com On Mon, Mar 8, 2010 at 5:37 PM, John Williams j...@37signals.com wrote: Good afternoon. We have been experiencing an odd issue with one of our Solr nodes. Upon startup or when bringing in a new index we get a CPU spike for 5 minutes or so. I have attached a graph of this spike. During this time simple queries return without a problem but more complex queries do not return. Here are some more details about the instance: Index Size: ~16G Max Heap: 6144M GC Option: -XX:+UseConcMarkSweepGC System Memory: 16G We have a very similar instance to this one but with a much larger index that we are not seeing this sort of issue. Your help is greatly appreciated. Let me know if you need any additional information. Thanks, John -- John Williams System Administrator 37signals
RE: Exception during Solr startup
Hi Yonik Erik, Thanks to both of you. It seems like our container had some issues and was causing this problem. Thanks, Raghu -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Yonik Seeley Sent: Wednesday, August 13, 2008 10:57 AM To: solr-user@lucene.apache.org Subject: Re: Exception during Solr startup On Wed, Aug 13, 2008 at 10:55 AM, Kashyap, Raghu [EMAIL PROTECTED] wrote: SEVERE: java.lang.UnsupportedClassVersionError: Bad version number in .class file This is normally a mismatch between java compiler and runtime (like using Java6 to compile and Java5 to try and run). -Yonik
Exception during Solr startup
Hi, Today I started seeing this exception when I started solr instance. Any ideas what might be causing this problem? INFO: xsltCacheLifetimeSeconds=5 Aug 13, 2008 9:20:45 AM org.apache.solr.common.SolrException log SEVERE: java.lang.UnsupportedClassVersionError: Bad version number in .class file at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:620) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124) at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappCla ssLoader.java:1819) at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader .java:872) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader .java:1327) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader .java:1206) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:242) at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.jav a:239) at org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.j ava:260) at org.apache.solr.util.plugin.AbstractPluginLoader.create(AbstractPluginLo ader.java:83) at org.apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoad er.java:140) at org.apache.solr.core.SolrCore.loadSearchComponents(SolrCore.java:622) at org.apache.solr.core.SolrCore.init(SolrCore.java:408) at org.apache.solr.core.MultiCore.create(MultiCore.java:255) at org.apache.solr.core.MultiCore.load(MultiCore.java:139) at org.apache.solr.servlet.SolrDispatchFilter.initMultiCore(SolrDispatchFil ter.java:147) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java: 72) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFi lterConfig.java:275) at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(Applicatio nFilterConfig.java:397) at org.apache.catalina.core.ApplicationFilterConfig.init(ApplicationFilte rConfig.java:108) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.jav a:3709) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4356 ) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) at org.apache.catalina.core.StandardHost.start(StandardHost.java:719) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443) at org.apache.catalina.startup.Embedded.start(Embedded.java:825) at com.orbitz.container.app.application.EmbeddedTomcat.startTomcat(Embedded Tomcat.java:284) at com.orbitz.container.app.application.EmbeddedTomcat.start(EmbeddedTomcat .java:111) at com.orbitz.container.app.application.WebappStarter$Starter.run(WebappSta rter.java:68) at java.lang.Thread.run(Thread.java:595) Thanks, Raghu
Re: Exception during Solr startup
Can you tell us a little bit more about your situation? What changed today? New Solr WAR? What version of Solr are you using? -Grant On Aug 13, 2008, at 10:55 AM, Kashyap, Raghu wrote: Hi, Today I started seeing this exception when I started solr instance. Any ideas what might be causing this problem? INFO: xsltCacheLifetimeSeconds=5 Aug 13, 2008 9:20:45 AM org.apache.solr.common.SolrException log SEVERE: java.lang.UnsupportedClassVersionError: Bad version number in .class file at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:620) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java: 124) at org .apache.catalina.loader.WebappClassLoader.findClassInternal(WebappCla ssLoader.java:1819) at org .apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader .java:872) at org .apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader .java:1327) at org .apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader .java:1206) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java: 319) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:242) at org .apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.jav a:239) at org .apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.j ava:260) at org .apache.solr.util.plugin.AbstractPluginLoader.create(AbstractPluginLo ader.java:83) at org .apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoad er.java:140) at org.apache.solr.core.SolrCore.loadSearchComponents(SolrCore.java:622) at org.apache.solr.core.SolrCore.init(SolrCore.java:408) at org.apache.solr.core.MultiCore.create(MultiCore.java:255) at org.apache.solr.core.MultiCore.load(MultiCore.java:139) at org .apache.solr.servlet.SolrDispatchFilter.initMultiCore(SolrDispatchFil ter.java:147) at org .apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java: 72) at org .apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFi lterConfig.java:275) at org .apache.catalina.core.ApplicationFilterConfig.setFilterDef(Applicatio nFilterConfig.java:397) at org .apache.catalina.core.ApplicationFilterConfig.init(ApplicationFilte rConfig.java:108) at org .apache.catalina.core.StandardContext.filterStart(StandardContext.jav a:3709) at org.apache.catalina.core.StandardContext.start(StandardContext.java: 4356 ) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) at org.apache.catalina.core.StandardHost.start(StandardHost.java:719) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443) at org.apache.catalina.startup.Embedded.start(Embedded.java: 825) at com .orbitz.container.app.application.EmbeddedTomcat.startTomcat(Embedded Tomcat.java:284) at com .orbitz.container.app.application.EmbeddedTomcat.start(EmbeddedTomcat .java:111) at com.orbitz.container.app.application.WebappStarter $Starter.run(WebappSta rter.java:68) at java.lang.Thread.run(Thread.java:595) Thanks, Raghu
RE: Exception during Solr startup
I am still trying to figure out what changed. From the SCM nothing seems to have changed from yesterday. We are using the nightly build of solr from 07/22 -Raghu -Original Message- From: Grant Ingersoll [mailto:[EMAIL PROTECTED] Sent: Wednesday, August 13, 2008 10:35 AM To: solr-user@lucene.apache.org Subject: Re: Exception during Solr startup Can you tell us a little bit more about your situation? What changed today? New Solr WAR? What version of Solr are you using? -Grant On Aug 13, 2008, at 10:55 AM, Kashyap, Raghu wrote: Hi, Today I started seeing this exception when I started solr instance. Any ideas what might be causing this problem? INFO: xsltCacheLifetimeSeconds=5 Aug 13, 2008 9:20:45 AM org.apache.solr.common.SolrException log SEVERE: java.lang.UnsupportedClassVersionError: Bad version number in .class file at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:620) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java: 124) at org .apache.catalina.loader.WebappClassLoader.findClassInternal(WebappCla ssLoader.java:1819) at org .apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader .java:872) at org .apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader .java:1327) at org .apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader .java:1206) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java: 319) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:242) at org .apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.jav a:239) at org .apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.j ava:260) at org .apache.solr.util.plugin.AbstractPluginLoader.create(AbstractPluginLo ader.java:83) at org .apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoad er.java:140) at org.apache.solr.core.SolrCore.loadSearchComponents(SolrCore.java:622) at org.apache.solr.core.SolrCore.init(SolrCore.java:408) at org.apache.solr.core.MultiCore.create(MultiCore.java:255) at org.apache.solr.core.MultiCore.load(MultiCore.java:139) at org .apache.solr.servlet.SolrDispatchFilter.initMultiCore(SolrDispatchFil ter.java:147) at org .apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java: 72) at org .apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFi lterConfig.java:275) at org .apache.catalina.core.ApplicationFilterConfig.setFilterDef(Applicatio nFilterConfig.java:397) at org .apache.catalina.core.ApplicationFilterConfig.init(ApplicationFilte rConfig.java:108) at org .apache.catalina.core.StandardContext.filterStart(StandardContext.jav a:3709) at org.apache.catalina.core.StandardContext.start(StandardContext.java: 4356 ) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) at org.apache.catalina.core.StandardHost.start(StandardHost.java:719) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443) at org.apache.catalina.startup.Embedded.start(Embedded.java: 825) at com .orbitz.container.app.application.EmbeddedTomcat.startTomcat(Embedded Tomcat.java:284) at com .orbitz.container.app.application.EmbeddedTomcat.start(EmbeddedTomcat .java:111) at com.orbitz.container.app.application.WebappStarter $Starter.run(WebappSta rter.java:68) at java.lang.Thread.run(Thread.java:595) Thanks, Raghu
Re: Exception during Solr startup
Is this perhaps due to the renaming of multicore.xml to solr.xml? Erik On Aug 13, 2008, at 10:55 AM, Kashyap, Raghu wrote: Hi, Today I started seeing this exception when I started solr instance. Any ideas what might be causing this problem? INFO: xsltCacheLifetimeSeconds=5 Aug 13, 2008 9:20:45 AM org.apache.solr.common.SolrException log SEVERE: java.lang.UnsupportedClassVersionError: Bad version number in .class file at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:620) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java: 124) at org .apache.catalina.loader.WebappClassLoader.findClassInternal(WebappCla ssLoader.java:1819) at org .apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader .java:872) at org .apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader .java:1327) at org .apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader .java:1206) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java: 319) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:242) at org .apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.jav a:239) at org .apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.j ava:260) at org .apache.solr.util.plugin.AbstractPluginLoader.create(AbstractPluginLo ader.java:83) at org .apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoad er.java:140) at org.apache.solr.core.SolrCore.loadSearchComponents(SolrCore.java:622) at org.apache.solr.core.SolrCore.init(SolrCore.java:408) at org.apache.solr.core.MultiCore.create(MultiCore.java:255) at org.apache.solr.core.MultiCore.load(MultiCore.java:139) at org .apache.solr.servlet.SolrDispatchFilter.initMultiCore(SolrDispatchFil ter.java:147) at org .apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java: 72) at org .apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFi lterConfig.java:275) at org .apache.catalina.core.ApplicationFilterConfig.setFilterDef(Applicatio nFilterConfig.java:397) at org .apache.catalina.core.ApplicationFilterConfig.init(ApplicationFilte rConfig.java:108) at org .apache.catalina.core.StandardContext.filterStart(StandardContext.jav a:3709) at org.apache.catalina.core.StandardContext.start(StandardContext.java: 4356 ) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) at org.apache.catalina.core.StandardHost.start(StandardHost.java:719) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443) at org.apache.catalina.startup.Embedded.start(Embedded.java: 825) at com .orbitz.container.app.application.EmbeddedTomcat.startTomcat(Embedded Tomcat.java:284) at com .orbitz.container.app.application.EmbeddedTomcat.start(EmbeddedTomcat .java:111) at com.orbitz.container.app.application.WebappStarter $Starter.run(WebappSta rter.java:68) at java.lang.Thread.run(Thread.java:595) Thanks, Raghu
Re: Exception during Solr startup
On Wed, Aug 13, 2008 at 10:55 AM, Kashyap, Raghu [EMAIL PROTECTED] wrote: SEVERE: java.lang.UnsupportedClassVersionError: Bad version number in .class file This is normally a mismatch between java compiler and runtime (like using Java6 to compile and Java5 to try and run). -Yonik
