Re: Latest trunk locks execution thread in SolrCore.getSearcher()
Solr now starts up fine in Tomcat with both JMX and ReplicationHandler enabled if I construct my working copy as follows: download solr-r815830 reverse merge r815587 (SOLR-1427: fixed registry MBean issue) apply 1427.afterlatch.patch I haven't tried applying 1427.afterlatch.patch to the svn head, but I assume the results would be the same. 2009/9/20 Chris Hostetter hossman_luc...@fucit.org: : instance, which is based on Solr SVN r815830. This patch did not seem : to solve the hang problem; once I reenabled JMX, then the process : would hang at the same spot, i.e. right after Yeah, I believe the hang is really orthoginal to the SearchComponents issue and seems to be because of the types of Statistics ReplicationHandler wants to expose and the way certain JMX implementaions deal with a newly registered MBean -- the changes made in SOLR-1427 (which yonik has since reverted) just exposed this problem by registering hte MBeans at an inopportune time. I'd apprecaite it if anyone who observed this problem in their version of Java could try out this patch... https://issues.apache.org/jira/secure/attachment/12420151/SOLR-1427.afterlatch.patch ...with both the ReplicationHandler and JMX turned on. The patch can be applied to the current trunk (where yonik reverted the previous code that caused the hang) If things still lock up even with that patch, a full threaddump would be much apprecaited. -Hoss
Re: Latest trunk locks execution thread in SolrCore.getSearcher()
: instance, which is based on Solr SVN r815830. This patch did not seem : to solve the hang problem; once I reenabled JMX, then the process : would hang at the same spot, i.e. right after Yeah, I believe the hang is really orthoginal to the SearchComponents issue and seems to be because of the types of Statistics ReplicationHandler wants to expose and the way certain JMX implementaions deal with a newly registered MBean -- the changes made in SOLR-1427 (which yonik has since reverted) just exposed this problem by registering hte MBeans at an inopportune time. I'd apprecaite it if anyone who observed this problem in their version of Java could try out this patch... https://issues.apache.org/jira/secure/attachment/12420151/SOLR-1427.afterlatch.patch ...with both the ReplicationHandler and JMX turned on. The patch can be applied to the current trunk (where yonik reverted the previous code that caused the hang) If things still lock up even with that patch, a full threaddump would be much apprecaited. -Hoss
RE: Latest trunk locks execution thread in SolrCore.getSearcher()
HURAH! Works for me! Thank you! -Original Message- From: Chris Hostetter [mailto:hossman_luc...@fucit.org] Sent: Sunday, September 20, 2009 6:49 PM To: solr-user@lucene.apache.org Subject: Re: Latest trunk locks execution thread in SolrCore.getSearcher() : instance, which is based on Solr SVN r815830. This patch did not seem : to solve the hang problem; once I reenabled JMX, then the process : would hang at the same spot, i.e. right after Yeah, I believe the hang is really orthoginal to the SearchComponents issue and seems to be because of the types of Statistics ReplicationHandler wants to expose and the way certain JMX implementaions deal with a newly registered MBean -- the changes made in SOLR-1427 (which yonik has since reverted) just exposed this problem by registering hte MBeans at an inopportune time. I'd apprecaite it if anyone who observed this problem in their version of Java could try out this patch... https://issues.apache.org/jira/secure/attachment/12420151/SOLR-1427.afte rlatch.patch ...with both the ReplicationHandler and JMX turned on. The patch can be applied to the current trunk (where yonik reverted the previous code that caused the hang) If things still lock up even with that patch, a full threaddump would be much apprecaited. -Hoss
Re: Latest trunk locks execution thread in SolrCore.getSearcher()
Also, do you have any custom components or anything that implements SolrInfoMBean? On Sep 18, 2009, at 8:16 AM, Grant Ingersoll wrote: Can you try the patch I just put up on https://issues.apache.org/jira/browse/SOLR-1427 and let me know if it works when JMX is enabled? Also, do you have warming queries setup? On Sep 17, 2009, at 12:46 PM, Chris Harris wrote: It looks like this works as a fix for me as well. (I'm not currently using JMX for anything anyway.) Curiously, the single-core example solrconfig.xml also has jmx /, but it doesn't seem to be a problem there. 2009/9/17 Dadasheva, Olga olga_dadash...@harvard.edu: Hi, FWIW: disabling jmx/ fixed this problem for me. Thanks you! -Olga -Original Message- From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley Sent: Thursday, September 17, 2009 1:09 PM To: solr-user@lucene.apache.org Subject: Re: Latest trunk locks execution thread in SolrCore.getSearcher() Interesting... I still haven't been able to reproduce a hang with either jetty or tomcat. I enabled replication and JMX... still nothing. -Yonik http://www.lucidimagination.com On Thu, Sep 17, 2009 at 12:35 PM, Chris Harris rygu...@gmail.com wrote: I found what looks like the same issue when I tried to install r815830 under Tomcat. (It works ok with the normal Jetty example/ start.jar.) I haven't checked the stack trace, but Tomcat would hang right after the message INFO: Adding debug component:org.apache.solr.handler.component.debugcompon...@1904e0d showed up in the log. I have a little more evidence about Yonik's theory that SOLR-1427 is part of the cause. In particular, when I reverse-merged r815587 (the commit for SOLR-1427) into (out of?) my r815830-based working copy, then Tomcat was able to load Solr normally. 2009/9/16 Yonik Seeley yo...@lucidimagination.com: On a quick look, it looks like this was caused (or at least triggered by) https://issues.apache.org/jira/browse/SOLR-1427 Registering the bean in the SolrCore constructor causes it to immediately turn around and ask for the stats which asks for a searcher, which blocks. -Yonik http://www.lucidimagination.com On Wed, Sep 16, 2009 at 9:34 PM, Dadasheva, Olga olga_dadash...@harvard.edu wrote: Hi, I am testing EmbeddedSolrServer vs StreamingUpdateSolrServer for my crawlers using more or less recent Solr code and everything was fine till today when I took the latest trunk code. When I start my crawler I see a number of INFO outputs 2009-09-16 21:08:29,399 INFO Adding component:org.apache.solr.handler.component.HighlightComponent @36ae8 3 (SearchHandler.java:132) - [main] 2009-09-16 21:08:29,400 INFO Adding component:org.apache.solr.handler.component.StatsComponent @1fb24d3 (SearchHandler.java:132) - [main] 2009-09-16 21:08:29,401 INFO Adding component:org.apache.solr.handler.component.TermVectorComponent @14ba 9a2 (SearchHandler.java:132) - [main] 2009-09-16 21:08:29,402 INFO Adding debug component:org.apache.solr.handler.component.DebugComponent @12ea1dd (SearchHandler.java:137) - [main] and then the log/program stops. -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using Solr/Lucene: http://www.lucidimagination.com/search -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using Solr/Lucene: http://www.lucidimagination.com/search
Re: Latest trunk locks execution thread in SolrCore.getSearcher()
No, I'm pretty sure nothing implements SolrInfoMBean. I applied the new 1K version of SOLR-1427.patch from https://issues.apache.org/jira/browse/SOLR-1427 (which appears to be a secondary patch, to be applied once the main SOLR-1427 patch has already been applied) to my problematic Solr instance, which is based on Solr SVN r815830. This patch did not seem to solve the hang problem; once I reenabled JMX, then the process would hang at the same spot, i.e. right after INFO: Adding debug component:org.apache.solr.handler.component.debugcompon...@1d7b222 appeared in the Tomcat log. When Solr/Tomcat are hung, there are two Solr-related threads that show up in a thread dump. I'll paste those stack traces below: pool-1-thread-1 prio=6 tid=0x0b1ef800 nid=0xdc8 waiting on condition [0x0b68f000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x035f3e60 (a java.util.concurrent.CountDownLatch$Sync) at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(Unknown Source) at java.util.concurrent.CountDownLatch.await(Unknown Source) at org.apache.solr.core.SolrCore$1.call(SolrCore.java:559) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Thread-1 prio=6 tid=0x00c92c00 nid=0xf14 in Object.wait() [0x0b19e000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x035d2ab0 (a java.lang.Object) at java.lang.Object.wait(Object.java:485) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:994) - locked 0x035d2ab0 (a java.lang.Object) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:907) at org.apache.solr.handler.ReplicationHandler.getIndexVersion(ReplicationHandler.java:472) at org.apache.solr.handler.ReplicationHandler.getStatistics(ReplicationHandler.java:490) at org.apache.solr.core.JmxMonitoredMap$SolrDynamicMBean.getMBeanInfo(JmxMonitoredMap.java:224) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassName(Unknown Source) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(Unknown Source) at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(Unknown Source) at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:137) at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:47) at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:446) at org.apache.solr.core.SolrCore.init(SolrCore.java:578) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:425) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:278) at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:117) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:83) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:275) snip 2009/9/18 Grant Ingersoll gsing...@apache.org: Also, do you have any custom components or anything that implements SolrInfoMBean? On Sep 18, 2009, at 8:16 AM, Grant Ingersoll wrote: Can you try the patch I just put up on https://issues.apache.org/jira/browse/SOLR-1427 and let me know if it works when JMX is enabled? Also, do you have warming queries setup? On Sep 17, 2009, at 12:46 PM, Chris Harris wrote: It looks like this works as a fix for me as well. (I'm not currently using JMX for anything anyway.) Curiously, the single-core example solrconfig.xml also has jmx /, but it doesn't seem to be a problem there. 2009/9/17 Dadasheva, Olga olga_dadash...@harvard.edu: Hi, FWIW: disabling jmx/ fixed this problem for me. Thanks you! -Olga -Original Message- From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley Sent: Thursday, September 17, 2009 1:09 PM To: solr-user@lucene.apache.org Subject: Re: Latest trunk locks execution thread in SolrCore.getSearcher() Interesting... I still haven't been able to reproduce a hang with either jetty or tomcat. I enabled replication and JMX... still nothing. -Yonik http://www.lucidimagination.com On Thu, Sep 17, 2009 at 12:35 PM, Chris Harris rygu...@gmail.com wrote: I found what looks like the same
Re: Latest trunk locks execution thread in SolrCore.getSearcher()
Forgot to answer this one. Yes, I do have a warming query to get the sort caches up to speed. I think it takes a while to run; my guess would be 30 seconds or so. 2009/9/18 Grant Ingersoll gsing...@apache.org: Also, do you have warming queries setup? On Sep 17, 2009, at 12:46 PM, Chris Harris wrote: It looks like this works as a fix for me as well. (I'm not currently using JMX for anything anyway.) Curiously, the single-core example solrconfig.xml also has jmx /, but it doesn't seem to be a problem there. 2009/9/17 Dadasheva, Olga olga_dadash...@harvard.edu: Hi, FWIW: disabling jmx/ fixed this problem for me.
Re: Latest trunk locks execution thread in SolrCore.getSearcher()
I found what looks like the same issue when I tried to install r815830 under Tomcat. (It works ok with the normal Jetty example/start.jar.) I haven't checked the stack trace, but Tomcat would hang right after the message INFO: Adding debug component:org.apache.solr.handler.component.debugcompon...@1904e0d showed up in the log. I have a little more evidence about Yonik's theory that SOLR-1427 is part of the cause. In particular, when I reverse-merged r815587 (the commit for SOLR-1427) into (out of?) my r815830-based working copy, then Tomcat was able to load Solr normally. 2009/9/16 Yonik Seeley yo...@lucidimagination.com: On a quick look, it looks like this was caused (or at least triggered by) https://issues.apache.org/jira/browse/SOLR-1427 Registering the bean in the SolrCore constructor causes it to immediately turn around and ask for the stats which asks for a searcher, which blocks. -Yonik http://www.lucidimagination.com On Wed, Sep 16, 2009 at 9:34 PM, Dadasheva, Olga olga_dadash...@harvard.edu wrote: Hi, I am testing EmbeddedSolrServer vs StreamingUpdateSolrServer for my crawlers using more or less recent Solr code and everything was fine till today when I took the latest trunk code. When I start my crawler I see a number of INFO outputs 2009-09-16 21:08:29,399 INFO Adding component:org.apache.solr.handler.component.highlightcompon...@36ae83 (SearchHandler.java:132) - [main] 2009-09-16 21:08:29,400 INFO Adding component:org.apache.solr.handler.component.statscompon...@1fb24d3 (SearchHandler.java:132) - [main] 2009-09-16 21:08:29,401 INFO Adding component:org.apache.solr.handler.component.termvectorcompon...@14ba9a2 (SearchHandler.java:132) - [main] 2009-09-16 21:08:29,402 INFO Adding debug component:org.apache.solr.handler.component.debugcompon...@12ea1dd (SearchHandler.java:137) - [main] and then the log/program stops.
Re: Latest trunk locks execution thread in SolrCore.getSearcher()
: Interesting... I still haven't been able to reproduce a hang with : either jetty or tomcat. : I enabled replication and JMX... still nothing. I haven't tried to reproduce the problem, and i don't even have a concrete theory as to what the problem is ... but i did want to point out something that smells fishy... looking over the patch in SOLR-1427, i notice that the patch adds new code so that SolrResourceLoader adds everything it instantiates to the registry map -- but there are no lines in that patch removing any code that was already adding stuff to the infoRegistry ... which means any bean types that were already getting instantiated by the ResourceLoader, and explicitly added to hte registry are probably now getting added to the registry twice ... for a normal Map that's not a big deal, but perhaps the JMX backed map has some problems with that? (it's not likely something that we ever tested extensively) Alternately: maybe the problem isn't with adding to the JMX backed map twice ... maybe the issue is what state the beans are in each time we add them to the map ... the first time requesthandlers are added to the infoRegistry it's prior to the inform(SolrCore) call so they havne't been fully initalized yet -- but the way SOlrResourceLoader adds them now (the second time) is *after* inform(SolrCore) has been called and in between the creation/countDown of the CountDownLatch for the searchExecutor ... ...hmmm, actually this starting to sound like a concrete theory, ReplicationHandler.getStatistics has very different behavior before/after inform(SolrCore) has been called Maybe moving the resourceLoader.inform(infoRegistry) call PRIOR to resourceLoader.inform( this ) *or* AFTER latch.countDown() would solve this problem? -Hoss
Re: Latest trunk locks execution thread in SolrCore.getSearcher()
: Maybe moving the resourceLoader.inform(infoRegistry) call PRIOR to
: resourceLoader.inform( this ) *or* AFTER latch.countDown() would solve
: this problem?
If this really is the problem, then a more general purpose solution to
future proof us against similar problems down the road would probably be
to get rid of the current if (config.jmxConfig.enabled) logic for
initializing SolrCore.infoRegistry, and instead us something like...
infoRegistry = new ConcurrentHashMapString, SolrInfoMBean();
...
// do all initialization, add things to infoRegistry as needed
...
// done with all initilation, core and all MBeans are fully functional
if (config.jmxConfig.enabled) {
Map tmp = infoRegistry;
infoRegistry = new JmxMonitoredMapString, SolrInfoMBean(name,
config.jmxConfig);
infoRegistry.putAll(tmp);
} else {
log.info(JMX monitoring not detected for core: + name);
}
...that way we wouldn't have to worry about any other type of resource
contention that might happen from the JMX monitor wanting ot get info from
the MBeans as soon as they are added to the registry.
-Hoss
Re: Latest trunk locks execution thread in SolrCore.getSearcher()
On Sep 16, 2009, at 6:54 PM, Yonik Seeley wrote: On a quick look, it looks like this was caused (or at least triggered by) https://issues.apache.org/jira/browse/SOLR-1427 Registering the bean in the SolrCore constructor causes it to immediately turn around and ask for the stats which asks for a searcher, which blocks. Hmm, yeah that sounds likely. We can revert if need be and push to 1.5 -Yonik http://www.lucidimagination.com On Wed, Sep 16, 2009 at 9:34 PM, Dadasheva, Olga olga_dadash...@harvard.edu wrote: Hi, I am testing EmbeddedSolrServer vs StreamingUpdateSolrServer for my crawlers using more or less recent Solr code and everything was fine till today when I took the latest trunk code. When I start my crawler I see a number of INFO outputs 2009-09-16 21:08:29,399 INFO Adding component:org.apache.solr.handler.component.highlightcompon...@36ae83 (SearchHandler.java:132) - [main] 2009-09-16 21:08:29,400 INFO Adding component:org.apache.solr.handler.component.statscompon...@1fb24d3 (SearchHandler.java:132) - [main] 2009-09-16 21:08:29,401 INFO Adding component:org.apache.solr.handler.component.TermVectorComponent @14ba9a2 (SearchHandler.java:132) - [main] 2009-09-16 21:08:29,402 INFO Adding debug component:org.apache.solr.handler.component.debugcompon...@12ea1dd (SearchHandler.java:137) - [main] and then the log/program stops. The thread dump reveals the following: main prio=3 tid=0x0003 nid=0x2 in Object.wait() [0xfe67c000..0xfe67fd80] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0xeaaf6b10 (a java.lang.Object) at java.lang.Object.wait(Object.java:485) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java: 991) - locked 0xeaaf6b10 (a java.lang.Object) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java: 904) at org.apache.solr.handler.ReplicationHandler.getIndexVersion (ReplicationHa ndler.java:472) at org.apache.solr.handler.ReplicationHandler.getStatistics (ReplicationHand ler.java:490) at org.apache.solr.core.JmxMonitoredMap$SolrDynamicMBean.getMBeanInfo (JmxMo nitoredMap.java:224) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassNa me(DefaultMBeanServerInterceptor.java:321) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean (Defa ultMBeanServerInterceptor.java:307) at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean (JmxMBeanServer.java :482) at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:137) at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:47) at org.apache.solr.core.SolrResourceLoader.inform (SolrResourceLoader.java:4 46) at org.apache.solr.core.SolrCore.init(SolrCore.java:578) at harvard.solr.search.service.EmbeddedSearchService.setSolrHome (EmbeddedSe archService.java:47) The same is happening for the StreamingUpdateSolrServer. Do you think it's a bug? Thank you for looking into it, -Olga -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using Solr/Lucene: http://www.lucidimagination.com/search
Re: Latest trunk locks execution thread in SolrCore.getSearcher()
Ok, let me revise my previous report:
It turns out if I take the single-core or multicore example stuff from
running ant example on r815830 and plop them into Tomcat, this
doesn't hang. However, if I plop Solr as I configure it for production
use into Tomcat, it does hang.
The main ways I can think of how my Solr conf differs from the example
configs are the use of ExtractingRequestHandler and of the Solr qsol
patch (https://issues.apache.org/jira/browse/SOLR-896). As an
experiment, I tried removing apache-solr-cell-1.4-dev.jar and
apache-solr-qsol-1.4-dev.jar from my Solr lib directory, and removing
reference to qsol from my solrconfig.xml, but this does not prevent
Solr from hanging in Tomcat; in this case Tomcat hung in the same spot
as I reported before.
I did find one way to get my production-like Solr configuration not to
hang on startup. That was to 1) remove apache-solr-qsol-1.4-dev.jar
from my lib/ directory, but to 2) still leave this config option in my
solrconfig.xml:
queryParser name=qsol class=org.apache.solr.search.QsolQParserPlugin/
Something about the class loading exception seems to get around the bean issue.
Here's an excerpt from my Tomcat log:
...
Sep 17, 2009 12:01:22 PM org.apache.solr.core.SolrCore init
INFO: [filingcore] Opening new SolrCore at
c:\tomcat-solr\solrme\filingcore\,
dataDir=c:\tomcat-solr\solrme\filingcore\data\
Sep 17, 2009 12:01:22 PM org.apache.solr.core.JmxMonitoredMap init
INFO: JMX monitoring is enabled. Adding Solr mbeans to JMX Server:
com.sun.jmx.mbeanserver.jmxmbeanser...@ecb67f
Sep 17, 2009 12:01:22 PM org.apache.solr.core.SolrCore initListeners
INFO: [filingcore] Added SolrEventListener:
org.apache.solr.core.QuerySenderListener{queries=[{q=*:*,sort=date
asc,companynameforsorting asc,formtype asc,start=0,rows=10}]}
Sep 17, 2009 12:01:22 PM org.apache.solr.core.SolrCore initListeners
INFO: [filingcore] Added SolrEventListener:
org.apache.solr.core.QuerySenderListener{queries=[{q=*:*,sort=date
asc,companynameforsorting asc,formtype asc,start=0,rows=10}]}
Sep 17, 2009 12:01:22 PM org.apache.solr.request.XSLTResponseWriter init
INFO: xsltCacheLifetimeSeconds=5
Sep 17, 2009 12:01:22 PM org.apache.solr.core.SolrCore finalize
SEVERE: REFCOUNT ERROR: unreferenced
org.apache.solr.core.solrc...@4eeaaf (filingcore) has a reference
count of 1
Sep 17, 2009 12:01:22 PM org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: Error loading class
'org.apache.solr.search.QsolQParserPlugin'
at
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:311)
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:411)
at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:1479)
at org.apache.solr.core.SolrCore.initQParsers(SolrCore.java:1433)
at org.apache.solr.core.SolrCore.init(SolrCore.java:533)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:425)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:278)
at
org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:117)
at
org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:83)
at
org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:275)
at
org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:397)
at
org.apache.catalina.core.ApplicationFilterConfig.init(ApplicationFilterConfig.java:108)
at
org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3800)
at
org.apache.catalina.core.StandardContext.start(StandardContext.java:4450)
at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:526)
at
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:630)
at
org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:556)
at
org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:491)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1206)
at
org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:314)
at
org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:722)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
at
org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
at
org.apache.catalina.core.StandardService.start(StandardService.java:516)
at
org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
at
RE: Latest trunk locks execution thread in SolrCore.getSearcher()
Hi, FWIW: disabling jmx/ fixed this problem for me. Thanks you! -Olga -Original Message- From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley Sent: Thursday, September 17, 2009 1:09 PM To: solr-user@lucene.apache.org Subject: Re: Latest trunk locks execution thread in SolrCore.getSearcher() Interesting... I still haven't been able to reproduce a hang with either jetty or tomcat. I enabled replication and JMX... still nothing. -Yonik http://www.lucidimagination.com On Thu, Sep 17, 2009 at 12:35 PM, Chris Harris rygu...@gmail.com wrote: I found what looks like the same issue when I tried to install r815830 under Tomcat. (It works ok with the normal Jetty example/start.jar.) I haven't checked the stack trace, but Tomcat would hang right after the message INFO: Adding debug component:org.apache.solr.handler.component.debugcompon...@1904e0d showed up in the log. I have a little more evidence about Yonik's theory that SOLR-1427 is part of the cause. In particular, when I reverse-merged r815587 (the commit for SOLR-1427) into (out of?) my r815830-based working copy, then Tomcat was able to load Solr normally. 2009/9/16 Yonik Seeley yo...@lucidimagination.com: On a quick look, it looks like this was caused (or at least triggered by) https://issues.apache.org/jira/browse/SOLR-1427 Registering the bean in the SolrCore constructor causes it to immediately turn around and ask for the stats which asks for a searcher, which blocks. -Yonik http://www.lucidimagination.com On Wed, Sep 16, 2009 at 9:34 PM, Dadasheva, Olga olga_dadash...@harvard.edu wrote: Hi, I am testing EmbeddedSolrServer vs StreamingUpdateSolrServer for my crawlers using more or less recent Solr code and everything was fine till today when I took the latest trunk code. When I start my crawler I see a number of INFO outputs 2009-09-16 21:08:29,399 INFO Adding component:org.apache.solr.handler.component.highlightcompon...@36ae8 3 (SearchHandler.java:132) - [main] 2009-09-16 21:08:29,400 INFO Adding component:org.apache.solr.handler.component.statscompon...@1fb24d3 (SearchHandler.java:132) - [main] 2009-09-16 21:08:29,401 INFO Adding component:org.apache.solr.handler.component.termvectorcompon...@14ba 9a2 (SearchHandler.java:132) - [main] 2009-09-16 21:08:29,402 INFO Adding debug component:org.apache.solr.handler.component.debugcompon...@12ea1dd (SearchHandler.java:137) - [main] and then the log/program stops.
Latest trunk locks execution thread in SolrCore.getSearcher()
Hi, I am testing EmbeddedSolrServer vs StreamingUpdateSolrServer for my crawlers using more or less recent Solr code and everything was fine till today when I took the latest trunk code. When I start my crawler I see a number of INFO outputs 2009-09-16 21:08:29,399 INFO Adding component:org.apache.solr.handler.component.highlightcompon...@36ae83 (SearchHandler.java:132) - [main] 2009-09-16 21:08:29,400 INFO Adding component:org.apache.solr.handler.component.statscompon...@1fb24d3 (SearchHandler.java:132) - [main] 2009-09-16 21:08:29,401 INFO Adding component:org.apache.solr.handler.component.termvectorcompon...@14ba9a2 (SearchHandler.java:132) - [main] 2009-09-16 21:08:29,402 INFO Adding debug component:org.apache.solr.handler.component.debugcompon...@12ea1dd (SearchHandler.java:137) - [main] and then the log/program stops. The thread dump reveals the following: main prio=3 tid=0x0003 nid=0x2 in Object.wait() [0xfe67c000..0xfe67fd80] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0xeaaf6b10 (a java.lang.Object) at java.lang.Object.wait(Object.java:485) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:991) - locked 0xeaaf6b10 (a java.lang.Object) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:904) at org.apache.solr.handler.ReplicationHandler.getIndexVersion(ReplicationHa ndler.java:472) at org.apache.solr.handler.ReplicationHandler.getStatistics(ReplicationHand ler.java:490) at org.apache.solr.core.JmxMonitoredMap$SolrDynamicMBean.getMBeanInfo(JmxMo nitoredMap.java:224) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassNa me(DefaultMBeanServerInterceptor.java:321) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(Defa ultMBeanServerInterceptor.java:307) at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java :482) at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:137) at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:47) at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:4 46) at org.apache.solr.core.SolrCore.init(SolrCore.java:578) at harvard.solr.search.service.EmbeddedSearchService.setSolrHome(EmbeddedSe archService.java:47) The same is happening for the StreamingUpdateSolrServer. Do you think it's a bug? Thank you for looking into it, -Olga
Re: Latest trunk locks execution thread in SolrCore.getSearcher()
On a quick look, it looks like this was caused (or at least triggered by) https://issues.apache.org/jira/browse/SOLR-1427 Registering the bean in the SolrCore constructor causes it to immediately turn around and ask for the stats which asks for a searcher, which blocks. -Yonik http://www.lucidimagination.com On Wed, Sep 16, 2009 at 9:34 PM, Dadasheva, Olga olga_dadash...@harvard.edu wrote: Hi, I am testing EmbeddedSolrServer vs StreamingUpdateSolrServer for my crawlers using more or less recent Solr code and everything was fine till today when I took the latest trunk code. When I start my crawler I see a number of INFO outputs 2009-09-16 21:08:29,399 INFO Adding component:org.apache.solr.handler.component.highlightcompon...@36ae83 (SearchHandler.java:132) - [main] 2009-09-16 21:08:29,400 INFO Adding component:org.apache.solr.handler.component.statscompon...@1fb24d3 (SearchHandler.java:132) - [main] 2009-09-16 21:08:29,401 INFO Adding component:org.apache.solr.handler.component.termvectorcompon...@14ba9a2 (SearchHandler.java:132) - [main] 2009-09-16 21:08:29,402 INFO Adding debug component:org.apache.solr.handler.component.debugcompon...@12ea1dd (SearchHandler.java:137) - [main] and then the log/program stops. The thread dump reveals the following: main prio=3 tid=0x0003 nid=0x2 in Object.wait() [0xfe67c000..0xfe67fd80] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0xeaaf6b10 (a java.lang.Object) at java.lang.Object.wait(Object.java:485) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:991) - locked 0xeaaf6b10 (a java.lang.Object) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:904) at org.apache.solr.handler.ReplicationHandler.getIndexVersion(ReplicationHa ndler.java:472) at org.apache.solr.handler.ReplicationHandler.getStatistics(ReplicationHand ler.java:490) at org.apache.solr.core.JmxMonitoredMap$SolrDynamicMBean.getMBeanInfo(JmxMo nitoredMap.java:224) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassNa me(DefaultMBeanServerInterceptor.java:321) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(Defa ultMBeanServerInterceptor.java:307) at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java :482) at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:137) at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:47) at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:4 46) at org.apache.solr.core.SolrCore.init(SolrCore.java:578) at harvard.solr.search.service.EmbeddedSearchService.setSolrHome(EmbeddedSe archService.java:47) The same is happening for the StreamingUpdateSolrServer. Do you think it's a bug? Thank you for looking into it, -Olga
