RE: Solr 4.3 Startup with Multiple Cores Hangs on Registering Core

2013-09-06 Thread Austin Rasmussen
Thanks for clearing that up Erick.  The updateLog XML element isn't present in 
any of the solrconfig.xml files, so I don't believe this is enabled.  

I posted the directory listing of all of the core data directories in a prior 
post, but there are no files/folders found that contain tlog in the name of 
them.

-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com] 
Sent: Friday, September 06, 2013 9:18 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr 4.3 Startup with Multiple Cores Hangs on Registering Core

bq: I'm actually not using the transaction log (or the 
NRTCachingDirectoryFactory); it's currently set up to use the 
MMapDirectoryFactory,

This isn't relevant to whether you're using the update log or not, this is just 
how the index is handled. Look for something in your solrconfig.xml
like:
 updateLog
  str name=dir${solr.ulog.dir:}/str
/updateLog

The other thing to check is if you have files in a tlog directory that's a 
sibling to your index directory as Hoss suggested.

You may well NOT have any transaction log, but it's something to check.



RE: Solr 4.3 Startup with Multiple Cores Hangs on Registering Core

2013-09-06 Thread Austin Rasmussen
: Do all of your cores have newSearcher event listners configured or just
: 2 (i'm trying to figure out if it's a timing fluke that these two are 
stalled, or if it's something special about the configs)

All of my cores have both the newSearcher and firstSearcher event listeners 
configured. (The firstSearcher actually doesn't have any queries configured 
against it, so it probably should just be removed altogether)

: Can you try removing the newSearcher listners to confirm that that does in 
fact make the problem go away?

Removing the newSearcher listeners does not make the problem go away; 
however, removing the firstSearcher listener (even if the newSearcher 
listener is still configured) does make the problem go away.

: With the newSearcher listeners in place, Can you try setting 
spellcheck=false as a query param on the newSearcher listeners you have 
configured and 
: see if that works arround the problem?

Adding the spellcheck=false param to the firstSearcher listener does appear 
to work around the problem.

: Assuming it's just 2 cores using these listeners: can you reproduce this 
problem with a simpler seup where only one of the affected cores is in use?

Since it's not just these two cores, I'm not sure how to produce much of a 
simpler setup.  I did attempt to limit how many cores are loaded in the 
solr.xml, and found that if I cut it down to 56, it was able to load 
successfully (without any of the above config changed).

If I cut it down to 57 cores, it doesn't hang at registering core any more, 
it actually gets as far as  QuerySenderListener sending requests to 
Searcher@2f28849 main{StandardDirectoryReader(...

If 58+ cores are loaded at start up, that's when it begins to hang at 
registering core.  However, it always hangs on the *last* core configured in 
the solr.xml, regardless of how many cores are being loaded.


: can you reproduce using Solr 4.4?
: It would be helpful if you could create a jira and attach...
: * your complete configs -- or at least some configs similar to yours that are 
complete enough to reproduce the startup problem.  
: * some sample data (based on
: your initial description, i'm guessing there at least needs to be a handful 
of docs in the index -- and most likelye they need to match your warming query 
-: - but we don't need your actual indexes, just some docs that will work with 
your configs that we can index 
:  restart to see the problem. 
: * these thread dumps.

I can likely get to this early next week, both checking into how this behaves 
using Solr 4.4 and submitting a JIRA with your requested info.


Solr 4.3 Startup with Multiple Cores Hangs on Registering Core

2013-09-05 Thread Austin Rasmussen
Hello,

I currently have Solr 4.3 set up with about 400 cores set to load upon start 
up.  When starting Solr with an empty index for each core, Solr is able to load 
all of the cores and start up normally as expected.  However, after running a 
dataimport on all cores and restarting Solr, it hangs at 
org.apache.solr.core.CoreContainer; registering core: ... without any type of 
error message in the log.  The process still exists at this point, but doesn't 
make any progress even if left for a period of time.  Prior to the restart, 
Solr continues to function normally, and is searchable.

Solr is currently running in master-slave replication, and this same, exact 
behavior occurs on the master and both slaves.

I've checked all of the system log files and am also unable to find any errors 
or messages that would point to a particular problem.  Originally, I had 
thought it may have been related to an open file limit, but I also tried 
raising the limit to 65k, and Solr continued to hang at the same spot.  It does 
appear to be related to files to an extent, since removing the index/data 
directory of half of the cores does allow Solr to start up normally.

Any help or suggestions are appreciated.

Thanks!