Thanks for your reply Daniel. The dataDir is set in each solrconfig.xml; each one has been checked to ensure it points to its corresponding location. The error we see is that on machine reboot not all of the shards start successfully, and if the fail was to be a leader the replicas can't take its place (presumably because the leader incorrect data directory is inconsistent with their own).
More detail that I can add is that the catalina.out log for failed shards reports: May 15, 2013 5:56:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/solr] created a ThreadLocal with key of type [org.apache.solr.schema.DateField.ThreadLocalDateFormat] (value [org.apache.solr.schema.DateField$ThreadLocalDateFormat@524e13f6]) and a value of type [org.apache.solr.schema.DateField.ISO8601CanonicalDateFormat] (value [org.apache.solr.schema.DateField$ISO8601CanonicalDateFormat@6b2ed43a]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. This doesn't (to me) relate to the problem, but that doesn't necessarily mean it's not. Plus, it's the only SEVERE reported and only reported in the failed shard catalina.out log. Checking the zookeeper logs, we're seeing: 2013-05-16 13:25:46,839 [myid:1] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@762] - Connection broken for id 3, my id = 1, error = java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(Quoru mCnxManager.java:747) 2013-05-16 13:25:46,841 [myid:1] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@765] - Interrupting SendWorker 2013-05-16 13:25:46,842 [myid:1] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@679] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.re portInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw aitNanos(AbstractQueuedSynchronizer.java:2095) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389 ) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(Quorum CnxManager.java:831) at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnx Manager.java:62) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(Quoru mCnxManager.java:667) 2013-05-16 13:25:46,843 [myid:1] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@688] - Send worker leaving thread This is I think as separate issue in that this happens immediately after I restart a zookeeper. (I.e., I see this in a log, restart that zookeeper, and immediately see a similar issue in one of the other two zookeeper logs). -----Original Message----- From: Daniel Collins [mailto:danwcoll...@gmail.com] Sent: 16 May 2013 13:28 To: solr-user@lucene.apache.org Subject: Re: Solr 4.3.0: Shard instances using incorrect data directory on machine boot What actual error do you see in Solr? Is there an exception and if so, can you post that? As I understand it, datatDir is set from the solrconfig.xml file, so either your instances are picking up the "wrong" file, or you have some override which is incorrect? Where do you set solr.data.dir, at the environment when you start Solr or in solrconfig? On 16 May 2013 12:23, Hoggarth, Gil <gil.hogga...@bl.uk> wrote: > Hi all, I hope you can advise a solution to our incorrect data > directory issue. > > > > We have 2 physical servers using Solr 4.3.0, each with 24 separate > tomcat instances (RedHat 6.4, java 1.7.0_10-b18, tomcat 7.0.34) with a > solr shard in each. This configuration means that each shard has its > own data directory declared. (Server OS, tomcat and solr, including > shards, created via automated builds.) > > > > That is, for example, > > - tomcat instance, /var/local/tomcat/solrshard3/, port 8985 > > - corresponding solr instance, /usr/local/solrshard3/, with > /usr/local/solrshard3/collection1/conf/solrconfig.xml > > - corresponding solr data directory, > /var/local/solrshard3/collection1/data/ > > > > We process ~1.5 billion documents, which is why we use so 48 shards > (24 leaders, 24 replicas). These physical servers are rebooted > regularly to fsck their drives. When rebooted, we always see several > (~10-20) shards failing to start (UI cloud view shows them as 'Down' or 'Recovering' > though they never recover without intervention), though there is not a > pattern to which shards fail to start - we haven't recorded any that > always or never fail. On inspection, the UI dashboard for these failed > shards displays, for example: > > - Host Server1 > > - Instance /usr/local/sholrshard3/collection1 > > - Data /var/local/solrshard6/collection1/data > > - Index /var/local/solrshard6/collection1/data/index > > > > To fix such failed shards, I manually restart the shard leader and > replicas, which fixes the issue. However, of course, I would like to > know a permanent cure for this, not a remedy. > > > > We use a separate zookeeper service, spread across 3 Virtual Machines > within our private network of ~200 servers (physical and virtual). > Network traffic is constant but relatively little across 1GB bandwidth. > > > > Any advice or suggestions greatly appreciated. > > Gil > > > > Gil Hoggarth > > Web Archiving Engineer > > The British Library, Boston Spa, West Yorkshire, LS23 7BQ > > > >