hello all, environment: solr 3.5
1 - master 2 - slave slaves are set to poll master every 10 minutes. i have had replication running on one master and two slaves - for a few weeks now. these boxes are not production boxes - just QA/test boxes. right after i started a re-index on the master - i started to see the following errors on both of the slave boxes. in previous test runs - i have not noticed any errors. can someone help me understand what is causing these errors? thank you, 2012-06-03 19:30:23,104 INFO [org.apache.solr.update.UpdateHandler] (pool-16-thread-1) start commit(optimize=false,waitFlush=true,waitSearcher=true,expungeDeletes=false) 2012-06-03 19:30:23,164 SEVERE [org.apache.solr.handler.ReplicationHandler] (pool-16-thread-1) SnapPull failed org.apache.solr.common.SolrException: Index fetch failed : at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:331) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:268) at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.lang.RuntimeException: java.io.IOException: read past EOF: MMapIndexInput(path="/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kgm.fdx") at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1103) at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:418) at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:470) at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:321) ... 11 more Caused by: java.io.IOException: read past EOF: MMapIndexInput(path="/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kgm.fdx") at org.apache.lucene.store.MMapDirectory$MMapIndexInput.readByte(MMapDirectory.java:279) at org.apache.lucene.store.DataInput.readInt(DataInput.java:84) at org.apache.lucene.store.MMapDirectory$MMapIndexInput.readInt(MMapDirectory.java:315) at org.apache.lucene.index.FieldsReader.<init>(FieldsReader.java:138) at org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:212) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:117) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:93) at org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:235) at org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:34) at org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:506) at org.apache.lucene.index.DirectoryReader.access$000(DirectoryReader.java:45) at org.apache.lucene.index.DirectoryReader$2.doBody(DirectoryReader.java:498) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:754) at org.apache.lucene.index.DirectoryReader.doOpenNoWriter(DirectoryReader.java:493) at org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:450) at org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:396) at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:520) at org.apache.lucene.index.IndexReader.reopen(IndexReader.java:697) at org.apache.solr.search.SolrIndexReader.reopen(SolrIndexReader.java:414) at org.apache.solr.search.SolrIndexReader.reopen(SolrIndexReader.java:425) at org.apache.solr.search.SolrIndexReader.reopen(SolrIndexReader.java:35) at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:501) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1083) ... 14 more 2012-06-03 19:30:23,197 INFO [org.apache.solr.handler.SnapPuller] (pool-12-thread-1) Skipping download for /appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kiq.tis 2012-06-03 19:30:23,198 INFO [org.apache.solr.handler.SnapPuller] (pool-12-thread-1) Skipping download for /appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kit.tis 2012-06-03 19:30:23,198 INFO [org.apache.solr.handler.SnapPuller] (pool-12-thread-1) Skipping download for /appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kgm.fdt 2012-06-03 19:30:23,198 INFO [org.apache.solr.handler.SnapPuller] (pool-12-thread-1) Skipping download for /appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5jra.prx 2012-06-03 19:30:23,198 INFO [org.apache.solr.handler.SnapPuller] (pool-12-thread-1) Skipping download for /appl/solr/stress/partcatalog/index/core1/index.20120514101522/segments_50qw 2012-06-03 19:30:23,198 INFO [org.apache.solr.handler.SnapPuller] (pool-12-thread-1) Skipping download for /appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kiq.tii 2012-06-03 19:30:23,198 INFO [org.apache.solr.handler.SnapPuller] (pool-12-thread-1) Total time taken for download : 23 secs 2012-06-03 19:30:23,201 SEVERE [org.apache.solr.handler.ReplicationHandler] (pool-12-thread-1) SnapPull failed org.apache.solr.common.SolrException: Index fetch failed : at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:331) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:268) at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.io.FileNotFoundException: File does not exist /appl/solr/stress/partcatalog/index/core1/index.20120603192959/_5kgm.fdx at org.apache.solr.common.util.FileUtils.sync(FileUtils.java:64) at org.apache.solr.handler.SnapPuller$FileFetcher$1.run(SnapPuller.java:923) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more 2012-06-03 19:39:59,904 INFO [org.apache.solr.handler.SnapPuller] (pool-12-thread-1) Master's version: 1326238466864, generation: 235126 2012-06-03 19:39:59,904 INFO [org.apache.solr.handler.SnapPuller] (pool-12-thread-1) Slave's version: 1326238462878, generation: 233332 2012-06-03 19:39:59,905 INFO [org.apache.solr.handler.SnapPuller] (pool-12-thread-1) St -- View this message in context: http://lucene.472066.n3.nabble.com/seeing-errors-during-replication-process-on-slave-boxes-read-past-EOF-tp3987489.html Sent from the Solr - User mailing list archive at Nabble.com.