old index files not deleted on slave
------------------------------------
Key: SOLR-2329
URL: https://issues.apache.org/jira/browse/SOLR-2329
Project: Solr
Issue Type: Bug
Components: replication (java)
Affects Versions: 4.0
Environment: centos 5.5
ext3 file system
Reporter: Edwin Khodabakchian
I have set up index replication (triggered on optimize). The problem I
am having is the old index files are not being deleted on the slave.
After each replication, I can see the old files still hanging around
as well as the files that have just been pulled. This causes the data
directory size to increase by the index size every replication until
the disk fills up.
I am running 4.0 rev 993367 with patch SOLR-1316. Otherwise, the setup
is pretty vanilla. I can reproduce this on multiple slaves.
Checking the logs, I see the following error:
SEVERE: SnapPull failed
org.apache.solr.common.SolrException: Index fetch failed :
at
org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:329)
at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:265)
at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock
obtain timed out:
NativeFSLock@/var/solrhome/data/index/lucene-cdaa80c0fefe1a7dfc7aab89298c614c-write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:84)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1065)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:954)
at
org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:192)
at
org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:99)
at
org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:173)
at
org.apache.solr.update.DirectUpdateHandler2.forceOpenWriter(DirectUpdateHandler2.java:376)
at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:471)
at
org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:319)
... 11 more
lsof reveals that the file is still opened from the java process.
Contents of the index data dir:
master:
-rw-rw-r-- 1 feeddo feeddo 191 Dec 14 01:06 _1lg.fnm
-rw-rw-r-- 1 feeddo feeddo 26M Dec 14 01:07 _1lg.fdx
-rw-rw-r-- 1 feeddo feeddo 1.9G Dec 14 01:07 _1lg.fdt
-rw-rw-r-- 1 feeddo feeddo 474M Dec 14 01:12 _1lg.tis
-rw-rw-r-- 1 feeddo feeddo 15M Dec 14 01:12 _1lg.tii
-rw-rw-r-- 1 feeddo feeddo 144M Dec 14 01:12 _1lg.prx
-rw-rw-r-- 1 feeddo feeddo 277M Dec 14 01:12 _1lg.frq
-rw-rw-r-- 1 feeddo feeddo 311 Dec 14 01:12 segments_1ji
-rw-rw-r-- 1 feeddo feeddo 23M Dec 14 01:12 _1lg.nrm
-rw-rw-r-- 1 feeddo feeddo 191 Dec 18 01:11 _24e.fnm
-rw-rw-r-- 1 feeddo feeddo 26M Dec 18 01:12 _24e.fdx
-rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 01:12 _24e.fdt
-rw-rw-r-- 1 feeddo feeddo 483M Dec 18 01:23 _24e.tis
-rw-rw-r-- 1 feeddo feeddo 15M Dec 18 01:23 _24e.tii
-rw-rw-r-- 1 feeddo feeddo 146M Dec 18 01:23 _24e.prx
-rw-rw-r-- 1 feeddo feeddo 283M Dec 18 01:23 _24e.frq
-rw-rw-r-- 1 feeddo feeddo 311 Dec 18 01:24 segments_1xz
-rw-rw-r-- 1 feeddo feeddo 23M Dec 18 01:24 _24e.nrm
-rw-rw-r-- 1 feeddo feeddo 191 Dec 18 13:15 _25z.fnm
-rw-rw-r-- 1 feeddo feeddo 26M Dec 18 13:16 _25z.fdx
-rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 13:16 _25z.fdt
-rw-rw-r-- 1 feeddo feeddo 484M Dec 18 13:35 _25z.tis
-rw-rw-r-- 1 feeddo feeddo 15M Dec 18 13:35 _25z.tii
-rw-rw-r-- 1 feeddo feeddo 146M Dec 18 13:35 _25z.prx
-rw-rw-r-- 1 feeddo feeddo 284M Dec 18 13:35 _25z.frq
-rw-rw-r-- 1 feeddo feeddo 20 Dec 18 13:35 segments.gen
-rw-rw-r-- 1 feeddo feeddo 311 Dec 18 13:35 segments_1y1
-rw-rw-r-- 1 feeddo feeddo 23M Dec 18 13:35 _25z.nrm
slave:
-rw-rw-r-- 1 feeddo feeddo 20 Dec 13 17:54 segments.gen
-rw-rw-r-- 1 feeddo feeddo 191 Dec 15 01:07 _1mk.fnm
-rw-rw-r-- 1 feeddo feeddo 26M Dec 15 01:08 _1mk.fdx
-rw-rw-r-- 1 feeddo feeddo 1.9G Dec 15 01:08 _1mk.fdt
-rw-rw-r-- 1 feeddo feeddo 476M Dec 15 01:18 _1mk.tis
-rw-rw-r-- 1 feeddo feeddo 15M Dec 15 01:18 _1mk.tii
-rw-rw-r-- 1 feeddo feeddo 144M Dec 15 01:18 _1mk.prx
-rw-rw-r-- 1 feeddo feeddo 278M Dec 15 01:18 _1mk.frq
-rw-rw-r-- 1 feeddo feeddo 312 Dec 15 01:18 segments_1kj
-rw-rw-r-- 1 feeddo feeddo 23M Dec 15 01:18 _1mk.nrm
-rw-rw-r-- 1 feeddo feeddo 0 Dec 15 01:19
lucene-cdaa80c0fefe1a7dfc7aab89298c614c-write.lock
-rw-rw-r-- 1 feeddo feeddo 191 Dec 15 13:14 _1qu.fnm
-rw-rw-r-- 1 feeddo feeddo 26M Dec 15 13:16 _1qu.fdx
-rw-rw-r-- 1 feeddo feeddo 1.9G Dec 15 13:16 _1qu.fdt
-rw-rw-r-- 1 feeddo feeddo 477M Dec 15 13:28 _1qu.tis
-rw-rw-r-- 1 feeddo feeddo 15M Dec 15 13:28 _1qu.tii
-rw-rw-r-- 1 feeddo feeddo 144M Dec 15 13:28 _1qu.prx
-rw-rw-r-- 1 feeddo feeddo 278M Dec 15 13:28 _1qu.frq
-rw-rw-r-- 1 feeddo feeddo 311 Dec 15 13:28 segments_1oe
-rw-rw-r-- 1 feeddo feeddo 23M Dec 15 13:28 _1qu.nrm
-rw-rw-r-- 1 feeddo feeddo 191 Dec 17 01:12 _222.fnm
-rw-rw-r-- 1 feeddo feeddo 26M Dec 17 01:15 _222.fdx
-rw-rw-r-- 1 feeddo feeddo 1.9G Dec 17 01:15 _222.fdt
-rw-rw-r-- 1 feeddo feeddo 481M Dec 17 01:36 _222.tis
-rw-rw-r-- 1 feeddo feeddo 15M Dec 17 01:36 _222.tii
-rw-rw-r-- 1 feeddo feeddo 145M Dec 17 01:36 _222.prx
-rw-rw-r-- 1 feeddo feeddo 281M Dec 17 01:36 _222.frq
-rw-rw-r-- 1 feeddo feeddo 23M Dec 17 01:36 _222.nrm
-rw-rw-r-- 1 feeddo feeddo 311 Dec 17 01:36 segments_1xv
-rw-rw-r-- 1 feeddo feeddo 191 Dec 17 13:10 _233.fnm
-rw-rw-r-- 1 feeddo feeddo 26M Dec 17 13:13 _233.fdx
-rw-rw-r-- 1 feeddo feeddo 1.9G Dec 17 13:13 _233.fdt
-rw-rw-r-- 1 feeddo feeddo 482M Dec 17 13:31 _233.tis
-rw-rw-r-- 1 feeddo feeddo 15M Dec 17 13:31 _233.tii
-rw-rw-r-- 1 feeddo feeddo 146M Dec 17 13:31 _233.prx
-rw-rw-r-- 1 feeddo feeddo 282M Dec 17 13:31 _233.frq
-rw-rw-r-- 1 feeddo feeddo 311 Dec 17 13:31 segments_1xx
-rw-rw-r-- 1 feeddo feeddo 23M Dec 17 13:31 _233.nrm
-rw-rw-r-- 1 feeddo feeddo 191 Dec 18 01:11 _24e.fnm
-rw-rw-r-- 1 feeddo feeddo 26M Dec 18 01:12 _24e.fdx
-rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 01:12 _24e.fdt
-rw-rw-r-- 1 feeddo feeddo 483M Dec 18 01:23 _24e.tis
-rw-rw-r-- 1 feeddo feeddo 15M Dec 18 01:23 _24e.tii
-rw-rw-r-- 1 feeddo feeddo 146M Dec 18 01:23 _24e.prx
-rw-rw-r-- 1 feeddo feeddo 283M Dec 18 01:23 _24e.frq
-rw-rw-r-- 1 feeddo feeddo 311 Dec 18 01:24 segments_1xz
-rw-rw-r-- 1 feeddo feeddo 23M Dec 18 01:24 _24e.nrm
-rw-rw-r-- 1 feeddo feeddo 191 Dec 18 13:15 _25z.fnm
-rw-rw-r-- 1 feeddo feeddo 26M Dec 18 13:16 _25z.fdx
-rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 13:16 _25z.fdt
-rw-rw-r-- 1 feeddo feeddo 484M Dec 18 13:35 _25z.tis
-rw-rw-r-- 1 feeddo feeddo 15M Dec 18 13:35 _25z.tii
-rw-rw-r-- 1 feeddo feeddo 146M Dec 18 13:35 _25z.prx
-rw-rw-r-- 1 feeddo feeddo 284M Dec 18 13:35 _25z.frq
-rw-rw-r-- 1 feeddo feeddo 311 Dec 18 13:35 segments_1y1
-rw-rw-r-- 1 feeddo feeddo 23M Dec 18 13:35 _25z.nrm
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
