Hello,
We are using Lucene 6.4.2 with a file-based index, Oracle JDK 1.8.0_121,
Windows 10.
We found that the following steps generate unrecoverable error (we have to
restart our JVM in order to resume normal work which is not acceptable):
1. Create an index directory, open IndexWriter, generate the index data and
close the writer.
2. Perform search on the index. It will open DirectoryReader and we leave it
open so it can be re-used.
3. We run nightly job that will regenerate the index because some data there
is time-sensitive. When the job is trying to open an IndexWriter again we are
getting the following error:
java.lang.IllegalArgumentException: Directory MMapDirectory@C:\...\INDEXNAME
lockFactory=org.apache.lucene.store.NativeFSLockFactory@10159324 still has
pending deleted files; cannot initialize IndexWriter
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:795)
Our understanding of the problem is that the IndexWriter is waiting for some
old files (like _*.cfs) to be deleted but the files are still locked by
operation system because the directory reader is still open. We found by
debugging FSDirectory class code that all attempts to delete those files fail
with java.nio.file.AccessDeniedException that is not re-thrown by the code. It
creates a deadlock that can be only resolved by restarting java VM.
We found a workaround (by overriding FSDirectory. checkPendingDeletions() to
return 'false' always and opening new reader after re-indexing so the old
reader is getting closed and the old files are getting unlocked and deleted
eventually) but I think it's a serious problem and it has to be resolved by
Lucene developers.
Regards,
Vitaly Stroutchkov
Senior software developer at Kloudville Inc.
