Hello all,
For quite some time, we've been experiencing intermittent indexing failures
that look similar to the following:
java.io.IOException: couldn't rename segments.new to segments
at
org.apache.lucene.store.FSDirectory.renameFile(Ljava.lang.String;Ljava.lang.
String)V(Unknown Source)
at
org.apache.lucene.index.SegmentInfos.write(Lorg.apache.lucene.store.Director
y)V(Unknown Source)
This is caused by File.renameTo failing. From previous run-ins with that
method, it's actually fairly flaky, especially under various Windows JVMs.
The fix is pretty easy -- if File.renameTo fails, manually copy the file
from the old to new destination.
Old code (FSDirectory.renameFile):
------------------------------
if (!old.renameTo(nu))
throw new IOException("couldn't rename " + from + " to " + to);
------------------------------
New code ("buf" variable is declared at top of file as null):
------------------------------
// Rename the old file to the new one. Unfortunately, the renameTo()
method doesn't
// work reliably under some JVMs. Therefore, if the rename fails, we
manually rename by
// copying the old file to the new one.
if (!old.renameTo(nu)) {
try {
java.io.InputStream in = new FileInputStream(old);
java.io.OutputStream out = new FileOutputStream(nu);
int len;
// See if the buf needs to be initialized. Initialization is only
done on-demand
// since many VM's will never run into the renameTo bug and hence
shouldn't waste
// 1K of mem for no reason.
if (buf == null) {
buf = new byte[1024];
}
while ((len = in.read(buf)) >= 0) {
out.write(buf, 0, len);
}
in.close();
out.close();
// Delete the old file.
old.delete();
}
catch (IOException ioe) {
throw new IOException("couldn't rename " + from + " to " + to);
}
}
------------------------------
Advantages to this fix:
* Makes indexing more reliable, especially under VM's where the renameTo
method doesn't always work.
* Uses no additional memory or resources when File.renameTo works normally.
Disadvantages:
* It always sucks to add workarounds to bugs in other libraries, but what
is one to do?
* We've run into File.renameTo issues on multiple machines and VM versions,
but it would be good if others could confirm they've had problems as well.
I'd suggest this fix as a good candidate for Lucene 1.3 unless others see
problems with it.
Regards,
Matt
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
