Woops, you're right: this is a bug. I'll open an issue, fold in your
nice test case & fix it. Thanks Jed!
On hitting OOM, IndexWriter marks that its internal state (buffered
documents, deletions) may be corrupt and so it rollsback to the last
commit instead of flushing a new segment.
To workaround this, on catching an OOME on any of IndexWriter's
methods, you should 1) forcibly remove the write lock
(IndexWriter.unlock static method) and then 2) not call any methods on
the old writer. Even if the old writer has concurrent merges running,
they will refuse to commit on seeing that an OOM had occurred.
Mike
Jed Wesley-Smith wrote:
All,
We have seen the following stacktrace in production with Lucene 2.3.2:
java.lang.IllegalStateException: abort() can only be called when
IndexWriter was opened with autoCommit=false
at org.apache.lucene.index.IndexWriter.abort(IndexWriter.java:2009)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1175)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1154)
This is caused by some IndexWriter method catching an
OutOfMemoryError previously and then aborting the close.
My question is twofold. Firstly, does it make any sense for this to
happen (it feels like a bug, shouldn't close not call abort if
autoCommit=true?). Secondly, is there anything I can do to recover
(should I call flush()?), or can I just ignore it if autoCommit is
true?
Following is a test that reproduces the problem:
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.store.RAMDirectory;
import org.junit.Assert;
import org.junit.Test;
import java.io.ByteArrayOutputStream;
import java.io.PrintStream;
import java.util.concurrent.atomic.AtomicBoolean;
public class TestIndexWriter {
@Test public void testOutOfMemoryErrorCausesCloseToFail() throws
Exception {
final AtomicBoolean throwFirst = new AtomicBoolean(true);
final IndexWriter writer = new IndexWriter(new RAMDirectory(),
new StandardAnalyzer()) {
@Override public void message(final String message) {
if (message.startsWith("now flush at close") &&
throwFirst.getAndSet(false)) {
throw new OutOfMemoryError(message);
}
}
};
// need to set an info stream so message is called
writer.setInfoStream(new PrintStream(new
ByteArrayOutputStream())); //or better, use NullOS from commons-io
try {
writer.close();
Assert.fail("OutOfMemoryError expected");
}
catch (final OutOfMemoryError expected) {}
// throws IllegalStateEx
writer.close();
}
}
thanks for any help,
jed.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
