I don't think that's the problem -- mergeMiddle shouldn't need any
sync'ing, because getNextMerge (which is sync'd) will have already
advanced the merge into runningMerges.
Ie, finishMerges(false) tells all runningMerges to abort, and then
waits for them to do so. During this time, because stopMerges is
true, no further merges are allowed to be added to pendingMerges, so
once all runningMerges complete there should be no more outstanding
merges. Yet somehow there are... I don't see it yet. And I can't
repro the failure locally so far...
Mike
On Sun, Aug 2, 2009 at 2:51 PM, Shai Erera<ser...@gmail.com> wrote:
> Thanks Mike for the clarification.
>
> Almost all merge methods are synced on IndexWriter, except for mergeMiddle.
> So it is possible that you call abort, after the merge thread has already
> checked if the merge was aborted ...
>
> Maybe if we do:
> sync (this) {
> merge.checkAborted(directory);
> new SegmentMerger(...);
> }
>
> We'll protect against this particular exception? Because if merge.abort()
> wasn't called yet, the sync clause makes sure it won't be called until we
> get the SegmentMerger, otherwise we'll know the merge was aborted and so
> won't create SM at all.
>
> I looked at the rest of the method and this looks to be the only unprotected
> place. Although it is a loooong method, so I may have missed a spot.
>
> Shai
>
> On Sun, Aug 2, 2009 at 8:45 PM, Michael McCandless
> <luc...@mikemccandless.com> wrote:
>>
>> The exceptions aren't expected.
>>
>> When you call IndexWriter.close(false), it's supposed to notify all
>> running merges to abort, wait until they in fact do stop, and then
>> return.
>>
>> But it looks as if there's a silly thread hazard, whereby IndexWriter
>> believes all merges have aborted, but in fact those 4 were just about
>> to get started and "missed" the abort request, and then tried to start
>> and hit the ACE.
>>
>> Mike
>>
>> On Sun, Aug 2, 2009 at 6:49 AM, Shai Erera<ser...@gmail.com> wrote:
>> > Isn't this expected? I mean, if you close IndexWriter, but there are
>> > merges
>> > run in the background, aren't they expected to hit an
>> > AlreadyClosedException?
>> >
>> > Perhaps we should fix the test to check whether CMS hit this exception
>> > and
>> > if so clear it (as AFAIU, it's expected), and leave any other
>> > exceptions?
>> > This will require some change in CMS (to expose that information,
>> > through
>> > e.g. getUnhandledExceptions).
>> >
>> > Or add to CMS.Thread.run() a catch for AlreadyClosedException and ignore
>> > it,
>> > as we do for MergeAbortedException?
>> >
>> > Or, stop doing any merges if CMS.close() has been called?
>> >
>> > On Sun, Aug 2, 2009 at 12:49 AM, Michael Busch <busch...@gmail.com>
>> > wrote:
>> >>
>> >> While running 'ant test' I just got this failure (on Mac OS 10.5.7).
>> >> Even
>> >> worse, the next time I ran it the test passed, so this might be hard to
>> >> reproduce.
>> >> I haven't tried yet to find out what caused this.
>> >>
>> >> [junit] Exception in thread "Lucene Merge Thread #1"
>> >> org.apache.lucene.index.MergePolicy$MergeException:
>> >> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is
>> >> closed
>> >> [junit] at
>> >>
>> >> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
>> >> [junit] at
>> >>
>> >> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
>> >> [junit] Caused by: org.apache.lucene.store.AlreadyClosedException:
>> >> this
>> >> IndexWriter is closed
>> >> [junit] at
>> >> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:791)
>> >> [junit] at
>> >> org.apache.lucene.index.IndexWriter.getDirectory(IndexWriter.java:2303)
>> >> [junit] at
>> >> org.apache.lucene.index.SegmentMerger.<init>(SegmentMerger.java:88)
>> >> [junit] at
>> >> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4942)
>> >> [junit] at
>> >> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4598)
>> >> [junit] at
>> >>
>> >> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
>> >> [junit] at
>> >>
>> >> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
>> >> [junit] Exception in thread "Lucene Merge Thread #2"
>> >> org.apache.lucene.index.MergePolicy$MergeException:
>> >> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is
>> >> closed
>> >> [junit] at
>> >>
>> >> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
>> >> [junit] at
>> >>
>> >> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
>> >> [junit] Caused by: org.apache.lucene.store.AlreadyClosedException:
>> >> this
>> >> IndexWriter is closed
>> >> [junit] at
>> >> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:791)
>> >> [junit] at
>> >> org.apache.lucene.index.IndexWriter.getDirectory(IndexWriter.java:2303)
>> >> [junit] at
>> >> org.apache.lucene.index.SegmentMerger.<init>(SegmentMerger.java:88)
>> >> [junit] at
>> >> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4942)
>> >> [junit] at
>> >> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4598)
>> >> [junit] at
>> >>
>> >> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
>> >> [junit] at
>> >>
>> >> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
>> >> [junit] Exception in thread "Lucene Merge Thread #3"
>> >> org.apache.lucene.index.MergePolicy$MergeException:
>> >> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is
>> >> closed
>> >> [junit] at
>> >>
>> >> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
>> >> [junit] at
>> >>
>> >> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
>> >> [junit] Caused by: org.apache.lucene.store.AlreadyClosedException:
>> >> this
>> >> IndexWriter is closed
>> >> [junit] at
>> >> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:791)
>> >> [junit] at
>> >> org.apache.lucene.index.IndexWriter.getDirectory(IndexWriter.java:2303)
>> >> [junit] at
>> >> org.apache.lucene.index.SegmentMerger.<init>(SegmentMerger.java:88)
>> >> [junit] at
>> >> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4942)
>> >> [junit] at
>> >> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4598)
>> >> [junit] at
>> >>
>> >> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
>> >> [junit] at
>> >>
>> >> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
>> >> [junit] Exception in thread "Lucene Merge Thread #4"
>> >> org.apache.lucene.index.MergePolicy$MergeException:
>> >> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is
>> >> closed
>> >> [junit] at
>> >>
>> >> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
>> >> [junit] at
>> >>
>> >> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
>> >> [junit] Caused by: org.apache.lucene.store.AlreadyClosedException:
>> >> this
>> >> IndexWriter is closed
>> >> [junit] at
>> >> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:791)
>> >> [junit] at
>> >> org.apache.lucene.index.IndexWriter.getDirectory(IndexWriter.java:2303)
>> >> [junit] at
>> >> org.apache.lucene.index.SegmentMerger.<init>(SegmentMerger.java:88)
>> >> [junit] at
>> >> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4942)
>> >> [junit] at
>> >> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4598)
>> >> [junit] at
>> >>
>> >> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
>> >> [junit] at
>> >>
>> >> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
>> >> [junit] ------------- ---------------- ---------------
>> >> [junit] Testcase:
>> >> testNoWaitClose(org.apache.lucene.index.TestIndexWriter): FAILED
>> >> [junit] ConcurrentMergeScheduler hit unhandled exceptions
>> >> [junit] junit.framework.AssertionFailedError:
>> >> ConcurrentMergeScheduler
>> >> hit unhandled exceptions
>> >> [junit] at
>> >> org.apache.lucene.util.LuceneTestCase.tearDown(LuceneTestCase.java:54)
>> >> [junit] Test org.apache.lucene.index.TestIndexWriter FAILED
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
>> >> For additional commands, e-mail: java-dev-h...@lucene.apache.org
>> >>
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: java-dev-h...@lucene.apache.org
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
