Hi,
today I was able to run the build at work on a RedHat x64 + JDK6
(1.6.0_22), 64 bits and the build went ok. I'll try in the next days with a
32bit JDK to see if I can reproduce the exception, although I believe I
won't have any problems.. I've been digging a bit through lucene src and
found that it behaves here and there differently depending on 32 or 64bit,
OS, and JDK -don't know why I'm having the exceptions, and being the only
one with them :-? Most probably something with my environment, as I get the
exceptions with a clean build.
regards,
juan pablo
On Fri, Jun 22, 2012 at 1:38 AM, Juan Pablo Santos Rodríguez <
juanpablo.san...@gmail.com> wrote:
> Hallo,
>
> @Harry: I've been comparing bot setInfoStream outputs. They're basically
> the same output, only a couple of differences
> - when running all 6 tests I only get up to 15 IW, instead of 18
> - the exceptions either when writing the segments file or when closing the
> index (and therefore committing changes)
> IW 8 [Fri Jun 22 00:36:22 CEST 2012; JSPWiki Lucene Indexer]: commit:
> wrote segments file "segments_1"
> IFD [Fri Jun 22 00:36:22 CEST 2012; JSPWiki Lucene Indexer]: now
> checkpoint "segments_1" [0 segments ; isCommit = true]
> IFD [Fri Jun 22 00:36:22 CEST 2012; JSPWiki Lucene Indexer]: delete
> pending file _0.tis
> IFD [Fri Jun 22 00:36:22 CEST 2012; JSPWiki Lucene Indexer]: delete
> "_0.tis"
> IFD [Fri Jun 22 00:36:22 CEST 2012; JSPWiki Lucene Indexer]: unable to
> remove file "_0.tis": java.io.IOException: Cannot delete
> E:\tmp\testworkdir\lucene\_0.tis; Will re-try later.
> IFD [Fri Jun 22 00:36:22 CEST 2012; JSPWiki Lucene Indexer]: delete
> pending file _0.nrm
> IFD [Fri Jun 22 00:36:22 CEST 2012; JSPWiki Lucene Indexer]: delete
> "_0.nrm"
> IFD [Fri Jun 22 00:36:22 CEST 2012; JSPWiki Lucene Indexer]: unable to
> remove file "_0.nrm": java.io.IOException: Cannot delete
> E:\tmp\testworkdir\lucene\_0.nrm; Will re-try later.
>
> My environment: Win 7 x64, with or without cygwin, with JDK 1.6.0_19 first
> and 1.7.0_3 secondly, 64 bit edition of both JDKs. I can try to run the ant
> tests at work (JDK 1.6, 32-bit) but most probably not until next Monday /
> Tuesday. Also, I'll try to run with newer JDK6/7 versions but again, not
> until next Monday/Tuesday.
>
> @Florian: it seems to me that the problem is caused due to a concurrency
> issue, as the index is updated in a separate thread, so it would be
> difficult to serialize its execution. Also, being a concurrency issue makes
> it hard to reproduce in a unit test.
>
> The singleton approach would be feasible too and most probably would also
> get rid off the (my) exceptions problem. I preferred the re-schedule
> approach because if an exception occurs while doing the index operation, we
> are now logging it and then acting as if nothing had happened, with the
> page not being indexed; re-scheduling the index operation simply places the
> WikiPage back in the index queue (as the indexing hasn't been performed) so
> the index operation can be retried later on.
>
>
> rgrds,
> juan pablo
>
>
>
> On Thu, Jun 21, 2012 at 10:28 PM, Florian Holeczek <flor...@holeczek.de>wrote:
>
>> Hi Juan Pablo,
>>
>> I'd rather not fix working code only because some tests are not working
>> due to (test-)architectural issues.
>> Not sure if I understood the problem completely though.
>> Wouldn't the right way be serializing the tests or using a singleton for
>> the Lucene index?
>>
>> Regards
>> Florian
>>
>>
>> ----- Ursprüngliche Mail -----
>> Von: "Juan Pablo Santos Rodríguez" <juanpablo.san...@gmail.com>
>> An: jspwiki-dev@incubator.apache.org
>> Gesendet: Donnerstag, 21. Juni 2012 18:55:35
>> Betreff: Re: 3 out of 6 tests from SearchManagerTest failing if run all
>> together
>>
>> Ok, so finally, I think I got it.
>>
>> Updates to lucene Index are done in a separate thread, via the
>> LuceneSearchProvider.LuceneUpdater inner class, which is executed on a
>> different thread. This may lead to situations where there are more than
>> one
>> IndexWriter instances working on the same directory, provoking the
>> exception noted on the first email.
>>
>> The proposed fix is relatively easy, on
>> LuceneSearchProvider.luceneIndexPage( WikiPage page, String text,
>> IndexWriter writer ) method, if an update index operation fails, simply
>> re-schedule the index operation:
>>
>> protected synchronized void updateLuceneIndex( WikiPage page, String
>> text )
>> {
>> [...]
>> try
>> {
>> [...]
>> luceneIndexPage( page, text, writer );
>> }
>> catch ( IOException e )
>> {
>> log.error("Unable to update page '" + page.getName() + "' from
>> Lucene index", e);
>> reindexPage( page );
>> }
>> catch( Exception e )
>> {
>> log.error("Unexpected Lucene exception - please check
>> configuration!",e);
>> reindexPage( page );
>> }
>> finally
>> [...]
>>
>> This approach could cause some sort of infinite loop if there's something
>> severe happening (i.e.: disk full, no permission to write on the work dir,
>> etc.), but I think that those kind of situations would be noted throughout
>> the whole application, so there would be little to do on those cases
>> anyway.
>>
>> WDYT about the suggested change?
>>
>>
>> br,
>> juan pablo
>>
>>
>> On Thu, Jun 21, 2012 at 1:00 AM, Juan Pablo Santos Rodríguez <
>> juanpablo.san...@gmail.com> wrote:
>>
>> > Hello Harry,
>> >
>> > I've increased the time up to 100s and still the same tests failing /
>> > exceptions. I think it has to be something related to index updating, as
>> > the IOExceptions noted by enabling writer.setInfoStream( System.out );
>> > occur every time a page is updated.
>> >
>> > Also, the exceptions are caught always at
>> > LuceneSearchProvider.LuceneUpdater threads so maybe they are trying to
>> open
>> > an IndexWriter while the JUnit test has his own IndexWriter opened.
>> Anyway,
>> > I'll continue doing more tests to see if I can isolate the cause of the
>> > exception.
>> >
>> >
>> > regards,
>> > juan pablo
>> >
>> >
>> >
>> > On Wed, Jun 20, 2012 at 6:21 AM, Harry Metske <harry.met...@gmail.com
>> >wrote:
>> >
>> >> Wat happens of you increase the waittime specified in the
>> >> SearchManagerTest?
>> >>
>> >> I think I had these ones when this time was too short.
>> >>
>> >> regards,
>> >> Harry
>> >> Op 20 jun. 2012 00:45 schreef "Juan Pablo Santos Rodríguez" <
>> >> juanpablo.san...@gmail.com> het volgende:
>> >>
>> >> > Hi,
>> >> >
>> >> > I'm stuck running the tests from SearchManagerTest. If I run them
>> >> > individually, then all is OK.
>> >> >
>> >> > However, if I run them all together (via Ant task or via Eclipse),
>> then
>> >> the
>> >> > following tests fail: testSimplesearch3, testTitleSearch,
>> >> testTitleSearch2,
>> >> > at the assertEquals() regarding res.size()
>> >> >
>> >> > jspwikitests.log yields several exceptions like this one:
>> >> >
>> >> > java.io.IOException: Cannot overwrite:
>> E:\tmp\testworkdir\lucene\_0.fdt
>> >> > at
>> >> >
>> org.apache.lucene.store.FSDirectory.ensureCanWrite(FSDirectory.java:316)
>> >> > at
>> >> >
>> org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:305)
>> >> > at
>> org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:83)
>> >> > at
>> >> >
>> >> >
>> >>
>> org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:64)
>> >> > at
>> >> >
>> >> >
>> >>
>> org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:107)
>> >> > at
>> >> >
>> >> >
>> >>
>> org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:151)
>> >> > at
>> >> >
>> >> >
>> >>
>> org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1404)
>> >> > at
>> >> >
>> >> >
>> >>
>> org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1424)
>> >> > at
>> >> >
>> >> >
>> >>
>> org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1043)
>> >> > at
>> >> >
>> >> >
>> >>
>> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:772)
>> >> > at
>> >> >
>> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2060)
>> >> > at
>> >> >
>> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2034)
>> >> > at
>> >> >
>> >> >
>> >>
>> org.apache.wiki.search.LuceneSearchProvider.luceneIndexPage(LuceneSearchProvider.java:485)
>> >> > at
>> >> >
>> >> >
>> >>
>> org.apache.wiki.search.LuceneSearchProvider.updateLuceneIndex(LuceneSearchProvider.java:377)
>> >> > [...]
>> >> >
>> >> > Also, if I place writer.setInfoStream( System.out ); just after the
>> >> writer
>> >> > is instantiated, I get several messages similar to this one:
>> >> > IFD [Wed Jun 20 00:20:07 CEST 2012; JSPWiki Lucene Indexer]: unable
>> to
>> >> > remove file "_0.fdt": java.io.IOException: Cannot delete
>> >> > E:\tmp\testworkdir\lucene\_0.fdt; Will re-try later.
>> >> >
>> >> > Done a little googling and found that may be related to opening an
>> >> > IndexWriter on the index while there's another one open [1] or
>> removing
>> >> > files while the index is open at that directory [2]. I'll continue
>> >> looking
>> >> > at this tomorrow, but I was wondering if anyone else is experiencing
>> >> this
>> >> > problem too.
>> >> >
>> >> >
>> >> > br,
>> >> > juan pablo
>> >> >
>> >> > [1]: http://www.gossamer-threads.com/lists/lucene/java-user/92262
>> >> > [2]:
>> http://www.gossamer-threads.com/lists/lucene/java-user/62201#62201
>> >> >
>> >>
>> >
>> >
>>
>
>
