I was trying to run the whole suite of randomized tests, wrapped inside my own implementation.
I wasn't sure how my stuff will behave against MMap, NIO, RAM, NRT implementations, so I thought running fully randomized tests would work to find edge cases in my implementation. By randomizing everything I found MockDirectoryWrapper with this issue. I'll retry my tests without MockDirectoryWrapper. On Tue, Aug 21, 2012 at 12:18 PM, Uwe Schindler <[email protected]> wrote: > I don’t understand what you are doing! Do you wrap MockIndexInputWrapper > again with your wrapper? This can of course not work, as > MockIndexInputWrapper is an internal test-only class, so the big question > is:**** > > ** ** > > What are you trying to do?**** > > ** ** > > -----**** > > Uwe Schindler**** > > H.-H.-Meier-Allee 63, D-28213 Bremen**** > > http://www.thetaphi.de**** > > eMail: [email protected]**** > > ** ** > > *From:* Danil ŢORIN [mailto:[email protected]] > *Sent:* Tuesday, August 21, 2012 11:13 AM > > *To:* [email protected] > *Subject:* Re: Custom Directory and IndexInput**** > > ** ** > > Strange thing:**** > > If I wrap all "reasonable" directories into my directory, my tests pass.** > ** > > However if I wrap MockIndexInputWrapper, it calls "checkIndex()" after > close on delegated DataInput, and obviously fails to read proper data.**** > > ** ** > > CheckIndex failed**** > > ERROR: could not read any segments file in directory**** > > org.apache.lucene.index.IndexFormatTooNewException: Format version is not > supported (resource: > MockIndexInputWrapper(RAMInputStream(name=segments.gen))): -495571168 > (needs to be between -2 and -2)**** > > at > org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:692) > **** > > at > org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:601) > **** > > at > org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:327)**** > > at > org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:350)**** > > at > org.apache.lucene.util._TestUtil.checkIndex(_TestUtil.java:192)**** > > at > org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:580) > **** > > ** ** > > Is there a clean way to force checkIndex on the top-level Directory > involved in tests, not on one of the wrappers below?**** > > ** ** > > On Tue, Aug 21, 2012 at 11:26 AM, Uwe Schindler <[email protected]> wrote:** > ** > > In my commit I also solved the casting problem with covariant override. > But DataInput.clone() was always public.**** > > **** > > -----**** > > Uwe Schindler**** > > H.-H.-Meier-Allee 63, D-28213 Bremen**** > > http://www.thetaphi.de**** > > eMail: [email protected]**** > > **** > > *From:* Danil ŢORIN [mailto:[email protected]] > *Sent:* Tuesday, August 21, 2012 9:55 AM**** > > > *To:* [email protected] > *Subject:* Re: Custom Directory and IndexInput**** > > **** > > My bad, few casts solved the issue.**** > > On Tue, Aug 21, 2012 at 10:46 AM, Danil ŢORIN <[email protected]> wrote:* > *** > > One tricky thing, I'm doing a lot of delegation, I can't call "clone()" on > the DataInput I'm delegating to.**** > > (unfortunately in my case it's delegation, not inheritance)**** > > **** > > The fact that the delegate implements Clonable, doesn't help, the clone() > method is still protected :(.**** > > Any idea how to circumvent this?**** > > **** > > On Tue, Aug 21, 2012 at 10:37 AM, Uwe Schindler <[email protected]> wrote:** > ** > > Thanks for „reporting“ lack of documentation, I will shortly commit some > additional specs to IndexInput’s javadocs!**** > > **** > > Thanks,**** > > Uwe**** > > **** > > -----**** > > Uwe Schindler**** > > H.-H.-Meier-Allee 63, D-28213 Bremen**** > > http://www.thetaphi.de**** > > eMail: [email protected]**** > > **** > > *From:* Danil ŢORIN [mailto:[email protected]] > *Sent:* Tuesday, August 21, 2012 8:59 AM > *To:* [email protected] > *Subject:* Re: Custom Directory and IndexInput**** > > **** > > Thanks, that's exactly the response I was looking for.**** > > On Tue, Aug 21, 2012 at 9:31 AM, Uwe Schindler <[email protected]> wrote:*** > * > > Hi,**** > > **** > > It is required that every thread uses its own IndexInput. This is why e.g. > every search calls IndexInput.clone() several times while executing a > search query. Because of that there are explicietly no thread safety > requirements in IndexInput, but clone() must be implemented as a cheap > method, cloning the IndexInput and keeping the state. It is not required to > recreate file descriptors or like that, but it must make sure that the new > IndexInput can be used independent, although on same file/same file > descriptor like another one.**** > > **** > > Uwe**** > > **** > > -----**** > > Uwe Schindler**** > > H.-H.-Meier-Allee 63, D-28213 Bremen**** > > http://www.thetaphi.de**** > > eMail: [email protected]**** > > **** > > *From:* Danil ŢORIN [mailto:[email protected]] > *Sent:* Tuesday, August 21, 2012 8:22 AM > *To:* [email protected] > *Subject:* Custom Directory and IndexInput**** > > **** > > > I'm trying to build a custom directory implementation, actually the > directory itself just fallback to one of the existing directory > implementations, so it's actually more about IndexInput/IndexOutput. > > I have some concerns about my implementation, especially related to > multithreading:**** > > - IndexOutput writeByte/writeBytes should be no problem since only one > thread will write to specific output at any time**** > - IndexInput on the other hand could be called from multiple threads*** > * > - most implementations of IndexInput (including mine) keep some > internal state like current position in the file/buffer/etc**** > - if one of the threads is calling readShort()/readLong()/readVInt() > (default implementations from DataInput) while some other thread is calling > readByte() bad stuff will happen**** > - still in most existing implementations I don't see any > synchronization code on reads**** > - is this problem solved somewhere on a higher level, like > IndexReader/IndexSearcher, so I don't have to worry about it?**** > - cause I'm really not sure how to solve it without forcing > synchronized on all read methods and I'd really hate that.**** > > How this issue is solved in Lucene? what should I do to make sure that my > implementation doing right thing?**** > > **** > > Currently I'm on Lucene 4.0.0 BETA**** > > **** > > **** > > **** > > **** > > ** ** >
