Hi,
OK. If you test against only Lucene Core’s directories you are fine. Wrapping the special wrappers from test-framework is not needed and wrong, they are just for testing Lucene itsself. In fact when running all Lucene tests against you own directory (-Dtests.directory=…), the test framework will take care and wrap YOUR Directory with MockDirectoryWrapper to assert some special things (like Lucene code closes everything correctly, Index is correct,…) Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen <http://www.thetaphi.de/> http://www.thetaphi.de eMail: [email protected] From: Danil ŢORIN [mailto:[email protected]] Sent: Tuesday, August 21, 2012 11:34 AM To: [email protected] Subject: Re: Custom Directory and IndexInput I was trying to run the whole suite of randomized tests, wrapped inside my own implementation. I wasn't sure how my stuff will behave against MMap, NIO, RAM, NRT implementations, so I thought running fully randomized tests would work to find edge cases in my implementation. By randomizing everything I found MockDirectoryWrapper with this issue. I'll retry my tests without MockDirectoryWrapper. On Tue, Aug 21, 2012 at 12:18 PM, Uwe Schindler <[email protected]> wrote: I don’t understand what you are doing! Do you wrap MockIndexInputWrapper again with your wrapper? This can of course not work, as MockIndexInputWrapper is an internal test-only class, so the big question is: What are you trying to do? ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de <http://www.thetaphi.de/> eMail: [email protected] From: Danil ŢORIN [mailto:[email protected]] Sent: Tuesday, August 21, 2012 11:13 AM To: [email protected] Subject: Re: Custom Directory and IndexInput Strange thing: If I wrap all "reasonable" directories into my directory, my tests pass. However if I wrap MockIndexInputWrapper, it calls "checkIndex()" after close on delegated DataInput, and obviously fails to read proper data. CheckIndex failed ERROR: could not read any segments file in directory org.apache.lucene.index.IndexFormatTooNewException: Format version is not supported (resource: MockIndexInputWrapper(RAMInputStream(name=segments.gen))): -495571168 (needs to be between -2 and -2) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:692) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:601) at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:327) at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:350) at org.apache.lucene.util._TestUtil.checkIndex(_TestUtil.java:192) at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:580) Is there a clean way to force checkIndex on the top-level Directory involved in tests, not on one of the wrappers below? On Tue, Aug 21, 2012 at 11:26 AM, Uwe Schindler <[email protected]> wrote: In my commit I also solved the casting problem with covariant override. But DataInput.clone() was always public. ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de <http://www.thetaphi.de/> eMail: [email protected] From: Danil ŢORIN [mailto:[email protected]] Sent: Tuesday, August 21, 2012 9:55 AM To: [email protected] Subject: Re: Custom Directory and IndexInput My bad, few casts solved the issue. On Tue, Aug 21, 2012 at 10:46 AM, Danil ŢORIN <[email protected]> wrote: One tricky thing, I'm doing a lot of delegation, I can't call "clone()" on the DataInput I'm delegating to. (unfortunately in my case it's delegation, not inheritance) The fact that the delegate implements Clonable, doesn't help, the clone() method is still protected :(. Any idea how to circumvent this? On Tue, Aug 21, 2012 at 10:37 AM, Uwe Schindler <[email protected]> wrote: Thanks for „reporting“ lack of documentation, I will shortly commit some additional specs to IndexInput’s javadocs! Thanks, Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de <http://www.thetaphi.de/> eMail: [email protected] From: Danil ŢORIN [mailto:[email protected]] Sent: Tuesday, August 21, 2012 8:59 AM To: [email protected] Subject: Re: Custom Directory and IndexInput Thanks, that's exactly the response I was looking for. On Tue, Aug 21, 2012 at 9:31 AM, Uwe Schindler <[email protected]> wrote: Hi, It is required that every thread uses its own IndexInput. This is why e.g. every search calls IndexInput.clone() several times while executing a search query. Because of that there are explicietly no thread safety requirements in IndexInput, but clone() must be implemented as a cheap method, cloning the IndexInput and keeping the state. It is not required to recreate file descriptors or like that, but it must make sure that the new IndexInput can be used independent, although on same file/same file descriptor like another one. Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de <http://www.thetaphi.de/> eMail: [email protected] From: Danil ŢORIN [mailto:[email protected]] Sent: Tuesday, August 21, 2012 8:22 AM To: [email protected] Subject: Custom Directory and IndexInput I'm trying to build a custom directory implementation, actually the directory itself just fallback to one of the existing directory implementations, so it's actually more about IndexInput/IndexOutput. I have some concerns about my implementation, especially related to multithreading: * IndexOutput writeByte/writeBytes should be no problem since only one thread will write to specific output at any time * IndexInput on the other hand could be called from multiple threads * most implementations of IndexInput (including mine) keep some internal state like current position in the file/buffer/etc * if one of the threads is calling readShort()/readLong()/readVInt() (default implementations from DataInput) while some other thread is calling readByte() bad stuff will happen * still in most existing implementations I don't see any synchronization code on reads * is this problem solved somewhere on a higher level, like IndexReader/IndexSearcher, so I don't have to worry about it? * cause I'm really not sure how to solve it without forcing synchronized on all read methods and I'd really hate that. How this issue is solved in Lucene? what should I do to make sure that my implementation doing right thing? Currently I'm on Lucene 4.0.0 BETA
