Hi Jennifer, Could you reproduce it on your side by doing the same kind of systems tests again? If you could then I'd be very happy if you could try a patched version that we have been working on and see if that fixes the issue.
Cheers, Tobias On Tue, May 17, 2011 at 2:49 AM, Jennifer Hickey <[email protected]> wrote: > Hi Tobias, > Unfortunately I don't have an isolated test case, as I was doing a fairly > involved system test at the time. I may be able to have a colleague work on > reproducing it at a later date (I've been diverted to something else for the > moment). > > I was remote debugging with Eclipse, so I toggled a method breakpoint on > Thread.interrupt() and then inspected the stack once the breakpoint was hit. > > Sorry I don't have more information at the moment. I agree that > eliminating the interrupts sounds like the best approach, if possible. > > Thanks, > Jennifer > ________________________________________ > From: [email protected] [[email protected]] On > Behalf Of Tobias Ivarsson [[email protected]] > Sent: Thursday, April 28, 2011 6:23 AM > To: Neo4j user discussions > Subject: Re: [Neo4j] ClosedChannelExceptions in highly concurrent > environment > > Hi Jennifer, > > I'd first like to thank you for the testing and analysis you've done. Very > useful stuff. Do you think you could send some test code our way that > reproduces this issue? > > This is actually the first time this issue has been reported, so I wouldn't > say it is a common issue. My guess is that your thread volume triggered a > rare condition that wouldn't be encountered otherwise. > > I'm also curious to know how you found the source of the interruptions. > When > I debug thread interruptions I've never been able to find out where the > thread got interrupted from without doing tedious procedures of breakpoint > + > logging + trying to match thread ids. If you have a better method for doing > that I'd very much like to know. > > I think we should focus the effort on fixing the interruption issue if we > can. And I believe we would be able to do that if the interruptions do in > fact originate from where you say they do. But the suggestion of being able > to switch the lucene directory implementation is still interesting, but as > you point out since it has issues on some platforms it would be better if > we > could be rid of the interruption issue. > > Cheers, > Tobias > > On Thu, Apr 28, 2011 at 12:41 AM, Jennifer Hickey <[email protected] > >wrote: > > > Hello, > > I've been running some tests w/approx 400 threads reading various indexed > > property values. I'm running on 64 bit Linux. I was frequently seeing > the > > ClosedChannelException below. The javadoc on Lucene's NIOFSDirectory > states > > that "Accessing this class either directly or indirectly from a thread > while > > it's interrupted can close the underlying file descriptor immediately if > at > > the same time the thread is blocked on IO. The file descriptor will > remain > > closed and subsequent access to {@link NIOFSDirectory} will throw a > {@link > > ClosedChannelException}. If your application uses either {@link > > Thread#interrupt()} or {@link Future#cancel(boolean)} you should use > {@link > > SimpleFSDirectory} in favor of {@link NIOFSDirectory}." > > > > A bit of debugging revealed that the Thread.interrupts were coming from > > Neo4j, specifically in RWLock and MappedPersistenceWindow. So it seems > like > > this would be a common problem, though perhaps I am missing something? > > > > SimpleFSDirectory seems a bit of a performance bottleneck, so I switched > to > > MMapDirectory and the problem did go away. I didn't see a way to switch > > implementations w/out modifying neo4j code, so I changed LuceneDataSource > as > > follows: > > > > static Directory getDirectory( String storeDir, > > IndexIdentifier identifier ) throws IOException > > { > > MMapDirectory dir=new MMapDirectory(getFileDirectory( storeDir, > > identifier), null); > > if(MMapDirectory.UNMAP_SUPPORTED) { > > dir.setUseUnmap(true); > > } > > return dir; > > } > > > > So I'm wondering if others have seen this problem and/or if there is a > > recommended solution? Our product runs on quite a few different > operating > > systems, so I have some reservations about using MMapDirectory as well > > (javadoc speaks of a few caveats on Windows, 64 vs 32, etc). Also, I'd > > rather not maintain a patched version of the neo4j code if avoidable. > > > > Thanks! > > Jennifer > > > > Exception: > > Caused by: java.nio.channels.ClosedChannelException > > at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:88) > > at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:613) > > at > > > org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:161) > > at > > > org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:139) > > at > > > org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(CompoundFileReader.java:285) > > at > > > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:160) > > at > > > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:39) > > at org.apache.lucene.store.DataInput.readVInt(DataInput.java:86) > > at > > > org.apache.lucene.index.codecs.DeltaBytesReader.read(DeltaBytesReader.java:40) > > at > > > org.apache.lucene.index.codecs.PrefixCodedTermsReader$FieldReader$SegmentTermsEnum.next(PrefixCodedTermsReader.java:469) > > at > > > org.apache.lucene.index.codecs.PrefixCodedTermsReader$FieldReader$SegmentTermsEnum.seek(PrefixCodedTermsReader.java:385) > > at org.apache.lucene.index.TermsEnum.seek(TermsEnum.java:68) > > at org.apache.lucene.index.Terms.docFreq(Terms.java:53) > > at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:898) > > at org.apache.lucene.index.IndexReader.docFreq(IndexReader.java:882) > > at > > org.apache.lucene.index.DirectoryReader.docFreq(DirectoryReader.java:687) > > > > > > _______________________________________________ > > Neo4j mailing list > > [email protected] > > https://lists.neo4j.org/mailman/listinfo/user > > > > > > -- > Tobias Ivarsson <[email protected]> > Hacker, Neo Technology > www.neotechnology.com > Cellphone: +46 706 534857 > _______________________________________________ > Neo4j mailing list > [email protected] > https://lists.neo4j.org/mailman/listinfo/user > _______________________________________________ > Neo4j mailing list > [email protected] > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson <[email protected]> Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
