I'd like to learn more about this issue as well, since we also deploy on both Windows and Linux systems, and are heavily leveraging Lucene indexing with Neo4J.
Thanks. Rick ________________________________________ From: [email protected] [[email protected]] On Behalf Of Jennifer Hickey [[email protected]] Sent: Wednesday, April 27, 2011 3:41 PM To: [email protected] Subject: [Neo4j] ClosedChannelExceptions in highly concurrent environment Hello, I've been running some tests w/approx 400 threads reading various indexed property values. I'm running on 64 bit Linux. I was frequently seeing the ClosedChannelException below. The javadoc on Lucene's NIOFSDirectory states that "Accessing this class either directly or indirectly from a thread while it's interrupted can close the underlying file descriptor immediately if at the same time the thread is blocked on IO. The file descriptor will remain closed and subsequent access to {@link NIOFSDirectory} will throw a {@link ClosedChannelException}. If your application uses either {@link Thread#interrupt()} or {@link Future#cancel(boolean)} you should use {@link SimpleFSDirectory} in favor of {@link NIOFSDirectory}." A bit of debugging revealed that the Thread.interrupts were coming from Neo4j, specifically in RWLock and MappedPersistenceWindow. So it seems like this would be a common problem, though perhaps I am missing something? SimpleFSDirectory seems a bit of a performance bottleneck, so I switched to MMapDirectory and the problem did go away. I didn't see a way to switch implementations w/out modifying neo4j code, so I changed LuceneDataSource as follows: static Directory getDirectory( String storeDir, IndexIdentifier identifier ) throws IOException { MMapDirectory dir=new MMapDirectory(getFileDirectory( storeDir, identifier), null); if(MMapDirectory.UNMAP_SUPPORTED) { dir.setUseUnmap(true); } return dir; } So I'm wondering if others have seen this problem and/or if there is a recommended solution? Our product runs on quite a few different operating systems, so I have some reservations about using MMapDirectory as well (javadoc speaks of a few caveats on Windows, 64 vs 32, etc). Also, I'd rather not maintain a patched version of the neo4j code if avoidable. Thanks! Jennifer Exception: Caused by: java.nio.channels.ClosedChannelException at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:88) at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:613) at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:161) at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:139) at org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(CompoundFileReader.java:285) at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:160) at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:39) at org.apache.lucene.store.DataInput.readVInt(DataInput.java:86) at org.apache.lucene.index.codecs.DeltaBytesReader.read(DeltaBytesReader.java:40) at org.apache.lucene.index.codecs.PrefixCodedTermsReader$FieldReader$SegmentTermsEnum.next(PrefixCodedTermsReader.java:469) at org.apache.lucene.index.codecs.PrefixCodedTermsReader$FieldReader$SegmentTermsEnum.seek(PrefixCodedTermsReader.java:385) at org.apache.lucene.index.TermsEnum.seek(TermsEnum.java:68) at org.apache.lucene.index.Terms.docFreq(Terms.java:53) at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:898) at org.apache.lucene.index.IndexReader.docFreq(IndexReader.java:882) at org.apache.lucene.index.DirectoryReader.docFreq(DirectoryReader.java:687) _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
