Interesting ... whats the OS? On Sun, May 2, 2010 at 7:12 PM, John Wang <[email protected]> wrote:
> 12 concurrent search threads in a stress environment. (with about 5M doc > index) > > -John > > On Sun, May 2, 2010 at 3:13 PM, Mark Miller <[email protected]> wrote: > >> Perfectly safe to use SimpleFSDirectory. When you measure the performance, >> are you measuring a lot of concurrent requests? That's where you should see >> the win. >> >> - Mark >> >> >> On 5/1/10 6:38 PM, John Wang wrote: >> >>> We are seeing this issue as well in your production. (using Zoie on top >>> of lucene 2.9.1) >>> After some performance comparisons, we do NOT see performance gain with >>> NIO, rather these nasty ClosedChannelExceptions. >>> >>> I think the performance gains ppl are seeing with 2.9.1 can be due to >>> many different things. From what we seen, they are not related to >>> NIOFSDirectory. >>> >>> Our solution is to avoid calling FSDirectory.open(), instead just call >>> new SimpleFSDirectory(). Is this safe? >>> >>> -John >>> >>> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Perhaps - one of the things they are supposed to be addressing is >>> extendability. >>> >>> nio2 does have FileSystemProvider, which would actually allow you to >>> create a custom channel ! >>> >>> I have not dug in enough to know much more than that though. >>> >>> *But*, another really interesting thing is that in Java 7, >>> FileDescriptors are ref counted ! (though users can't inc/dec). >>> >>> But, FileInputStream and OutputStream have a new constructor that >>> takes >>> a FileDescriptor. >>> >>> So possibly, you could just make one that sits around to keep the >>> FileDescriptor valid, and get your channel off >>> FileInputStream/FileOutputStream? >>> >>> And then if it goes down, make a new one using the FileDescriptor >>> which >>> was not actually closed because there was a still a ref to it. >>> >>> Possibly .... ;) >>> >>> Michael McCandless wrote: >>> > Does anyone know if nio2 has improved this...? >>> > >>> > Mike >>> > >>> > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen >>> > <[email protected] <mailto:[email protected]>> >>> >>> wrote: >>> > >>> >> Defaulting NIOFSDir could account for some of the recent speed >>> >> improvements users have been reporting in Lucene 2.9. So >>> removing it >>> >> as a default could reverse those and people could then report >>> Lucene >>> >> 3.X has slowed... >>> >> >>> >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless >>> >> <[email protected] <mailto:[email protected]>> >>> >>> wrote: >>> >> >>> >>> Bummer. >>> >>> >>> >>> So the only viable workarounds are 1) don't use >>> Thread.interrupt (nor, >>> >>> things like Future.cancel, which in turn use Thread.interrupt) >>> with >>> >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the >>> app must >>> >>> make a deletion policy that keeps a commit alive if any reader is >>> >>> using it. Or, 3) don't use NIOFSDir! >>> >>> >>> >>> Mike >>> >>> >>> >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer >>> >>> <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> >>> >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless >>> >>>> <[email protected] <mailto:[email protected]>> >>> >>> wrote: >>> >>>> >>> >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler >>> <[email protected] <mailto:[email protected]>> wrote: >>> >>>>> >>> >>>>> >>> >>>>>> So I checked the code of NIOFSIndexInput, my last comment >>> was not really correct: >>> >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens >>> the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open >>> until the file is closed (and also the channel). >>> >>>>>> >>> >>>>>> So it's really simple to fix in my opinion, just call >>> getChannel() again on this exception. Because the RAF should still >>> be open? >>> >>>>>> >>> >>>> Short answer: >>> >>>> public final FileChannel getChannel() { >>> >>>> synchronized (this) { >>> >>>> if (channel == null) >>> >>>> channel = FileChannelImpl.open(fd, true, rw, >>> this); >>> >>>> return channel; >>> >>>> } >>> >>>> } >>> >>>> >>> >>>> this is not gonna work I tried it before. The RandomAccessFile >>> buffers >>> >>>> the channel!! >>> >>>> >>> >>>> simon >>> >>>> >>> >>>>> I think we need a definitive answer on what happens to the >>> RAF when >>> >>>>> the FileChannel was closed by Thread.Interrupt. Simon can >>> you test >>> >>>>> this? >>> >>>>> >>> >>>>> Mike >>> >>>>> >>> >>>>> >>> --------------------------------------------------------------------- >>> >>>>> To unsubscribe, e-mail: >>> [email protected] >>> <mailto:[email protected]> >>> >>> >>>>> For additional commands, e-mail: >>> [email protected] <mailto: >>> [email protected]> >>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>> >>> --------------------------------------------------------------------- >>> >>>> To unsubscribe, e-mail: [email protected] >>> <mailto:[email protected]> >>> >>> >>>> For additional commands, e-mail: >>> [email protected] <mailto: >>> [email protected]> >>> >>> >>>> >>> >>>> >>> >>>> >>> >>> >>> --------------------------------------------------------------------- >>> >>> To unsubscribe, e-mail: [email protected] >>> <mailto:[email protected]> >>> >>> >>> For additional commands, e-mail: >>> [email protected] <mailto: >>> [email protected]> >>> >>> >>> >>> >>> >>> >>> >>> >> >>> --------------------------------------------------------------------- >>> >> To unsubscribe, e-mail: [email protected] >>> <mailto:[email protected]> >>> >>> >> For additional commands, e-mail: [email protected] >>> <mailto:[email protected]> >>> >>> >> >>> >> >>> >> >>> > >>> > >>> --------------------------------------------------------------------- >>> > To unsubscribe, e-mail: [email protected] >>> <mailto:[email protected]> >>> >>> > For additional commands, e-mail: [email protected] >>> <mailto:[email protected]> >>> >>> > >>> > >>> >>> >>> -- >>> - Mark >>> >>> http://www.lucidimagination.com >>> >>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> <mailto:[email protected]> >>> >>> For additional commands, e-mail: [email protected] >>> <mailto:[email protected]> >>> >>> >>> >> >> -- >> - Mark >> >> http://www.lucidimagination.com >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> > -- - Mark http://www.lucidimagination.com
