Solaris and MacOs On Sun, May 2, 2010 at 4:47 PM, Mark Miller <[email protected]> wrote:
> Interesting ... whats the OS? > > > On Sun, May 2, 2010 at 7:12 PM, John Wang <[email protected]> wrote: > >> 12 concurrent search threads in a stress environment. (with about 5M doc >> index) >> >> -John >> >> On Sun, May 2, 2010 at 3:13 PM, Mark Miller <[email protected]>wrote: >> >>> Perfectly safe to use SimpleFSDirectory. When you measure the >>> performance, are you measuring a lot of concurrent requests? That's where >>> you should see the win. >>> >>> - Mark >>> >>> >>> On 5/1/10 6:38 PM, John Wang wrote: >>> >>>> We are seeing this issue as well in your production. (using Zoie on top >>>> of lucene 2.9.1) >>>> After some performance comparisons, we do NOT see performance gain with >>>> NIO, rather these nasty ClosedChannelExceptions. >>>> >>>> I think the performance gains ppl are seeing with 2.9.1 can be due to >>>> many different things. From what we seen, they are not related to >>>> NIOFSDirectory. >>>> >>>> Our solution is to avoid calling FSDirectory.open(), instead just call >>>> new SimpleFSDirectory(). Is this safe? >>>> >>>> -John >>>> >>>> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>> Perhaps - one of the things they are supposed to be addressing is >>>> extendability. >>>> >>>> nio2 does have FileSystemProvider, which would actually allow you to >>>> create a custom channel ! >>>> >>>> I have not dug in enough to know much more than that though. >>>> >>>> *But*, another really interesting thing is that in Java 7, >>>> FileDescriptors are ref counted ! (though users can't inc/dec). >>>> >>>> But, FileInputStream and OutputStream have a new constructor that >>>> takes >>>> a FileDescriptor. >>>> >>>> So possibly, you could just make one that sits around to keep the >>>> FileDescriptor valid, and get your channel off >>>> FileInputStream/FileOutputStream? >>>> >>>> And then if it goes down, make a new one using the FileDescriptor >>>> which >>>> was not actually closed because there was a still a ref to it. >>>> >>>> Possibly .... ;) >>>> >>>> Michael McCandless wrote: >>>> > Does anyone know if nio2 has improved this...? >>>> > >>>> > Mike >>>> > >>>> > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen >>>> > <[email protected] <mailto:[email protected]>> >>>> >>>> wrote: >>>> > >>>> >> Defaulting NIOFSDir could account for some of the recent speed >>>> >> improvements users have been reporting in Lucene 2.9. So >>>> removing it >>>> >> as a default could reverse those and people could then report >>>> Lucene >>>> >> 3.X has slowed... >>>> >> >>>> >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless >>>> >> <[email protected] <mailto:[email protected]>> >>>> >>>> wrote: >>>> >> >>>> >>> Bummer. >>>> >>> >>>> >>> So the only viable workarounds are 1) don't use >>>> Thread.interrupt (nor, >>>> >>> things like Future.cancel, which in turn use Thread.interrupt) >>>> with >>>> >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the >>>> app must >>>> >>> make a deletion policy that keeps a commit alive if any reader >>>> is >>>> >>> using it. Or, 3) don't use NIOFSDir! >>>> >>> >>>> >>> Mike >>>> >>> >>>> >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer >>>> >>> <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>> >>>> >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless >>>> >>>> <[email protected] <mailto:[email protected]>> >>>> >>>> wrote: >>>> >>>> >>>> >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler >>>> <[email protected] <mailto:[email protected]>> wrote: >>>> >>>>> >>>> >>>>> >>>> >>>>>> So I checked the code of NIOFSIndexInput, my last comment >>>> was not really correct: >>>> >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens >>>> the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open >>>> until the file is closed (and also the channel). >>>> >>>>>> >>>> >>>>>> So it's really simple to fix in my opinion, just call >>>> getChannel() again on this exception. Because the RAF should still >>>> be open? >>>> >>>>>> >>>> >>>> Short answer: >>>> >>>> public final FileChannel getChannel() { >>>> >>>> synchronized (this) { >>>> >>>> if (channel == null) >>>> >>>> channel = FileChannelImpl.open(fd, true, rw, >>>> this); >>>> >>>> return channel; >>>> >>>> } >>>> >>>> } >>>> >>>> >>>> >>>> this is not gonna work I tried it before. The RandomAccessFile >>>> buffers >>>> >>>> the channel!! >>>> >>>> >>>> >>>> simon >>>> >>>> >>>> >>>>> I think we need a definitive answer on what happens to the >>>> RAF when >>>> >>>>> the FileChannel was closed by Thread.Interrupt. Simon can >>>> you test >>>> >>>>> this? >>>> >>>>> >>>> >>>>> Mike >>>> >>>>> >>>> >>>>> >>>> --------------------------------------------------------------------- >>>> >>>>> To unsubscribe, e-mail: >>>> [email protected] >>>> <mailto:[email protected]> >>>> >>>> >>>>> For additional commands, e-mail: >>>> [email protected] <mailto: >>>> [email protected]> >>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>> >>>> --------------------------------------------------------------------- >>>> >>>> To unsubscribe, e-mail: [email protected] >>>> <mailto:[email protected]> >>>> >>>> >>>> For additional commands, e-mail: >>>> [email protected] <mailto: >>>> [email protected]> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>> >>>> --------------------------------------------------------------------- >>>> >>> To unsubscribe, e-mail: [email protected] >>>> <mailto:[email protected]> >>>> >>>> >>> For additional commands, e-mail: >>>> [email protected] <mailto: >>>> [email protected]> >>>> >>>> >>> >>>> >>> >>>> >>> >>>> >> >>>> --------------------------------------------------------------------- >>>> >> To unsubscribe, e-mail: [email protected] >>>> <mailto:[email protected]> >>>> >>>> >> For additional commands, e-mail: [email protected] >>>> <mailto:[email protected]> >>>> >>>> >> >>>> >> >>>> >> >>>> > >>>> > >>>> --------------------------------------------------------------------- >>>> > To unsubscribe, e-mail: [email protected] >>>> <mailto:[email protected]> >>>> >>>> > For additional commands, e-mail: [email protected] >>>> <mailto:[email protected]> >>>> >>>> > >>>> > >>>> >>>> >>>> -- >>>> - Mark >>>> >>>> http://www.lucidimagination.com >>>> >>>> >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] >>>> <mailto:[email protected]> >>>> >>>> For additional commands, e-mail: [email protected] >>>> <mailto:[email protected]> >>>> >>>> >>>> >>> >>> -- >>> - Mark >>> >>> http://www.lucidimagination.com >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> >> > > > -- > - Mark > > http://www.lucidimagination.com > >
