You only have to create the deletion policy (merging uses it). Mike
On Wed, Jan 20, 2010 at 11:27 AM, Sertic Mirko, Bedag <mirko.ser...@bedag.ch> wrote: > Ok, so does the merging go thru the IndexDeletionPolicy, or do I have to deal > with the MergePolicy to take care of merging? > > Regards > Mirko > > -----Ursprüngliche Nachricht----- > Von: Michael McCandless [mailto:luc...@mikemccandless.com] > Gesendet: Mittwoch, 20. Januar 2010 17:12 > An: java-user@lucene.apache.org > Betreff: Re: NFS, Stale File Handle Problem and my thoughts.... > > Yes, normal merging will cause this problem as well. > > Generally you should always use IndexReader.reopen -- it gives much > better reopen speed, less resources used, less GC, etc. > > Mike > > On Wed, Jan 20, 2010 at 10:49 AM, Sertic Mirko, Bedag > <mirko.ser...@bedag.ch> wrote: >> Mike >> >> Thank you so much for your feedback! >> >> Will the new IndexDeletionPolicy also be considered when segments are >> merged? Does merging also affect the NFS problem? >> >> Should I use IndexReader.reOpen() or just create a new IndexReader? >> >> Thanks in advance >> Mirko >> >> -----Ursprüngliche Nachricht----- >> Von: Michael McCandless [mailto:luc...@mikemccandless.com] >> Gesendet: Mittwoch, 20. Januar 2010 16:14 >> An: java-user@lucene.apache.org >> Betreff: Re: NFS, Stale File Handle Problem and my thoughts.... >> >> Right, it's only machine A that needs the deletion policy. All >> read-only machines just reopen on their schedule (or you can use some >> communication means a Shai describes to have lower latency reopen >> after the writer commits). >> >> Also realize that doing searching over NFS does not usually give very >> good performance... (though I've heard that mounting read-only can >> improve performance). >> >> Mike >> >> On Wed, Jan 20, 2010 at 9:05 AM, Sertic Mirko, Bedag >> <mirko.ser...@bedag.ch> wrote: >>> Hi Mike >>> >>> Thank you for your feedback! >>> >>> So I would need the following setup: >>> >>> a) Machine A with custom IndexDeletionPolicy and single IndexReader instance >>> b) Machine B with custom IndexDeletionPolicy and single IndexReader instance >>> c) Machine A and B periodically check if the index needs to be reopened, at >>> least at 12 o'clock >>> d) Machine A runs an Index update and optimization at 8 o'clock, using the >>> IndexDeletionPolicy. The IndexDeletionPolicy keeps track of the files to be >>> deleted. >>> e) On Machine A, the no longer needed files are taken from the >>> IndexDeletionPolicy, and deleted at 12:30. At this point the files to be >>> deleted should no longer be required by any IndexReader and can be safely >>> deleted. >>> >>> So the IndexDeletionPolicy should be a kind of Singleton, and in fact would >>> only be needed on Machine A, as only here index modifications are made. >>> Machine B has read only access. >>> >>> Would this be a valid setup? The only limitation is there is only ONE >>> IndexWriter box, and multiple IndexReader boxes. Based on our requirements, >>> this should fix very well. I really want to avoid some kind of index >>> replication between the boxes... >>> >>> Regards >>> Mirko >>> >>> >>> >>> -----Ursprüngliche Nachricht----- >>> Von: Michael McCandless [mailto:luc...@mikemccandless.com] >>> Gesendet: Mittwoch, 20. Januar 2010 14:45 >>> An: java-user@lucene.apache.org >>> Betreff: Re: NFS, Stale File Handle Problem and my thoughts.... >>> >>> Right, you just need to make a custom IndexDeletionPolicy. NFS makes >>> no effort to protect deletion of still-open files. >>> >>> A simple approach is one that only deletes a commit if it's more than >>> XXX minutes/hours old, such that XXX is set higher than the frequency >>> that IndexReaders are guaranteed to have reopened. >>> >>> The TestDeletionPolicy unit test in Lucene's sources has the >>> ExperiationTimeDeletionPolicy that you can use (but scrutinize!). >>> >>> Another alternative is to simply reopen the reader whenever you hit >>> Stale file handle (think of it as NFS's means of notifying you that >>> your reader is no longer current ;) ). But, if reopen time is >>> potentially long for your app, it's no good to make queries pay that >>> cost and the deletion policy is better. >>> >>> Mike >>> >>> On Wed, Jan 20, 2010 at 8:29 AM, Sertic Mirko, Bedag >>> <mirko.ser...@bedag.ch> wrote: >>>> h...@all >>>> >>>> >>>> >>>> We are using Lucene 2.4.1 on Debian Linux with 2 boxes. The index is >>>> stored on a common NFS share. Every box has a single IndexReader >>>> instance, and one Box has an IndexWriter instance, adding new documents >>>> or deleting existing documents at a given point in time. After adding or >>>> deleting the documents, a IndexWriter.optimize() is called. Every box >>>> checks periodically with IndexReader.isCurrent if the index needs to be >>>> reopened. >>>> >>>> >>>> >>>> Now, we are encountering a "Stale file handle" error on box b after the >>>> index was modified and optimized by box a. >>>> >>>> >>>> >>>> As far as i understand the problem with NFS is that box b tries to >>>> open/access a file that was deleted by box a on the NFS share. >>>> >>>> >>>> >>>> The question is now, when are files deleted? Does only the index >>>> optimization delete files, or can files be deleted just by adding or >>>> removing documents from an existing index? >>>> >>>> >>>> >>>> I now that there might be a better setup with Lucene and index >>>> replication, but this is an existing application and we cannot change >>>> the architecture now. So what would be the best solution? >>>> >>>> >>>> >>>> Can I just "change" the way Lucene deletes files? I think that just >>>> renaming no longer needed files would be good on NFS. After every >>>> IndexReader has reopened the index, the renamed files can be safely >>>> deleted, as they are definitely no longer needed. Where would be the >>>> hook point? I heard something about IndexDeletionPolicy.... >>>> >>>> >>>> >>>> Thanks in advance! >>>> >>>> >>>> >>>> Mirko >>>> >>>> >>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
