My original question is if there exists a way to configure writer when to writer to FSDirectory. I think there may be something in the IndexWriterConfig that can helps.
On Mon, Feb 6, 2012 at 11:50 PM, Ian Lea <ian....@gmail.com> wrote: > Well, yes. What would you expect? From the javadocs for > IndexWriter.commit() > > Commits all pending changes (added & deleted documents, segment > merges, added indexes, etc.) to the index, and syncs all referenced > index files ... This may be a costly operation, so you should test the > cost in your application and do it only when really necessary. > > If you are using NRTManager why do you care how long this takes? How > often are you calling it? Why? > > > -- > Ian. > > > On Mon, Feb 6, 2012 at 3:45 PM, Cheng <zhoucheng2...@gmail.com> wrote: > > Uwe, when I meant speed is slow, I didn't refer to instant visibility of > > changes, but that the changes may be synchronized with FSDirectory when I > > use writer.commit(). > > > > When I use RAMDirectory, the writer.commit() seems much faster than using > > NRTManager built upon FSDirectory. So, I am guessing the difference is > the > > index synchronization. > > > > > > > > On Mon, Feb 6, 2012 at 11:40 PM, Uwe Schindler <u...@thetaphi.de> wrote: > > > >> Please review the following articles about NRT, absolutely instant > updates > >> that are visible as they are done are almost impossible (even with > >> RAMDirectory): > >> > >> http://goo.gl/mzAHt > >> http://goo.gl/5RoPx > >> http://goo.gl/vSJ7x > >> > >> Uwe > >> > >> ----- > >> Uwe Schindler > >> H.-H.-Meier-Allee 63, D-28213 Bremen > >> http://www.thetaphi.de > >> eMail: u...@thetaphi.de > >> > >> > -----Original Message----- > >> > From: Cheng [mailto:zhoucheng2...@gmail.com] > >> > Sent: Monday, February 06, 2012 4:27 PM > >> > To: java-user@lucene.apache.org > >> > Subject: Re: Configure writer to write to FSDirectory? > >> > > >> > Ian, > >> > > >> > I encountered an issue that I need to frequently update the index. The > >> > NRTManager seems not very helpful on this front as the speed is slower > >> than > >> > RAMDirectory is used. > >> > > >> > Any improvement advice? > >> > > >> > > >> > > >> > On Mon, Feb 6, 2012 at 10:24 PM, Cheng <zhoucheng2...@gmail.com> > wrote: > >> > > >> > > That really helps! I will try it out. > >> > > > >> > > Thanks. > >> > > > >> > > > >> > > On Mon, Feb 6, 2012 at 10:12 PM, Ian Lea <ian....@gmail.com> wrote: > >> > > > >> > >> You would use NRTManagerReopenThread as a standalone thread, not > >> > >> plugged into your Executor stuff. It is a utility class which you > >> > >> don't have to use. See the javadocs. > >> > >> > >> > >> But in your case I'd use it, to start with anyway. Fire it up with > >> > >> suitable settings and forget about it, except to call close() > >> > >> eventually. Once you've got things up and running you can tweak > >> > >> things as much as you want but you appear to be having trouble > >> > >> getting up and running. > >> > >> > >> > >> So ... somewhere in the initialisation code of your app, create an > >> > >> IndexWriter, NRTManager + ReopenThread and SearcherManager as > >> > >> outlined before. Then pass the NRTManager to any/all write methods > >> > >> or threads and the SearcherManager instance to any/all search > methods > >> > >> or threads and you're done. If you want to use threads that are > part > >> > >> of your ExecutorService, fine. Just wrap it all together in > whatever > >> > >> combination of Thread or Runnable instances you want. > >> > >> > >> > >> > >> > >> Does that help? > >> > >> > >> > >> > >> > >> -- > >> > >> Ian. > >> > >> > >> > >> > >> > >> > I don't understand this following portion: > >> > >> > > >> > >> > IndexWriter iw = new IndexWriter(whatever - some standard disk > >> > >> > index); NRTManager nrtm = new NRTManager(iw, null); > >> > >> > NRTManagerReopenThread ropt = new NRTManagerReopenThread(nrtm, > >> > >> > ...); ropt.setXxx(...); .... > >> > >> > ropt.start(); > >> > >> > > >> > >> > I have a java ExecutorServices instance running which take care > of > >> > >> > my > >> > >> own > >> > >> > applications. I don't know how this NRTManagerReopenThread works > >> > >> > with my own ExecutorService instance. > >> > >> > > >> > >> > Can both work together? How can the NRTManagerReopenThread > >> > instance > >> > >> ropt be > >> > >> > plugged into my own multithreading framework? > >> > >> > > >> > >> > On Mon, Feb 6, 2012 at 8:17 PM, Ian Lea <ian....@gmail.com> > wrote: > >> > >> > > >> > >> >> If you can use NRTManager and SearcherManager things should be > >> > >> >> easy and blazingly fast rather than unbearably slow. The latter > >> > >> >> phrase is not one often associated with lucene. > >> > >> >> > >> > >> >> IndexWriter iw = new IndexWriter(whatever - some standard disk > >> > >> >> index); NRTManager nrtm = new NRTManager(iw, null); > >> > >> >> NRTManagerReopenThread ropt = new > >> > NRTManagerReopenThread(nrtm, > >> > >> >> ...); ropt.setXxx(...); ... > >> > >> >> ropt.start(); > >> > >> >> > >> > >> >> SearcherManager srchm = nrtm.getSearcherManager(b); > >> > >> >> > >> > >> >> Then add docs to your index via nrtm.addDocument(d), update with > >> > >> >> nrtm.updateDocument(...), and to search use > >> > >> >> > >> > >> >> IndexSearcher searcher = srchm.acquire(); try { search ... > >> > >> >> } finally { > >> > >> >> srchm.release(searcher); > >> > >> >> } > >> > >> >> > >> > >> >> All thread safe so you don't have to worry about any > complications > >> > >> >> there. And I bet it'll be blindingly fast. > >> > >> >> > >> > >> >> Don't forget to close() things down at the end. > >> > >> >> > >> > >> >> > >> > >> >> -- > >> > >> >> Ian. > >> > >> >> > >> > >> >> > >> > >> >> > >> > >> >> On Mon, Feb 6, 2012 at 12:15 AM, Cheng <zhoucheng2...@gmail.com > > > >> > >> wrote: > >> > >> >> > I was trying to, but don't know how to even I read some of > your > >> > >> blogs. > >> > >> >> > > >> > >> >> > On Sun, Feb 5, 2012 at 10:22 PM, Michael McCandless < > >> > >> >> > luc...@mikemccandless.com> wrote: > >> > >> >> > > >> > >> >> >> Are you using near-real-time readers? > >> > >> >> >> > >> > >> >> >> (IndexReader.open(IndexWriter)) > >> > >> >> >> > >> > >> >> >> Mike McCandless > >> > >> >> >> > >> > >> >> >> http://blog.mikemccandless.com > >> > >> >> >> > >> > >> >> >> On Sun, Feb 5, 2012 at 9:03 AM, Cheng < > zhoucheng2...@gmail.com> > >> > >> wrote: > >> > >> >> >> > Hi Uwe, > >> > >> >> >> > > >> > >> >> >> > My challenge is that I need to update/modify the indexes > >> > >> frequently > >> > >> >> while > >> > >> >> >> > providing the search capability. I was trying to use > >> > >> >> >> > FSDirectory, > >> > >> but > >> > >> >> >> found > >> > >> >> >> > out that the reading and writing from/to FSDirectory is > >> > >> >> >> > unbearably > >> > >> >> slow. > >> > >> >> >> So > >> > >> >> >> > I now am trying the RAMDirectory, which is fast. > >> > >> >> >> > > >> > >> >> >> > I don't know of MMapDirectory, and wonder if it is as fast > >> > >> >> >> > as > >> > >> >> >> RAMDirectory. > >> > >> >> >> > > >> > >> >> >> > > >> > >> >> >> > On Sun, Feb 5, 2012 at 4:14 PM, Uwe Schindler > >> > >> >> >> > <u...@thetaphi.de> > >> > >> >> wrote: > >> > >> >> >> > > >> > >> >> >> >> Hi Cheng, > >> > >> >> >> >> > >> > >> >> >> >> It seems that you use a RAMDirectory for *caching*, > >> > >> >> >> >> otherwise it > >> > >> >> makes > >> > >> >> >> no > >> > >> >> >> >> sense to write changes back. In recent Lucene versions, > this > >> > >> >> >> >> is > >> > >> not a > >> > >> >> >> good > >> > >> >> >> >> idea, especially for large indexes (RAMDirectory eats your > >> > >> >> >> >> heap > >> > >> >> space, > >> > >> >> >> >> allocates millions of small byte[] arrays,...). If you > need > >> > >> something > >> > >> >> >> like > >> > >> >> >> >> a > >> > >> >> >> >> caching Directory and you are working on a 64bit platform, > >> > >> >> >> >> you > >> > >> can > >> > >> >> use > >> > >> >> >> >> MMapDirectory (where the operating system kernel manages > the > >> > >> >> read/write > >> > >> >> >> >> between disk an memory). MMapDirectory is returned by > >> > >> >> >> >> default for > >> > >> >> >> >> FSDirectory.open() on most 64 bit platforms. The good > thing: > >> > >> >> >> >> the > >> > >> >> >> "caching" > >> > >> >> >> >> space is outside your JVM heap, so does not slowdown the > >> > >> >> >> >> garbage > >> > >> >> >> collector. > >> > >> >> >> >> So be sure to *not* allocate too much heap space (-Xmx) to > >> > >> >> >> >> your > >> > >> >> search > >> > >> >> >> app, > >> > >> >> >> >> only the minimum needed to execute it and leave the rest > of > >> > >> >> >> >> your > >> > >> RAM > >> > >> >> >> >> available for the OS kernel to manage FS cache. > >> > >> >> >> >> > >> > >> >> >> >> Uwe > >> > >> >> >> >> > >> > >> >> >> >> ----- > >> > >> >> >> >> Uwe Schindler > >> > >> >> >> >> H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > >> > >> >> >> >> eMail: u...@thetaphi.de > >> > >> >> >> >> > >> > >> >> >> >> > >> > >> >> >> >> > -----Original Message----- > >> > >> >> >> >> > From: Cheng [mailto:zhoucheng2...@gmail.com] > >> > >> >> >> >> > Sent: Sunday, February 05, 2012 7:56 AM > >> > >> >> >> >> > To: java-user@lucene.apache.org > >> > >> >> >> >> > Subject: Configure writer to write to FSDirectory? > >> > >> >> >> >> > > >> > >> >> >> >> > Hi, > >> > >> >> >> >> > > >> > >> >> >> >> > I build an RAMDirectory on a FSDirectory, and would like > >> the > >> > >> writer > >> > >> >> >> >> associated > >> > >> >> >> >> > with the RAMDirectory to periodically write to hard > drive. > >> > >> >> >> >> > > >> > >> >> >> >> > Is this achievable? > >> > >> >> >> >> > > >> > >> >> >> >> > Thanks. > >> > >> >> >> >> > >> > >> >> >> >> > >> > >> >> >> >> > >> > >> > --------------------------------------------------------------------- > >> > >> >> >> >> To unsubscribe, e-mail: > >> java-user-unsubscr...@lucene.apache.org > >> > >> >> >> >> For additional commands, e-mail: > >> > >> java-user-h...@lucene.apache.org > >> > >> >> >> >> > >> > >> >> >> >> > >> > >> >> >> > >> > >> >> >> > >> > >> > --------------------------------------------------------------------- > >> > >> >> >> To unsubscribe, e-mail: > java-user-unsubscr...@lucene.apache.org > >> > >> >> >> For additional commands, e-mail: > >> java-user-h...@lucene.apache.org > >> > >> >> >> > >> > >> >> >> > >> > >> >> > >> > >> >> > >> --------------------------------------------------------------------- > >> > >> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >> > >> >> For additional commands, e-mail: > java-user-h...@lucene.apache.org > >> > >> >> > >> > >> >> > >> > >> > >> > >> > --------------------------------------------------------------------- > >> > >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >> > >> For additional commands, e-mail: java-user-h...@lucene.apache.org > >> > >> > >> > >> > >> > > > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >> For additional commands, e-mail: java-user-h...@lucene.apache.org > >> > >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >
