It hasn't been committed yet ... it's a patch on https://issues.apache.org/jira/browse/LUCENE-4123
Mike McCandless http://blog.mikemccandless.com On Sat, Jun 30, 2012 at 4:24 AM, Cheng <zhoucheng2...@gmail.com> wrote: > Hi, > > I can't find the CachingRAMDirectory in Lucene 3.6. Is this decaperated? > > Thanks > > On Mon, Jun 18, 2012 at 7:32 PM, Michael McCandless < > luc...@mikemccandless.com> wrote: > >> 9 fold improvement using RAMDir over MMapDir is much more than I've >> seen (~30-40% maybe) in the past. >> >> Can you explain how you are using Lucene? >> >> You may also want to try the CachingRAMDirectory patch on >> https://issues.apache.org/jira/browse/LUCENE-4123 >> >> Mike McCandless >> >> http://blog.mikemccandless.com >> >> On Sat, Jun 16, 2012 at 7:18 AM, Cheng <zhoucheng2...@gmail.com> wrote: >> > After a number of test, the performance of MMapDirectory is not even >> close >> > to that of RAMDirectory, in terms of speed. >> > >> > My application w/ the former can only deal with 10 tasks per round while >> it >> > could handle over 90 w/ RAMDirectory. >> > >> > I use the application in Linux. >> > >> > What can be the reasons? >> > >> > Thanks. >> > >> > >> > On Tue, Jun 5, 2012 at 7:53 AM, Uwe Schindler <u...@thetaphi.de> wrote: >> > >> >> This is managed by your operating system. In general OS kernels like >> Linux >> >> or Windows use all free memory to cache disk accesses. >> >> >> >> ----- >> >> Uwe Schindler >> >> H.-H.-Meier-Allee 63, D-28213 Bremen >> >> http://www.thetaphi.de >> >> eMail: u...@thetaphi.de >> >> >> >> >> >> > -----Original Message----- >> >> > From: Cheng [mailto:zhoucheng2...@gmail.com] >> >> > Sent: Monday, June 04, 2012 6:10 PM >> >> > To: java-user@lucene.apache.org >> >> > Subject: Re: RAMDirectory unexpectedly slows >> >> > >> >> > Can I control the size of ram given to either MMapDirectory or >> >> > ByteBufferDirectory? >> >> > >> >> > On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler <u...@thetaphi.de> >> wrote: >> >> > >> >> > > Hi, >> >> > > >> >> > > If you are using MMapDirectory or this ByteBufferDirectory (which is >> >> > > similar to the first) the used RAM is outside JVM heap, it is in the >> >> > > FS cache of the OS kernel. Giving too much memory to the JVM >> penalizes >> >> > > the OS cache, so give only as much as the App needs. Lucene and the >> OS >> >> > > kernel will then utilize the remaining memory for caching. >> >> > > >> >> > > Please read docs of MMapDirectory and inform yourself about mmap in >> >> e.g. >> >> > > Wikipedia. >> >> > > >> >> > > Uwe >> >> > > -- >> >> > > Uwe Schindler >> >> > > H.-H.-Meier-Allee 63, 28213 Bremen >> >> > > http://www.thetaphi.de >> >> > > >> >> > > >> >> > > >> >> > > Cheng <zhoucheng2...@gmail.com> schrieb: >> >> > > >> >> > > Please shed more insight into the difference between JVM heap size >> and >> >> > > the memory size used by Lucene. >> >> > > >> >> > > What I am getting at is that no matter however much ram I give my >> >> > > apps, Lucene can't utilize it. Is that right? >> >> > > >> >> > > What about the ByteBufferDirectory? Can this specific directory >> >> > > utilize the 2GB memory I grant to the app? >> >> > > >> >> > > On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen < >> >> > > jason.rutherg...@gmail.com> wrote: >> >> > > >> >> > > > If you want the index to be stored completely in RAM, there is the >> >> > > > ByteBuffer directory [1]. Though I do not see the point in putting >> >> > > > an index in RAM, it will be cached in RAM regardless in the OS >> >> > > > system IO cache. >> >> > > > >> >> > > > 1. >> >> > > > >> >> > > >> https://github.com/elasticsearch/elasticsearch/blob/master/src/main/ja >> >> > > va/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java >> >> > > > >> >> > > > On Mon, Jun 4, 2012 at 10:55 AM, Cheng <zhoucheng2...@gmail.com> >> >> > wrote: >> >> > > > > My indexes are 500MB+. So it seems like that RAMDirectory is not >> >> > > > > good >> >> > > for >> >> > > > > that big a size. >> >> > > > > >> >> > > > > My challenge, on the other side, is that I need to update the >> >> > > > > indexes >> >> > > > very >> >> > > > > frequently. So, do you think MMapDirectory is the solution? >> >> > > > > >> >> > > > > Thanks. >> >> > > > > >> >> > > > > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky < >> >> > > j...@basetechnology.com >> >> > > > >wrote: >> >> > > > > >> >> > > > >> From the javadoc for RAMDirectory: >> >> > > > >> >> >> > > > >> "Warning: This class is not intended to work with huge indexes. >> >> > > > Everything >> >> > > > >> beyond several hundred megabytes will waste resources (GC >> >> > > > >> cycles), >> >> > > > because >> >> > > > >> it uses an internal buffer size of 1024 bytes, producing >> millions >> >> > > > >> of byte[1024] arrays. This class is optimized for small >> >> > > > >> memory-resident indexes. It also has bad concurrency on >> >> multithreaded >> >> > environments. >> >> > > > >> >> >> > > > >> It is recommended to materialize large indexes on disk and use >> >> > > > >> MMapDirectory, which is a high-performance directory >> >> > > > >> implementation >> >> > > > working >> >> > > > >> directly on the file system cache of the operating system, so >> >> > > > >> copying >> >> > > > data >> >> > > > >> to Java heap space is not useful." >> >> > > > >> >> >> > > > >> -- Jack Krupansky >> >> > > > >> >> >> > > > >> -----Original Message----- From: Cheng >> >> > > > >> Sent: Monday, June 04, 2012 10:08 AM >> >> > > > >> To: java-user@lucene.apache.org >> >> > > > >> Subject: RAMDirectory unexpectedly slows >> >> > > > >> >> >> > > > >> >> >> > > > >> Hi, >> >> > > > >> >> >> > > > >> My apps need to read from and write to some big indexes >> >> frequently. >> >> > > So I >> >> > > > >> use RAMDirectory instead of FSDirectory, and give JVM about 2GB >> >> > > > >> memory size. >> >> > > > >> >> >> > > > >> I notice that the speed of reading and writing unexpectedly >> slows >> >> > > > >> as >> >> > > the >> >> > > > >> size of the indexes increases. Since the usage of RAM is less >> >> > > > >> than >> >> > > 20%, >> >> > > > I >> >> > > > >> think by default the RAMDirectory doesn't take advantage of the >> >> > > memory I >> >> > > > >> assigned to JVM. >> >> > > > >> >> >> > > > >> What are the steps to improve the reading and writing speed of >> >> > > > >> RAMDirectory? >> >> > > > >> >> >> > > > >> Thanks! >> >> > > > >> Jeff >> >> > > > >> >> >> > > > >> >> >> > > >_____________________________________________ >> >> > > **_____________________________________________ >> >> > > **--------- >> >> > > > >> To unsubscribe, e-mail: >> >> > > > >> java-user-unsubscribe@lucene.**apache.org< >> >> > > > java-user-unsubscr...@lucene.apache.org> >> >> > > > >> For additional commands, e-mail: >> >> > > > >> java-user-help@lucene.apache.**org< >> >> > > > java-user-h...@lucene.apache.org> >> >> > > > >> >> >> > > > >> >> >> > > > >> >> > > >_____________________________________________ >> >> > > >> >> > > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> >> > > > For additional commands, e-mail: java-user-h...@lucene.apache.org >> >> > > > >> >> > > > >> >> > > >> >> > > >> >> >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> >> --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
