Hi, There is a misunderstanding: Just by allocating a direct buffer, there is still no difference to a heap buffer in the workflow!
NIO will read the data from file, copy it to FS cache and then the positional read() system call (used by NIO) copies the FS cache contents to the direct buffer, no real difference to a heap buffer. So it is still the same: data needs to be copied. Please note: Direct buffers have nothing to do with file system cache, they don't share the same memory areas! Hotspot allocates direct buffers using malloc() outside of Java heap, so not really useful here. The backside of using a non-heap buffer as target for the copy operation is the fact, that direct buffers are approx. 2 times slower when accessed from Java code (because they are outside java heap, the VM has to do extra work to prevent illegal accesses: so you have the same time for copy but slower access from Java. The buffers allocated by NIO are small so it does not matter for performance where they are. So heap is better. MappedByteBuffers are also direct buffers (they have the same base class), so there is still the overhead when accessing them from Java code, but to get rid of the additional copy, use MMapDirectory. To conclude: - MMapDirectory: No copying of the data from FS-cache to heap or direct buffers needed, which wastes most of the time. Access times to MappedByteBuffer from Java code is slower, but the spared data copy makes it much better for large files as used by Lucene. - NIOFSDirectory with direct buffers: Needs to copy data from FS cache to direct buffer memory (outside heap). Access times slower to direct buffers than to heap buffers -> 2 times bad - NIOFSDirectory with heap buffers: Needs to copy data from FS cache to heap. Access time from java code is very good! Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -----Original Message----- > From: wangzhijiang...@yahoo.com.cn > [mailto:wangzhijiang...@yahoo.com.cn] > Sent: Wednesday, July 31, 2013 11:59 AM > To: java-user@lucene.apache.org > Subject: why lucene not use DirectByteBuffer in NIOFSDirectory > > I read this article "Use Lucene's MMapDirectory on 64bit platforms, please!" > and it said the MMapDirectory is better than other Directory because it will > void copy data between file system cache and java heap. > > I checked the source code of NIOFSDirectory, and in new Buffer method it > called "ByteBuffer.wrap(newBuffer)", the generated ByteBuffer is > HeapByteBuffer. And it will indeed copy data between file system cache and > java heap. Why not use ByteBuffer.allocateDirect to generate > DirectyByteBuffer, and it will store data directly in file sysytem cache, not > java heap. If in this case , what is the different in performance between NIO > and MMap? Or allocate directy memory in still slower than Mmap? > > Maybe I made some misunderstanding of lucene code, thank you for any > suggestion in advance. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
