> On Sun, Jul 12, 2009 at 10:51 AM, eks dev<eks...@yahoo.co.uk> wrote: > > > MMapDirectory has support for chunking (Ineteger.MAX_VALUE) anyhow... > maybe for such cases this threshold can become user settable. I will try > to experiment with it (I am talking about MMapDirectory -> private final > int MAX_BBUF = Integer.MAX_VALUE;) > > This chunking doesn't reduce virtual memory usage, because the buffers > are all mapped up front on opening the IndexInput. > > It's only done because of a limitation of ByteBuffer (it uses int not > long to address the values).
The idea of chunking is because of address space fragmentation. If the Java VM allocates or frees memory (e.g. when expanding the heap from Xms to Xmx), the address space of the process gets fragmented. If the index is very large, the byte buffer must fit in complete as one fragment into the address space, which may easily fail when you only have 32 bit. If the file is mapped in different chunks, each chunk can be somewhere else in address space (e.g., not only one 1 Gig junk is needed, there can be ten 100 Meg junks at different places). Uwe --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
