I'm not an expert, so take my answer with a few grains of salt. Also the following applies to Linux; I don't know anything about other OSs.
On the physical memory level, memory for the array doesn't need to be contiguous. That is the whole point of having virtual memory in the first place. On the virtual memory level, the memory for array needs to be contiguous. So what is likely to happen is that an anonymous memory mapping for a particular region is created. This memory mapping is initialized with the zero page (has 0's as content), so all page table entries for this region are pointing to the same chunk of physical memory. Only when there is a write on this zero page, the copy on write mechanism kicks in and a page frame (physical memory) is allocated and assigned to that page table entry. Afaik ByteBuffer.allocate direct just forwards the request to the OS to do an anonymous memory mapping. And since the virtual address space is huge, it should be easy to find a contiguous region of virtual memory for that array. When you use new Object[...] you deal with the memory management from the JVM and then it depends on the GC algorithm. For example with CMS, there needs to be an entry in the free list large enough to hold that array (it needs to be contiguous). With G1 AFAIK it is less of an issue; if it is a large array, one or more non-contiguous humongous regions can be used. With parallel/serial, it should also not be an issue since these compact the memory and therefore there is no fragmentation. Which GC algorithm are you using? On Thursday, October 1, 2020 at 9:43:59 PM UTC+3 Shevek wrote: > When I do new byte[N], I get OutOfMemoryError, despite that the VM > claims to have more than enough free space (according to MemoryMXBean, > Runtime.freeMemory, visualvm, etc). > > My working assumption is that while I have enough free memory, I don't > have enough contiguous free memory. Is there a solution to this? Will I > get better results from any of: > > * ByteBuffer.allocateDirect() - presumably yes, but has other issues > relating to overall memory usage on the system > * G1GC (or other GC which allocates (relocatable?) regions) - this is a > deep hole I haven't yet explored. > * Calling System.gc() before allocating a contiguous region [apparently > doesn't help]. > * Other? > > If we do follow a strategy using allocateDirect, will we end up with the > same fragmentation issue in the native heap, along with committed > off-heap memory which we can no longer effectively use, or is the > off-heap memory managed in some manner which avoids this problem? > > Thank you. > > S. > -- You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web, visit https://groups.google.com/d/msgid/mechanical-sympathy/99daf015-ae67-4fb7-948f-cfb4a2cc00d6n%40googlegroups.com.
