I should have not opened my mouth :) I thought multiple humongous regions for a single array did not need to be contiguous. But apparently this is incorrect:
https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/g1_gc_tuning.html On Fri, Oct 2, 2020 at 8:19 AM Peter Veentjer <[email protected]> wrote: > I forget a very important part. > > There is a memory allocator used by the process. And this memory allocator > either makes use of the program break or the mmap for an anonymous memory > mapping. > > The ByteBuffer.allocateDirect forwards allocation requests to this memory > allocator; it will not directly allocate memory by changing the program > break or add an anonymous memory mapping region (so what I said before was > incorrect). > > This memory allocator could also have a big impact on how the system deals > with fragmentation. > > On Fri, Oct 2, 2020 at 8:11 AM [email protected] <[email protected]> > wrote: > >> I'm not an expert, so take my answer with a few grains of salt. Also the >> following applies to Linux; I don't know anything about other OSs. >> >> On the physical memory level, memory for the array doesn't need to be >> contiguous. That is the whole point of having virtual memory in the first >> place. >> >> On the virtual memory level, the memory for array needs to be contiguous. >> So what is likely to happen is that an anonymous memory mapping for a >> particular region is created. This memory mapping is initialized with the >> zero page (has 0's as content), so all page table entries for this region >> are pointing to the same chunk of physical memory. Only when there is a >> write on this zero page, the copy on write mechanism kicks in and a page >> frame (physical memory) is allocated and assigned to that page table entry. >> >> Afaik ByteBuffer.allocate direct just forwards the request to the OS to >> do an anonymous memory mapping. And since the virtual address space is >> huge, it should be easy to find a contiguous region of virtual memory for >> that array. >> >> When you use new Object[...] you deal with the memory management from the >> JVM and then it depends on the GC algorithm. For example with CMS, there >> needs to be an entry in the free list large enough to hold that array (it >> needs to be contiguous). With G1 AFAIK it is less of an issue; if it is a >> large array, one or more non-contiguous humongous regions can be used. With >> parallel/serial, it should also not be an issue since these compact the >> memory and therefore there is no fragmentation. >> >> Which GC algorithm are you using? >> >> On Thursday, October 1, 2020 at 9:43:59 PM UTC+3 Shevek wrote: >> >>> When I do new byte[N], I get OutOfMemoryError, despite that the VM >>> claims to have more than enough free space (according to MemoryMXBean, >>> Runtime.freeMemory, visualvm, etc). >>> >>> My working assumption is that while I have enough free memory, I don't >>> have enough contiguous free memory. Is there a solution to this? Will I >>> get better results from any of: >>> >>> * ByteBuffer.allocateDirect() - presumably yes, but has other issues >>> relating to overall memory usage on the system >>> * G1GC (or other GC which allocates (relocatable?) regions) - this is a >>> deep hole I haven't yet explored. >>> * Calling System.gc() before allocating a contiguous region [apparently >>> doesn't help]. >>> * Other? >>> >>> If we do follow a strategy using allocateDirect, will we end up with the >>> same fragmentation issue in the native heap, along with committed >>> off-heap memory which we can no longer effectively use, or is the >>> off-heap memory managed in some manner which avoids this problem? >>> >>> Thank you. >>> >>> S. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "mechanical-sympathy" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web, visit >> https://groups.google.com/d/msgid/mechanical-sympathy/99daf015-ae67-4fb7-948f-cfb4a2cc00d6n%40googlegroups.com >> <https://groups.google.com/d/msgid/mechanical-sympathy/99daf015-ae67-4fb7-948f-cfb4a2cc00d6n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web, visit https://groups.google.com/d/msgid/mechanical-sympathy/CAGuAWdAR-wDMhgiCx-ZaY2c1GBuqZKkYxd0ELJFqwgTY%3DGpjxA%40mail.gmail.com.
