On Thu, 5 Dec 2024 14:32:53 GMT, Thomas Stuefe <[email protected]> wrote:
>> Hi @stooke !
>>
>>> Hello, @tstuefe , and thanks for your comments. I'll address a few here
>>> while I work on the others. I have changed the os-specific names to
>>> lowercase, but I don't think it makes them stand out more. The square
>>> brackets were intended to do that. Might I change this back?
>>
>> Sure, if it looks worse. I just wanted to make sure we can cleanly
>> distinguish NMT sections from OS sections.
>>
>>>
>>> I think there is only one JAVAHEAP segment because due to an issue with my
>>> build[1] there was no CDS archive available.
>>
>> Has nothing to do with CDS. The heap consists of committed and reserved
>> areas. Committed areas have backing swap space allocated for them, and are
>> accessible. Reserved areas have not and are generally not. API wise the
>> difference is that Reserved sections set the MAP_NORESERVE flag for mmap,
>> and are generally allocated with PROT_NONE.
>>
>> So, the heap should show up with several neighboring sections, some
>> committed, some just reserved. Similar how most of the stacks should show up
>> with two entries, one for the writable stack, one for the guard page that is
>> protected.
>>
>> ---
>>
>>
>> Simple test I did on MacOS with your patch: I reserve 1G of memory at
>> startup, uncommitted (added to os::init_2)
>>
>>
>> if (UseNewCode) {
>> char* p = os::reserve_memory(G, false, mtInternal);
>> tty->print_cr("Pointer is %p", p);
>> }
>>
>>
>>
>> vmmap shows:
>>
>>
>> VM_ALLOCATE 10ccb4000-14ccb4000 [ 1.0G 0K 0K
>> 0K] ---/rwx SM=NUL
>>
>>
>> so, looks good. 1GB, with all protection flags cleared. But System.map shows
>> nothing for this address range.
>>
>>
>> Now, I commit the second half of the range:
>>
>>
>>
>> if (UseNewCode) {
>> char* p = os::reserve_memory(G, false, mtInternal);
>> tty->print_cr("Pointer is %p", p);
>> bool b = os::commit_memory(p + (512 * M), 512 * M, false);
>> assert(b,"???");
>> }
>>
>>
>> vmmap shows only the committed part now, omitting the still uncommitted
>> first half. But it gets the protection flags right again (rw now):
>>
>>
>> VM_ALLOCATE (reserved) 148000000-168000000 [512.0M 0K 0K
>> 0K] rw-/rwx SM=NUL reserved VM address space (unallocated)
>>
>>
>> System.map shows nothing.
>>
>>
>> What goes on? Is the OS lying to us? Do we have an error? Both vmmap and
>> System.map seem to struggle, with vmmap being somewhat more correct.
>
>> @tstuefe I've look into your test, and I will modify the PR to display these
>> regions - it was incorrectly identifying them as "free". As to the strange
>> vmmap behaviour, I found that the two sections appeared in different places:
>> the uncommitted spaces appeared in "==== Non-writable regions for process":
>> `VM_ALLOCATE 300000000-320000000 [512.0M 0K 0K 0K] ---/rwx SM=NUL ` and the
>> committed spaces in "==== Writable regions for process": `VM_ALLOCATE
>> (reserved) 320000000-340000000 [512.0M 0K 0K 0K] rw-/rwx SM=NUL reserved VM
>> address space (unallocated) ` I have made a few changes, track reserved and
>> committed memory better, and uploaded an updated sample output.
>> [vm_memory_map_89174.txt](https://github.com/user-attachments/files/18013640/vm_memory_map_89174.txt)
>
> Yes, this is better.
>
> Metaspace sections look like this:
>
>
> 0x000130000000-0x000130010000 65536 rw-/rwx pvt 0
> META
> 0x000130010000-0x000130020000 65536 rw-/rwx pvt 0
> META
> 0x000130020000-0x000130400000 4063232 ---/rwx --- 0x20000
> META
> 0x000130400000-0x000130410000 65536 rw-/rwx pvt 0
> META
> 0x000130410000-0x000134000000 62849024 ---/rwx --- 0x410000
> META
>
>
> A single 64MB space node. First three entries together are the initial 4MB
> chunk the boot class loader uses. Forth line, together with some space from
> the fifth line will belong to the next chunk of the next class loader.
>
> Class space is still a bit weird:
>
>
> 0x018001000000-0x018001010000 65536 rw-/rwx pvt 0
> CLASS
> 0x018001010000-0x018001040000 196608 ---/rwx --- 0x1010000
> CLASS
> 0x018001040000-0x018001050000 65536 rw-/rwx pvt 0
> CLASS
> 0x018001050000-0x018008000000 117112832 ---/rwx --- 0x1050000
> CLASS
> 0x018008000000-0x018010000000 134217728 ---/rwx --- 0
> CLASS
> 0x018010000000-0x018018000000 134217728 ---/rwx --- 0
> CLASS
> 0x018018000000-0x018020000000 134217728 ---/rwx --- 0
> CLASS
> 0x018020000000-0x018028000000 134217728 ---/rwx --- 0
> CLASS
> 0x018028000000-0x018030000000 134217728 ---/rwx --- 0
> CLASS
> 0x018030000000-0x018038000000 134217728 ---/rwx --- 0
> CLASS
> 0x018038000000-0x018040000000 134217728 ---/rwx --- 0
> CLASS
> 0x018040000000-0x018041000000 16777216 ---/rwx --- ...
@tstuefe I ran an experiment with raw mmap, and there's no way to differentiate
between one large allocation of 5*128MB and 5 smaller allocations of 128MB. I
_could_ add code to fold these, but we risk loosing information.
**I have found that the combination of mach_make_memory_entry_64() +
mach_vm_map() (instead of using mmap(), which is just a compatibility layer on
top of mach system calls) causes the kernel to treat the allocation as one
region, and report it as such! This may be a topic of a further PR**
-------------
PR Comment: https://git.openjdk.org/jdk/pull/20953#issuecomment-2520662304
