On Mon, Jul 9, 2012 at 7:17 AM, Eric B Munson <emun...@mgebm.net> wrote:
> I can't help much with checking the actual hardware support, but we are
> currently seeing trouble with malloc backed
> by huge pages. At some point, glibc started ignoring the __morecore hook
> and I have not yet tracked down the how/why
> of it. For testing 1GB pages, I recommend you try one of the functions in
> libhugetlbfs that gives direct access to
> huge pages (see get_huge_pages()). You can check if it is using a 1GB
> page by reading /proc/meminfo while the program
> is active, alternatively there is a helper in our test suite that will
> read and parse /proc/self/smaps to determine if
> a given address is mapped to a huge page.
>
> Eric
>
Thanks for the suggestions Eric.
I can see the HugePages_Free in /proc/meminfo go down when the program
runs, and the count in
/sys/kernel/mm/hugepages/hugepages-1048576kB/free_hugepages too.
I'm not sure which test program you meant, but /proc/$PID/smaps looks
pretty self-explanatory.
My test program prints out the address it gets back from malloc, and
looking that up in smaps says there's a 1 gig page there:
130 09:43:11 seb@seb-X8DTU-6 /home/seb/tlb-plots>
> LD_PRELOAD=/usr/lib/libhugetlbfs.so HUGETLB_MORECORE=1g HUGETLB_VERBOSE=99
> HUGETLB_DEBUG=1 perf stat --append -r 1 -e dTLB-load-misses -e dTLB-loads
> ./random-memory-access --size=$((1024*1024*1024))
> --walking-read-shuffled=100000000000
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: Found pagesize 1048576 kB
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: Detected page sizes:
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: Size: 1048576 kB (default)
> Mount: /libhugetlbfs-1g
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: Parsed kernel version: [3] . [2] .
> [0]
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: Feature private_reservations is
> present in this kernel
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: Feature noreserve_safe is present
> in this kernel
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: Feature map_hugetlb is present in
> this kernel
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: Kernel has MAP_PRIVATE
> reservations. Disabling heap prefaulting.
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: Kernel supports MAP_HUGETLB
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: HUGETLB_SHARE=0, sharing disabled
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: HUGETLB_NO_RESERVE=no,
> reservations enabled
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: No segments were appropriate for
> remapping
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: setup_morecore(): heapaddr =
> 0x40000000
> libhugetlbfs [seb-X8DTU-6:31521]: INFO: Found pagesize 1048576 kB
> libhugetlbfs [seb-X8DTU-6:31521]: INFO: Detected page sizes:
> libhugetlbfs [seb-X8DTU-6:31521]: INFO: Size: 1048576 kB (default)
> Mount: /libhugetlbfs-1g
> libhugetlbfs [seb-X8DTU-6:31521]: INFO: Parsed kernel version: [3] . [2] .
> [0]
> libhugetlbfs [seb-X8DTU-6:31521]: INFO: Feature private_reservations is
> present in this kernel
> libhugetlbfs [seb-X8DTU-6:31521]: INFO: Feature noreserve_safe is present
> in this kernel
> libhugetlbfs [seb-X8DTU-6:31521]: INFO: Feature map_hugetlb is present in
> this kernel
> libhugetlbfs [seb-X8DTU-6:31521]: INFO: Kernel has MAP_PRIVATE
> reservations. Disabling heap prefaulting.
> libhugetlbfs [seb-X8DTU-6:31521]: INFO: Kernel supports MAP_HUGETLB
> libhugetlbfs [seb-X8DTU-6:31521]: INFO: HUGETLB_SHARE=0, sharing disabled
> libhugetlbfs [seb-X8DTU-6:31521]: INFO: HUGETLB_NO_RESERVE=no,
> reservations enabled
> libhugetlbfs [seb-X8DTU-6:31521]: INFO: No segments were appropriate for
> remapping
> libhugetlbfs [seb-X8DTU-6:31521]: INFO: setup_morecore(): heapaddr =
> 0x40000000
> libhugetlbfs [seb-X8DTU-6:31522]: INFO: Found pagesize 1048576 kB
> libhugetlbfs [seb-X8DTU-6:31522]: INFO: Detected page sizes:
> libhugetlbfs [seb-X8DTU-6:31522]: INFO: Size: 1048576 kB (default)
> Mount: /libhugetlbfs-1g
> libhugetlbfs [seb-X8DTU-6:31522]: INFO: Parsed kernel version: [3] . [2] .
> [0]
> libhugetlbfs [seb-X8DTU-6:31522]: INFO: Feature private_reservations is
> present in this kernel
> libhugetlbfs [seb-X8DTU-6:31522]: INFO: Feature noreserve_safe is present
> in this kernel
> libhugetlbfs [seb-X8DTU-6:31522]: INFO: Feature map_hugetlb is present in
> this kernel
> libhugetlbfs [seb-X8DTU-6:31522]: INFO: Kernel has MAP_PRIVATE
> reservations. Disabling heap prefaulting.
> libhugetlbfs [seb-X8DTU-6:31522]: INFO: Kernel supports MAP_HUGETLB
> libhugetlbfs [seb-X8DTU-6:31522]: INFO: HUGETLB_SHARE=0, sharing disabled
> libhugetlbfs [seb-X8DTU-6:31522]: INFO: HUGETLB_NO_RESERVE=no,
> reservations enabled
> libhugetlbfs [seb-X8DTU-6:31522]: INFO: No segments were appropriate for
> remapping
> libhugetlbfs [seb-X8DTU-6:31522]: INFO: setup_morecore(): heapaddr =
> 0x40000000
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: Found pagesize 1048576 kB
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: Detected page sizes:
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: Size: 1048576 kB (default)
> Mount: /libhugetlbfs-1g
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: Parsed kernel version: [3] . [2] .
> [0]
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: Feature private_reservations is
> present in this kernel
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: Feature noreserve_safe is present
> in this kernel
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: Feature map_hugetlb is present in
> this kernel
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: Kernel has MAP_PRIVATE
> reservations. Disabling heap prefaulting.
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: Kernel supports MAP_HUGETLB
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: HUGETLB_SHARE=0, sharing disabled
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: HUGETLB_NO_RESERVE=no,
> reservations enabled
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: No segments were appropriate for
> remapping
> libhugetlbfs [seb-X8DTU-6:31520]: INFO: setup_morecore(): heapaddr =
> 0x40000000
> libhugetlbfs [seb-X8DTU-6:31523]: INFO: Found pagesize 1048576 kB
> libhugetlbfs [seb-X8DTU-6:31523]: INFO: Detected page sizes:
> libhugetlbfs [seb-X8DTU-6:31523]: INFO: Size: 1048576 kB (default)
> Mount: /libhugetlbfs-1g
> libhugetlbfs [seb-X8DTU-6:31523]: INFO: Parsed kernel version: [3] . [2] .
> [0]
> libhugetlbfs [seb-X8DTU-6:31523]: INFO: Feature private_reservations is
> present in this kernel
> libhugetlbfs [seb-X8DTU-6:31523]: INFO: Feature noreserve_safe is present
> in this kernel
> libhugetlbfs [seb-X8DTU-6:31523]: INFO: Feature map_hugetlb is present in
> this kernel
> libhugetlbfs [seb-X8DTU-6:31523]: INFO: Kernel has MAP_PRIVATE
> reservations. Disabling heap prefaulting.
> libhugetlbfs [seb-X8DTU-6:31523]: INFO: Kernel supports MAP_HUGETLB
> libhugetlbfs [seb-X8DTU-6:31523]: INFO: HUGETLB_SHARE=0, sharing disabled
> libhugetlbfs [seb-X8DTU-6:31523]: INFO: HUGETLB_NO_RESERVE=no,
> reservations enabled
> libhugetlbfs [seb-X8DTU-6:31523]: INFO: No segments were appropriate for
> remapping
> libhugetlbfs [seb-X8DTU-6:31523]: INFO: setup_morecore(): heapaddr =
> 0x40000000
> pid=31523
> allocating 1073741824 bytes (268435456 4-byte objects), aligned on a
> 1073741824-byte boundary
> libhugetlbfs [seb-X8DTU-6:31523]: INFO: hugetlbfs_morecore(2684223488) =
> ...
> libhugetlbfs [seb-X8DTU-6:31523]: INFO: heapbase = 0x40000000, heaptop =
> 0x40000000, mapsize = 0, delta=2684223488
> libhugetlbfs [seb-X8DTU-6:31523]: INFO: Attempting to map 3221225472 bytes
> libhugetlbfs [seb-X8DTU-6:31523]: INFO: ... = 0x40000000
> libhugetlbfs [seb-X8DTU-6:31523]: INFO: hugetlbfs_morecore(135168) = ...
> libhugetlbfs [seb-X8DTU-6:31523]: INFO: heapbase = 0x40000000, heaptop =
> 0xdffe0000, mapsize = c0000000, delta=-536866816
> libhugetlbfs [seb-X8DTU-6:31523]: INFO: ... = 0xdffe0000
> memblob_mother=0x40000010, alignment=0x0000000040000000,
> offset=0x0000000000000010, memblob=0x80000000
> test 'walking random memory read'
> doing prep
> starting test
> 4 wrap-arounds!
Here's the smaps entry for 0x40000010:
40000000-100000000 rw-p 00000000 00:0c 742560
> /anon_hugepage (deleted)
> Size: 3145728 kB
> Rss: 0 kB
> Pss: 0 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 0 kB
> Private_Clean: 0 kB
> Private_Dirty: 0 kB
> Referenced: 0 kB
> Anonymous: 0 kB
> AnonHugePages: 0 kB
> Swap: 0 kB
> KernelPageSize: 1048576 kB
> MMUPageSize: 1048576 kB
> Locked: 0 kB
When I don't use those funny environment variables to turn on 1 gig pages,
I see 4k pages in the smaps entry for the pointer I get from malloc().
I'll try get_huge_pages().
More later. Thanks for the help.
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Libhugetlbfs-devel mailing list
Libhugetlbfs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libhugetlbfs-devel
