More info seems like relinking is kinda a must when trying to build
with large page support for the code segments.
So I've attempted to relink the big uber application but its still
segfaulting when trying to load the code into large pages.
I got a core dump with a backtrace in libhugetlbfs initialization somewhere.
(gdb) bt
#0 0x0000000002e88496 in munmap ()
#1 0x00002ac20494b30a in remap_segments (seg=0x2ac204b555a0, num=2)
at elflink.c:1111
#2 0x00002ac20494bd0d in __lh_hugetlbfs_setup_elflink () at elflink.c:1313
#3 0x00002ac20494e76e in setup_libhugetlbfs () at init.c:30
#4 0x00002ac2049512c6 in __do_global_ctors_aux ()
from /usr/lib64/libhugetlbfs.so
#5 0x00002ac204948adb in _init () from /usr/lib64/libhugetlbfs.so
#6 0x00002ac204825988 in ?? ()
#7 0x00000033adc0d22b in call_init () from /lib64/ld-linux-x86-64.so.2
#8 0x00000033adc0d335 in _dl_init_internal () from /lib64/ld-linux-x86-64.so.2
#9 0x00000033adc00aaa in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#10 0x0000000000000002 in ?? ()
#11 0x00007fffaa51a6e1 in ?? ()
#12 0x00007fffaa51a709 in ?? ()
#13 0x0000000000000000 in ?? ()
(gdb)
(gdb) up
#1 0x00002ac20494b30a in remap_segments (seg=0x2ac204b555a0, num=2)
at elflink.c:1111
1111 munmap((void *) start, mapsize);
(gdb) print seg[0]
$1 = {vaddr = 0x400000, filesz = 51650212, memsz = 51650212, extrasz = 0,
prot = 5, fd = 4, index = 2, page_size = 2097152}
(gdb) print seg[1]
$2 = {vaddr = 0x3741ea8, filesz = 14561904, memsz = 99200296,
extrasz = 84638392, prot = 3, fd = 3, index = 3, page_size = 2097152}
(gdb) print seg[2]
$3 = {vaddr = 0x0, filesz = 0, memsz = 0, extrasz = 0, prot = 0, fd = 0,
index = 0, page_size = 0}
(gdb)
Why would the call to munmap segfault? did the function get munmaped?
Thanks,
- David Brown
On Thu, Jun 25, 2009 at 12:10 PM, David Brown<dmlb2...@gmail.com> wrote:
> Okay nm the first part, it does actually allocate large pages and use
> them in the code segment now, didn't relink the binary I was using.
> Output now looks like
>
> $ hugectl --text --bss --data --heap --shm --share-text -v -v -v
> ./test 100000000
> hugectl: INFO: HUGETLB_VERBOSE='5'
> hugectl: INFO: LD_LIBRARY_PATH='/usr/lib32:/usr/lib64:'
> hugectl: INFO: HUGETLB_ELFMAP='R:W'
> hugectl: INFO: HUGETLB_MORECORE='yes'
> hugectl: INFO: HUGETLB_SHM='yes'
> hugectl: WARNING: LD_PRELOAD not appropriate for this map combination
> hugectl: INFO: HUGETLB_SHARE='1'
> libhugetlbfs [dmlb2000ts:31341]: INFO: Parsed kernel version: [2] . [6] . [30]
> libhugetlbfs [dmlb2000ts:31341]: INFO: Feature private_reservations is
> present in this kernel
> libhugetlbfs [dmlb2000ts:31341]: INFO: Kernel has MAP_PRIVATE
> reservations. Disabling heap prefaulting.
> libhugetlbfs [dmlb2000ts:31341]: INFO: HUGETLB_SHARE=1, sharing
> enabled for only read-only segments
> libhugetlbfs [dmlb2000ts:31341]: INFO: Segment 0 (phdr 2):
> 0x400000-0x4008fc (filesz=0x8fc) (prot = 0x5)
> libhugetlbfs [dmlb2000ts:31341]: INFO: Segment 1 (phdr 3):
> 0x800900-0x80800b60 (filesz=0x228) (prot = 0x3)
> libhugetlbfs [dmlb2000ts:31341]: INFO: libhugetlbfs version: 2.4
> libhugetlbfs [dmlb2000ts:31342]: WARNING: Unable to verify address
> range 0x800000 - 0x800000. Not empty?
> libhugetlbfs [dmlb2000ts:31342]: INFO: Mapped hugeseg at
> 0x2aaaaac00000. Copying 0x228 bytes and 0 extra bytes from
> 0x800900...done
> libhugetlbfs [dmlb2000ts:31341]: INFO: Prepare succeeded
> libhugetlbfs [dmlb2000ts:31341]: INFO: setup_morecore(): heapaddr = 0x82000000
> 0
>
> However, I try this same binary on an older kernel and it doesn't look the
> same.
>
> $ hugectl -v -v -v --text --bss --heap --shm --data ./test 100000000
> hugectl: INFO: HUGETLB_VERBOSE='5'
> hugectl: INFO:
> LD_LIBRARY_PATH='/usr/local/lib:/usr/local/lib64:/usr/mpi/gcc/mvapich2-1.2p1/lib:'
> hugectl: INFO: HUGETLB_ELFMAP='R:W'
> hugectl: INFO: HUGETLB_MORECORE='yes'
> hugectl: INFO: HUGETLB_SHM='yes'
> hugectl: WARNING: LD_PRELOAD not appropriate for this map combination
> libhugetlbfs [cut1n10:24262]: INFO: Parsed kernel version: [2] . [6] . [18]
> libhugetlbfs [cut1n10:24262]: INFO: HUGETLB_SHARE=0, sharing disabled
> libhugetlbfs [cut1n10:24262]: INFO: Segment 0 (phdr 2):
> 0x400000-0x4008bc (filesz=0x8bc) (prot = 0x5)
> libhugetlbfs [cut1n10:24262]: INFO: Segment 1 (phdr 3):
> 0x5008c0-0x80500af0 (filesz=0x220) (prot = 0x3)
> libhugetlbfs [cut1n10:24262]: WARNING: Layout problem with segments 0 and 1:
> Segments would overlap
> libhugetlbfs [cut1n10:24262]: INFO: No segments were appropriate for remapping
> libhugetlbfs [cut1n10:24262]: INFO: setup_morecore(): heapaddr = 0x98000000
> 0
>
> When was large page support for code segments supported?
>
> Thanks,
> - David Brown
>
> On Thu, Jun 25, 2009 at 10:31 AM, David Brown<dmlb2...@gmail.com> wrote:
>> I fixed the memory issue and the application is running but its not
>> allocating large pages.
>>
>> $ hugectl --text --bss --data --heap --shm -v -v -v ./test 100000000
>> hugectl: INFO: HUGETLB_VERBOSE='5'
>> hugectl: INFO: LD_LIBRARY_PATH='/usr/lib32:/usr/lib64:'
>> hugectl: INFO: HUGETLB_ELFMAP='R:W'
>> hugectl: INFO: HUGETLB_MORECORE='yes'
>> hugectl: INFO: HUGETLB_SHM='yes'
>> hugectl: WARNING: LD_PRELOAD not appropriate for this map combination
>> libhugetlbfs [dmlb2000ts:15538]: INFO: Parsed kernel version: [2] . [6] .
>> [30]
>> libhugetlbfs [dmlb2000ts:15538]: INFO: Feature private_reservations is
>> present in this kernel
>> libhugetlbfs [dmlb2000ts:15538]: INFO: HUGETLB_SHARE=0, sharing disabled
>> libhugetlbfs [dmlb2000ts:15538]: INFO: Segment 0 (phdr 2):
>> 0x400000-0x4008bc (filesz=0x8bc) (prot = 0x5)
>> libhugetlbfs [dmlb2000ts:15538]: INFO: Segment 1 (phdr 3):
>> 0x5008c0-0x80500af0 (filesz=0x220) (prot = 0x3)
>> libhugetlbfs [dmlb2000ts:15538]: WARNING: Layout problem with segments 0 and
>> 1:
>> Segments would overlap
>> libhugetlbfs [dmlb2000ts:15538]: INFO: No segments were appropriate
>> for remapping
>> libhugetlbfs [dmlb2000ts:15538]: INFO: Kernel has MAP_PRIVATE
>> reservations. Disabling heap prefaulting.
>> libhugetlbfs [dmlb2000ts:15538]: INFO: setup_morecore(): heapaddr =
>> 0x80e00000
>> 0
>>
>>
>> So I'm guessing make check is supposed to be run as root? since there
>> seems to be mount commands and echoing stuff out to proc that is
>> performed.
>>
>> I'd be nice if if you could just use some of the same checks in
>> libhugetlbfs.so to find a mount and see if you can mmap in it then run
>> the tests if you aren't root.
>>
>> ( as me )
>> ********** TEST SUMMARY
>> * 2M
>> * 32-bit 64-bit
>> * Total testcases: 0 98
>> * Skipped: 0 0
>> * PASS: 0 68
>> * FAIL: 0 5
>> * Killed by signal: 0 0
>> * Bad configuration: 0 11
>> * Expected FAIL: 0 0
>> * Unexpected PASS: 0 0
>> * Strange test result: 0 14
>> **********
>>
>> ( as root )
>> ********** TEST SUMMARY
>> * 2M
>> * 32-bit 64-bit
>> * Total testcases: 0 98
>> * Skipped: 0 0
>> * PASS: 0 81
>> * FAIL: 0 0
>> * Killed by signal: 0 0
>> * Bad configuration: 0 3
>> * Expected FAIL: 0 0
>> * Unexpected PASS: 0 0
>> * Strange test result: 0 14
>> **********
>>
>> Hmmm, not sure what tests were what do you want the full output?
>>
>> Thanks,
>> - David Brown
>>
>> On Wed, Jun 24, 2009 at 9:09 PM, David
>> Gibson<da...@gibson.dropbear.id.au> wrote:
>>> On Thu, Jun 25, 2009 at 03:50:14AM +0000, David Brown wrote:
>>>> I've been playing around with large pages and libhugetlbfs and have
>>>> been having no such luck getting my code segment to be backed by large
>>>> pages.
>>>>
>>>> I have a small example code that doesn't seem to be loading.
>>>
>>> First, does the libhugetlbfs pass ("make check").
>>>
>>> [snip]
>>>> $ hugectl -v -v --text --bss --heap --shm --data ./test 10000000
>>>> hugectl: INFO: HUGETLB_VERBOSE='4'
>>>> hugectl: INFO: LD_LIBRARY_PATH='/usr/local/lib:/usr/local/lib64:'
>>>> hugectl: INFO: HUGETLB_ELFMAP='R:W'
>>>> hugectl: INFO: HUGETLB_MORECORE='yes'
>>>> hugectl: INFO: HUGETLB_SHM='yes'
>>>> hugectl: WARNING: LD_PRELOAD not appropriate for this map combination
>>>> libhugetlbfs [uber-gamer:31047]: INFO: Parsed kernel version: [2] . [6] .
>>>> [30]
>>>> libhugetlbfs [uber-gamer:31047]: INFO: Feature private_reservations is
>>>> present in this kernel
>>>> libhugetlbfs [uber-gamer:31047]: INFO: Kernel has MAP_PRIVATE
>>>> reservations. Disabling heap prefaulting.
>>>> libhugetlbfs [uber-gamer:31047]: INFO: HUGETLB_SHARE=0, sharing disabled
>>>> libhugetlbfs [uber-gamer:31047]: INFO: Segment 0 (phdr 2):
>>>> 0x400000-0x40089c (filesz=0x89c) (prot = 0x5)
>>>> libhugetlbfs [uber-gamer:31047]: INFO: Segment 1 (phdr 3):
>>>> 0x8008a0-0x40800ae0 (filesz=0x220) (prot = 0x3)
>>>> libhugetlbfs [uber-gamer:31047]: INFO: libhugetlbfs version: 2.4
>>>> libhugetlbfs [uber-gamer:31048]: INFO: Mapped hugeseg at
>>>> 0x2aaaaac00000. Copying 0x89c bytes and 0 extra bytes from
>>>> 0x400000...done
>>>> libhugetlbfs [uber-gamer:31047]: INFO: Prepare succeeded
>>>> libhugetlbfs [uber-gamer:31049]: WARNING: Unable to verify address
>>>> range 0x800000 - 0x800000. Not empty?
>>>> libhugetlbfs [uber-gamer:31049]: INFO: Mapped hugeseg at
>>>> 0x2aaaaac00000. Copying 0x220 bytes and 0 extra bytes from
>>>> 0x8008a0...done
>>>> libhugetlbfs [uber-gamer:31047]: INFO: Prepare succeeded
>>>> Failed to map hugepage segment 1: 800000-40a00000 (errno=12)
>>>> Aborted
>>>
>>> errno 12 is ENOMEM. This suggests you don't have enough hugepages
>>> allocated to back the program's segments.
>>>
>>> --
>>> David Gibson | I'll have my music baroque, and my code
>>> david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
>>> | _way_ _around_!
>>> http://www.ozlabs.org/~dgibson
>>>
>>
>
------------------------------------------------------------------------------
_______________________________________________
Libhugetlbfs-devel mailing list
Libhugetlbfs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libhugetlbfs-devel
