Hi Eric, Thanks for the quick response and sorry for the delay on getting back on this issue.
> From: Eric B Munson [[email protected]] On Behalf Of Eric B Munson > [[email protected]] > Sent: Friday, January 22, 2010 9:02 AM > To: Gilmore, Doug > Cc: '[email protected]' > Subject: Re: [Libhugetlbfs-devel] problem that can occur when mapping a large > data + BSS section > > Doug, > > MAP_PRIVATE is used because we want to make sure that changes > made to the data in the mapping are not readable by other processes. > MAP_SHARED was used previously because older kernels did not reserve > MAP_PRIVATE mappings properly, now that this has been fixed in kernel, > we prefer to use MAP_PRIVATE. I was curious how the data could be read by other processes. Isn't the hugetlbfs file unlinked when the file is created? > Try this same test using the new > relinking method described in the HOWTO and let me know if that works. > > Eric Here is the typescript of the link and failure using method from the HOWTO file: $ gcc smain.c sl.so -B /home/dgilmore/libhugetlbfs-2.7/usr/local/share/libhugetlbfs -Wl,--hugetlbfs-align $ HUGETLB_ELFMAP=RW HUGETLB_VERBOSE=99 LD_LIBRARY_PATH=.:/home/dgilmore/libhugetlbfs-2.7/usr/local/lib64 ./a.out libhugetlbfs [pcetilapia04:26624]: INFO: Parsed kernel version: [2] . [6] . [27] [post-release: 39] libhugetlbfs [pcetilapia04:26624]: INFO: Feature private_reservations is present in this kernel libhugetlbfs [pcetilapia04:26624]: INFO: Kernel has MAP_PRIVATE reservations. Disabling heap prefaulting. libhugetlbfs [pcetilapia04:26624]: INFO: HUGETLB_SHARE=0, sharing disabled libhugetlbfs [pcetilapia04:26624]: INFO: Segment 0 (phdr 2): 0x400000-0x4008dc (filesz=0x8dc) (prot = 0x5) libhugetlbfs [pcetilapia04:26624]: INFO: Segment 1 (phdr 3): 0xbffdf8-0x26400060 (filesz=0x230) (prot = 0x3) libhugetlbfs [pcetilapia04:26624]: INFO: libhugetlbfs version: 2.7 (modified) libhugetlbfs [pcetilapia04:26625]: INFO: Mapped hugeseg at 0x2aaaaac00000. Copying 0x8dc bytes and 0 extra bytes from 0x400000...done libhugetlbfs [pcetilapia04:26624]: INFO: Prepare succeeded libhugetlbfs [pcetilapia04:26626]: INFO: Mapped hugeseg at 0x2aaaaac00000. Copying 0x230 bytes and 0x25800038 extra bytes from 0xbffdf8...done libhugetlbfs [pcetilapia04:26624]: INFO: Prepare succeeded Failed to map hugepage segment 1: a00000-26600000 (errno=12) Aborted Doug > > On Thu, 21 Jan 2010, Gilmore, Doug wrote: > > > While testing applications here at AMD we discovered that > > one of programs that was linked with BDT hugepage mappings > > failed to load on SLES11 but not on SLES10. The issue was > > caused by having symbols in the dynamic symbol table that > > are associated with very large addresses. This causes > > prepare_segment() in elflink.c to create a very large > > libhugetlbfs file in which the data + BSS segments are > > copied, which can cause the mapping of the associated > > hugetlbfs file to the original address to fail since the > > flags argument to mmap() contains MAP_PRIVATE instead of > > MAP_SHARED. (Newer kernels reserve more huge pages to > > ensure that there will be pages available for any COW > > operations.) > > > > The following a type script of a simple test that > > demonstrates the problem: > > > > SLES10: > > $ cat /etc/issue > > > > Welcome to SUSE Linux Enterprise Server 10 SP2 (x86_64) - Kernel \r (\l). > > > > > > $ cat sl.c > > extern int v[]; > > > > int *addr_of_v() > > { > > return v; > > } > > $ cat smain.c > > #include <stdio.h> > > > > extern int *addr_of_v(void); > > > > #define NN 600 > > int v[NN*1024*1024/sizeof(int)]; > > > > int main() > > { > > printf("addr_of_v returned %p\n", addr_of_v()); > > return 0; > > } > > $ gcc -shared -fPIC sl.c -o sl.so > > $ gcc smain.c sl.so > > -Wl,--script=/home/dgilmore/libhugetlbfs-2.7/local/usr/local/share/libhugetlbfs/ldscripts/elf_x86_64.xBDT > > $ HUGETLB_VERBOSE=99 > > LD_LIBRARY_PATH=.:/home/dgilmore/libhugetlbfs-2.7/local/usr/local/lib64 > > ./a.out > > libhugetlbfs [pcetilapia03:16539]: INFO: Parsed kernel version: [2] . [6] . > > [16] [post-release: 60] > > libhugetlbfs [pcetilapia03:16539]: INFO: HUGETLB_SHARE=0, sharing disabled > > libhugetlbfs [pcetilapia03:16539]: INFO: Segment 0 (phdr 2): > > 0x1000000-0x100081c (filesz=0x81c) (prot = 0x5) > > libhugetlbfs [pcetilapia03:16539]: INFO: Segment 1 (phdr 3): > > 0x2000000-0x27800260 (filesz=0x228) (prot = 0x7) > > libhugetlbfs [pcetilapia03:16539]: INFO: libhugetlbfs version: 2.7 > > libhugetlbfs [pcetilapia03:16540]: INFO: Mapped hugeseg at 0x2aaaaac00000. > > Copying 0x81c bytes and 0 extra bytes from 0x1000000...done > > libhugetlbfs [pcetilapia03:16539]: INFO: Prepare succeeded > > libhugetlbfs [pcetilapia03:16541]: INFO: Mapped hugeseg at 0x2aaaaac00000. > > Copying 0x228 bytes and 0x25800038 extra bytes from 0x2000000...done > > libhugetlbfs [pcetilapia03:16539]: INFO: Prepare succeeded > > addr_of_v returned 0x2000260 > > $ cat /proc/sys/vm/nr_hugepages > > 450 > > $ > > > > SLES11: > > $ cat /etc/issue > > > > Welcome to SUSE Linux Enterprise Server 11 (x86_64) - Kernel \r (\l). > > > > > > $ HUGETLB_VERBOSE=99 > > LD_LIBRARY_PATH=.:/home/dgilmore/libhugetlbfs-2.7/local/usr/local/lib64 > > ./a.out > > libhugetlbfs [pcetilapia04:22891]: INFO: Parsed kernel version: [2] . [6] . > > [27] [post-release: 39] > > libhugetlbfs [pcetilapia04:22891]: INFO: Feature private_reservations is > > present in this kernel > > libhugetlbfs [pcetilapia04:22891]: INFO: Kernel has MAP_PRIVATE > > reservations. Disabling heap prefaulting. > > libhugetlbfs [pcetilapia04:22891]: INFO: HUGETLB_SHARE=0, sharing disabled > > libhugetlbfs [pcetilapia04:22891]: INFO: Segment 0 (phdr 2): > > 0x1000000-0x100081c (filesz=0x81c) (prot = 0x5) > > libhugetlbfs [pcetilapia04:22891]: INFO: Segment 1 (phdr 3): > > 0x2000000-0x27800260 (filesz=0x228) (prot = 0x7) > > libhugetlbfs [pcetilapia04:22891]: INFO: libhugetlbfs version: 2.7 > > libhugetlbfs [pcetilapia04:22895]: INFO: Mapped hugeseg at 0x2aaaaac00000. > > Copying 0x81c bytes and 0 extra bytes from 0x1000000...done > > libhugetlbfs [pcetilapia04:22891]: INFO: Prepare succeeded > > libhugetlbfs [pcetilapia04:22896]: INFO: Mapped hugeseg at 0x2aaaaac00000. > > Copying 0x228 bytes and 0x25800038 extra bytes from 0x2000000...done > > libhugetlbfs [pcetilapia04:22891]: INFO: Prepare succeeded > > Failed to map hugepage segment 1: 2000000-27a00000 (errno=12) > > Aborted > > $ cat /proc/sys/vm/nr_hugepages > > 450 > > $ > > > > However on SLES11, if I rebuild libhugetlbfs with the following patch, the > > mapping will succeed. > > > > $ diff -u /home/dgilmore/libhugetlbfs-2.7/elflink.c{.1,} > > --- /home/dgilmore/libhugetlbfs-2.7/elflink.c.1 2009-12-20 > > 01:40:19.000000000 -0800 > > +++ /home/dgilmore/libhugetlbfs-2.7/elflink.c 2010-01-21 15:06:50.124600000 > > -0800 > > @@ -1120,7 +1120,7 @@ > > start = ALIGN_DOWN((unsigned long)seg[i].vaddr, hpage_size); > > offset = (unsigned long)(seg[i].vaddr - start); > > mapsize = ALIGN(offset + seg[i].memsz, hpage_size); > > - mmap_flags = MAP_PRIVATE|MAP_FIXED; > > + mmap_flags = MAP_SHARED|MAP_FIXED; > > > > /* > > * If this is a read-only mapping whose contents are > > $ HUGETLB_VERBOSE=99 > > LD_LIBRARY_PATH=.:/home/dgilmore/libhugetlbfs-2.7/usr/local/lib64 ./a.out > > libhugetlbfs [pcetilapia04:23286]: INFO: Parsed kernel version: [2] . [6] . > > [27] [post-release: 39] > > libhugetlbfs [pcetilapia04:23286]: INFO: Feature private_reservations is > > present in this kernel > > libhugetlbfs [pcetilapia04:23286]: INFO: Kernel has MAP_PRIVATE > > reservations. Disabling heap prefaulting. > > libhugetlbfs [pcetilapia04:23286]: INFO: HUGETLB_SHARE=0, sharing disabled > > libhugetlbfs [pcetilapia04:23286]: INFO: Segment 0 (phdr 2): > > 0x1000000-0x10008b4 (filesz=0x8b4) (prot = 0x5) > > libhugetlbfs [pcetilapia04:23286]: INFO: Segment 1 (phdr 3): > > 0x2000000-0x27800260 (filesz=0x230) (prot = 0x7) > > libhugetlbfs [pcetilapia04:23286]: INFO: libhugetlbfs version: 2.7 > > (modified) > > libhugetlbfs [pcetilapia04:23287]: INFO: Mapped hugeseg at 0x2aaaaac00000. > > Copying 0x8b4 bytes and 0 extra bytes from 0x1000000...done > > libhugetlbfs [pcetilapia04:23286]: INFO: Prepare succeeded > > libhugetlbfs [pcetilapia04:23288]: INFO: Mapped hugeseg at 0x2aaaaac00000. > > Copying 0x230 bytes and 0x25800030 extra bytes from 0x2000000...done > > libhugetlbfs [pcetilapia04:23286]: INFO: Prepare succeeded > > addr_of_v returned 0x2000260 > > $ > > > > So far my testing hasn't uncovered issues with this change. Does > > anyone know why the final mmap() calls use use MAP_PRIVATE instead of > > MAP_SHARED? > > > > Doug > > > ... > > -- > Eric B Munson > IBM Linux Technology Center > [email protected] > ------------------------------------------------------------------------------ The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com _______________________________________________ Libhugetlbfs-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/libhugetlbfs-devel
