Doug, MAP_PRIVATE is used because we want to make sure that changes made to the data in the mapping are not readable by other processes. MAP_SHARED was used previously because older kernels did not reserve MAP_PRIVATE mappings properly, now that this has been fixed in kernel, we prefer to use MAP_PRIVATE. Try this same test using the new relinking method described in the HOWTO and let me know if that works.
Eric
On Thu, 21 Jan 2010, Gilmore, Doug wrote:
> While testing applications here at AMD we discovered that
> one of programs that was linked with BDT hugepage mappings
> failed to load on SLES11 but not on SLES10. The issue was
> caused by having symbols in the dynamic symbol table that
> are associated with very large addresses. This causes
> prepare_segment() in elflink.c to create a very large
> libhugetlbfs file in which the data + BSS segments are
> copied, which can cause the mapping of the associated
> hugetlbfs file to the original address to fail since the
> flags argument to mmap() contains MAP_PRIVATE instead of
> MAP_SHARED. (Newer kernels reserve more huge pages to
> ensure that there will be pages available for any COW
> operations.)
>
> The following a type script of a simple test that
> demonstrates the problem:
>
> SLES10:
> $ cat /etc/issue
>
> Welcome to SUSE Linux Enterprise Server 10 SP2 (x86_64) - Kernel \r (\l).
>
>
> $ cat sl.c
> extern int v[];
>
> int *addr_of_v()
> {
> return v;
> }
> $ cat smain.c
> #include <stdio.h>
>
> extern int *addr_of_v(void);
>
> #define NN 600
> int v[NN*1024*1024/sizeof(int)];
>
> int main()
> {
> printf("addr_of_v returned %p\n", addr_of_v());
> return 0;
> }
> $ gcc -shared -fPIC sl.c -o sl.so
> $ gcc smain.c sl.so
> -Wl,--script=/home/dgilmore/libhugetlbfs-2.7/local/usr/local/share/libhugetlbfs/ldscripts/elf_x86_64.xBDT
> $ HUGETLB_VERBOSE=99
> LD_LIBRARY_PATH=.:/home/dgilmore/libhugetlbfs-2.7/local/usr/local/lib64
> ./a.out
> libhugetlbfs [pcetilapia03:16539]: INFO: Parsed kernel version: [2] . [6] .
> [16] [post-release: 60]
> libhugetlbfs [pcetilapia03:16539]: INFO: HUGETLB_SHARE=0, sharing disabled
> libhugetlbfs [pcetilapia03:16539]: INFO: Segment 0 (phdr 2):
> 0x1000000-0x100081c (filesz=0x81c) (prot = 0x5)
> libhugetlbfs [pcetilapia03:16539]: INFO: Segment 1 (phdr 3):
> 0x2000000-0x27800260 (filesz=0x228) (prot = 0x7)
> libhugetlbfs [pcetilapia03:16539]: INFO: libhugetlbfs version: 2.7
> libhugetlbfs [pcetilapia03:16540]: INFO: Mapped hugeseg at 0x2aaaaac00000.
> Copying 0x81c bytes and 0 extra bytes from 0x1000000...done
> libhugetlbfs [pcetilapia03:16539]: INFO: Prepare succeeded
> libhugetlbfs [pcetilapia03:16541]: INFO: Mapped hugeseg at 0x2aaaaac00000.
> Copying 0x228 bytes and 0x25800038 extra bytes from 0x2000000...done
> libhugetlbfs [pcetilapia03:16539]: INFO: Prepare succeeded
> addr_of_v returned 0x2000260
> $ cat /proc/sys/vm/nr_hugepages
> 450
> $
>
> SLES11:
> $ cat /etc/issue
>
> Welcome to SUSE Linux Enterprise Server 11 (x86_64) - Kernel \r (\l).
>
>
> $ HUGETLB_VERBOSE=99
> LD_LIBRARY_PATH=.:/home/dgilmore/libhugetlbfs-2.7/local/usr/local/lib64
> ./a.out
> libhugetlbfs [pcetilapia04:22891]: INFO: Parsed kernel version: [2] . [6] .
> [27] [post-release: 39]
> libhugetlbfs [pcetilapia04:22891]: INFO: Feature private_reservations is
> present in this kernel
> libhugetlbfs [pcetilapia04:22891]: INFO: Kernel has MAP_PRIVATE reservations.
> Disabling heap prefaulting.
> libhugetlbfs [pcetilapia04:22891]: INFO: HUGETLB_SHARE=0, sharing disabled
> libhugetlbfs [pcetilapia04:22891]: INFO: Segment 0 (phdr 2):
> 0x1000000-0x100081c (filesz=0x81c) (prot = 0x5)
> libhugetlbfs [pcetilapia04:22891]: INFO: Segment 1 (phdr 3):
> 0x2000000-0x27800260 (filesz=0x228) (prot = 0x7)
> libhugetlbfs [pcetilapia04:22891]: INFO: libhugetlbfs version: 2.7
> libhugetlbfs [pcetilapia04:22895]: INFO: Mapped hugeseg at 0x2aaaaac00000.
> Copying 0x81c bytes and 0 extra bytes from 0x1000000...done
> libhugetlbfs [pcetilapia04:22891]: INFO: Prepare succeeded
> libhugetlbfs [pcetilapia04:22896]: INFO: Mapped hugeseg at 0x2aaaaac00000.
> Copying 0x228 bytes and 0x25800038 extra bytes from 0x2000000...done
> libhugetlbfs [pcetilapia04:22891]: INFO: Prepare succeeded
> Failed to map hugepage segment 1: 2000000-27a00000 (errno=12)
> Aborted
> $ cat /proc/sys/vm/nr_hugepages
> 450
> $
>
> However on SLES11, if I rebuild libhugetlbfs with the following patch, the
> mapping will succeed.
>
> $ diff -u /home/dgilmore/libhugetlbfs-2.7/elflink.c{.1,}
> --- /home/dgilmore/libhugetlbfs-2.7/elflink.c.1 2009-12-20
> 01:40:19.000000000 -0800
> +++ /home/dgilmore/libhugetlbfs-2.7/elflink.c 2010-01-21 15:06:50.124600000
> -0800
> @@ -1120,7 +1120,7 @@
> start = ALIGN_DOWN((unsigned long)seg[i].vaddr, hpage_size);
> offset = (unsigned long)(seg[i].vaddr - start);
> mapsize = ALIGN(offset + seg[i].memsz, hpage_size);
> - mmap_flags = MAP_PRIVATE|MAP_FIXED;
> + mmap_flags = MAP_SHARED|MAP_FIXED;
>
> /*
> * If this is a read-only mapping whose contents are
> $ HUGETLB_VERBOSE=99
> LD_LIBRARY_PATH=.:/home/dgilmore/libhugetlbfs-2.7/usr/local/lib64 ./a.out
> libhugetlbfs [pcetilapia04:23286]: INFO: Parsed kernel version: [2] . [6] .
> [27] [post-release: 39]
> libhugetlbfs [pcetilapia04:23286]: INFO: Feature private_reservations is
> present in this kernel
> libhugetlbfs [pcetilapia04:23286]: INFO: Kernel has MAP_PRIVATE reservations.
> Disabling heap prefaulting.
> libhugetlbfs [pcetilapia04:23286]: INFO: HUGETLB_SHARE=0, sharing disabled
> libhugetlbfs [pcetilapia04:23286]: INFO: Segment 0 (phdr 2):
> 0x1000000-0x10008b4 (filesz=0x8b4) (prot = 0x5)
> libhugetlbfs [pcetilapia04:23286]: INFO: Segment 1 (phdr 3):
> 0x2000000-0x27800260 (filesz=0x230) (prot = 0x7)
> libhugetlbfs [pcetilapia04:23286]: INFO: libhugetlbfs version: 2.7 (modified)
> libhugetlbfs [pcetilapia04:23287]: INFO: Mapped hugeseg at 0x2aaaaac00000.
> Copying 0x8b4 bytes and 0 extra bytes from 0x1000000...done
> libhugetlbfs [pcetilapia04:23286]: INFO: Prepare succeeded
> libhugetlbfs [pcetilapia04:23288]: INFO: Mapped hugeseg at 0x2aaaaac00000.
> Copying 0x230 bytes and 0x25800030 extra bytes from 0x2000000...done
> libhugetlbfs [pcetilapia04:23286]: INFO: Prepare succeeded
> addr_of_v returned 0x2000260
> $
>
> So far my testing hasn't uncovered issues with this change. Does
> anyone know why the final mmap() calls use use MAP_PRIVATE instead of
> MAP_SHARED?
>
> Doug
>
> ------------------------------------------------------------------------------
> Throughout its 18-year history, RSA Conference consistently attracts the
> world's best and brightest in the field, creating opportunities for Conference
> attendees to learn about information security's most important issues through
> interactions with peers, luminaries and emerging and established companies.
> http://p.sf.net/sfu/rsaconf-dev2dev
> _______________________________________________
> Libhugetlbfs-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/libhugetlbfs-devel
>
--
Eric B Munson
IBM Linux Technology Center
[email protected]
signature.asc
Description: Digital signature
------------------------------------------------------------------------------ Throughout its 18-year history, RSA Conference consistently attracts the world's best and brightest in the field, creating opportunities for Conference attendees to learn about information security's most important issues through interactions with peers, luminaries and emerging and established companies. http://p.sf.net/sfu/rsaconf-dev2dev
_______________________________________________ Libhugetlbfs-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/libhugetlbfs-devel
