On Mon, Feb 26, 2007 at 11:13:24AM -0800, Nishanth Aravamudan wrote:
> We've found that on Power some 32-bit binaries, when relinked via
> libhugetlbfs, run out of address space. Add some heuristics to warn the
> user if the memory size of the segments (plus the size of the largest
> segment) exceeds 80% of addressable memory.
Sorry, didn't really look at this closely before.
At what point exactly are we running out of address space? As we
actually remap the segments? Or later, when the program uses malloc()
or other mappings? Either way I assume it's the extra buffer space we
need between hugepage and normal page segments that's causing us to
run out of address space where we didn't before.
I'm just a little uneasy about heuristic thresholds like this, so I'd
like to understand the situation better to see if we can have a more
precise warning condition.
> diff --git a/elflink.c b/elflink.c
> index a53c649..bed8ef6 100644
> --- a/elflink.c
> +++ b/elflink.c
> @@ -55,6 +55,9 @@
> #define ELF_ST_TYPE(x) ELF64_ST_TYPE(x)
> #endif
>
> +/* 90% of 32-bit addressable memory */
> +#define MEMSZ_THRESHOLD_32 0xCCCCCCCCUL
Not necessarily. That's correct for 32-bit processes running on a
64-bit powerpc kernel, but not always on other archs. In particular
if the kernel is 32-bit, there will generally only be 2-3G of
addressable memory, because the kernel inhabits every process's
address space.
This will be a little tricky to fix, since on a number of archs
(including i386) the amount of addressable memory can depend on the
kernel config. We'll need to find a way to determine the address
space limit at run time, I don't know how off the top of my head.
> +
> #ifdef __syscall_return
> #ifdef __i386__
> /* The normal i386 syscall macros don't work with -fPIC :( */
> @@ -231,6 +234,63 @@ static void assemble_path(char *dst, const char *fmt,
> ...)
> }
> }
>
> +#if defined(__i386__) || defined(__powerpc32__)
We can use __LP64__ to check for a non-64-bit system here instead of
the individual architecture #defines, we already do that elsewhere in
elflink.c
Although come to that, since we'll need to determine the threshold at
run-time anyway, we might as well always run this check, even on
64-bit systems. It would just be very unlikely to trigger there,
because the threshold would be so high.
> +static long read_vsize()
> +{
> + FILE *filp;
> + unsigned long vsize;
> + int ret;
> +
> + filp = fopen("/proc/self/stat", "r");
> + if (!filp) {
> + ERROR("Couldn't open /proc/self/stat: %s\n", strerror(errno));
> + return -1;
> + }
> +
> + ret = fscanf(filp, "%*d %*s %*c %*d %*d %*d %*d %*d %*u %*u %*u %*u %*u
> %*u %*u %*d %*d %*d %*d %*d %*d %*u %lu", &vsize);
> + if (ret != 1) {
> + ERROR("Couldn't find vsize in /proc/self/stat\n");
> + return -1;
> + }
> + fclose(filp);
> +
> + return vsize;
> +}
> +
> +static void check_memsz()
> +{
> + int i;
> + unsigned long memsz_total = 0, memsz_max = 0;
> + if (htlb_num_segs == 0)
> + return;
> + if (__debug) {
> + /* parse /proc/self/statm */
> + memsz_total = read_vsize();
> + } else {
> + /*
> + * rough heuristic to see if we'll run out of address
> + * space
> + */
> + for (i = 0; i < htlb_num_segs; i++) {
> + memsz_total += htlb_seg_table[i].memsz;
> + if (htlb_seg_table[i].memsz > memsz_max)
> + memsz_max = htlb_seg_table[i].memsz;
> + }
> + }
> + /* avoid overflow checking by using two checks */
> + if (memsz_total > MEMSZ_THRESHOLD_32 ||
> + memsz_total + memsz_max > MEMSZ_THRESHOLD_32) {
> + WARNING("The memory requirements of this binary's "
> + "segments may be too large for a 32-bit "
> + "address space. This limitation can be "
> + "removed by rebuilding the binary as "
> + "64-bit.\n");
> + }
> +}
> +#else
> +static inline void check_memsz() { }
> +#endif
> +
> /**
> * find_or_create_share_path - obtain a directory to store the shared
> * hugetlbfs files
> @@ -568,6 +628,7 @@ static void parse_elf(Elf_Ehdr *ehdr)
> ehdr->e_phnum);
> htlb_num_segs++;
> }
> + check_memsz();
> }
>
> /*
>
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Libhugetlbfs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/libhugetlbfs-devel
