On Thu, 2014-04-03 at 07:21 -0700, janjust wrote: > Philippe, > Thanks a lot for helping with this. > > I ran the code as you suggested. > aprun is a job submission system for our cluster machines. > > Attached is a file with all the output in order: > code, native, strace native, valgrind, strace -f valgrind > > If you look at the strace -f valgrind output, at the bottom you'll see a > mmap fail with EINVAL return code. Ok, I think I have an hypothesis (the below is pure guess work): I guess that the file on the hugetlbfs is special (as it is on this special mounted "huge page" file system). This file provides huge pages, which must (probably) respect some constraints such as: it must be mapped at a multiple of a huge page (1M ? 4M ? or whatever) and/or it must be in a specific part of the adress space and/or ...
Valgrind adress space manager does not understand the notion of huge page. What valgrind does is: it maintains a list of unused "address space zone". To do an mmap, valgrind decides at which adress the mmap will be done. and then asks a fixed mapping at this address to the kernel. If this fixed mapping address is done in a way which is incompatible with the constraints to have a huge page, the kernel makes the mmap call fail. In the below strace extracts, you see that under valgrind, the mmap call is using a first argument different of NULL, and has added a MAP_FIXED argument : strace native: mmap(NULL, 4194304, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = 0x2aaaaac00000 strace valgrind: mmap(0x4801000, 4194304, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_FIXED, 3, 0) = -1 EINVAL (Invalid argument) The logic for all this is in syswrap-generic.c around line 2146 (SVN trunk). What we might maybe do is to have yet another refinement, which is: when the mmap fails, try again, but without any MAP_FIXED arg, and afterwards, verify that the address decided by the kernel is ok. In other words, in the syscall handling for the client, the idea is to introduce a kludge similar to what is done in aspacemgr-linux.c around line 2460 and following. This looks not too difficult to do (and I could even test this on the gcc110 compile farm system, that has a huge fs file system :). How to confirm the hypothesis ? I suggest you run several times the small test program natively. If the map is always at the same address, modify the small program to pass as first argument this address (and maybe add MAP_FIXED). If afterwards, the small program succeeds both natively and under valgrind, it looks like the hypothesis is somewhat confirmed. If the above hypothesis looks correct and/or you can confirm it, then I suggest you file a bug in bugzilla (and do not hesitate to try to prepare the patch described above :). Philippe ------------------------------------------------------------------------------ _______________________________________________ Valgrind-users mailing list Valgrind-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/valgrind-users
