Mandeep has solved it for me. I was able to set /proc/sys/vm/nr_hugepages to 12288, corresponding to 24GiB of memory, and make a corresponding mmap request. It takes a while -- a very noticeable couple of seconds -- but it works.
There are a few things still to be worked out, like why that setting is reduced when my little program runs. So that it has to be re-established in order to run it again. But it gets me started. On Mon, Aug 18, 2014 at 1:25 PM, Mandeep Sandhu <[email protected] > wrote: > I sent the previous mail too early! > > I got it to work for root user by setting "/proc/sys/vm/nr_hugepages" to > 20 as suggested in Documentation/vm/hugetlbpage.txt. Now I'm able to mmap > it and it reflects the same in /proc/meminfo as well! :) > > HTH, > -mandeep > > > > On Mon, Aug 18, 2014 at 1:23 PM, Mandeep Sandhu < > [email protected]> wrote: > >> I tried your program and got the same result, i.e "Resource temporarily >> unavailable" for an ordinary user and "Cannot allocate memory" if run as >> root. >> >> However, reading the Huge TLB pages doc >> (Documentation/vm/hugetlbpage.txt), I saw the following para which might >> explain the error: >> ... >> Once a number of huge pages have been pre-allocated to the kernel huge >> page >> pool, a user with appropriate privilege can use either the mmap system >> call >> or shared memory system calls to use the huge pages. See the discussion of >> Using Huge Pages, below. >> >> The administrator can allocate persistent huge pages on the kernel boot >> command line by specifying the "hugepages=N" parameter, where 'N' = the >> number of huge pages requested. This is the most reliable method of >> allocating huge pages as memory has not yet become fragmented. >> ... >> ... >> >> I don't have my kernel booted with it, so that might explain why mmap fails >> for MAP_HUGETLB. >> >> I can't restart my machine with the new flag right now, but maybe you can >> try. >> >> HTH, >> >> -mandeep >> >> >> >> >> On Mon, Aug 18, 2014 at 11:30 AM, Kevin O'Gorman <[email protected]> >> wrote: >> >>> Now I'm befuddled. I modified the source so that it was easy to try >>> different combinations. If I sudo it, I can map a 1GiB (1<<30) private >>> anonymous region if I sudo the program. I cannot find any combination with >>> MAP_HUGETLB that works at all. >>> >>> Here it is in working form, with an anonymous map and no backing file. >>> The mmap request is for a single byte, so it works for ordinary users Try >>> uncommenting the MAP_HUGETLB line, and I'd love to hear about a version >>> that works, even if it's only for root. >>> >>> Since I'm aiming for a huge memory region, I figured HUGETLB would make >>> sense to limit the overhead. >>> >>> ============================ CUT HERE ================ >>> /** >>> * @file >>> * <pre>"Find out the limits on locked memory" >>> * Last Modified: Mon Aug 18 11:21:23 PDT 2014 >>> * @author Kevin O'Gorman >>> */ >>> >>> #define _GNU_SOURCE /* enable some of the mmap flags */ >>> #include <unistd.h> >>> #include <stdlib.h> >>> #include <stdio.h> >>> #include <string.h> >>> #include <sys/mman.h> >>> >>> /* for getrlimit(2) */ >>> #include <sys/time.h> >>> #include <sys/resource.h> >>> >>> /* for open(2) */ >>> #include <sys/types.h> >>> #include <sys/stat.h> >>> #include <sys/fcntl.h> >>> >>> int >>> main(int argc, char *argv[]) >>> { >>> void *where; >>> size_t length = 1; >>> struct rlimit rlim; >>> int flags = 0; >>> int fd = -1; >>> >>> // fd = open("solvedtaq", O_RDWR); if (fd == -1) perror("open"); >>> printf("fd is %d\n", fd); >>> >>> // flags |= MAP_HUGETLB; >>> flags |= MAP_PRIVATE; >>> flags |= MAP_LOCKED; >>> flags |= MAP_ANONYMOUS; >>> >>> if (getrlimit(RLIMIT_MEMLOCK, &rlim) == 0) { >>> printf("RLIMIT_MEMLOCK: hard: %lld, soft: %lld\n", (long >>> long)rlim.rlim_max, (long long)rlim.rlim_cur); >>> } else { >>> perror("getrlimit failed"); >>> } >>> >>> where = mmap(NULL, length, PROT_READ | PROT_WRITE, flags, fd, 0); >>> if (where != MAP_FAILED) { >>> printf("Mapped at %p\n", where); >>> } else { >>> perror("Mapping failed"); >>> } >>> >>> return EXIT_SUCCESS; >>> } >>> ============================ CUT HERE ================ >>> >>> >>> >>> On Mon, Aug 18, 2014 at 10:24 AM, Kevin O'Gorman <[email protected]> >>> wrote: >>> >>>> >>>> On Mon, Aug 18, 2014 at 9:46 AM, <[email protected]> wrote: >>>> >>>>> On Mon, 18 Aug 2014 08:15:12 -0700, "Kevin O'Gorman" said: >>>>> >>>>> > maybe someone here can help me figure out how to map a big (really >>>>> big) >>>>> > work area and lock it in memory. >>>>> >>>>> What problem are you trying to solve by doing this? It usually >>>>> doesn't make >>>>> sense to lock more than a small chunk in memory - that's mostly for >>>>> crypto >>>>> programs like PGP that are paranoid about keys being written to swap >>>>> spaces, >>>>> and a *very* few highly performance oriented programs. 99.8% of the >>>>> time, >>>>> the kernel will do a reasonable job of making sure the right pages are >>>>> in >>>>> memory. >>>>> >>>>> Plus, programmers almost always over-estimate what they *really* need >>>>> to lock >>>>> down. The end result is that performance ends up being *worse*, >>>>> because they >>>>> lock out more than needed, leaving all the rest of the processes on >>>>> the system >>>>> fighting over a too-small pool of pages. Really sucks when you >>>>> actually have >>>>> to read from disk every I/O because you no longer have an in-core >>>>> file cache :) >>>>> >>>>> I'm solving a huge alpha-beta search problem, and want a huge cache >>>> table tocheck permutations. That's why I increased RAM from 8 to 32 GB, >>>> and would >>>> be happy to use the extra 24GB all for the cache. (Cache table is like >>>> a hash table but with collisions causing replacement rather than chaining). >>>> >>>> Even in my small tests, the cache is giving me speedups in the >>>> neighborhood of >>>> a million, so I'm going big. Because this is about speed, I do not >>>> *ever* want it to >>>> swap. >>>> >>>> >>>>> > I wrote a little test program to try this out, and it fails. As a >>>>> regular >>>>> > user, perror() tells me some "resource is temporarily unavailable". >>>>> As >>>>> > root, it says it "cannot allocate memory". I'm only asking for 1 >>>>> byte. >>>>> >>>>> > where = mmap(NULL, length, PROT_READ | PROT_WRITE, MAP_PRIVATE | >>>>> > MAP_ANONYMOUS | MAP_HUGETLB | MAP_LOCKED | MAP_POPULATE, -1, 0); >>>>> >>>>> Remember that no matter what "length" you pass, even if it's 1 byte, >>>>> the >>>>> kernel has to allocate at least enough integer pages to cover the >>>>> request. >>>>> You ask for 4100 bytes, it has to allocate 2 pages because 4096 isn't >>>>> quite big enough. >>>>> >>>>> And if you ask for MAP_HUGETLB, you're going to get at least 1 page. >>>>> Which >>>>> is probably 2M in size. And that can be painful if the default max >>>>> lockable >>>>> memory is less than 2M (at least on my Fedora Rawhide box, it's all of >>>>> 64K). There's also some interaction danger with MAP_POPULATE and >>>>> memory >>>>> fragmentation. But both POPULATE and HUGETLB have a high chance of >>>>> not playing well with LOCKED. >>>>> >>>> >>>> Good catch. Indeed, removing MAP_HUGETLB allows the program to >>>> succeed. I'll try increasing the resource limit, but may have to have to >>>> sudo it, do the mmap() and then chusr to my real ID. >>>> >>>> The fd of -1 that some folks thought might be a problem is suggested in >>>> the man page for use with MAP_ANONYMOUS. That flag means I'm not using a >>>> backing file, after all. >>>> >>>> -- >>>> Kevin O'Gorman >>>> >>>> programmer, n. an organism that transmutes caffeine into software. >>>> Please consider the environment before printing this email. >>>> >>> >>> >>> >>> -- >>> Kevin O'Gorman >>> >>> programmer, n. an organism that transmutes caffeine into software. >>> Please consider the environment before printing this email. >>> >>> _______________________________________________ >>> Kernelnewbies mailing list >>> [email protected] >>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies >>> >>> >> > -- Kevin O'Gorman programmer, n. an organism that transmutes caffeine into software. Please consider the environment before printing this email.
_______________________________________________ Kernelnewbies mailing list [email protected] http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
