Hello everybody, I have a question regarding behavior of the memory pinning using ibv_reg_mr() on RHEL/Centos 5.3/OFED-1.4.1. This may be more a Linux feature/issue, and I apologize if that is the case. Here is the scenario
1. An application runs which allocates a significant fraction of the system memory as RDMA buffers (e.g. 3GB of ram on a 4GB system). These are setup using ibv_reg_mr(). It is clear that the pages are pinned from the kernel perspective. With just this program running, the resident set size of the program approaches the allocation size. 2. If other memory-intensive processes are now started, the resident set size of the RDMA-using program shrinks dramatically. 3. Even if the other memory-intensive programs are stopped, and the RDMA-using program is forced to read its memory, the resident set never grows to a ³reasonable² size again. My potentially foolish assumptions are/were that: i) Since the memory is pinned anyway, it would be locked into the process address space, and ii) even if that were not the case, that the process would be able to regain a large RSS when any competing processes stopped. For ii), it almost seems that the VM doesn¹t realize that the pages it would be grabbing back are already resident and therefore won¹t actually take any more memory. Clearly, I can use mlock() to avoid the issue, but I was wondering if I have missed something obvious here. Any clues/brickbats gratefully received! Regards, Tim Wright -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
