Hi All, > The documentation about mmap > <http://pubs.opengroup.org/onlinepubs/007908799/xsh/mmap.html> > indicates that MAP_PRIVATE and MAP_SHARED affects the write operations > only. I am under the impression that for read-only file maps, the > system will automatically share the content -- therefore nothing is > said about this case.
> As long as nobody is changing to mapped region (and fastbit does not) mapping > a file via mmap from different processes will result in page tables of all > processes to point to the same memory pages in buffer/memory buffer of the > buffer cache. Sorry, I'have misunderstood the documentation. After re-read, how mmap reuse memory segment, work like PHP do when allocating memory for inter-variable substitution (copy-on-write). > As a consequence the process will share the same physical memory pages, but > the virtual memory size of each process will count them individually. > Therefore your system should be configured to allow enough overcommit of > memory, because if you mmap the same huge files in a lot of processes your > virtual memory reported by the system tools (ps, top) will be MUCH higher > than the physical memory. Can we check the real number of memory usage? On 2/11/15, Norbert Heußer <[email protected]> wrote: > Hi, > > under most modern operation system (windows, mac, linux) the flag the file > buffer and the memory buffer cache is a shared resource. For this resource > the flag indicating MAP_PRIVATE and MAP_SHARED is really relevant for write > operations only. > > As long as nobody is changing to mapped region (and fastbit does not) > mapping a file via mmap from different processes will result in page tables > of all processes to point to the same memory pages in buffer/memory buffer > of the buffer cache. > > More detailed description for linux in the following, but it should hold for > most modern systems, too. > > As a consequence the process will share the same physical memory pages, but > the virtual memory size of each process will count them individually. > Therefore your system should be configured to allow enough overcommit of > memory, because if you mmap the same huge files in a lot of processes your > virtual memory reported by the system tools (ps, top) will be MUCH higher > than the physical memory. > As a second consequence the use counter of all 4k-pages in the memory buffer > for each process touching the page (means reading) will be increased. This > results in the memory manager considered the page to be „more important“ and > the probability of the physical page being used for something else will > decrease. > > Best, > Norbert > > >> Am 10.02.2015 um 15:15 schrieb Aris Setyawan <[email protected]>: >> >> I have checked the code. Yes, the mmap opened readonly, "read_only (@c >> opt = 0) mode". >> But this lead to call mmap using MAP_PRIVATE flag (not shared). >> >> To share mmap between process, one must mmap using MAP_SHARED. >> So I think this mmap wouldn't be shared between process. >> >> Is this expected behavior? >> >> -Aris >> >> On 2/9/15, K. John Wu <[email protected]> wrote: >>> Hi, Aris, >>> >>> In all cases, FastBit is doing memory map on read-only content. >>> Therefore, the system should be able to safely share the content among >>> different processes. Hope this helps. >>> >>> John >>> >>> >>> On 2/8/15 7:17 PM, Aris Setyawan wrote: >>>> Hi John, >>>> >>>> Is fastbit memory mapped file, shared between process? >>>> >>>> I plan to use fastbit from php-apache, which have multiple process >>>> architecture. >>>> If the memory mapped file shared, I can save a lot of memory. >>>> >>>> >>>> Regards, >>>> -aris. >>>> _______________________________________________ >>>> FastBit-users mailing list >>>> [email protected] >>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>>> >>> _______________________________________________ >>> FastBit-users mailing list >>> [email protected] >>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>> >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
