On Dec 7, 2010, at 9:53 AM, Francesc Alted wrote:
> A Tuesday 07 December 2010 17:07:30 Philip Winston escrigué:
>> On Tue, Dec 7, 2010 at 6:28 AM, Francesc Alted <[email protected]>
> wrote:
>>> In addition, memory mapping also has drawbacks, being
>>> the most important one the inability to map files that are larger
>>> than your available virtual memory, which renders this technology
>>> inadequate for many uses.
>>
>> I guess I knew about the VM limitation but it hadn't completely sunk
>> in.
>>
>> So in fact mmap's limit is the same as RAM:
>> RAM -> big initial load -> VM limited
>> mmap -> zero initial load -> VM limited
>> disk -> zero initial load -> disk limited
>>
>> That is interesting. For us VM limit will probably last us a long
>> time, if we crank up the swap size on our machines. But maybe not
>> forever, probably a fully disk-based solution is the most future
>> proof.
>
> One final piece of warning: when you use mmap beyond the extend of your
> RAM, you will end swapping out many data (shared libraries, other
> processes) that might be important for the performance of your computer.
>
> This is another reason why I don't personally like the mmap approach. I
> find much better to let the kernel to decide which data in the
> filesystem should be cached in-memory, and not the user app (but YMMV).
I wonder if some mechanism for just mmapping a portion of the file
(particularly a dataset's elements) would be valuable?
Quincey
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
