>>>>> "BT" == Ben Tilly <[EMAIL PROTECTED]> writes:
BT> How was I confusing issues? What I meant is that calling mmap does BT> not use significant amounts of RAM. (The OS needs some to track BT> that the mapping exists, but that should be it.) Once you actually use BT> the data that you mmapped in, file contents will be swapped in, and BT> RAM will be taken, but not until then. mmap uses virtual ram which means physical ram and swap space. so mmap can suck up as much physical ram as you have if you allocate it. BT> As for a 3 GB limit, now that you mention it, I heard something BT> about that. But I didn't pay attention since I don't need it right now. BT> I've also heard about Intel's large addressing extensions (keep 2GB BT> in normal address space, page around the top 2 GB, you get 64 GB BT> of addressible memory). I'm curious about how (or if) the two can BT> cooperate. eww, that sounds like manual management of extended memory. like the days of overlays or even pdp-11 extended memory (which i used to access 22 bit address (yes, 22 bit PHYSICAL space) from a 16 bit cpu.). not something you want to deal with unless you have to. as for the original problem, i keep saying that mmap will give little help if the input matrix won't fit into physical ram. once you start swapping (manually or virtually) all bets are off on speed of any transpostion algorithm. you have to start counting disk accesses and once you do, who care how it was done (mmap or whatever)? >> To be clear, though, if you had 10MB of RAM, you could still mmap a 3GB >> file, assuming you allowed for over-committed allocation in the kernel >> (assuming Linux... filthy habit, I know). BT> Exactly what I was referring to. BT> However the over-committed allocation comment confuses me. BT> Why would a single mmap result in over committing memory? you can allocate all the virtual space allowed when you don't need it. then mmap gains you little. mmap is best used as a window into a large file. note that page faults when using mmap will block a process until that page is swapped in. so some systems use small processes to touch shared (mmaped) virtual ram so that block and when they wake up the main process can use the real ram. BT> I was thinking that you'd use something like Sys::Mmap's mmap BT> call directly so that there is a Perl variable that Perl thinks is a BT> regular variable but which at a C level has its data at an mmapped BT> location. Fragile, I know (because Perl doesn't know that it cannot BT> reallocate the variable), but as long as you are careful to not cause BT> it to be reallocated or copied, there should be no limitations on BT> what you can do. but those are bad limitations. i wouldn't ever design in such a beast. i would use other methods of IPC than mmap in perl. BT> In Perl I'd expect it to be possible but fragile. If Parrot could make BT> it possible and not fragile, that would be great. parrot will have mmap in bytecode which will be nice and fast and shared. i dunno about mmaped data as that is always an issue and with garbage collection it can be tricky. uri -- Uri Guttman ------ [EMAIL PROTECTED] -------- http://www.stemsystems.com --Perl Consulting, Stem Development, Systems Architecture, Design and Coding- Search or Offer Perl Jobs ---------------------------- http://jobs.perl.org _______________________________________________ Boston-pm mailing list [EMAIL PROTECTED] http://mail.pm.org/mailman/listinfo/boston-pm
