Thanks for that thorough explanation! We're going to try using mmap(). It will be interesting to do this from Swift.
> On May 4, 2017, at 10:04 , David Duncan <[email protected]> wrote: > > >> On May 3, 2017, at 11:51 PM, Rick Mann <[email protected]> wrote: >> >>> >>> On May 3, 2017, at 23:27 , Doug Hill <[email protected]> wrote: >>> >>> >>>> On May 3, 2017, at 6:21 PM, Rick Mann <[email protected]> wrote: >>>> >>>> Our iOS app works with very large data buffers (hundreds of MB). As you >>>> can imagine, we run into issues at times. >>>> >>>> I saw some sample code that used this technique, and it got me wondering >>>> if this actually works around the 600 MB limitation of some iOS devices >>>> (which begs another question: doss the large iPad Pro with 3GB of RAM also >>>> limit apps to 600 MB?). >>>> >>>> -- >>>> Rick Mann >>>> [email protected] >>> >>> The limits of mmap should be the limits of the file system. Well, >>> technically it’s limited by the size_t parameter which I guess is OS >>> specific. But these days it should be what the VFS can handle. >>> >>> I just tried out this code snippet: >>> >>> http://www.linuxquestions.org/questions/programming-9/mmap-tutorial-c-c-511265/ >>> >>> <http://www.linuxquestions.org/questions/programming-9/mmap-tutorial-c-c-511265/> >>> >>> and set the size of the mapped file to 32GB Works fine on a Mac with 16GB >>> of RAM. >>> >>> It is SLOOOOOOWWWWWW however. You have to move all that data across a bus >>> after all. If you need to load 3GB of data from your file, I recommend >>> making it happen on a background thread. And make some lights blink and >>> play a happy tune while the user waits. >>> >>> Not sure what the limitation on some iOS devices you’re referring to. Is it >>> dependent on the installed RAM? Are there other artificial limits? >> >> Yeah, at least on 1 GB iOS devices, apps are limited to 600 MB (even if >> there's more available). I think there are subtleties to that (e.g. maybe >> texture memory isn't counted), I'm not sure. > > iOS tracks memory per-process primarily in 3 buckets – Dirty, Resident, and > Virtual. The hierarchy is also in that order, all Dirty memory is Resident, > and all Resident memory is Virtual. Virtual just means that address space is > allocated – there is a hard limit on this that is below the theoretical > address space (significantly so in the case of iOS), in the 2-4GB range (may > vary by device installed memory, but I don’t recall). Resident memory is > memory that has physical RAM allocated to it and so the CPU can manipulate > it. This is the 600MB limit that you mentioned (at least on 1GB devices, > again I don’t recall if it is different or what it is on larger devices). > Dirty memory is memory that has been written to. Because iOS does not have a > demand paging memory system, this memory typically has no where to go if you > hit the Resident memory limits. > > This is where mmap() comes into play – it allows you to create a Virtual > allocation, whose Resident memory is backed by a file, just like in a demand > paged system. So you can write to that memory, and if your app uses too much > memory, the OS can attempt to recover by writing it to disk on your behalf > (turning Resident-Dirty memory into Virtual). Sweet right? > > But remember that hard Virtual limit I mentioned earlier – that means that if > you try to mmap() too much you end up running out of Virtual memory, and > you’ll likely get abort()’d by some subsystem attempting to allocate memory. > > And so even though iOS is a 64-bit OS, due to other performance > considerations, the virtual address space is more akin to a 32-bit OS. This > is of course an implementation detail that can be expanded in the future, but > something to keep in mind if you plan to use mmap() a lot. > > From the sounds of what our app does, keeping around a single mmap()’s > scratch space may be reasonable. You probably still want to keep it at a > modest size, but it should help you process those large data buffers without > getting your app killed. > -- > David Duncan > -- Rick Mann [email protected] _______________________________________________ Cocoa-dev mailing list ([email protected]) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to [email protected]
