On Tue, Mar 11, 2008 at 9:54 AM, Carl E. McIntosh <[EMAIL PROTECTED]> wrote: > Can you please give advice about handling large data files with memory > management techniques? I am attempting to read three large files (1 > GB, 208 MB, 725 MB) sequentially and place the data into arrays for > processing. Here is my psuedocode: > > 1) Import a file into NSString. > NSString *aFileString = [NSString stringWithContentsOfFile: > fileLocation]; // Convert file at path to myFileString text holder;
Depending on the encoding of the file itself, this could end up allocating a block of memory twice the size of the file. > 2) Use NSScanner pull out integers and floats > NSScanner *aFileScanner = [[NSScanner alloc] initWithString: > aFileString]; This has the potential of copying the passed in string (not likely to be a problem, as you string is immutable, but is still something to keep in mind). > > 3) Store values into arrays. > float myFloats [100000][2000]; or > float myInts [100000][2000]; Again, *another* huge block of memory. > 4) repeat three times with 3 different files. Depending on how this is done (how tight your autorelease pools are, etc.), the old string may not have been deallocated yet. > This algorithm works for smaller files but chokes on the larger files > and I get malloc errors. I've attempted to use NSZone's to the same > failure. > > Can you please give advice about handling large data files with memory > management techniques? You'll have to use less memory. Remember, that on a 32-bit machine, you will never be able to allocate more than 4GB of memory in your address space. On top of that, much of that memory is already used by the system's frameworks (usually between 1 and 2 GB). > I have 4 GB ram and can hog off 2 - 3 GBs for the process. I don't > know how to explicitly allocate real memory. I'd rather not use > virtual memory. Any references or examples would be appreciated. Memory doesn't work like that (i.e you cannot allocate "real" memory). The closest that you can get is to allocate memory and to "wire" it down; however, this is a horrible idea for allocations this large (you'll still run out of address space as before). Your best bet is to change your algorithm to not have to load the entire file at once (perhaps load it a line at a time, or in blocks of 4K or so). -- Clark S. Cox III [EMAIL PROTECTED] _______________________________________________ Cocoa-dev mailing list ([email protected]) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to [EMAIL PROTECTED]
