Is the real problem that process_file($file) will break on a single
file that is larger than 4GB and not in how you go through the sequence
of files?
Or is if you process 10 files, each .4GB, you die on the 10th file?
If the former then you might be slurping the file in its entirity and
need to make process_file work record-at-a-time (or similar) and clear
unnecessary state between lines.
If the latter then you've got a memory leak by not clearing state
between files (and you might still have the former problem).
Cheers,
Richard
Brian Raven wrote:
Craig Cardimon wrote:
I'm working with large numbers of files that can vary in size up to 4
GB and more. Every time I try tackling them straight on I run out of
virtual memory. The files are sequentially numbered somewhat like SS
numbers -- they contain hyphens. The file numbers are unique as far
as I can tell.
I don't know what an SS number is, and its probably not relevant. I just
hope that its explanation doesn't invoke Godwin's law. :-)
Is there a loop I can use that will process the first 10,000 files,
then exit; then the next 10,000 files, then exit; and so on until all
files have been processed? The exiting is so I can run a SQL Server
job to insert the processed files into SQL Server.
The loop itself is fairly simple, for example:
my $number_of_files = get_number_of_files_to_process();
for (1..$number_of_files) {
my $file = get_next_filename();
last unless defined $file;
process_file($file);
mark_file_as_processed($file);
}
As long as the list of files to be processed does not change during
processing, then a simple solution involves 2 files. One containing a
the names of files to be processed, and the other containing the name of
the last file processed.
HTH
--
*** Richard A. Wells, [EMAIL PROTECTED], +1.978.371.7425
*** Reality And Wonder, http://www.raw.com/
|
_______________________________________________
ActivePerl mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
