Steve Dawson wrote:
Quoting Craig Cardimon <[EMAIL PROTECTED]>:
processing almost 70,000 files. Eventually it ran out of memory and
crashed.
I am clearing every variable as soon as I am done with it, but it's
simply not enough. I have found that if I run the script on 20 or 30,000
files at a time, stopping completely between runs, everything is fine.
I'm trying to come up with a loop that will process the first 20,000
files and then stop. When I run the script again, it will process the
next 20,000 files and then stop. And so on until it has processed all
the files in the current directory. The directory changes with each
monthly load. The script will keep track the files in that directory so
there are no duplicates.
Are you using Win32::ODBC? There's a memory leak in that modules which added
about 6K to one of my script every time I created and disposed of a odbc
object. I ended up creating the object once and reusing it for the life of the
script.
I was also keeping the scope of file related variables very tight to makesure
that memory was freed up and reused on each file processing cycle.
SD
Yes, I am using Win32::ODBC. I thought create and destroy was the way to
go. Perhaps it isn't. I make several different ODBC connections due to
the nature of the script and how the data gets munged. Perhaps I should
create them all at the beginning and keep them open for the duration of
the script. Is there a limit to the number of ODBC connections that can
be open at the same time?
-- Craig
---
avast! Antivirus: Outbound message clean.
Virus Database (VPS): 0537-0, 09/12/2005
Tested on: 9/13/2005 7:43:55 AM
avast! - copyright (c) 1988-2004 ALWIL Software.
http://www.avast.com
_______________________________________________
ActivePerl mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
