nikhil anand wrote:
Hi,
is there any perl module which shows how much memory
and time my perl script took to run?
Also, is there a way to increase the temporary buffer
size which perl uses.
I want to parse a very big file. Parsing the file line
by line is a slow process. But when I try directly to
read in a huge chunk and do regular expression
matching, I am running out of memory. What is the
maximum sized chunk of data on which I can do regular
expression matching at a time.
what better alternativesto I have when parsin files to
while ($line = <FILE>)
to make my script parse the file faster?
I'm having a similar sort of problem. I must parse ASCII files that can
be very large.
First, files are searched for certain patterns.
Second, if those patterns are found, the file is prepared for insertion
into a SQL Server 2000 database.
Third, the prepared file is inserted into SQL Server.
The script itself is fairly simple, but the data load is killing it. The
script ran for about three or four hours without stopping because it was
processing almost 70,000 files. Eventually it ran out of memory and
crashed.
I am clearing every variable as soon as I am done with it, but it's
simply not enough. I have found that if I run the script on 20 or 30,000
files at a time, stopping completely between runs, everything is fine.
I'm trying to come up with a loop that will process the first 20,000
files and then stop. When I run the script again, it will process the
next 20,000 files and then stop. And so on until it has processed all
the files in the current directory. The directory changes with each
monthly load. The script will keep track the files in that directory so
there are no duplicates.
-- Craig
---
avast! Antivirus: Outbound message clean.
Virus Database (VPS): 0536-5, 09/09/2005
Tested on: 9/12/2005 8:51:27 AM
avast! - copyright (c) 1988-2004 ALWIL Software.
http://www.avast.com
_______________________________________________
ActivePerl mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
