If you gave some code with your question, I would have a better idea what is
taking so long. I will venture a guess, only because I know what happened
when I first started working with large files. The first thing to check is
if you have any code that looks like this:
---------------------------
open(INFILE,"bigfile.log");
my @infile = <INFILE>;
foreach(@infile){
do something...
}
---------------------------
If you do, you'll want to change it to something like this:
---------------------------
open(INFILE,"bigfile.log");
while(<INFILE>){
do something...
}
---------------------------
The differenc maye seem like a small one when you are working with smaller
files, but if you are working with large files, you will soon find that
loading 30MB into RAM before even starting to parse the file is very costly
in terms of resources, and can make your scripts much longer than they
really have to be. This might not even be your problem, though. The best
way to get a good answer is to ask a good question, and if you want to know
a better way to code something you should provide some of your code as a
reference.
-----Original Message-----
From: Kevin Old
To: [EMAIL PROTECTED]
Sent: 3/11/02 5:42 PM
Subject: Process large files quickly...how to?
Hello all,
I have a few files that are on average 30MB and need to be processed
through
a perl script. The script ends up taking almost and hour. Thing is,
the
script cannot run more than an hour cause another script is kicked off
(same
script different data every hour).
What is the best way to read the data in and process it? Read each line
of
the file into an array and then process each line or just process the
line
directly?
Anyone have any scripts that process large files quickly? I'd love to
see
examples of how you did it.
Thanks,
Kevin
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--------------------------------------------------------------------------------
This email may contain confidential and privileged
material for the sole use of the intended recipient.
If you are not the intended recipient, please contact
the sender and delete all copies.
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
