I suppose it depends on what kind of processing you are doing. Can you give us examples?
For that size of data, I would make sure you are reading the file in line by line (as opposed to reading it to an array). Also, anything that prints to the screen will slow things down considerably. I know it's nice to see the line count continuing to increase so you know something is happening, but printing even a single character adds a lot of overhead. Save most of the printing until the end to dump your results. If I want a status, I usually print a single char (like a "-") for every 1,000 lines processed, and a "+" at the 10,000 marks. Or...depending on the data...I'll check the file size ahead of time then create an estimated finish time. I know, based on historical runs that I can process about $X MB/minute. If the file is $Y MB, it should take $Y/$X minutes to complete. Print this info up front, then a status line every so often... -rjf > -----Original Message----- > From: Kevin Old [mailto:[EMAIL PROTECTED]] > Sent: Monday, March 11, 2002 19:42 > To: [EMAIL PROTECTED] > Subject: Process large files quickly...how to? > > > Hello all, > > I have a few files that are on average 30MB and need to be > processed through a perl script. The script ends up taking > almost and hour. Thing is, the script cannot run more than > an hour cause another script is kicked off (same script > different data every hour). > > What is the best way to read the data in and process it? > Read each line of the file into an array and then process > each line or just process the line directly? > > Anyone have any scripts that process large files quickly? > I'd love to see examples of how you did it. > > Thanks, > Kevin > > > > > -- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
