I wouldn't read in the contents of a file all at once like that. If anything you can optimize the code by checking against a size threshold. If the size is what you would consider to be large, read in content line by line. If the size is what you would consider to be small to medium, read in one lump some. Maybe that will help balance things out.
Donald S. -----Original Message----- From: Craig Cardimon [mailto:[EMAIL PROTECTED] Sent: Tuesday, March 15, 2005 3:28 PM To: [email protected]; ActivePerl Subject: Keyword search is dragging I'm searching large ASCII files for keywords. The keywords are part of section headings. These headings are in all caps on lines by themselves. The files sometimes contain HTML tags. My logic handles this well enough, but combs through the HTML very slowly. I'm dealing with tens of thousands of files, so speed counts. I thought I'd get around this by using HTML::TokeParser to remove any HTML before I searched each file. But now the script processes EVERY file slowly, taking a few seconds for each. Any suggestions on how I might optimize the following code, or what I could be doing better? -- Craig # slurp file into variable { local $/; $wholefile = <IN>; } # remove HTML tags from variable, leaving only text my $parser = HTML::TokeParser->new (\$wholefile); while (my $token = $parser->get_token) { next unless $token->[0] eq 'T'; $wholefile2 = $wholefile2 . $token->[1]; } foreach $keyword (@all_keywords) { my $re = qr { ( # start of $1 variable ( # start of a group (\w+[A-Z])+ # one or more words in caps \s+ # one or more spaces )* # zero or more groups $keyword # the $keyword variable \s+ # one or more spaces AGREEMENT # the word "AGREEMENT" ) # end of $1 variable }x; my $wholeRE = qr{^\s*$re\s*$}; if($wholefile2 =~ /$wholeRE/gm) { # proceed } } --- avast! Antivirus: Outbound message clean. Virus Database (VPS): 0511-0, 03/15/2005 Tested on: 3/15/2005 3:28:18 PM avast! - copyright (c) 1988-2004 ALWIL Software. http://www.avast.com _______________________________________________ ActivePerl mailing list [email protected] To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs ----------- The information contained in this e-mail message, and any attachment, is confidential and for use solely by the intended recipient. If you have received this message in error, please delete this message immediately. Although Moody's KMV makes every effort to protect its computing environment from malicious code, Moody's KMV is not responsible for any virus or other type of suspect code that may be transferred via this e-mail message. _______________________________________________ ActivePerl mailing list [email protected] To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
