If my memory serves me with some degree of faith, I seem to remember reading once -- I think in these missives, or perhaps the Perl Monastery -- that slurping in a file can, in certain circumstances, actually be slower than reading/processing line-by-line.
HTH,
Deane
| Craig Cardimon <[EMAIL PROTECTED]>
Sent by: [EMAIL PROTECTED] 03/15/2005 14:28
|
To: [email protected], ActivePerl <[email protected]> cc: Subject: Keyword search is dragging |
I'm searching large ASCII files for keywords. The keywords are part of
section headings. These headings are in all caps on lines by themselves.
The files sometimes contain HTML tags. My logic handles this well
enough, but combs through the HTML very slowly. I'm dealing with tens of
thousands of files, so speed counts.
I thought I'd get around this by using HTML::TokeParser to remove any
HTML before I searched each file. But now the script processes EVERY
file slowly, taking a few seconds for each.
Any suggestions on how I might optimize the following code, or what I
could be doing better?
-- Craig
# slurp file into variable
{
local $/;
$wholefile = <IN>;
}
# remove HTML tags from variable, leaving only text
my $parser = HTML::TokeParser->new (\$wholefile);
while (my $token = $parser->get_token)
{
next unless $token->[0] eq 'T';
$wholefile2 = $wholefile2 . $token->[1];
}
foreach $keyword (@all_keywords)
{
my $re = qr
{
( # start of $1 variable
( # start of a group (\w+[A-Z])+ # one or more words in caps
\s+ # one or more spaces
)* # zero or more groups
$keyword # the $keyword variable
\s+ # one or more spaces
AGREEMENT # the word "AGREEMENT"
) # end of $1 variable
}x;
my $wholeRE = qr{^\s*$re\s*$};
if($wholefile2 =~ /$wholeRE/gm)
{
# proceed
}
}
---
avast! Antivirus: Outbound message clean.
Virus Database (VPS): 0511-0, 03/15/2005
Tested on: 3/15/2005 3:28:18 PM
avast! - copyright (c) 1988-2004 ALWIL Software.
http://www.avast.com
_______________________________________________
ActivePerl mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
_______________________________________________ ActivePerl mailing list [email protected] To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
