|
In a message dated 4/22/2006 9:10:58 P.M. Eastern Standard Time,
[EMAIL PROTECTED] writes:
> hello all,
> I have a text file and i want to get the total number of occurences of > each unigrams that is one word , and the bigram the simultaneous > occurence of 2 words and same for 3 words as trigrams. > I want to store them in hash tables but am not able to get how to > parse through the text file and add them to hash . Can anybody help me > out.. > > ex. if text is hello how are you all hello again. > i need > unigrams as > hello 2 > how 1 > you 1....so on > > bigrams as > hello how 1 > how are 1 > are you 1 > > trigram as > hello how are 1 > how are you 1 > .... > > Please help.. > > -thanks hi amit --
the code below should be close to what you need. please let me
know how it
works out for you.
i hope this isn't just your homework; if it is, i'm not really helping
you!
hth -- bill walters
-------------- code begins ------------------
use strict;
use warnings; # this bit is a little fragile. note that the separator string must be # composed of characters that do NOT match the unigram characters! my $unigram = qr/ \w+ /x; # what a unigram looks like my $non_unigram = qr/ \W+ /x; # what anything that's NOT a unigram looks like my $separator = ' '; # what to substitute for non-unigram stuff $separator !~ $unigram or die "separator looks like a unigram";
while (defined(my $line = <DATA>)) {
$line = canonicalize($line, $non_unigram,
$separator);
print_grams('unigram', extract_grams($line, 1, $unigram,
$separator));
print_grams('bigram', extract_grams($line, 2, $unigram, $separator)); print_grams('trigram', extract_grams($line, 3, $unigram, $separator)); }
$! and die "reading test lines: $!";
sub extract_grams { my ($line, $n, $unigram_regex, $separator) = @_;
my %grams;
my $remaining_line = $line;
# do NOT use /x switch or extra space outside @{[]} in
next regex!
my $n_gram_regex = qr/@{[ join $separator, ($unigram_regex) x $n ]}/; while ($remaining_line =~ /($n_gram_regex)/x) { #
capture n_gram
$grams{$1}++; # count captured n_gram (autovivifiying) $remaining_line =~ s/ $unigram_regex //x; # break off 1st unigram } return $line, %grams; # return line, hash with
captured, counted n_grams
}
sub print_grams { my ($gram_name, $line, %grams) = @_;
printf "%d %ss of ``%s'' \n", scalar(keys %grams),
$gram_name, $line;
for my $key (sort keys %grams)
{
printf "%6d %s \n", $grams{$key}, $key; } }
sub canonicalize { my ($line, $non_unigram_regex, $separator_string) =
@_;
# kill leading and trailing non-unigram
stuff
$line =~ s/ ^ $non_unigram_regex | $non_unigram_regex $ //xg; # replace embedded non-unigram stuff with uniform string $line =~ s/ $non_unigram_regex /$separator_string/xg; return $line;
}
__DATA__ fee fie foe fum foo bar baz hello good fellows, hello good friends now is the time for now is the time, the time is now ;;;all-good... men to.come,,to''' ... the aid.of,their party |
_______________________________________________ ActivePerl mailing list [email protected] To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
