hello ,
Thanks for the help .
But my problem is that i need to get 3 different hashes for all uni bi
tri grams ..
coz later in my code i have to calculate the value as
trigram{hello how are} /bigram{hello how}
so how do i relate the 2 values in the hashes ...
i used a code as :
my($prev1,$prev2) = ('','');
while (<>) {
@words = split;
foreach my $word (@words) {
$unigram{$word}++;
$bigram{"$prev1 $word}++;
$trigram{"$prev2 $prev1 $word}++;
$prev2 = $prev1;
$prev1 = $word;
}
}
but not able to process it further ....
On 4/23/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
>
>
> In a message dated 4/22/2006 9:10:58 P.M. Eastern Standard Time,
> [EMAIL PROTECTED] writes:
>
> > hello all,
> > I have a text file and i want to get the total number of occurences of
> > each unigrams that is one word , and the bigram the simultaneous
> > occurence of 2 words and same for 3 words as trigrams.
> > I want to store them in hash tables but am not able to get how to
> > parse through the text file and add them to hash . Can anybody help me
> > out..
> >
> > ex. if text is hello how are you all hello again.
> > i need
> > unigrams as
> > hello 2
> > how 1
> > you 1....so on
> >
> > bigrams as
> > hello how 1
> > how are 1
> > are you 1
> >
> > trigram as
> > hello how are 1
> > how are you 1
> > ....
> >
> > Please help..
> >
> > -thanks
>
> hi amit --
>
> the code below should be close to what you need. please let me know how it
> works out for you.
>
> i hope this isn't just your homework; if it is, i'm not really helping you!
>
>
> hth -- bill walters
>
> -------------- code begins ------------------
> use strict;
> use warnings;
>
>
> # this bit is a little fragile. note that the separator string must be
> # composed of characters that do NOT match the unigram characters!
> my $unigram = qr/ \w+ /x; # what a unigram looks like
> my $non_unigram = qr/ \W+ /x; # what anything that's NOT a unigram looks
> like
> my $separator = ' '; # what to substitute for non-unigram stuff
>
> $separator !~ $unigram or die "separator looks like a unigram";
>
> while (defined(my $line = <DATA>)) {
>
> $line = canonicalize($line, $non_unigram, $separator);
>
> print_grams('unigram', extract_grams($line, 1, $unigram, $separator));
> print_grams('bigram', extract_grams($line, 2, $unigram, $separator));
> print_grams('trigram', extract_grams($line, 3, $unigram, $separator));
>
> }
>
> $! and die "reading test lines: $!";
>
>
> sub extract_grams {
>
> my ($line, $n, $unigram_regex, $separator) = @_;
>
> my %grams;
>
> my $remaining_line = $line;
>
> # do NOT use /x switch or extra space outside @{[]} in next regex!
> my $n_gram_regex = qr/@{[ join $separator, ($unigram_regex) x $n ]}/;
>
> while ($remaining_line =~ /($n_gram_regex)/x) { # capture n_gram
> $grams{$1}++; # count captured n_gram (autovivifiying)
> $remaining_line =~ s/ $unigram_regex //x; # break off 1st unigram
> }
>
> return $line, %grams; # return line, hash with captured, counted
> n_grams
>
> }
>
>
> sub print_grams {
>
> my ($gram_name, $line, %grams) = @_;
>
> printf "%d %ss of ``%s'' \n", scalar(keys %grams), $gram_name, $line;
>
> for my $key (sort keys %grams) {
> printf "%6d %s \n", $grams{$key}, $key;
> }
>
> }
>
>
> sub canonicalize {
>
> my ($line, $non_unigram_regex, $separator_string) = @_;
>
> # kill leading and trailing non-unigram stuff
> $line =~ s/ ^ $non_unigram_regex | $non_unigram_regex $ //xg;
> # replace embedded non-unigram stuff with uniform string
> $line =~ s/ $non_unigram_regex /$separator_string/xg;
>
> return $line;
>
> }
>
>
> __DATA__
> fee fie foe fum foo bar baz
>
> hello good fellows, hello good friends
> now is the time for
> now is the time, the time is now
> ;;;all-good...
> men to.come,,to'''
> ... the aid.of,their
> party
>
_______________________________________________
ActivePerl mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
