On Fri, Mar 21, 2008 at 11:31 AM, YORO BA <[EMAIL PROTECTED]> wrote:
>
>
> Hello Ted,
> thanks you a lot. This help me very well. Is it possible to measure
> the first sentence of file1 and the first of file2 and give the values
> and do the same for the next sentences until the end ?
>
> Thanks you
Hi Yoro,
As of now, text_compare.pl and getSimilarity( ) only support file to
file comparisons. So, you
would need to break your input files into separate files, one per
sentence, in order to do
a sentence by sentence comparison. I should caution you that for many
documents you won't
actually find a one to one correspondence between sentences, but, if
you have that, then
something like I show below should work pretty well. If you don't have
that, you might want
to consider paragraph by paragraph comparisons, or some larger unit.
# you may want to measure the similarity of a document
# sentence by sentence - the below example shows you
# how - suppose you have two text files file1.txt and
# file2.txt - each having the same number of sentences.
# convert those files into multiple files, where each
# sentence from each file is in a separate file.
# if file1.txt and file3.txt each have three sentences,
# filex.txt will become sentx1.txt sentx2.txt sentx3.txt
# this just calls getSimilarity( ) for each pair of sentences
use Text::Similarity::Overlaps;
my %options = ('normalize' => 1);
my $mod = Text::Similarity::Overlaps->new (\%options);
defined $mod or die "Construction of
Text::Similarity::Overlaps failed";
@file1_sentences = qw / sent11.txt sent12.txt sent13.txt /;
@file2_sentences = qw / sent21.txt sent22.txt sent23.txt /;
# assumes that both documents have same number of sentences
for ($i=0; $i <= $#file1_sentences; $i++) {
my $score = $mod->getSimilarity
($file1_sentences[$i], $file2_sentences[$i]);
print "The similarity of $file1_sentences[$i] and
$file2_sentences[$i] is : $score\n";
}
my $score = $mod->getSimilarity ('file1.txt', 'file2.txt');
print "The similarity of the two files is : $score\n";
When I ran this on my example file from my previous message (the short
paragraphs about cricket),
I got the following results - note that I manually mixed the sentence
pairs with their output, but that
would be fairly easy to add to a program like this...
----------------------------------------------------------------------
marimba(1865): cat sent11.txt
The Asian Cricket Council (ACC) confirmed on Thursday that Pakistan
would host the Asia Cup one-day tournament from June 24 to July 6.
marimba(1866): cat sent21.txt
The Asian Cricket Council has confirmed Pakistan will host this year.s
Asia Cup.
The similarity of sent11.txt and sent21.txt is : 0.486486486486487
----------------------------------------------------------------------
marimba(1867): cat sent12.txt
ACC backing for Pakistan came the week after Australia postponed their
tour, which was due to start at the end of March, because of the
troubled security situation.
marimba(1868): cat sent22.txt
ACC officials met Pakistan Cricket Board bosses on Thursday and decided
to hold the tournament in Lahore and Karachi from June 25 to July 6.
The similarity of sent12.txt and sent22.txt is : 0.150943396226415
----------------------------------------------------------------------
marimba(1869): cat sent13.txt
ACC chief executive Ashraf-ul-Haque told a news conference Pakistan and
India would play in the same group in the six-team tournament which will
be held in Karachi and Lahore.
marimba(1870): cat sent23.txt
It will be hoped the decision will seek to dispel security fears, which
led Cricket Australia to postpone their tour to the country and which
have been raised over Pakistan.s hosting of the ICC Champions Trophy in
September.
The similarity of sent13.txt and sent23.txt is : 0.235294117647059
----------------------------------------------------------------------
Here's the actual program output:
The similarity of sent11.txt and sent21.txt is : 0.486486486486487
The similarity of sent12.txt and sent22.txt is : 0.150943396226415
The similarity of sent13.txt and sent23.txt is : 0.235294117647059
The similarity of the two files is : 0.506329113924051
If you look through this example, you can actually see some of the
possible pitfalls of measuring sentence by sentence, for example in one
article they mention security issues in sentence 2 but in the other
in sentence 3. I think that's an illustration of why the overall document
similarity is great than those of any of the individual sentences. Also,
I didn't use a stoplist here so much of the similarity does come from
stop words. Anyway, you get the idea.
Finally, if you'd rather use text_compare.pl you can write a very similar
script that calls it with system( ) rather than what I've done above.
So, hopefully this gives you some ideas about how to proceed!
Good luck,
Ted
--
Ted Pedersen
http://www.d.umn.edu/~tpederse
