On Fri, Mar 21, 2008 at 8:11 AM, YORO BA <[EMAIL PROTECTED]> wrote: > > Hello all, > i am tried to use text-similarity to measure two text. Text1.txt with > 800 sentences and text2.txt with also 800 sentences. How can i use > text-similarity to do the measurement of score, f-measure, precision, > recall etc.... > Thanks you very much
Greetings Yoro, If you are using the command line program, text_compare.pl, you can see these values using the --verbose option. If you don't use this, it will return a single number which is the F-measure. For example, suppose my two input files are as follows : :::::::::::::: file1.txt :::::::::::::: The Asian Cricket Council (ACC) confirmed on Thursday that Pakistan would host the Asia Cup one-day tournament from June 24 to July 6. ACC backing for Pakistan came the week after Australia postponed their tour, which was due to start at the end of March, because of the troubled security situation. ACC chief executive Ashraf-ul-Haque told a news conference Pakistan and India would play in the same group in the six-team tournament which will be held in Karachi and Lahore. :::::::::::::: file2.txt :::::::::::::: The Asian Cricket Council has confirmed Pakistan will host this year.s Asia Cup. ACC officials met Pakistan Cricket Board bosses on Thursday and decided to hold the tournament in Lahore and Karachi from June 25 to July 6. It will be hoped the decision will seek to dispel security fears, which led Cricket Australia to postpone their tour to the country and which have been raised over Pakistan.s hosting of the ICC Champions Trophy in September. To see all the different measures for these two files, you would run like this: marimba(360): text_compare.pl --verbose --type Text::Similarity::Overlaps file1.txt file2.txt And you would get the following output: keys: 22 -->'acc' len(1) cnt(1) -->'and' len(1) cnt(2) -->'asia cup' len(2) cnt(1) -->'australia' len(1) cnt(1) -->'confirmed' len(1) cnt(1) -->'from june' len(2) cnt(1) -->'host' len(1) cnt(1) -->'in' len(1) cnt(2) -->'karachi' len(1) cnt(1) -->'lahore' len(1) cnt(1) -->'of the' len(2) cnt(1) -->'on thursday' len(2) cnt(1) -->'pakistan' len(1) cnt(3) -->'security' len(1) cnt(1) -->'the' len(1) cnt(3) -->'the asian cricket council' len(4) cnt(1) -->'their tour' len(2) cnt(1) -->'to' len(1) cnt(1) -->'to july 6' len(3) cnt(1) -->'tournament' len(1) cnt(1) -->'which' len(1) cnt(2) -->'will be' len(2) cnt(1) wc 1: 80 wc 2: 78 Raw score: 40 Precision: 0.512820512820513 Recall : 0.5 F-measure: 0.506329113924051 E-measure: 0.493670886075949 Cosine : 0.506369683541833 0.506329113924051 'keys' tells us the number of overlapping words or phrases, and the 'raw score' is simply the sum of the products of the length * frequency (len and cnt) for each overlap. Now, this 'raw score' is of course something you might want to vary, and in fact that's what we do with the lesk measure in WordNet-Similarity, we square the lengths in order to weight them more heavily. wc1 and wc2 are the lengths of the two files (word counts). You'll note that Text::Similarity makes all text lower case and removes some punctuation. That's something that's coded into the module now, although I'm thinking about making that an option, so you could either let us clean the text or leave it as is (so you could clean yourself, or not, depending on what you are doing.) Finally, you might want to remove some stopwords from your matches. That's pretty easy to do via the --stoplist option. You just create a file of stop words, 1 word per line. Please note that right now Text::Similarity is very fussy about blank spaces, so if your line has a blank space rather than a carriage return after the word, it will treat your stop word as 'of ' rather than 'of', and not remove as you expect. In any case, suppose my stoplist.txt file looks like this: will be on a i in of the to and which Note that these words should be lower case as by the time stoplist handling occurs the text has been cleaned.... marimba(362): text_compare.pl --verbose --stoplist stoplist.txt --type Text::Similarity::Overlaps file1.txt file2.txt keys: 15 -->'acc' len(1) cnt(1) -->'asia cup' len(2) cnt(1) -->'asian cricket council' len(3) cnt(1) -->'australia' len(1) cnt(1) -->'confirmed' len(1) cnt(1) -->'from june' len(2) cnt(1) -->'host' len(1) cnt(1) -->'july 6' len(2) cnt(1) -->'karachi' len(1) cnt(1) -->'lahore' len(1) cnt(1) -->'pakistan' len(1) cnt(3) -->'security' len(1) cnt(1) -->'their tour' len(2) cnt(1) -->'thursday' len(1) cnt(1) -->'tournament' len(1) cnt(1) wc 1: 58 wc 2: 55 Raw score: 23 Precision: 0.418181818181818 Recall : 0.396551724137931 F-measure: 0.407079646017699 E-measure: 0.592920353982301 Cosine : 0.407223183282994 0.407079646017699 So, that gives you a sense of what you can do with text_compare.pl - if you are programming Text::Similarity yourself via the method calls, you can pass the verbose option as the value of a hash entry and see this same kind of output. There is a small glitch that affects perl 5.6 that I'm working on today, and I should have a release that fixes that later today (in the unlikely event you are still using perl 5.6 - if you are, just upgrade to 5.10.0 for goodness sake :) As a part of 0.04 I'll be improving the documentation for text_compare.pl and adding some test cases - the test cases (found in /t) are actually a nice way to get ideas of how to use the code as well, so make sure to check those out. I hope this helps get you going, and it's good to hear you are using Text-Similarity! Cordially, Ted -- Ted Pedersen http://www.d.umn.edu/~tpederse
