I am still trying to fully understand minHash algorithm and I had the same results like below when running the MinHashDriver.
I have a use case wherein I need to determine the content similarity of 2 documents like what's been described in Andrei Broder's paper - 'Identifying and Filtering Near-Duplicate Documents' (http://dl.acm.org/citation.cfm?id=736184). I started dissecting the clusters generated by Mahout's MinHashDriver to compare document content equality and to determine how accurate the clustering was? I do see that the first 2 files from the output below were put in the same cluster 106460162-207863047; thought the actual text content in both the files is different. How? I am assuming that the NGram attribute was set to the default value of 1 when creating the tf-idf vectors from sequence files. Suneel ________________________________ From: Grant Ingersoll <[email protected]> To: [email protected] Sent: Tuesday, October 25, 2011 5:55 AM Subject: Re: MinHash Clustering in Mahout On Oct 19, 2011, at 11:38 AM, Varun Thacker wrote: > I was trying to run the MinHash algorithm on the Reuters data set, so I did > the following before running MinHashDriver > > - Get the Reuters dataset > - Run org.apache.lucene.benchmark.utils.ExtractReuters to generate > reuters-out from reuters-sgm(the downloaded archive) > - Run seqdirectory to convert reuters-out to SequenceFile format > - Run seq2sparse to convert SequenceFiles to sparse vector format > > I used these instructions from the K-means clustering wiki page. > > This is the command I used to run MinHashDriver > > ./mahout org.apache.mahout.clustering.minhash.MinHashDriver --input > /home/varun/mahout/sparse/tfidf-vectors/ -o /home/varun/mahout/minhash > > The output file looks something like this: > > 106460162-207863047 /reut2-015.sgm-653.txt > 106460162-207863047 /reut2-021.sgm-7.txt > 106460162-207863047 /reut2-013.sgm-307.txt > 106460162-207863047 /reut2-013.sgm-306.txt > 106460162-207863047 /reut2-014.sgm-786.txt > 106460162-207863047 /reut2-013.sgm-304.txt > 106460162-207863047 /reut2-013.sgm-303.txt > 106460162-207863047 /reut2-021.sgm-230.txt > 106460162-207863047 /reut2-012.sgm-548.txt > 106460162-207863047 /reut2-020.sgm-161.txt > 106460162-207863047 /reut2-021.sgm-553.txt > 106460162-207863047 /reut2-013.sgm-299.txt > 106460162-207863047 /reut2-015.sgm-284.txt > 106460162-207863047 /reut2-013.sgm-996.txt > 106460162-207863047 /reut2-021.sgm-441.txt > 106460162-207863047 /reut2-013.sgm-298.txt > 106460162-207863047 /reut2-013.sgm-995.txt > 106460162-207863047 /reut2-015.sgm-521.txt > 106460162-207863047 /reut2-020.sgm-162.txt > 106460162-207863047 /reut2-020.sgm-163.txt > 106460162-207863047 /reut2-013.sgm-296.txt > ... > ... > > > Is this the correct way of running MinHash. > > If yes then I would update the wiki page > https://cwiki.apache.org/confluence/display/MAHOUT/Minhash+Clustering with > the instructions. > > Otherwise if someone could tell me on what am I doing wrong. I haven't looked into the code, but I get similar outputs, so I assume it is working. Might be good to incorporate this into the build-reuters.sh as well as try it on some other input. -Grant
