Space: Apache Mahout (https://cwiki.apache.org/confluence/display/MAHOUT)
Page: Quick tour of Mahout text processing from the command line
(https://cwiki.apache.org/confluence/display/MAHOUT/Quick+tour+of+Mahout+text+processing+from+the+command+line)
Added by Pat Ferrel:
---------------------------------------------------------------------
h1. {color:#000000}{*}Quick tour of Mahout text processing from the command
line{*}{color}
{color:#000000}This is a concise quick tour of using the mahout command line to
generate text analysis data. It follows examples from the{color} [Mahout in
Action|http://manning.com/owen/]{color:#000000}book and uses the Reuters-21578
data set. This is one simple path through vectorizing text, creating clusters
and calculating similar documents. The examples will work locally or
distributed on a hadoop cluster. With the small data set provided a local
installation is probably fast enough.{color}
h1. {color:#000000}{*}Generate Mahout sequence files from text{*}{color}
{color:#000000}Get the{color}
[Reuters-21578|http://www.daviddlewis.com/resources/testcollections/reuters21578/]
{color:#000000}(http://www.daviddlewis.com/resources/testcollections/reuters21578/)
files and extract them in “./reuters”. They are in SGML format.{color}#
{color:#000000}We first convert from SGML to text:{color}
{color:#000000}mvn \-e \-q exec:java
\-Dexec.mainClass="org.apache.lucene.benchmark.utils.ExtractReuters"
\-Dexec.args="reuters/ reuters-extracted/"{color}
{color:#000000}If you plan to run this example on a hadoop cluster you will
need to copy the files to HDFS, which is not covered here.{color}
# {color:#000000}Now turn raw text in a directory into mahout sequence
files:{color}
{color:#000000}mahout seqdirectory \{color}
{color:#000000} -c UTF-8 \{color}
{color:#000000} -i examples/reuters-extracted/ \{color}
{color:#000000} -o reuters-seqfiles{color}
# {color:#000000}Examine the sequence files with seqdumper:{color}
{color:#000000}mahout seqdumper \-s reuters-seqfiles/chunk-0 \| more{color}
{color:#000000}you should see something like this:{color}
{color:#000000}Input Path: reuters-seqfiles/chunk-0{color}
{color:#000000}Key class: class org.apache.hadoop.io.Text Value Class: class
org.apache.hadoop.io.Text{color}
{color:#000000}Key: /-tmp/reut2-000.sgm-0.txt: Value: 26-FEB-1987
15:01:01.79{color}
{color:#000000}BAHIA COCOA REVIEW{color}
{color:#000000}Showers continued throughout the week in the Bahia cocoa zone,
alleviating the drought since early January and improving prospects for the
coming temporao, although normal ...{color}
# {color:#000000}Create tfidf vectors.{color}
{color:#000000}mahout seq2sparse \{color}
{color:#000000} -i reuters-seqfiles/ \{color}
{color:#000000} -o reuters-vectors/ \{color}
{color:#000000} -ow \-chunk 100 \{color}
{color:#000000} -x 90 \{color}
{color:#000000} -seq \{color}
{color:#000000} -a
com.finderbots.analyzers.LuceneStemmingAnalyzer \{color}
{color:#000000} -ml 50 \{color}
{color:#000000} -n 2 \{color}
{color:#000000} -nv{color}
{color:#000000}This uses a custom lucene analyzer which incorporates several
token filters to stem, toss numbers, stop words (from a list), and small words.
n = 2 is best for cosine distance, which we are using in clustering and for
similarity. x is 90 meaning that if a token appears in 90% of the docs it is
considered a stop word. ml = 50 \-\- not sure what this does...{color}
{color:#000000}Note:{color} {color:#000000}get named vectors or it is difficult
to map docs to clusters{color}
# {color:#000000}Examine the vectors if you like but they are not really human
readable...{color}
{color:#000000}mahout seqdumper \-s reuters-seqfiles/part-r-00000{color}
# {color:#000000}Examine the tokenized docs to make sure the custom analyzer
did right:{color}
{color:#000000}mahout seqdumper \{color}
{color:#000000} -s
reuters-vectors/tokenized-documents/part-m-00000{color}
{color:#000000}This should show each doc with nice clean tokenized text with no
numbers, stemmed, etc.{color}
# {color:#000000}Make sure to look at the dictionary. It has every token with
the integer that references it. All the vectors will use the integer, not the
token so a lookup is required to see what is really inside a vector.{color}
{color:#000000}mahout seqdumper \{color}
{color:#000000} -s reuters-vectors/dictionary.file-0 \{color}
{color:#000000} \| more{color}
h1. {color:#000000}{*}Cluster the documents using kmeans{*}{color}
# {color:#000000}Calculate clusters and put document into the them.{color}
{color:#000000}mahout kmeans \{color}
{color:#000000} -i reuters-vectors/tfidf-vectors/ \{color}
{color:#000000} -c reuters-kmeans-centroids \{color}
{color:#000000} -cl \{color}
{color:#000000} -o reuters-kmeans-clusters \{color}
{color:#000000} -k 20 \{color}
{color:#000000} -ow \{color}
{color:#000000} -x 10 \{color}
{color:#000000} -dm
org.apache.mahout.common.distance.CosineDistanceMeasure{color}
{color:#000000}This calculates cluster centroids and puts them in the output
dir it then finds which vectors are included in the final clusters and puts
them in output/clusteredPoints. If you leav out \-cl you will not get the
mapping of doc to cluster.{color}
{color:#000000}Note: fuzzy kmeans it is pretty sensitive to the fuzzyness
measure so you can get meaningless clusters so look at the \-m parameter to
fkmeans before trying it. m = 2 produced garbage results.{color}
# {color:#000000}Examine the clusters and perhaps even do some anaylsis of how
good the clusters are:{color}
{color:#000000}mahout clusterdump \{color}
{color:#000000} -d reuters-vectors/dictionary.file-0 \{color}
{color:#000000} -dt sequencefile \{color}
{color:#000000} -s
reuters-kmeans-clusters/clusters-3-final/part-r-00000 \{color}
{color:#000000} -n 20 \{color}
{color:#000000} -b 100 \{color}
{color:#000000} -p
reuters-kmeans-clusters/clusteredPoints/{color}
# {color:#000000}The clusteredPoints dir has the docs mapped into clusters, and
if you created vectors with names (seq2sparse \-nv) you’ll see them. You also
have the distance from the centroid using the distance measure supplied to the
clustering driver. To look at this use seqdumper:{color}
{color:#000000}mahout seqdumper \{color}
{color:#000000} -s
reuters-kmeans-clusters/clusteredPoints/part-m-00000 \{color}
{color:#000000} \| more{color}
{color:#000000}You will see that the file contains{color}
{color:#000000} key: clusterid, value: wt = % likelihood the
vector is in cluster, distance from centroid, named vector belonging to the
cluster, vector data.{color}
{color:#000000}For kmeans the likelihood will be 1.0 or 0. For example:{color}
{color:#000000} Key: 21477: Value: wt: 1.0distance:
0.9420744909793364 vec: /-tmp/reut2-000.sgm-158.txt = \[372:0.318,
966:0.396, 3027:0.230, 8816:0.452, 8868:0.308, 13639:0.278, 13648:0.264,
14334:0.270, 14371:0.413\]{color}
{color:#000000}Clusters, of course, cannot have names. A simple solution is to
construct a name from the top terms in the centroid output from
clusterdump.{color}
h1. {color:#000000}{*}Calculate several similar docs to each doc in the
data{*}{color}
{color:#000000}This will take all docs in the data set then for each calculate
the 10 most similar docs. This is like “find more like this” type search but is
calculated in the background. This seems to be fast and requires only three
mapreduce passes.{color}# {color:#000000}First create a matrix from the
vectors:{color}
{color:#000000}mahout rowid \{color}
{color:#000000} -i
reuters-vectors/tfidf-vectors/part-r-00000{color}
{color:#000000} -o reuters-matrix{color}
{color:#000000}You’ll get output announcing the number of columns/dimensions in
the doc collection stored in the matrix. I looks like this:{color}
{color:#000000}Wrote out matrix with 21578 rows and 19515 columns to
reuters-matrix/matrix{color}
{color:#000000}Save the number of column since it is needed in the next step.
Also note that this creates a reuters-matrix/docIndex file where the rowids are
mapped to docids. In the case of this example it will be rowid-->file name
since we created named vectors in seq2sparse.{color}
{color:#000000}Note: This does not create a Mahout Matrix class but a sequence
file so use seqdumper to examine the results.{color}
# {color:#000000}Create a collection of similar docs to each row of the matrix
above:{color}
{color:#000000}mahout rowsimilarity \{color}
{color:#000000} -i reuters-named-matrix/matrix \{color}
{color:#000000} -o reuters-named-similarity \{color}
{color:#000000} -r 19515{color}
{color:#000000} --similarityClassname SIMILARITY_COSINE{color}
{color:#000000} -m 10{color}
{color:#000000} -ess{color}
{color:#000000}This will generate the 10 most similar docs to each doc in the
collection.{color}
# {color:#000000}Examine the similarity list:{color}
{color:#000000}mahout seqdumper \-s reuters-matrix/matrix \| more{color}
{color:#000000}Which should look something like this{color}
{color:#000000} {color}{color:#000000}Key: 0: Value:
{14458:0.2966480826934176,11399:0.30290014772966095,{color}
{color:#000000} 12793:0.22009858979452146,3275:0.1871791030103281,{color}
{color:#000000} 14613:0.3534278632679437,4411:0.2516380602790199,{color}
{color:#000000} 17520:0.3139731583634198,13611:0.18968888212315968,{color}
{color:#000000} 14354:0.17673965754661425,0:1.0000000000000004}{color}
{color:#000000}For each rowid there is a list of ten rowids and distances.
These corespond to documents and distance created by the
\--similarityCalssname. In this case they are cosines of the angle between doc
and similar doc. Look in the reuters-matrix/docIndex to find rowid to docid
mapping. It should look something like this:{color}
{color:#000000} Key: 0: Value: /-tmp/reut2-000.sgm-0.txt{color}
{color:#000000} Key: 1: Value: /-tmp/reut2-000.sgm-1.txt{color}
{color:#000000} Key: 2: Value:
/-tmp/reut2-000.sgm-10.txt{color}
{color:#000000} Key: 3: Value:
/-tmp/reut2-000.sgm-100.txt{color}
Change your notification preferences:
https://cwiki.apache.org/confluence/users/viewnotifications.action
