Space: Apache Lucene Mahout (http://cwiki.apache.org/confluence/display/MAHOUT) Page: TwentyNewsgroups (http://cwiki.apache.org/confluence/display/MAHOUT/TwentyNewsgroups)
Change Comment: --------------------------------------------------------------------- Clean up to (not including) "Running 20newsgroups example over hadoop cluster" Edited by Jerry Chen: --------------------------------------------------------------------- h1. Twenty Newsgroups Classification h2. Prerequisites * Mahout has been downloaded ([instructions here|http://cwiki.apache.org/confluence/display/MAHOUT/index#index-Installation%2FSetup]) * Maven is available * Your environment has the following variables: | {{$HADOOP_HOME}} | Environment variables refers to where Hadoop lives | | {{$MAHOUT_HOME}} | Environment variables refers to where Mahout lives | h2. Setting up Mahout After downloading the distribution, unzip/untar it into the directory of your choice and do: # In the trunk directory, compile everything and create the Hadoop job: {noformat}mvn install{noformat} h3. For Mahout 0.1: {HTMLcomment} # If you've run this before, you may want to rm -rf the work and temp directories {HTMLcomment} In {{$MAHOOT_HOME/examples}}: {noformat} $ cd examples $ ant -f build-deprecated.xml get-files $ mkdir lib $ mvn dependency:copy-dependencies -DoutputDirectory=lib $ ant -f build-deprecated.xml extract-20news-18828 -Ddest=target $ mv 20news-18828-collapse 20news-input {noformat} NOTE: After you have done this, skip to the hadoop section to run the 20newsgroups example in mahout 0.1 release h3. For Mahout 0.2+: # Change into {{$MAHOOT_HOME/examples}} directory {noformat}$ cd examples{noformat} # Download {{20news-18828.tar.gz}} from the [20newsgroups dataset|http://people.csail.mit.edu/jrennie/20Newsgroups/] # Extract dataset: {noformat}$ tar zxf 20news-18828.tar.gz{noformat} # Generate input dataset: {noformat} $ mkdir 20news-input $ mvn -e exec:java\ -Dexec.mainClass=org.apache.mahout.classifier.bayes.PrepareTwentyNewsgroups\ -Dexec.args="-p 20news-18828\ -o 20news-input\ -a org.apache.lucene.analysis.standard.StandardAnalyzer\ -c UTF-8" {noformat} h2. Running the example without hadoop This assumes you're running Mahout 0.2+: # Train the classifier: {noformat} $ mvn -e exec:java \ -Dexec.mainClass=org.apache.mahout.classifier.bayes.TrainClassifier \ -Dexec.args="-i 20news-input \ -o 20news-model \ -type cbayes \ -ng 1 \ -source hdfs" {noformat} # Test over input: {noformat} $ mvn -e exec:java \ -Dexec.mainClass=org.apache.mahout.classifier.bayes.TestClassifier \ -Dexec.args="-m 20news-model \ -d 20news-input \ -type cbayes \ -ng 1 \ -source hdfs \ -method sequential" {noformat} h2. Running 20newsgroups example over hadoop cluster h3. Set Up Hadoop Cluster # emacs $HADOOP_HOME/conf/hadoop-site.xml (add in local settings per [quickstart|http://hadoop.apache.org/core/docs/current/quickstart.html]) # $HADOOP_HOME/bin/hadoop namenode -format //Format the HDFS # $HADOOP_HOME/bin/start-all.sh //Start Hadoop # $HADOOP_HOME/bin/hadoop dfs -put $MAHOUT_HOME/examples/20news-input 20news-input //Copies the extracted text to HDFS Example: Train the Bayes Classifier using tri-grams: {noformat}$HADOOP_HOME/bin/hadoop jar $MAHOUT_HOME/examples/target/mahout-examples-0.1.job org.apache.mahout.classifier.bayes.TrainClassifier -i 20news-input -o newsmodel -ng 3 -type bayes{noformat} This will run 4 map reduce jobs on Hadoop to train the classifier and will take a while on a single node machine. You can monitor the status of these jobs by opening a web browser on your Job Tracker node: http://localhost:50030/jobtracker.jsp Test classifier over the input folder {noformat}$HADOOP_HOME/bin/hadoop jar $MAHOUT_HOME/examples/target/mahout-examples-0.1.job org.apache.mahout.classifier.bayes.TestClassifier -p newsmodel -t work/20news-input -ng 3 -type bayes{noformat} Output might look like: {noformat} 08/11/07 16:52:39 INFO bayes.TestClassifier: Done loading model: # labels: 20 08/11/07 16:52:39 INFO bayes.TestClassifier: Done generating Model 08/11/07 16:52:57 INFO bayes.TestClassifier: alt.atheism96.9962453066333775/799.0 08/11/07 16:53:15 INFO bayes.TestClassifier: comp.graphics99.28057553956835966/973.0 08/11/07 16:53:45 INFO bayes.TestClassifier: comp.os.ms-windows.misc96.95431472081218955/985.0 08/11/07 16:53:59 INFO bayes.TestClassifier: comp.sys.ibm.pc.hardware99.59266802443992978/982.0 08/11/07 16:54:10 INFO bayes.TestClassifier: comp.sys.mac.hardware99.47970863683663956/961.0 08/11/07 16:54:28 INFO bayes.TestClassifier: comp.windows.x99.59183673469387976/980.0 08/11/07 16:54:38 INFO bayes.TestClassifier: misc.forsale98.45679012345678957/972.0 08/11/07 16:54:50 INFO bayes.TestClassifier: rec.autos99.4949494949495985/990.0 08/11/07 16:55:04 INFO bayes.TestClassifier: rec.motorcycles100.0994/994.0 08/11/07 16:55:16 INFO bayes.TestClassifier: rec.sport.baseball99.89939637826961993/994.0 08/11/07 16:55:36 INFO bayes.TestClassifier: rec.sport.hockey99.89989989989989998/999.0 08/11/07 16:55:54 INFO bayes.TestClassifier: sci.crypt99.39455095862765985/991.0 08/11/07 16:56:05 INFO bayes.TestClassifier: sci.electronics98.98063200815494971/981.0 08/11/07 16:56:27 INFO bayes.TestClassifier: sci.med99.79797979797979988/990.0 08/11/07 16:56:44 INFO bayes.TestClassifier: sci.space99.3920972644377981/987.0 08/11/07 16:57:06 INFO bayes.TestClassifier: soc.religion.christian99.49849548645938992/997.0 08/11/07 16:57:24 INFO bayes.TestClassifier: talk.politics.guns99.45054945054945905/910.0 08/11/07 16:57:51 INFO bayes.TestClassifier: talk.politics.mideast98.82978723404256929/940.0 08/11/07 16:58:13 INFO bayes.TestClassifier: talk.politics.misc89.93548387096774697/775.0 08/11/07 16:58:25 INFO bayes.TestClassifier: talk.religion.misc61.78343949044586388/628.0 08/11/07 16:58:25 INFO bayes.TestClassifier: ======================================================= Summary ------------------------------------------------------- Correctly Classified Instances : 18369 97.5621% Incorrectly Classified Instances : 459 2.4379% Total Classified Instances : 18828 ======================================================= Confusion Matrix ------------------------------------------------------- a b c d e f g h i j k l m n o p q r s t <--Classified as 994 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 | 994 a = rec.motorcycles 0 976 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 2 1 | 980 b = comp.windows.x 7 0 929 1 0 0 0 0 0 0 0 0 1 0 2 0 0 0 0 0 | 940 c = talk.politics.mideast 0 0 0 905 0 0 1 0 0 0 0 0 0 0 0 0 3 0 1 0 | 910 d = talk.politics.guns 4 1 4 27 388 1 0 1 0 5 1 1 2 2 149 7 2 33 0 0 | 628 e = talk.religion.misc 3 0 0 0 0 985 0 1 0 0 0 0 0 1 0 0 0 0 0 0 | 990 f = rec.autos 0 0 0 0 0 0 993 1 0 0 0 0 0 0 0 0 0 0 0 0 | 994 g = rec.sport.baseball 0 0 0 0 0 0 1 998 0 0 0 0 0 0 0 0 0 0 0 0 | 999 h = rec.sport.hockey 0 0 0 0 0 0 0 0 956 0 2 0 0 0 0 0 0 0 2 1 | 961 i = comp.sys.mac.hardware 0 0 0 0 0 0 0 0 0 981 0 0 5 0 0 1 0 0 0 0 | 987 j = sci.space 0 0 0 0 0 0 0 0 0 0 978 0 1 0 0 0 0 0 2 1 | 982 k = comp.sys.ibm.pc.hardware 1 0 3 36 0 1 2 1 0 5 0 697 4 0 3 3 19 0 0 0 | 775 l = talk.politics.misc 0 2 0 0 0 0 0 0 0 0 2 0 966 0 0 0 0 0 2 1 | 973 m = comp.graphics 1 0 0 0 0 0 0 0 0 0 6 0 0 971 0 0 0 0 3 0 | 981 n = sci.electronics 1 0 0 0 0 0 0 0 1 0 0 0 0 0 992 1 0 1 0 1 | 997 o = soc.religion.christian 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 988 0 0 0 1 | 990 p = sci.med 0 0 0 2 0 0 0 0 0 0 0 0 2 1 0 0 985 0 1 0 | 991 q = sci.crypt 0 0 0 1 1 0 0 0 0 1 0 0 1 0 19 0 1 775 0 0 | 799 r = alt.atheism 1 0 0 0 0 3 1 2 0 0 3 0 0 5 0 0 0 0 957 0 | 972 s = misc.forsale 0 0 0 8 0 0 0 0 0 0 6 0 6 0 0 0 0 0 10 955 | 985 t = comp.os.ms-windows.misc {noformat} h2. Complementary Naive Bayes To Train a CBayes Classifier using bi-grams {noformat}$HADOOP_HOME/bin/hadoop jar $MAHOUT_HOME/examples/target/mahout-examples-0.1.job org.apache.mahout.classifier.bayes.TrainClassifier -i 20news-input -o 20news-model -ng 2 -type cbayes -source <hdfs|hbase>{noformat} To Test a CBayes Classifier using bi-grams {noformat}$HADOOP_HOME/bin/hadoop jar $MAHOUT_HOME/examples/target/mahout-examples-0.1.job org.apache.mahout.classifier.bayes.TestClassifier -p 20news-model -t work/20news-input -ng 2 -type cbayes -source <hdfs|hbase> -method <sequential|mapreduce>{noformat} Change your notification preferences: http://cwiki.apache.org/confluence/users/viewnotifications.action
