[
https://issues.apache.org/jira/browse/MAHOUT-9?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Ingersoll updated MAHOUT-9:
---------------------------------
Attachment: MAHOUT-9.patch
OK, here, finally, is a first draft. It is undoubtedly naive, as my Hadoop
skills were a bit rusty. At any rate, I haven't done NB before, so I used the
basic gist of Word Count and the example in "Programming Coll. Intell." and
wrote it up in Java on Hadoop. I also added a demo example using the 20
newsgroups data. See the TwentyNewsgroups.java file javadocs for details on
running. I added some ANT tasks to support the example as well. I will try to
document this up more.
At any rate, here are some things I am aware of:
1. the storage in the sequence file. I currently am using a String array as
features and not numerical features. I'm sure there is an optimization to be
had here (probably could use Lucene somehow, too), and probably could even make
it use generics, but I wanted to put something up first
2. I need to more thoroughly test the Classifier to make sure the probabilities
are right on. Am also debating adding thresholding capabilities as per the PCI
suggestion. For now, it returns the top category. I don't know what best
practice is here. Seems like it would be useful to have multiple categories
plus their scores so that an application could decide (i.e. push of the
thresholding problem to the app)
3. More documentation is needed. Javadocs and package level, as well as Wiki
updates
4. It doesn't do anything fancy in terms of validation, etc. Will need to read
up on some of the more advance techniques or leave that up to others.
5. I haven't tested at scale. I did test in both simple Hadoop mode and 1 node
distributed mode.
6. You need Lucene 2.3.1 (core and analyzers) for the example. I have added
this as a dep, but will upload here.
7. Ideally, the Twenty Newsgroups stuff would automatically create training
and test sets, but for now I just use it all to train. This is just some ant
monkey work IMO.
8. My Hadoop InputFormat is the KeyValueInputFormat. The BayesFileFormatter
can put files into this format. See the TwentyNewsgroups stuff for an example.
As an example of running, here's what I think I did (I'm tired and going from
memory):
>From the Mahout checkout core dir:
{quote}
ant extract-20news-18828
{quote}
This downloads and puts the 20 newsgroups stuff in a proper format in a ./work
directory
Then,
{quote}
ant examples-job
{quote}
Creates the Hadoop job jar.
Then, switch to your Hadoop download and basically follow the Hadoop quick
start guide (http://hadoop.apache.org/core/docs/r0.16.3/quickstart.html) but as:
{quote}
emacs conf/hadoop-site.xml //(add in local settings per quickstart)
bin/hadoop namenode -format //format your HDFS
bin/start-all.sh //Start Hadoop
bin/hadoop dfs -put <MAHOUT_HOME>/work/20news-18828-collapse 20newsInput
bin/hadoop jar <MAHOUT_HOME>/build/apache-mahout-0.1-dev-ex.jar
org.apache.mahout.classifiers.bayes.TwentyNewsgroups -t -i 20newsInput -o
20newsOutput
{quote}
MAHOUT_HOME is the location of where you did the ant commands above
Sorry for the long comment. Please let me know what to improve. I'm in no
hurry to commit, but would like to add it fairly soon.
> Implement MapReduce BayesianClassifier
> --------------------------------------
>
> Key: MAHOUT-9
> URL: https://issues.apache.org/jira/browse/MAHOUT-9
> Project: Mahout
> Issue Type: New Feature
> Reporter: Grant Ingersoll
> Assignee: Grant Ingersoll
> Priority: Minor
> Fix For: 0.1
>
> Attachments: MAHOUT-9.patch
>
>
> Implement a Bayesian classifier using M/R.
> I have a simple trainer done (not M/R) and will implement the classifier
> soon, then will upgrade it to use Hadoop.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
