[
https://issues.apache.org/jira/browse/MAHOUT-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szymon Chojnacki updated MAHOUT-588:
------------------------------------
Attachment: TamingDictionaryVectorizer.java
TamingGramKeyGroupComparator.java
TamingCollocMapper.java
TamingCollocDriver.java
TamingDictVect.java
TamingDictVect invokes TamingDictionaryVectorizer. Parameters are hard-coded.
It is assumed that the emails were tokenized with TamingToken into TamingVEC.
TamingDictionaryVectorizer is a modified DictionaryVectorizer used in
seq2sparse. Proposed implementation builds collocations only for words with a
parametrized frequency, which requires one more preprocessing MR-job. This
optimization is based on an observation that if we wish to restore n-grams with
minCount>100, than each token in the n-gram must have count>100. In practice we
can assume that the count of single tokens is few times higher than desired
n-gram count.
The implementation differences are as follows:
/* Modified DictionaryVectorizer, now collocations are built only for tokens
with defined minimum support. Most changes are enclosed by a commented
informative statement. Following classes where also modified:
1. TamingCollocDriver
- modified CollocDriver
- changed CollocMapper to TamingCollocMapper
- added frequent tokens to DistributedCache
2. TamingCollocMapper
- modified CollocMapper
- in setup() lilmited dictionary is loaded
- only terms with all tokens in the dictionary are sent forward to reducers
- used terminology: term is e.g. "Coca cola" tokens are "coca" "cola"
3. TamingGramKeyGroupComparator
- unmodified GramKeyGroupComparator
- just moved to current package to obtain visibility for TamingCollocDriver
*/
> Benchmark Mahout's clustering performance on EC2 and publish the results
> ------------------------------------------------------------------------
>
> Key: MAHOUT-588
> URL: https://issues.apache.org/jira/browse/MAHOUT-588
> Project: Mahout
> Issue Type: Task
> Reporter: Grant Ingersoll
> Attachments: SequenceFilesFromMailArchives.java,
> SequenceFilesFromMailArchives2.java, TamingAnalyzer.java,
> TamingCollocDriver.java, TamingCollocMapper.java, TamingDictVect.java,
> TamingDictionaryVectorizer.java, TamingGramKeyGroupComparator.java,
> TamingTokenizer.java, Top1000Tokens_maybe_stopWords, Uncompress.java,
> clusters_kMeans.txt, distcp_large_to_s3_failed.log,
> seq2sparse_small_failed.log, seq2sparse_xlarge_ok.log
>
>
> For Taming Text, I've commissioned some benchmarking work on Mahout's
> clustering algorithms. I've asked the two doing the project to do all the
> work in the open here. The goal is to use a publicly reusable dataset (for
> now, the ASF mail archives, assuming it is big enough) and run on EC2 and
> make all resources available so others can reproduce/improve.
> I'd like to add the setup code to utils (although it could possibly be done
> as a Vectorizer) and the publication of the results will be put up on the
> Wiki as well as in the book. This issue is to track the patches, etc.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
