[
https://issues.apache.org/jira/browse/MAHOUT-242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804191#action_12804191
]
Ted Dunning commented on MAHOUT-242:
------------------------------------
{quote}
Each of the existing attributes are serializable which suggests that there
might be a short path to Writable for these, I'll see if Avro offers any
shortcuts here. A subclass of IndexWriter may indeed be a very quick route to
this.
{quote}
Perhaps. But the fact that Avro data is self describing and highly portable is
probably as important as most other considerations. There are also lots of
concessions to Java that are in the Lucene document that are very different
from the concessions to good serialization that we need.
{quote}
I also wonder if flipping this around and implementing a LuceneIndexInputFormat
may be another option.
{quote}
Yes. Or at least a conversion system for spilling an index into SequenceFiles.
> LLR Collocation Identifier
> --------------------------
>
> Key: MAHOUT-242
> URL: https://issues.apache.org/jira/browse/MAHOUT-242
> Project: Mahout
> Issue Type: New Feature
> Affects Versions: 0.3
> Reporter: Drew Farris
> Priority: Minor
> Attachments: MAHOUT-242.patch, MAHOUT-242.patch,
> mahout-colloc.tar.gz, mahout-colloc.tar.gz
>
>
> Identifies interesting Collocations in text using ngrams scored via the
> LogLikelihoodRatio calculation.
> As discussed in:
> *
> http://www.lucidimagination.com/search/document/d051123800ab6ce7/collocations_in_mahout#26634d6364c2c0d2
> *
> http://www.lucidimagination.com/search/document/b8d5bb0745eef6e8/n_grams_for_terms#f16fa54417697d8e
> Current form is a tar of a maven project that depends on mahout. Build as
> usual with 'mvn clean install', can be executed using:
> {noformat}
> mvn -e exec:java -Dexec.mainClass="org.apache.mahout.colloc.CollocDriver"
> -Dexec.args="--input src/test/resources/article --colloc target/colloc
> --output target/output -w"
> {noformat}
> Output will be placed in target/output and can be viewed nicely using:
> {noformat}
> sort -rn -k1 target/output/part-00000
> {noformat}
> Includes rudimentary unit tests. Please review and comment. Needs more work
> to get this into patch state and integrate with Robin's document vectorizer
> work in MAHOUT-237
> Some basic TODO/FIXME's include:
> * use mahout math's ObjectInt map implementation when available
> * make the analyzer configurable
> * better input validation + negative unit tests.
> * more flexible ways to generate units of analysis (n-1)grams.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
