[
https://issues.apache.org/jira/browse/NUTCH-2209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15127634#comment-15127634
]
ASF GitHub Bot commented on NUTCH-2209:
---------------------------------------
Github user chrismattmann commented on a diff in the pull request:
https://github.com/apache/nutch/pull/87#discussion_r51521777
--- Diff:
src/plugin/scoring-similarity/src/java/org/apache/nutch/scoring/similarity/cosine/Model.java
---
@@ -0,0 +1,135 @@
+package org.apache.nutch.scoring.similarity.cosine;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.lucene.analysis.TokenStream;
+import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
+import
org.apache.nutch.scoring.similarity.util.LuceneAnalyzerUtil.StemFilterType;
+import org.apache.nutch.scoring.similarity.util.LuceneTokenizer;
+import
org.apache.nutch.scoring.similarity.util.LuceneTokenizer.TokenizerType;
+import org.apache.tika.Tika;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class creates a model used to store Document vector representation
of the corpus.
+ * @author Sujen Shah
--- End diff --
no author tags.
> Improved Tokenization for Similarity Scoring plugin
> ---------------------------------------------------
>
> Key: NUTCH-2209
> URL: https://issues.apache.org/jira/browse/NUTCH-2209
> Project: Nutch
> Issue Type: Improvement
> Components: scoring
> Reporter: Sujen Shah
> Assignee: Sujen Shah
> Labels: memex
> Fix For: 1.13
>
>
> This patch would add Lucene based tokenization to the cosine similarity
> plugin and clean up the code currently present.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
