NovelAnalyzer - wraps your choice of Lucene Analyzer and filters out all
"boilerplate" text
-------------------------------------------------------------------------------------------
Key: LUCENE-725
URL: http://issues.apache.org/jira/browse/LUCENE-725
Project: Lucene - Java
Issue Type: New Feature
Components: Analysis
Reporter: Mark Harwood
Priority: Minor
Attachments: NovelAnalyzer.java
This is a class I have found to be useful for analyzing small (in the hundreds)
collections of documents and removing any duplicate content such as standard
disclaimers or repeated text in an exchange of emails.
This has applications in sampling query results to identify key phrases,
improving speed-reading of results with similar content (eg email threads/forum
messages) or just removing duplicated noise from a search index.
To be more generally useful it needs to scale to millions of documents - in
which case an alternative implementation is required. See the notes in the
Javadocs for this class for more discussion on this
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
