Mark Harwood created LUCENE-6747:
------------------------------------
Summary: FingerprintFilter - a TokenFilter for clustering/linking
purposes
Key: LUCENE-6747
URL: https://issues.apache.org/jira/browse/LUCENE-6747
Project: Lucene - Core
Issue Type: New Feature
Components: modules/analysis
Reporter: Mark Harwood
Priority: Minor
A TokenFilter that emits a single token which is a sorted, de-duplicated set of
the input tokens.
This approach to normalizing text is used in tools like OpenRefine[1] and
elsewhere [2] to help in clustering or linking texts.
The implementation proposed here has a an upper limit on the size of the
combined token which is output.
[1] https://github.com/OpenRefine/OpenRefine/wiki/Clustering-In-Depth
[2]
https://rajmak.wordpress.com/2013/04/27/clustering-text-map-reduce-in-python/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
