[ https://issues.apache.org/jira/browse/LUCENE-6747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Harwood updated LUCENE-6747: --------------------------------- Attachment: fingerprintv2.patch Thanks for taking a look, Adrien. Added a v2 patch with following changes: 1) added call to input.end() to get final offset state 2) final state is retained using captureState() 3) Added a FingerprintFilterFactory class As for the alternative hashing idea : For speed reasons this would be a nice idea but reduces the read-ability of results if you want to debug any collisions or otherwise display connections. For compactness reasons (storing in doc values etc) it would always be possible to chain a conventional hashing algo in a TokenFilter on the end of this text-normalizing filter. (Do we already have a conventional hashing TokenFilter?) > FingerprintFilter - a TokenFilter for clustering/linking purposes > ----------------------------------------------------------------- > > Key: LUCENE-6747 > URL: https://issues.apache.org/jira/browse/LUCENE-6747 > Project: Lucene - Core > Issue Type: New Feature > Components: modules/analysis > Reporter: Mark Harwood > Priority: Minor > Attachments: fingerprintv1.patch, fingerprintv2.patch > > > A TokenFilter that emits a single token which is a sorted, de-duplicated set > of the input tokens. > This approach to normalizing text is used in tools like OpenRefine[1] and > elsewhere [2] to help in clustering or linking texts. > The implementation proposed here has a an upper limit on the size of the > combined token which is output. > [1] https://github.com/OpenRefine/OpenRefine/wiki/Clustering-In-Depth > [2] > https://rajmak.wordpress.com/2013/04/27/clustering-text-map-reduce-in-python/ -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org