[gate-cvs] SF.net SVN: gate:[14882] mimir/trunk/mimir-core/src/gate/mimir/util/ NormalizingTermProcessor.java

[markagreenwood]

Revision: 14882
          http://gate.svn.sourceforge.net/gate/?rev=14882&view=rev
Author:   markagreenwood
Date:     2012-01-05 12:55:46 +0000 (Thu, 05 Jan 2012)
Log Message:
-----------
added the MG4J term processor that does accent normalization

Added Paths:
-----------
    mimir/trunk/mimir-core/src/gate/mimir/util/NormalizingTermProcessor.java

Added: mimir/trunk/mimir-core/src/gate/mimir/util/NormalizingTermProcessor.java
===================================================================
--- mimir/trunk/mimir-core/src/gate/mimir/util/NormalizingTermProcessor.java    
                        (rev 0)
+++ mimir/trunk/mimir-core/src/gate/mimir/util/NormalizingTermProcessor.java    
2012-01-05 12:55:46 UTC (rev 14882)
@@ -0,0 +1,75 @@
+/*
+ *  NormalizingTermProcessor.java
+ *
+ *  Copyright (c) 2012, The University of Sheffield.
+ *
+ *  This file is part of GATE Mímir (see http://gate.ac.uk/family/mimir.html), 
+ *  and is free software, licenced under the GNU Lesser General Public License,
+ *  Version 3, June 2007 (also included with this distribution as file
+ *  LICENCE-LGPL3.html).
+ *
+ *  Mark A. Greenwood, 5 January 2012
+ *
+ *  $Id$
+ */
+package gate.mimir.util;
+
+import it.unimi.dsi.lang.MutableString;
+import it.unimi.dsi.mg4j.index.Index;
+import it.unimi.dsi.mg4j.index.TermProcessor;
+
+import java.io.InputStream;
+import java.lang.reflect.Method;
+import java.lang.reflect.UndeclaredThrowableException;
+import java.text.Normalizer;
+
+public class NormalizingTermProcessor implements TermProcessor
+{
+       private final static NormalizingTermProcessor INSTANCE = new 
NormalizingTermProcessor();
+
+       public final static TermProcessor getInstance()
+       {
+               return INSTANCE;
+       }
+
+       private NormalizingTermProcessor()
+       {
+               //nothing to do but this method forces people to use the 
getInstance() method;
+       }
+
+       @Override
+       public boolean processTerm(final MutableString term)
+       {
+               if (term == null) return false;
+
+               term.toLowerCase();
+
+               
//http://glaforge.appspot.com/article/how-to-remove-accents-from-a-string
+               term.replace(Normalizer.normalize(term, 
Normalizer.Form.NFD).replaceAll("\\p{InCombiningDiacriticalMarks}+", ""));
+
+               return true;
+       }
+
+       @Override
+       public boolean processPrefix(final MutableString prefix)
+       {
+               return processTerm(prefix);
+       }
+
+       private Object readResolve()
+       {
+               return INSTANCE;
+       }
+
+       @Override
+       public String toString()
+       {
+               return this.getClass().getName();
+       }
+
+       @Override
+       public NormalizingTermProcessor copy()
+       {
+               return this;
+       }
+}


Property changes on: 
mimir/trunk/mimir-core/src/gate/mimir/util/NormalizingTermProcessor.java
___________________________________________________________________
Added: svn:keywords
   + Id
Added: svn:eol-style
   + native

This was sent by the SourceForge.net collaborative development platform, the 
world's largest Open Source development site.


------------------------------------------------------------------------------
Ridiculously easy VDI. With Citrix VDI-in-a-Box, you don't need a complex
infrastructure or vast IT resources to deliver seamless, secure access to
virtual desktops. With this all-in-one solution, easily deploy virtual 
desktops for less than the cost of PCs and save 60% on VDI infrastructure 
costs. Try it free! http://p.sf.net/sfu/Citrix-VDIinabox
_______________________________________________
GATE-cvs mailing list
GATE-cvs@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/gate-cvs