I'm not sure what you mean. CLucene StandardTokenizer is meant for internal use only, and provides the calling Analyzer with a stream of identified tokens (it classifies the tokens, not just tokenizes them).
The ICU tokenizer is a general purpose tokenizer (like Boost's implementation is), with loads of extra functionality the CLucene one doesn't have or need. Itamar. -----Original Message----- From: Paul J. Lucas [mailto:p...@lucasmail.org] Sent: Tuesday, February 09, 2010 8:24 PM To: clucene-developers@lists.sourceforge.net Subject: [CLucene-dev] CLucene tokenizer vs ICU tokenizer How do they compare? - Paul ---------------------------------------------------------------------------- -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com _______________________________________________ CLucene-developers mailing list CLucene-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/clucene-developers ------------------------------------------------------------------------------ SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev _______________________________________________ CLucene-developers mailing list CLucene-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/clucene-developers
