Hi Adam, Did you see this? I haven't tested it. Just was curious to look around after your post.
http://stackoverflow.com/questions/1550950/detect-chinese-multibyte-character-in-the-string Matt Terenzio On Thu, Mar 24, 2011 at 10:50 AM, Adam Green <140...@gmail.com> wrote: > This has been a problem with collecting tweets from the API since I > started working with it. My users only want English tweets and they > view non-English tweets that I deliver to be a bug in my software. The > lang=en argument in the search API only filters a small percentage of > this, and I know of no way to do any filtering in the streaming API. I > started working with the PHP library call LanguageDetect a few days > ago, and it is doing a great job. > > http://pear.php.net/package/Text_LanguageDetect/ > > I tested it by filtering 40,000 recent tweets about @barackobama from > my 2012twit.com site, and it found almost 20% of the tweets to be non- > English. I screened the ones it found as non-English by hand, and > found less than a 1% false positive rate. That means I lost 0.2% of > the total flow to false positives to eliminate a 20% non-English rate. > Pretty good for a solution that is small, about 2,500 lines of code, > fast, open source, and free. I use it in my tweet parse phase of tweet > collection. First I gather tweets into a MySQL cache with Phirehose, > and then I parse the cached tweets into a normalized scheme. During > this parsing phase I screen each tweet with LanguageDetect. The > additional processing time of language detection is unnoticeable. > > The only limitation I found is that it doesn't detect Chinese or > Japanese, but I think I can find other solutions for this. If anyone > knows of a simple PHP detection algorithm for these languages, please > let me know. > > - Adam Green > Twitter API Developer > http://2012twit.com > http://140dev.com > @140dev > > -- > Twitter developer documentation and resources: http://dev.twitter.com/doc > API updates via Twitter: http://twitter.com/twitterapi > Issues/Enhancements Tracker: > http://code.google.com/p/twitter-api/issues/list > Change your membership to this group: > http://groups.google.com/group/twitter-development-talk > -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
