This has been a problem with collecting tweets from the API since I started working with it. My users only want English tweets and they view non-English tweets that I deliver to be a bug in my software. The lang=en argument in the search API only filters a small percentage of this, and I know of no way to do any filtering in the streaming API. I started working with the PHP library call LanguageDetect a few days ago, and it is doing a great job.
http://pear.php.net/package/Text_LanguageDetect/ I tested it by filtering 40,000 recent tweets about @barackobama from my 2012twit.com site, and it found almost 20% of the tweets to be non- English. I screened the ones it found as non-English by hand, and found less than a 1% false positive rate. That means I lost 0.2% of the total flow to false positives to eliminate a 20% non-English rate. Pretty good for a solution that is small, about 2,500 lines of code, fast, open source, and free. I use it in my tweet parse phase of tweet collection. First I gather tweets into a MySQL cache with Phirehose, and then I parse the cached tweets into a normalized scheme. During this parsing phase I screen each tweet with LanguageDetect. The additional processing time of language detection is unnoticeable. The only limitation I found is that it doesn't detect Chinese or Japanese, but I think I can find other solutions for this. If anyone knows of a simple PHP detection algorithm for these languages, please let me know. - Adam Green Twitter API Developer http://2012twit.com http://140dev.com @140dev -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
