Re: [twitter-dev] Quality Filter in Streaming API with follow predicates
Quoting John Kalucki j...@twitter.com: We've been pushing scale, operational and efficiency branches recently, and haven't exposed anything user-facing in quite some time. If we can't keep well ahead of the organic tweet growth, we're in a tough spot. At 50 million tweets a day, and an average tweet size (JSON, measured from sample) of about 1400 bytes, you're gaining about 70 GB of stuff a day without *any* geometric growth by my calculations. And if I read Kevin Weil's blog post correctly, that 50 million tweets a day is *after* spam removal! BTW, sample is delivering about 90,000 tweets per hour at the peak part of the day now. This mathematician hasn't had his coffee yet for 24 February 2010, so there are no theorems - just a simple wow! or perhaps OMG! ;-) -- M. Edward (Ed) Borasky borasky-research.net/m-edward-ed-borasky/ A mathematician is a device for turning coffee into theorems. ~ Paul Erdos
[twitter-dev] Quality Filter in Streaming API with follow predicates
From http://apiwiki.twitter.com/Streaming-API-Documentation#ResultQuality: Removing the quality filter from streams with follow predicates is, quite reasonably, an often requested feature. Does anyone have any ideas on the status of this feature or any suggested workarounds? We are working on a new version of TweetPo.st powered by the Streaming API. However, some of our beta testers are apparently being quality filtered and we are trying to figure out a way to get their tweets. They are OAuthed, so we can in theory poll their accounts directly for updates. However, we don't have a systematic way of figuring out whether they are being quality filtered in the first place or if that status changes subsequently. So, our specific questions are: A) Is there any update on the status of removing the quality filter from streams with follow predicates? B) Is there any way to programmatically check if a given user is being quality filtered? C) Is there any way to be notified when a given user's quality filter status changes? Any help here would be greatly appreciated as we would ideally like to use the Streaming API as much as possible and it's hard to explain to users when their tweets don't show up. Thanks! -jonathan = Jonathan Strauss, Co-Founder http://snowballfactory.com Campaign tracking for social media - http://awe.sm A smarter way to update Facebook from Twitter - http://tweetpo.st Sharecount button for Facebook - http://www.fbshare.me
Re: [twitter-dev] Quality Filter in Streaming API with follow predicates
This has been the next generally scheduled user-facing feature for quite some time on the Streaming API. We've been pushing scale, operational and efficiency branches recently, and haven't exposed anything user-facing in quite some time. If we can't keep well ahead of the organic tweet growth, we're in a tough spot. I hope that we can transition back to features soon enough, and this feature should be near, if not at the top of, the list. There's a git branch that has this feature largely working, but I'm afraid that the merge back to master is going to be a very painful one. The bits, they rot. As far as programmatic detection, there are significant policy issues in play around filtered users. Getting this feature shipped is the real solution. -John Kalucki http://twitter.com/jkalucki Infrastructure, Twitter Inc. On Tue, Feb 23, 2010 at 4:25 PM, Jonathan Strauss jonat...@snowballfactory.com wrote: From http://apiwiki.twitter.com/Streaming-API-Documentation#ResultQuality: Removing the quality filter from streams with follow predicates is, quite reasonably, an often requested feature. Does anyone have any ideas on the status of this feature or any suggested workarounds? We are working on a new version of TweetPo.st powered by the Streaming API. However, some of our beta testers are apparently being quality filtered and we are trying to figure out a way to get their tweets. They are OAuthed, so we can in theory poll their accounts directly for updates. However, we don't have a systematic way of figuring out whether they are being quality filtered in the first place or if that status changes subsequently. So, our specific questions are: A) Is there any update on the status of removing the quality filter from streams with follow predicates? B) Is there any way to programmatically check if a given user is being quality filtered? C) Is there any way to be notified when a given user's quality filter status changes? Any help here would be greatly appreciated as we would ideally like to use the Streaming API as much as possible and it's hard to explain to users when their tweets don't show up. Thanks! -jonathan = Jonathan Strauss, Co-Founder http://snowballfactory.com Campaign tracking for social media - http://awe.sm A smarter way to update Facebook from Twitter - http://tweetpo.st Sharecount button for Facebook - http://www.fbshare.me
