Re: [twitter-dev] Rate limiting for streaming API
On 2/19/11 1:49 PM, Paresh Nakhe wrote: Hi, From what i understand, there is no concept of rate limiting for streaming api. Actually it does make sense because if anyone is to use 'statuses/sample' method (say) the limit will soon be crossed. We are working on something that will heavily use the streaming api, so if rate limiting is imposed in future it could create some problems. Are there any chances of such a restriction being imposed? There's no limit on the amount of tweets you can receive, but there's a limit on the amount of searches you can do, and the amount of connections you can have open. Secondly, this api requires authentication unlike search api. Authentication for user streams is fine but I don't understand it's need for streaming api. There's probably no real technical reason for why this is required, but for properly being able to keep statistics etc, it's needed. That, and the fact that there are several different levels of access to the API. Thanks, Paresh -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Rate limiting for streaming API
On going through the documentation in more detail i found this: - The the track parameter (keywords), and the location parameter (geo) on the statuses/filter method are rate-limited predicates. - After the * limitation period* expires, all matching statuses will once again be delivered, along with a limit message that enumerates the total number of statuses that have been eliminated from the stream since the start of the connection. What exactly is the limitation period and what is one supposed to do after the limitation period expires? Destroy the current connection and create a new one? Paresh. On Sat, Feb 19, 2011 at 6:22 PM, Tom van der Woerdt i...@tvdw.eu wrote: On 2/19/11 1:49 PM, Paresh Nakhe wrote: Hi, From what i understand, there is no concept of rate limiting for streaming api. Actually it does make sense because if anyone is to use 'statuses/sample' method (say) the limit will soon be crossed. We are working on something that will heavily use the streaming api, so if rate limiting is imposed in future it could create some problems. Are there any chances of such a restriction being imposed? There's no limit on the amount of tweets you can receive, but there's a limit on the amount of searches you can do, and the amount of connections you can have open. Secondly, this api requires authentication unlike search api. Authentication for user streams is fine but I don't understand it's need for streaming api. There's probably no real technical reason for why this is required, but for properly being able to keep statistics etc, it's needed. That, and the fact that there are several different levels of access to the API. Thanks, Paresh -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- *What I have is not a dream, because I will make it a reality.* -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Rate limiting for streaming API
On Sat, Feb 19, 2011 at 7:52 AM, Tom van der Woerdt i...@tvdw.eu wrote: On 2/19/11 1:49 PM, Paresh Nakhe wrote: Hi, From what i understand, there is no concept of rate limiting for streaming api. Actually it does make sense because if anyone is to use 'statuses/sample' method (say) the limit will soon be crossed. We are working on something that will heavily use the streaming api, so if rate limiting is imposed in future it could create some problems. Are there any chances of such a restriction being imposed? There's no limit on the amount of tweets you can receive, but there's a limit on the amount of searches you can do, and the amount of connections you can have open. Responses here from Twitter staff and notes in the docs say that the sample (default) level of delivery is limited to 1% of all tweets. As long as your total flow is less than 1% of the firehose (all tweets) you are supposed to receive every tweet that matches your search terms. From my experience, you only receive about 95% of the tweets that match a set of search terms. I've never used a set of search terms that delivered anywhere near 1% of the total flow. Here are some numbers: - The total flow of all tweets is in the order of magnitude of 100,000,000 per day. - The sample stream should thus receive up to 1,000,000 tweets per day. - When I collect tweets based on search words that deliver about 10,000 tweets a day, I don't receive about 5% of the tweets that do match these terms. If getting *every* tweet that matches these terms is essential, I recommend combining streaming with backfilling. By backfilling I mean using the search API or timeline calls in the REST API to collect tweets that may have been missed by the streaming API. The result is real-time delivery of 95% of the tweets you need and eventual delivery of the remainder. Don't bother asking about future restrictions. That question will not be answered by Twitter HQ. -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Rate limiting for streaming API
On 2/19/11 2:23 PM, Paresh Nakhe wrote: On going through the documentation in more detail i found this: - The the track parameter (keywords), and the location parameter (geo) on the statuses/filter method are rate-limited predicates. You can't have an infinite number of search terms. - After the */limitation period/* expires, all matching statuses will once again be delivered, along with a limit message that enumerates the total number of statuses that have been eliminated from the stream since the start of the connection. As far as I know, this limitation is only for user streams. When you get more than 2 or 3 statuses per second, it wouldn't be readable for the user anyway so the extra statuses get discarded and you get a limitation notice. I don't think that this goes for the normal stream. What exactly is the limitation period and what is one supposed to do after the limitation period expires? Destroy the current connection and create a new one? Just keep listening :-) Paresh. On Sat, Feb 19, 2011 at 6:22 PM, Tom van der Woerdt i...@tvdw.eu mailto:i...@tvdw.eu wrote: On 2/19/11 1:49 PM, Paresh Nakhe wrote: Hi, From what i understand, there is no concept of rate limiting for streaming api. Actually it does make sense because if anyone is to use 'statuses/sample' method (say) the limit will soon be crossed. We are working on something that will heavily use the streaming api, so if rate limiting is imposed in future it could create some problems. Are there any chances of such a restriction being imposed? There's no limit on the amount of tweets you can receive, but there's a limit on the amount of searches you can do, and the amount of connections you can have open. Secondly, this api requires authentication unlike search api. Authentication for user streams is fine but I don't understand it's need for streaming api. There's probably no real technical reason for why this is required, but for properly being able to keep statistics etc, it's needed. That, and the fact that there are several different levels of access to the API. Thanks, Paresh -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- /What I have is not a dream, because I will make it a reality./ -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
