Thanks Chad - I will cut the numbers down as detailed. I have added
code to handle 503 responses and to wait for the time dictated by the
retry field, so if I *do* hit the buffers, at least it will be
handled. I'll log the fact as well, so that I can tune the pauses.
I'll give it a go and hope I avoid blowing my code up! :-)
Thanks for the response,
On Aug 4, 10:46 pm, Chad Etzel <c...@twitter.com> wrote:
> Hi Steve,
> This system sounds like will work well. Your current numbers as stated
> should stay within the rate limits. However, you should add logic to
> your code which will increase the pause lengths programmatically
> should you find that you are getting rate-limited responses.
> Twitter Platform Support
> On Tue, Aug 4, 2009 at 12:50 PM, steve<steveb...@googlemail.com> wrote:
> > There are a lot of messages and details around saying that the REST
> > API is 150 per hour, with whitelisting up to 20k per hour. The Search
> > API is "more than" the 150, but no specifics.
> >>> Note that the Search API is not limited by the same 150 requests per hour
> >>> limit as the REST API.
> >>> The number is quite a bit higher and we feel it is both liberal and
> >>> sufficient for most applications.
> > My question is this, I have just soft launchedwww.twitparade.co.uk,
> > and although the site is in early days, a lot of work is in the
> > scheduler that grabs, stores and publishes individual tweets.
> > The way I am doing it is as follows:
> > 1. Load a list of people in a specific time slice to check
> > 2. Loop through each person on list, pausing for 5 seconds after each
> > person (except the last)
> > 3. Pause for 20 seconds at the end of the list
> > 4. Pick up the next time slice and start again
> > The time slicing allows me to prioritise the people how have tweeted
> > more recently, by checking them more frequently.
> > With the pauses I am currently using, assuming each search is instant,
> > then in any 1 minute, I am carrying out a maximum of 12 searches,
> > equating to 720 an hour. If the minute spans a list change, then there
> > is a 20 second pause, so I would only carry out 8 searches, equating
> > to 480 an hour. This can mean that it takes 20 minutes for some Tweets
> > to be picked up, if that person hasn't tweeted for a while (as I check
> > them less often) - I would like to improve that.
> > The gatherer is desktop application, so doesn't have a referrer, but I
> > have set the User-Agent to list my app name and the URL of the final
> > site that the data is gathered for, so hopefully Twitter can ID my app
> > (aside: How can we tell that our User-Agent makes it through?). I am
> > also on a fixed IP address, so should be identifiable to the back-end
> > systems at Twitter's end.
> > So how aggressive with cutting my pauses can I be? The Search API
> > numbers are not publicized so I have no idea if I'm knocking on the
> > limits, or whether I can with much lower pauses.
> > If I cut step 2 down to 1 and step 3 to 5 seconds, then my max rate
> > would be 60 per minute = 3600 per hour, or 2700 per hour. Is this
> > within the unknown limits?
> > If someone from Twitter could confirm/deny that my use of caching,
> > user-agent and shorter pauses all works together, I'd appreciate it.
> > Thanks,
> > Steve
> > --
> > Quick Web Ltd
> > UK