Josh, It seems that you can accomplish most of your goals by using the / track feature in the Streaming API. You can then make far fewer calls to the Search API to cover dynamic cases, or fill in whatever else is left. I suspect you'll have a better user experience with far fewer coding and rate limiting hassles.
Let me know if you have any questions or issues with the Streaming API, or just post to this list. -John Kalucki http://twitter.com/jkalucki Services, Twitter Inc. On Aug 5, 12:11 pm, Josh Shabtai <joshshab...@gmail.com> wrote: > Hi there. I was just about to start a thread on this topic myself, as > I've developed a Web application that seems to be running into some > issues related to the search API. > > A disclaimer: I'm pretty inexperienced as a developer, so apologies > for any redundancy and/or misuse of terminology. > > I recently launchedhttp://www.twttrpoop.com, a Web application/parody > designed to apply relatively sophisticated search and analytical tools > to the basest of subjects (the URL is a dead giveaway). We've been > whitelisted, but recently, we experienced a surge in traffic and usage > that illuminated potential issues with our ability to access the > search and REST APIs. > > Some background on the app, before going into my questions... It > revolves around a few key modules: > > * A search engine that lets users compare the number of people > talking about #2 in the last 24 hours (according to a handful of > predetermined phrases) against any other keywords > * A real-time feed that pulls in live tweets using the same set of > predetermined keywords > * A leaderboard mechanism that identifies the most active keyword > 'abusers' on Twitter and scores their profiles according to frequency > of keyword usage > > Now, even though our taste in subject is questionable, we've made it a > priority to ensure that our search engine and leaderboard are as > accurate and useful as possible (ideally, we'd like to extend these > tools to other applications). To do this, we're making up to 19,000 > search API calls/hour. > > On the search engine front, we've built a database and cron job that > stores user-inputted keywords and publicly trending words that max out > at 1,500 results (around 1,000 different words). Then, we make a > search API call against each word every 5 minutes to ensure reasonably > accurate results. > > Our real-time feed makes a live search API call every 10 seconds (360 > API calls/hour) and we also make search API calls related to > approximately 20 distinct #2-focused keywords every 5 minutes (240 > search API calls/hour). > > After our initial surge in traffic, we've noticed some strange issues, > all of which seem to relate to us being unable to access data. So, > after that long-winded explanation, here are my questions: > > * First of all, are we within an acceptable rate limit for Search > API? What's the ballpark? > * We are using both REST and Search API calls to access Twitter > data. Does using both simultaneously (as we do) cause any problems you > are aware of? Do you have any known restrictions we may have missed? > * We make quite a few search API requests consecutively. (For > example, we will make simultaneous calls against various keywords.) > Is there a timing restriction that we should be aware of? > * Our site continually updates users who are talking about our > topic. Thus the page is almost always dynamic. However, almost always > when we refresh the page or reload the page we have problems fetching > data which of course distorts the page and often does not load > correctly. Could this be the result of using both methods REST and > Search to acquire data and doing it from the same IP address? If yes, > how do we solve this? If not, any ideas why this is happening? > > Thanks for the time and apologies for subjecting you to toilet humor.
