Thanks for the quick feedback, John. I'm checking it out and will let you know what happens.
On Aug 6, 10:15 am, John Kalucki <jkalu...@gmail.com> wrote: > Josh, > > It seems that you can accomplish most of your goals by using the / > track feature in the Streaming API. You can then make far fewer calls > to the Search API to cover dynamic cases, or fill in whatever else is > left. I suspect you'll have a better user experience with far fewer > coding and rate limiting hassles. > > Let me know if you have any questions or issues with the Streaming > API, or just post to this list. > > -John Kaluckihttp://twitter.com/jkalucki > Services, Twitter Inc. > > On Aug 5, 12:11 pm, Josh Shabtai <joshshab...@gmail.com> wrote: > > > Hi there. I was just about to start a thread on this topic myself, as > > I've developed a Web application that seems to be running into some > > issues related to the search API. > > > A disclaimer: I'm pretty inexperienced as a developer, so apologies > > for any redundancy and/or misuse of terminology. > > > I recently launchedhttp://www.twttrpoop.com, a Web application/parody > > designed to apply relatively sophisticated search and analytical tools > > to the basest of subjects (the URL is a dead giveaway). We've been > > whitelisted, but recently, we experienced a surge in traffic and usage > > that illuminated potential issues with our ability to access the > > search and REST APIs. > > > Some background on the app, before going into my questions... It > > revolves around a few key modules: > > > * A search engine that lets users compare the number of people > > talking about #2 in the last 24 hours (according to a handful of > > predetermined phrases) against any other keywords > > * A real-time feed that pulls in live tweets using the same set of > > predetermined keywords > > * A leaderboard mechanism that identifies the most active keyword > > 'abusers' on Twitter and scores their profiles according to frequency > > of keyword usage > > > Now, even though our taste in subject is questionable, we've made it a > > priority to ensure that our search engine and leaderboard are as > > accurate and useful as possible (ideally, we'd like to extend these > > tools to other applications). To do this, we're making up to 19,000 > > search API calls/hour. > > > On the search engine front, we've built a database and cron job that > > stores user-inputted keywords and publicly trending words that max out > > at 1,500 results (around 1,000 different words). Then, we make a > > search API call against each word every 5 minutes to ensure reasonably > > accurate results. > > > Our real-time feed makes a live search API call every 10 seconds (360 > > API calls/hour) and we also make search API calls related to > > approximately 20 distinct #2-focused keywords every 5 minutes (240 > > search API calls/hour). > > > After our initial surge in traffic, we've noticed some strange issues, > > all of which seem to relate to us being unable to access data. So, > > after that long-winded explanation, here are my questions: > > > * First of all, are we within an acceptable rate limit for Search > > API? What's the ballpark? > > * We are using both REST and Search API calls to access Twitter > > data. Does using both simultaneously (as we do) cause any problems you > > are aware of? Do you have any known restrictions we may have missed? > > * We make quite a few search API requests consecutively. (For > > example, we will make simultaneous calls against various keywords.) > > Is there a timing restriction that we should be aware of? > > * Our site continually updates users who are talking about our > > topic. Thus the page is almost always dynamic. However, almost always > > when we refresh the page or reload the page we have problems fetching > > data which of course distorts the page and often does not load > > correctly. Could this be the result of using both methods REST and > > Search to acquire data and doing it from the same IP address? If yes, > > how do we solve this? If not, any ideas why this is happening? > > > Thanks for the time and apologies for subjecting you to toilet humor.
