Hi there.  I was just about to start a thread on this topic myself, as
I've developed a Web application that seems to be running into some
issues related to the search API.

A disclaimer: I'm pretty inexperienced as a developer, so apologies
for any redundancy and/or misuse of terminology.

I recently launched http://www.twttrpoop.com, a Web application/parody
designed to apply relatively sophisticated search and analytical tools
to the basest of subjects (the URL is a dead giveaway).  We've been
whitelisted, but recently, we experienced a surge in traffic and usage
that illuminated potential issues with our ability to access the
search and REST APIs.

Some background on the app, before going into my questions... It
revolves around a few key modules:

    * A search engine that lets users compare the number of people
talking about #2 in the last 24 hours (according to a handful of
predetermined phrases) against any other keywords
    * A real-time feed that pulls in live tweets using the same set of
predetermined keywords
    * A leaderboard mechanism that identifies the most active keyword
'abusers' on Twitter and scores their profiles according to frequency
of keyword usage

Now, even though our taste in subject is questionable, we've made it a
priority to ensure that our search engine and leaderboard are as
accurate and useful as possible (ideally, we'd like to extend these
tools to other applications).  To do this, we're making up to 19,000
search API calls/hour.

On the search engine front, we've built a database and cron job that
stores user-inputted keywords and publicly trending words that max out
at 1,500 results (around 1,000 different words).  Then, we make a
search API call against each word every 5 minutes to ensure reasonably
accurate results.

Our real-time feed makes a live search API call every 10 seconds (360
API calls/hour) and we also make search API calls related to
approximately 20 distinct #2-focused keywords every 5 minutes (240
search API calls/hour).

After our initial surge in traffic, we've noticed some strange issues,
all of which seem to relate to us being unable to access data.  So,
after that long-winded explanation, here are my questions:

    * First of all, are we within an acceptable rate limit for Search
API? What's the ballpark?
    * We are using both REST and Search API calls to access Twitter
data. Does using both simultaneously (as we do) cause any problems you
are aware of? Do you have any known restrictions we may have missed?
    * We make quite a few search API requests consecutively.  (For
example, we will make simultaneous calls against various keywords.)
Is there a timing restriction that we should be aware of?
    * Our site continually updates users who are talking about our
topic. Thus the page is almost always dynamic. However, almost always
when we refresh the page or reload the page we have problems fetching
data which of course distorts the page and often does not load
correctly. Could this be the result of using both methods REST and
Search to acquire data and doing it from the same IP address? If yes,
how do we solve this? If not, any ideas why this is happening?

Thanks for the time and apologies for subjecting you to toilet humor.

Reply via email to