Neil:

If you mean a link to the Twitter doc on the search API, that is here:
http://dev.twitter.com/doc/get/search

I've also written a detailed comparison of the search API vs. the
streaming API that may help your coders pick the right solution:
http://140dev.com/twitter-api-programming-tutorials/aggregating-tweets-search-api-vs-streaming-api/

If you wanted a solution to the specific issue of filling in missing
tweets with the search API, I don't know of an example that covers
exactly that, but the basic idea is:
- Collect tweets with the streaming API and the track parameter for
your keywords. Make the id_str of each tweet the primary key of the
table where they are stored.
- Monitor the created_at value for the most recent tweet added to the
database with a cron job that runs every five minutes or so. If tweets
arrive less frequently for your keywords, make this a longer interval.
- If the monitor script finds that no new tweets have been stored
within your test interval, it can use the search API to gather tweets
based on the same keywords. The search API returns up to 15 pages of
results in reverse date order, with up to 100 tweets per page. So even
tweets for active topics will still available from search.
- As tweets arrive from search, compare the id_str of the new tweets
with those in the database. Add the ones that aren't already there.
Your search code should stop making requests when you get to older
tweets you already have in the DB.

This basically gives you real-time tweets from the streaming API, and
automatic back filling for missed tweets with the search API.

I've probably exhausted the patience of others on this topic, so
contact me directly if you need more help: 140...@gmail.com


On Thu, Nov 18, 2010 at 3:05 PM, Neil Sheth <sheth.n...@gmail.com> wrote:
> Hi Adam,
>
> Thanks for your advice, just wanted to ask if you have a link to the
> solution you mentioned for point 2 i.e. the Search API?
>
> Regards
> Neil
>
> On 18 November 2010 15:19, Adam Green <140...@gmail.com> wrote:
>>
>> On Thu, Nov 18, 2010 at 10:03 AM, Neil Sheth <sheth.n...@gmail.com> wrote:
>> > I have asked our developers to implement the statuses/filter API stream,
>> > however they have come back with the following:
>> >
>> > 1) We will not be able to fetch previous tweets for the accounts because
>> > count parameter can not be used for default role (but can be used for
>> > increased access level roles e.g. "Birddog, "Shadow").
>>
>> If by previous tweets you mean tweets in the past, that is never
>> available with the streaming API. Higher access levels won't give you
>> older tweets either. For past tweets you need to use the search API.
>> But the search API is rate limited, and it tends to fail if many
>> search terms are used, so that is also limited. Frankly, past tweets
>> are not available in large quantities with any portion of the API.
>>
>> >
>> > 2) Also, if streaming connections is closed (reason can be anything)
>> > than we
>> > will not be able to fetch the tweets for that connection closed period.
>> > So
>> > we do also need a monitoring job to monitor the daemon/backgroun process
>> > that fetch the tweets.
>> Yes, you should always have a monitor running to verify that the
>> streaming API collection process is still working. There are failures
>> on the order of once a month or more in my experience. This is much
>> better than in the past. These may crash your collection script.
>>
>> The answer is again the search API, which can be used to fill in the
>> missing tweets lost during the failure, but the limitations mentioned
>> above still apply.
>>
>> In general, if you are planning on capturing *all* tweets for a set of
>> words or users, and *never* losing any, you are setting an impossible
>> goal. Aiming for a very high level of accuracy is all you are going to
>> achieve. With the right coding 99% or better is possible.
>>
>> >
>> > Any help from anyone would be great
>> >
>> > Thanks
>> > Neil
>> >
>> > On 16 November 2010 22:19, Adam Green <140...@gmail.com> wrote:
>> >>
>> >> I'm using the streaming API to pull in tweets at that rate for several
>> >> sites with no problem. The default access level gives you all the
>> >> tweets for up to 5,000 users.
>> >> http://dev.twitter.com/pages/streaming_api_methods#statuses-filter
>> >>
>> >> From my experience, if you create a useful site for this level of
>> >> users, you can go back later with a much stronger argument for higher
>> >> access later.
>> >>
>> >> If you are using PHP, I recommend Fenn Bailey's Phirehose library to
>> >> manage the connection.
>> >> http://code.google.com/p/phirehose/
>> >>
>> >> If you architect your tweet collection code so that all tweets are put
>> >> into a normalized database, and the rest of the code only uses the
>> >> database and not the API, you can easily change to a new type of API
>> >> connection later. But if all you want now is tweets, the streaming API
>> >> is good for now.
>> >>
>> >> On Tue, Nov 16, 2010 at 5:06 PM, Neil Sheth <sheth.n...@gmail.com>
>> >> wrote:
>> >> > Great - thanks Tom.  For our site we will be pulling in quite few
>> >> > thousand
>> >> > tweets per hour.  Do you see any limitation with this API?
>> >> >
>> >> > On 16 November 2010 22:01, Thomas Mango <tsma...@gmail.com> wrote:
>> >> >>
>> >> >> Just wanted to chime in quickly. I've been using Site Streams in
>> >> >> production for over a month now and have found them to be absolutely
>> >> >> fantastic. Really rock solid. If Site Streams are indeed what you're
>> >> >> looking
>> >> >> for, I wouldn't let the beta tag scare you away.
>> >> >>
>> >> >> Taylor Singletary wrote:
>> >> >>>
>> >> >>> Hi Neil,
>> >> >>>
>> >> >>> What are you particularly trying to accomplish with your Twitter
>> >> >>> Integration? How are tweets used in the application? What APIs were
>> >> >>> you
>> >> >>> leveraging when you were planning a REST-only solution?
>> >> >>>
>> >> >>> While Site Streams is "officially beta" right now, it's very
>> >> >>> reliable
>> >> >>> --
>> >> >>> but whether it's the right solution for you really depends on what
>> >> >>> you're looking to accomplish.
>> >> >>>
>> >> >>> Thanks,
>> >> >>> Taylor
>> >> >>>
>> >> >>> On Tue, Nov 16, 2010 at 1:32 PM, Neil <sheth.n...@gmail.com
>> >> >>> <mailto:sheth.n...@gmail.com>> wrote:
>> >> >>>
>> >> >>>    We have previously raised a request to obtain twitter
>> >> >>> whitelisting
>> >> >>> but
>> >> >>>    have been told by Twitter (Brian) that we have built the wrong
>> >> >>>    solution.  Our developers are struggling to understand which
>> >> >>> solution
>> >> >>>    they need to build for our site www.mystweet.com
>> >> >>>    <http://www.mystweet.com> in order to get
>> >> >>>    whitelisted.  They have stated that they are unsure which one to
>> >> >>>    choose:
>> >> >>>
>> >> >>>    1) http://dev.twitter.com/pages/site_streams   - twitter
>> >> >>> recommend
>> >> >>>    this for the kind of solution which we want to implement, but
>> >> >>> this
>> >> >>> is
>> >> >>>    still in beta
>> >> >>>
>> >> >>>    2) http://dev.twitter.com/pages/user_streams_suggestions  - but
>> >> >>> this
>> >> >>>    is not what they would allow for our case.
>> >> >>>
>> >> >>>    Can you please advise what solution needs to be built?  We're
>> >> >>> hoping
>> >> >>>    to correct this before they go on their holidays
>> >> >>>
>> >> >>>    Thanks
>> >> >>>
>> >> >>>
>> >> >>>    PREVIOUS EMAIL FROM BRIAN
>> >> >>>
>> >> >>>    Hi Jessel,
>> >> >>>    Sorry about this! There is currently an issue that removes the
>> >> >>>    "rejection reason" from some whitelist emails. Your requests
>> >> >>> have
>> >> >>>    rejected because we encourage you to use our Streaming API
>> >> >>> instead
>> >> >>> to
>> >> >>>    accomplish your purposes. As described
>> >> >>>    onhttp://dev.twitter.com/pages/streaming_api_methods
>> >> >>>    <http://dev.twitter.com/pages/streaming_api_methods>
>> >> >>>    , you may use the statuses/filter method with the follow
>> >> >>> parameter
>> >> >>> to
>> >> >>>    receive a real-time stream of tweets from all the users you're
>> >> >>>    interested in. I apologize for any inconvenience this causes to
>> >> >>> your
>> >> >>>    project.
>> >> >>>
>> >> >>>    Thanks for your understanding,
>> >> >>>    Brian
>> >> >>>
>> >> >>>    --
>> >> >>>    Twitter developer documentation and resources:
>> >> >>>    http://dev.twitter.com/doc
>> >> >>>    API updates via Twitter: http://twitter.com/twitterapi
>> >> >>>    Issues/Enhancements Tracker:
>> >> >>>    http://code.google.com/p/twitter-api/issues/list
>> >> >>>    Change your membership to this group:
>> >> >>>    http://groups.google.com/group/twitter-development-talk
>> >> >>>
>> >> >>>
>> >> >>> --
>> >> >>> Twitter developer documentation and resources:
>> >> >>> http://dev.twitter.com/doc
>> >> >>> API updates via Twitter: http://twitter.com/twitterapi
>> >> >>> Issues/Enhancements Tracker:
>> >> >>> http://code.google.com/p/twitter-api/issues/list
>> >> >>> Change your membership to this group:
>> >> >>> http://groups.google.com/group/twitter-development-talk
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Thomas Mango
>> >> >> tsma...@gmail.com
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Twitter developer documentation and resources:
>> >> >> http://dev.twitter.com/doc
>> >> >> API updates via Twitter: http://twitter.com/twitterapi
>> >> >> Issues/Enhancements Tracker:
>> >> >> http://code.google.com/p/twitter-api/issues/list
>> >> >> Change your membership to this group:
>> >> >> http://groups.google.com/group/twitter-development-talk
>> >> >
>> >> > --
>> >> > Twitter developer documentation and resources:
>> >> > http://dev.twitter.com/doc
>> >> > API updates via Twitter: http://twitter.com/twitterapi
>> >> > Issues/Enhancements Tracker:
>> >> > http://code.google.com/p/twitter-api/issues/list
>> >> > Change your membership to this group:
>> >> > http://groups.google.com/group/twitter-development-talk
>> >> >
>> >>
>> >> --
>> >> Twitter developer documentation and resources:
>> >> http://dev.twitter.com/doc
>> >> API updates via Twitter: http://twitter.com/twitterapi
>> >> Issues/Enhancements Tracker:
>> >> http://code.google.com/p/twitter-api/issues/list
>> >> Change your membership to this group:
>> >> http://groups.google.com/group/twitter-development-talk
>> >
>> > --
>> > Twitter developer documentation and resources:
>> > http://dev.twitter.com/doc
>> > API updates via Twitter: http://twitter.com/twitterapi
>> > Issues/Enhancements Tracker:
>> > http://code.google.com/p/twitter-api/issues/list
>> > Change your membership to this group:
>> > http://groups.google.com/group/twitter-development-talk
>> >
>>
>>
>>
>> --
>> Adam Green
>> Twitter API Consultant and Trainer
>> http://140dev.com
>> @140dev
>>
>> --
>> Twitter developer documentation and resources: http://dev.twitter.com/doc
>> API updates via Twitter: http://twitter.com/twitterapi
>> Issues/Enhancements Tracker:
>> http://code.google.com/p/twitter-api/issues/list
>> Change your membership to this group:
>> http://groups.google.com/group/twitter-development-talk
>
> --
> Twitter developer documentation and resources: http://dev.twitter.com/doc
> API updates via Twitter: http://twitter.com/twitterapi
> Issues/Enhancements Tracker:
> http://code.google.com/p/twitter-api/issues/list
> Change your membership to this group:
> http://groups.google.com/group/twitter-development-talk
>

-- 
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group: 
http://groups.google.com/group/twitter-development-talk

Reply via email to