This is great advice, many thanks.
Will keep your blog and personal email address in mind for the future
Sent from my iPhone
On 18 Nov 2010, at 21:56, Adam Green <[email protected]> wrote:
Neil:
If you mean a link to the Twitter doc on the search API, that is here:
http://dev.twitter.com/doc/get/search
I've also written a detailed comparison of the search API vs. the
streaming API that may help your coders pick the right solution:
http://140dev.com/twitter-api-programming-tutorials/aggregating-tweets-search-api-vs-streaming-api/
If you wanted a solution to the specific issue of filling in missing
tweets with the search API, I don't know of an example that covers
exactly that, but the basic idea is:
- Collect tweets with the streaming API and the track parameter for
your keywords. Make the id_str of each tweet the primary key of the
table where they are stored.
- Monitor the created_at value for the most recent tweet added to the
database with a cron job that runs every five minutes or so. If tweets
arrive less frequently for your keywords, make this a longer interval.
- If the monitor script finds that no new tweets have been stored
within your test interval, it can use the search API to gather tweets
based on the same keywords. The search API returns up to 15 pages of
results in reverse date order, with up to 100 tweets per page. So even
tweets for active topics will still available from search.
- As tweets arrive from search, compare the id_str of the new tweets
with those in the database. Add the ones that aren't already there.
Your search code should stop making requests when you get to older
tweets you already have in the DB.
This basically gives you real-time tweets from the streaming API, and
automatic back filling for missed tweets with the search API.
I've probably exhausted the patience of others on this topic, so
contact me directly if you need more help: [email protected]
On Thu, Nov 18, 2010 at 3:05 PM, Neil Sheth <[email protected]>
wrote:
Hi Adam,
Thanks for your advice, just wanted to ask if you have a link to the
solution you mentioned for point 2 i.e. the Search API?
Regards
Neil
On 18 November 2010 15:19, Adam Green <[email protected]> wrote:
On Thu, Nov 18, 2010 at 10:03 AM, Neil Sheth
<[email protected]> wrote:
I have asked our developers to implement the statuses/filter API
stream,
however they have come back with the following:
1) We will not be able to fetch previous tweets for the accounts
because
count parameter can not be used for default role (but can be used
for
increased access level roles e.g. "Birddog, "Shadow").
If by previous tweets you mean tweets in the past, that is never
available with the streaming API. Higher access levels won't give
you
older tweets either. For past tweets you need to use the search API.
But the search API is rate limited, and it tends to fail if many
search terms are used, so that is also limited. Frankly, past tweets
are not available in large quantities with any portion of the API.
2) Also, if streaming connections is closed (reason can be
anything)
than we
will not be able to fetch the tweets for that connection closed
period.
So
we do also need a monitoring job to monitor the daemon/backgroun
process
that fetch the tweets.
Yes, you should always have a monitor running to verify that the
streaming API collection process is still working. There are
failures
on the order of once a month or more in my experience. This is much
better than in the past. These may crash your collection script.
The answer is again the search API, which can be used to fill in the
missing tweets lost during the failure, but the limitations
mentioned
above still apply.
In general, if you are planning on capturing *all* tweets for a
set of
words or users, and *never* losing any, you are setting an
impossible
goal. Aiming for a very high level of accuracy is all you are
going to
achieve. With the right coding 99% or better is possible.
Any help from anyone would be great
Thanks
Neil
On 16 November 2010 22:19, Adam Green <[email protected]> wrote:
I'm using the streaming API to pull in tweets at that rate for
several
sites with no problem. The default access level gives you all the
tweets for up to 5,000 users.
http://dev.twitter.com/pages/streaming_api_methods#statuses-filter
From my experience, if you create a useful site for this level of
users, you can go back later with a much stronger argument for
higher
access later.
If you are using PHP, I recommend Fenn Bailey's Phirehose
library to
manage the connection.
http://code.google.com/p/phirehose/
If you architect your tweet collection code so that all tweets
are put
into a normalized database, and the rest of the code only uses the
database and not the API, you can easily change to a new type of
API
connection later. But if all you want now is tweets, the
streaming API
is good for now.
On Tue, Nov 16, 2010 at 5:06 PM, Neil Sheth <[email protected]>
wrote:
Great - thanks Tom. For our site we will be pulling in quite few
thousand
tweets per hour. Do you see any limitation with this API?
On 16 November 2010 22:01, Thomas Mango <[email protected]>
wrote:
Just wanted to chime in quickly. I've been using Site Streams in
production for over a month now and have found them to be
absolutely
fantastic. Really rock solid. If Site Streams are indeed what
you're
looking
for, I wouldn't let the beta tag scare you away.
Taylor Singletary wrote:
Hi Neil,
What are you particularly trying to accomplish with your
Twitter
Integration? How are tweets used in the application? What
APIs were
you
leveraging when you were planning a REST-only solution?
While Site Streams is "officially beta" right now, it's very
reliable
--
but whether it's the right solution for you really depends on
what
you're looking to accomplish.
Thanks,
Taylor
On Tue, Nov 16, 2010 at 1:32 PM, Neil <[email protected]
<mailto:[email protected]>> wrote:
We have previously raised a request to obtain twitter
whitelisting
but
have been told by Twitter (Brian) that we have built the
wrong
solution. Our developers are struggling to understand which
solution
they need to build for our site www.mystweet.com
<http://www.mystweet.com> in order to get
whitelisted. They have stated that they are unsure which
one to
choose:
1) http://dev.twitter.com/pages/site_streams - twitter
recommend
this for the kind of solution which we want to implement,
but
this
is
still in beta
2) http://dev.twitter.com/pages/user_streams_suggestions
- but
this
is not what they would allow for our case.
Can you please advise what solution needs to be built?
We're
hoping
to correct this before they go on their holidays
Thanks
PREVIOUS EMAIL FROM BRIAN
Hi Jessel,
Sorry about this! There is currently an issue that removes
the
"rejection reason" from some whitelist emails. Your requests
have
rejected because we encourage you to use our Streaming API
instead
to
accomplish your purposes. As described
onhttp://dev.twitter.com/pages/streaming_api_methods
<http://dev.twitter.com/pages/streaming_api_methods>
, you may use the statuses/filter method with the follow
parameter
to
receive a real-time stream of tweets from all the users
you're
interested in. I apologize for any inconvenience this
causes to
your
project.
Thanks for your understanding,
Brian
--
Twitter developer documentation and resources:
http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker:
http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
--
Twitter developer documentation and resources:
http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker:
http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
--
Thomas Mango
[email protected]
--
Twitter developer documentation and resources:
http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker:
http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
--
Twitter developer documentation and resources:
http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker:
http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
--
Twitter developer documentation and resources:
http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker:
http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
--
Twitter developer documentation and resources:
http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker:
http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
--
Adam Green
Twitter API Consultant and Trainer
http://140dev.com
@140dev
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker:
http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker:
http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
