On Thu, Nov 18, 2010 at 10:03 AM, Neil Sheth <[email protected]> wrote: > I have asked our developers to implement the statuses/filter API stream, > however they have come back with the following: > > 1) We will not be able to fetch previous tweets for the accounts because > count parameter can not be used for default role (but can be used for > increased access level roles e.g. "Birddog, "Shadow").
If by previous tweets you mean tweets in the past, that is never available with the streaming API. Higher access levels won't give you older tweets either. For past tweets you need to use the search API. But the search API is rate limited, and it tends to fail if many search terms are used, so that is also limited. Frankly, past tweets are not available in large quantities with any portion of the API. > > 2) Also, if streaming connections is closed (reason can be anything) than we > will not be able to fetch the tweets for that connection closed period. So > we do also need a monitoring job to monitor the daemon/backgroun process > that fetch the tweets. Yes, you should always have a monitor running to verify that the streaming API collection process is still working. There are failures on the order of once a month or more in my experience. This is much better than in the past. These may crash your collection script. The answer is again the search API, which can be used to fill in the missing tweets lost during the failure, but the limitations mentioned above still apply. In general, if you are planning on capturing *all* tweets for a set of words or users, and *never* losing any, you are setting an impossible goal. Aiming for a very high level of accuracy is all you are going to achieve. With the right coding 99% or better is possible. > > Any help from anyone would be great > > Thanks > Neil > > On 16 November 2010 22:19, Adam Green <[email protected]> wrote: >> >> I'm using the streaming API to pull in tweets at that rate for several >> sites with no problem. The default access level gives you all the >> tweets for up to 5,000 users. >> http://dev.twitter.com/pages/streaming_api_methods#statuses-filter >> >> From my experience, if you create a useful site for this level of >> users, you can go back later with a much stronger argument for higher >> access later. >> >> If you are using PHP, I recommend Fenn Bailey's Phirehose library to >> manage the connection. >> http://code.google.com/p/phirehose/ >> >> If you architect your tweet collection code so that all tweets are put >> into a normalized database, and the rest of the code only uses the >> database and not the API, you can easily change to a new type of API >> connection later. But if all you want now is tweets, the streaming API >> is good for now. >> >> On Tue, Nov 16, 2010 at 5:06 PM, Neil Sheth <[email protected]> wrote: >> > Great - thanks Tom. For our site we will be pulling in quite few >> > thousand >> > tweets per hour. Do you see any limitation with this API? >> > >> > On 16 November 2010 22:01, Thomas Mango <[email protected]> wrote: >> >> >> >> Just wanted to chime in quickly. I've been using Site Streams in >> >> production for over a month now and have found them to be absolutely >> >> fantastic. Really rock solid. If Site Streams are indeed what you're >> >> looking >> >> for, I wouldn't let the beta tag scare you away. >> >> >> >> Taylor Singletary wrote: >> >>> >> >>> Hi Neil, >> >>> >> >>> What are you particularly trying to accomplish with your Twitter >> >>> Integration? How are tweets used in the application? What APIs were >> >>> you >> >>> leveraging when you were planning a REST-only solution? >> >>> >> >>> While Site Streams is "officially beta" right now, it's very reliable >> >>> -- >> >>> but whether it's the right solution for you really depends on what >> >>> you're looking to accomplish. >> >>> >> >>> Thanks, >> >>> Taylor >> >>> >> >>> On Tue, Nov 16, 2010 at 1:32 PM, Neil <[email protected] >> >>> <mailto:[email protected]>> wrote: >> >>> >> >>> We have previously raised a request to obtain twitter whitelisting >> >>> but >> >>> have been told by Twitter (Brian) that we have built the wrong >> >>> solution. Our developers are struggling to understand which >> >>> solution >> >>> they need to build for our site www.mystweet.com >> >>> <http://www.mystweet.com> in order to get >> >>> whitelisted. They have stated that they are unsure which one to >> >>> choose: >> >>> >> >>> 1) http://dev.twitter.com/pages/site_streams - twitter recommend >> >>> this for the kind of solution which we want to implement, but this >> >>> is >> >>> still in beta >> >>> >> >>> 2) http://dev.twitter.com/pages/user_streams_suggestions - but >> >>> this >> >>> is not what they would allow for our case. >> >>> >> >>> Can you please advise what solution needs to be built? We're >> >>> hoping >> >>> to correct this before they go on their holidays >> >>> >> >>> Thanks >> >>> >> >>> >> >>> PREVIOUS EMAIL FROM BRIAN >> >>> >> >>> Hi Jessel, >> >>> Sorry about this! There is currently an issue that removes the >> >>> "rejection reason" from some whitelist emails. Your requests have >> >>> rejected because we encourage you to use our Streaming API instead >> >>> to >> >>> accomplish your purposes. As described >> >>> onhttp://dev.twitter.com/pages/streaming_api_methods >> >>> <http://dev.twitter.com/pages/streaming_api_methods> >> >>> , you may use the statuses/filter method with the follow parameter >> >>> to >> >>> receive a real-time stream of tweets from all the users you're >> >>> interested in. I apologize for any inconvenience this causes to >> >>> your >> >>> project. >> >>> >> >>> Thanks for your understanding, >> >>> Brian >> >>> >> >>> -- >> >>> Twitter developer documentation and resources: >> >>> http://dev.twitter.com/doc >> >>> API updates via Twitter: http://twitter.com/twitterapi >> >>> Issues/Enhancements Tracker: >> >>> http://code.google.com/p/twitter-api/issues/list >> >>> Change your membership to this group: >> >>> http://groups.google.com/group/twitter-development-talk >> >>> >> >>> >> >>> -- >> >>> Twitter developer documentation and resources: >> >>> http://dev.twitter.com/doc >> >>> API updates via Twitter: http://twitter.com/twitterapi >> >>> Issues/Enhancements Tracker: >> >>> http://code.google.com/p/twitter-api/issues/list >> >>> Change your membership to this group: >> >>> http://groups.google.com/group/twitter-development-talk >> >> >> >> >> >> -- >> >> Thomas Mango >> >> [email protected] >> >> >> >> >> >> -- >> >> Twitter developer documentation and resources: >> >> http://dev.twitter.com/doc >> >> API updates via Twitter: http://twitter.com/twitterapi >> >> Issues/Enhancements Tracker: >> >> http://code.google.com/p/twitter-api/issues/list >> >> Change your membership to this group: >> >> http://groups.google.com/group/twitter-development-talk >> > >> > -- >> > Twitter developer documentation and resources: >> > http://dev.twitter.com/doc >> > API updates via Twitter: http://twitter.com/twitterapi >> > Issues/Enhancements Tracker: >> > http://code.google.com/p/twitter-api/issues/list >> > Change your membership to this group: >> > http://groups.google.com/group/twitter-development-talk >> > >> >> -- >> Twitter developer documentation and resources: http://dev.twitter.com/doc >> API updates via Twitter: http://twitter.com/twitterapi >> Issues/Enhancements Tracker: >> http://code.google.com/p/twitter-api/issues/list >> Change your membership to this group: >> http://groups.google.com/group/twitter-development-talk > > -- > Twitter developer documentation and resources: http://dev.twitter.com/doc > API updates via Twitter: http://twitter.com/twitterapi > Issues/Enhancements Tracker: > http://code.google.com/p/twitter-api/issues/list > Change your membership to this group: > http://groups.google.com/group/twitter-development-talk > -- Adam Green Twitter API Consultant and Trainer http://140dev.com @140dev -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
