Hi Adam, Thanks for your advice, just wanted to ask if you have a link to the solution you mentioned for point 2 i.e. the Search API?
Regards Neil On 18 November 2010 15:19, Adam Green <140...@gmail.com> wrote: > On Thu, Nov 18, 2010 at 10:03 AM, Neil Sheth <sheth.n...@gmail.com> wrote: > > I have asked our developers to implement the statuses/filter API stream, > > however they have come back with the following: > > > > 1) We will not be able to fetch previous tweets for the accounts because > > count parameter can not be used for default role (but can be used for > > increased access level roles e.g. "Birddog, "Shadow"). > > If by previous tweets you mean tweets in the past, that is never > available with the streaming API. Higher access levels won't give you > older tweets either. For past tweets you need to use the search API. > But the search API is rate limited, and it tends to fail if many > search terms are used, so that is also limited. Frankly, past tweets > are not available in large quantities with any portion of the API. > > > > > 2) Also, if streaming connections is closed (reason can be anything) than > we > > will not be able to fetch the tweets for that connection closed period. > So > > we do also need a monitoring job to monitor the daemon/backgroun process > > that fetch the tweets. > Yes, you should always have a monitor running to verify that the > streaming API collection process is still working. There are failures > on the order of once a month or more in my experience. This is much > better than in the past. These may crash your collection script. > > The answer is again the search API, which can be used to fill in the > missing tweets lost during the failure, but the limitations mentioned > above still apply. > > In general, if you are planning on capturing *all* tweets for a set of > words or users, and *never* losing any, you are setting an impossible > goal. Aiming for a very high level of accuracy is all you are going to > achieve. With the right coding 99% or better is possible. > > > > > Any help from anyone would be great > > > > Thanks > > Neil > > > > On 16 November 2010 22:19, Adam Green <140...@gmail.com> wrote: > >> > >> I'm using the streaming API to pull in tweets at that rate for several > >> sites with no problem. The default access level gives you all the > >> tweets for up to 5,000 users. > >> http://dev.twitter.com/pages/streaming_api_methods#statuses-filter > >> > >> From my experience, if you create a useful site for this level of > >> users, you can go back later with a much stronger argument for higher > >> access later. > >> > >> If you are using PHP, I recommend Fenn Bailey's Phirehose library to > >> manage the connection. > >> http://code.google.com/p/phirehose/ > >> > >> If you architect your tweet collection code so that all tweets are put > >> into a normalized database, and the rest of the code only uses the > >> database and not the API, you can easily change to a new type of API > >> connection later. But if all you want now is tweets, the streaming API > >> is good for now. > >> > >> On Tue, Nov 16, 2010 at 5:06 PM, Neil Sheth <sheth.n...@gmail.com> > wrote: > >> > Great - thanks Tom. For our site we will be pulling in quite few > >> > thousand > >> > tweets per hour. Do you see any limitation with this API? > >> > > >> > On 16 November 2010 22:01, Thomas Mango <tsma...@gmail.com> wrote: > >> >> > >> >> Just wanted to chime in quickly. I've been using Site Streams in > >> >> production for over a month now and have found them to be absolutely > >> >> fantastic. Really rock solid. If Site Streams are indeed what you're > >> >> looking > >> >> for, I wouldn't let the beta tag scare you away. > >> >> > >> >> Taylor Singletary wrote: > >> >>> > >> >>> Hi Neil, > >> >>> > >> >>> What are you particularly trying to accomplish with your Twitter > >> >>> Integration? How are tweets used in the application? What APIs were > >> >>> you > >> >>> leveraging when you were planning a REST-only solution? > >> >>> > >> >>> While Site Streams is "officially beta" right now, it's very > reliable > >> >>> -- > >> >>> but whether it's the right solution for you really depends on what > >> >>> you're looking to accomplish. > >> >>> > >> >>> Thanks, > >> >>> Taylor > >> >>> > >> >>> On Tue, Nov 16, 2010 at 1:32 PM, Neil <sheth.n...@gmail.com > >> >>> <mailto:sheth.n...@gmail.com>> wrote: > >> >>> > >> >>> We have previously raised a request to obtain twitter > whitelisting > >> >>> but > >> >>> have been told by Twitter (Brian) that we have built the wrong > >> >>> solution. Our developers are struggling to understand which > >> >>> solution > >> >>> they need to build for our site www.mystweet.com > >> >>> <http://www.mystweet.com> in order to get > >> >>> whitelisted. They have stated that they are unsure which one to > >> >>> choose: > >> >>> > >> >>> 1) http://dev.twitter.com/pages/site_streams - twitter > recommend > >> >>> this for the kind of solution which we want to implement, but > this > >> >>> is > >> >>> still in beta > >> >>> > >> >>> 2) http://dev.twitter.com/pages/user_streams_suggestions - but > >> >>> this > >> >>> is not what they would allow for our case. > >> >>> > >> >>> Can you please advise what solution needs to be built? We're > >> >>> hoping > >> >>> to correct this before they go on their holidays > >> >>> > >> >>> Thanks > >> >>> > >> >>> > >> >>> PREVIOUS EMAIL FROM BRIAN > >> >>> > >> >>> Hi Jessel, > >> >>> Sorry about this! There is currently an issue that removes the > >> >>> "rejection reason" from some whitelist emails. Your requests have > >> >>> rejected because we encourage you to use our Streaming API > instead > >> >>> to > >> >>> accomplish your purposes. As described > >> >>> onhttp://dev.twitter.com/pages/streaming_api_methods > >> >>> <http://dev.twitter.com/pages/streaming_api_methods> > >> >>> , you may use the statuses/filter method with the follow > parameter > >> >>> to > >> >>> receive a real-time stream of tweets from all the users you're > >> >>> interested in. I apologize for any inconvenience this causes to > >> >>> your > >> >>> project. > >> >>> > >> >>> Thanks for your understanding, > >> >>> Brian > >> >>> > >> >>> -- > >> >>> Twitter developer documentation and resources: > >> >>> http://dev.twitter.com/doc > >> >>> API updates via Twitter: http://twitter.com/twitterapi > >> >>> Issues/Enhancements Tracker: > >> >>> http://code.google.com/p/twitter-api/issues/list > >> >>> Change your membership to this group: > >> >>> http://groups.google.com/group/twitter-development-talk > >> >>> > >> >>> > >> >>> -- > >> >>> Twitter developer documentation and resources: > >> >>> http://dev.twitter.com/doc > >> >>> API updates via Twitter: http://twitter.com/twitterapi > >> >>> Issues/Enhancements Tracker: > >> >>> http://code.google.com/p/twitter-api/issues/list > >> >>> Change your membership to this group: > >> >>> http://groups.google.com/group/twitter-development-talk > >> >> > >> >> > >> >> -- > >> >> Thomas Mango > >> >> tsma...@gmail.com > >> >> > >> >> > >> >> -- > >> >> Twitter developer documentation and resources: > >> >> http://dev.twitter.com/doc > >> >> API updates via Twitter: http://twitter.com/twitterapi > >> >> Issues/Enhancements Tracker: > >> >> http://code.google.com/p/twitter-api/issues/list > >> >> Change your membership to this group: > >> >> http://groups.google.com/group/twitter-development-talk > >> > > >> > -- > >> > Twitter developer documentation and resources: > >> > http://dev.twitter.com/doc > >> > API updates via Twitter: http://twitter.com/twitterapi > >> > Issues/Enhancements Tracker: > >> > http://code.google.com/p/twitter-api/issues/list > >> > Change your membership to this group: > >> > http://groups.google.com/group/twitter-development-talk > >> > > >> > >> -- > >> Twitter developer documentation and resources: > http://dev.twitter.com/doc > >> API updates via Twitter: http://twitter.com/twitterapi > >> Issues/Enhancements Tracker: > >> http://code.google.com/p/twitter-api/issues/list > >> Change your membership to this group: > >> http://groups.google.com/group/twitter-development-talk > > > > -- > > Twitter developer documentation and resources: > http://dev.twitter.com/doc > > API updates via Twitter: http://twitter.com/twitterapi > > Issues/Enhancements Tracker: > > http://code.google.com/p/twitter-api/issues/list > > Change your membership to this group: > > http://groups.google.com/group/twitter-development-talk > > > > > > -- > Adam Green > Twitter API Consultant and Trainer > http://140dev.com > @140dev > > -- > Twitter developer documentation and resources: http://dev.twitter.com/doc > API updates via Twitter: http://twitter.com/twitterapi > Issues/Enhancements Tracker: > http://code.google.com/p/twitter-api/issues/list > Change your membership to this group: > http://groups.google.com/group/twitter-development-talk > -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
