This could help - http://www.flotzam.com/archivist/
On Jul 17, 6:57 am, CreativeEye <creativv...@gmail.com> wrote: > Myself and my friend are doing a research based on twitter. We need to > analyse each and every tweet real time. Can you guide how to approach > this. > > There could be 2 ways of doing this (without Firehose): > > 1) Get Twitter Public timeline repeatedly. > > Thankfully Twitter's caching has not been problem to me, they seem to > fetch me new data every request. But there are a lot of limitation for > this: > > According to TweeSpeed.com: > - Rate of New tweets in the Twitter Server is right now (Wed Jul 17 > 11:47:02 - GMT) at 9233 tweets/minute. > - Ranges between 7K to 20K on an average Weekday. > - On June 26 (MJ's death) - reached 25K tweets/minute. > > Let us now consider the limitation of API requests per hour. > - Currently @ 20K per hour. > - 1 Req = 20 Tweets > - Need 1K Req per minute = 60K req per hour. > > To Use 1K Requests per minute, we should be using around 17 requests > per second. But my server is able to process only 28-33 requests/ > minute. > > Is this the right way to proceed, or am I fundamentally wrong on the > approach. > > 2) Get follower network - user profiles and get their statuses. > Frequency of request their new status updates could be set against > their general update frequency. But this is Google-like old way of > indexing things, which does not quite stand today in the REAL TIME > twitter. > > I do know Firehose is an option, but that would again be something > like Approach 1. right? > > Please guide me how to proceed.
