I think this scenario is quite rare. Anyone who is following a thousand accounts is just sampling their feed, they don't care about every tweet. Also, a random population of 1000 accounts will not all tweet 5000 times a day, but a selected group might.
If someone insists on reading all of this, perhaps as a work function, they should use streaming. They can get all tweets from an arbitrary set public users with the follow function, or, on user streams, all tweets that the user follows. -John Kalucki http://twitter.com/jkalucki Infrastructure, Twitter Inc. On Sat, Jun 19, 2010 at 4:09 AM, Neuromaster <neuromas...@gmail.com> wrote: > Hi, > > Twitter is really a great tool for sharing and acquiring information. > Lately more and more people are starting to use it. That’s good news > because it becomes a trustable medium for information exchange. But, > paradoxically, it is bad news as well. While the number of users is > significantly increasing so is the number of possible candidates > (users) to follow. > > Empirical proof might help in understanding the problem: > > Suppose the user X is following 1000 users (which is not a rare > situation) > Suppose those users post in average 5 tweets/ day (again, a situation > not hard to accept) > User X is busy, but every morning for a limited period of time reads > the tweets of his friends. > That is ~5000 tweets. He chooses to skim through them and if something > good pops up he’ll pay more attention to it. > > At this moment the web site it’s not designed for this kind of > interaction. Pushing the “More” button it’s just not so practical. > One may think that the API comes to rescue, but actually it > doesn’t.While it comes with some features, those are limited > > The API is well suited for lightweight and mobile application where > the latest news is important. But for application that want to offer > some digging tools the API doesn’t offer much help. In the example > scenario the number of tweets accumulated in one day was ~5000 while > the limitation for the /statuses/home_timeline is ( at least in the > documentation) 3200. The problem here is that this limitation was ok > in the past, but now when more and more users are interacting through > tweeter this limitation becomes a real bottleneck. > > From what I’ve understood the reason for this behavior is server > overloading. The servers can’t handle this amount of data. > > > The question: > Is there a way to grant access to more data for developers? If not > directly at least through some kind of request (like in the case of > Rate Limiting) > > Thanks > > > On Jun 18, 11:36 pm, Taylor Singletary <taylorsinglet...@twitter.com> > wrote: >> Hi Folks, >> >> Many of you have noticed incorrect counts of how many tweets a given user >> has. In addition, you've noticed that statuses/user_timeline is only >> allowing around ~800 statuses to be returned in total. These issues should >> be resolved by the end of next week -- we'll be rebuilding the cached >> representations of these timelines and users which will repair the issue. >> This process takes some time and will be gradual and incremental. >> >> After consulting internally to get some clarity on the counts you should be >> expecting when using pagination, I wanted to share that while there's a >> maximum of about 3,200 statuses that you can retrieve for a given >> user_timeline (when under normal operating conditions), you'll find that >> statuses/home_timeline and statuses/friends_timeline do indeed have a >> maximum result set of around 800 tweets and have for some time. We'll be >> making sure the documentation is correct in this area where it's incorrect >> currently. >> >> Of course, all of these theoretical maximums have some variance as caches >> are re-aligned in the system. Sometimes you might find a little more >> available than these maximums. Sometimes a little less. Important to note >> that the total tweet count includes retweets, but some methods don't return >> retweets unless you explicitly request them with include_rts=true. >> >> Happy tweeting, >> Taylor >