Of course :) I'll arm chair conjecture a bit also, since while I do work here I'm definitely not the caliber of engineer as my colleagues, and certainly not very knowledge in what it takes to scale a service like Twitter:
Things like followers/ids and friends/ids are likely accessible easily because the social graph is a necessary component to delivering tweets properly: who gets what, in what way, on what timeline, etc. The entire historical archive of tweets for a given user is not required to remain in memory to perform these mission critical tasks -- whether that's just ids of tweets or otherwise. Again. Don't know nothing about nothing, but this conjecture seems reasonable. Taylor On Thu, Sep 30, 2010 at 2:35 PM, Abraham Williams <4bra...@gmail.com> wrote: > Us outsiders have to get our pokes and prods in while we can :-P > > Abraham > ------------- > Abraham Williams | Hacker Advocate | http://abrah.am > @abraham | http://projects.abrah.am | http://blog.abrah.am > This email is: [ ] shareable [x] ask first [ ] private. > > > On Thu, Sep 30, 2010 at 14:19, Taylor Singletary > <taylorsinglet...@twitter.com> wrote: >> >> Theorizing from the outside-in on our capacity issues aside, I'm a big >> advocate for a bulk status/show or lookup function. We're definitely >> giving that a lot of thought at the moment. >> >> Taylor >> >> On Thu, Sep 30, 2010 at 2:10 PM, Abraham Williams <4bra...@gmail.com> >> wrote: >> > Considering Twitter can support returning the ids of almost 300,000 >> > followers then 40,000 tweets should be easy. >> > http://api.twitter.com/1/followers/ids.json?screen_name=rsarver >> > >> > Abraham >> > ------------- >> > Abraham Williams | Hacker Advocate | http://abrah.am >> > @abraham | http://projects.abrah.am | http://blog.abrah.am >> > This email is: [ ] shareable [x] ask first [ ] private. >> > >> > >> > On Thu, Sep 30, 2010 at 14:04, Orian Marx (@orian) <or...@orianmarx.com> >> > wrote: >> >> >> >> This should probably be a separate thread, but... what about a bulk >> >> tweet lookup using status_ids for one very specific use case: turning >> >> search results into proper tweets and avoiding all the other issues >> >> that exist with the current implementation? I know bulk tweet lookup >> >> by id has been asked before and the response is it's the same capacity >> >> issue of accessing old tweets from storage. Okay, so what about a bulk >> >> tweet lookup limited to timestamps that fall within whatever range >> >> search capacity is currently at (ya know, like the last seven days)? I >> >> bet this could actually reduce the number of calls developers are >> >> making when dealing with converting search result items into proper >> >> status objects. >> >> >> >> On Sep 30, 4:57 pm, Taylor Singletary <taylorsinglet...@twitter.com> >> >> wrote: >> >> > I'll agree with all of you that'd it be valuable for us to do this. >> >> > >> >> > The current state of availability of tweets is a capacity issue. It's >> >> > not in anyway a deliberate prevention of access. >> >> > >> >> > As for since_id in this context -- it'd be great if it'd work to just >> >> > use since_id=1, but it doesn't. I don't even think it'd work if you >> >> > knew the very first status id that the user had ever posted.. because >> >> > in most cases, that status id would be unknown to the back end >> >> > transport layer (for the same reason it can't access the tweet >> >> > itself). >> >> > >> >> > If we could provide a means today to allow complete traversal of a >> >> > user_timeline for a specific user with over ~3200 tweets, we would -- >> >> > and I'm pretty confident we will. But not today and not tomorrow. >> >> > >> >> > Taylor >> >> > >> >> > On Thu, Sep 30, 2010 at 1:43 PM, M. Edward (Ed) Borasky >> >> > >> >> > >> >> > >> >> > <zn...@borasky-research.net> wrote: >> >> > > Retrieving one's GMail, for example, if one wants to delete the >> >> > > account, is >> >> > > a decidedly non-trivial exercise. Perhaps it's simple with Outlook, >> >> > > but >> >> > > KMail, Evolution and Thunderbird all gave me a ration of doo-doo in >> >> > > the >> >> > > process, and I'm not sure I did it right. >> >> > > -- >> >> > > M. Edward (Ed) Borasky >> >> > >http://borasky-research.nethttp://twitter.com/znmeb >> >> > >> >> > > "A mathematician is a device for turning coffee into theorems." - >> >> > > Paul >> >> > > Erdos >> >> > >> >> > > Quoting "Orian Marx (@orian)" <or...@orianmarx.com>: >> >> > >> >> > >> It remains a good idea. Imagine if Gmail only let you retrieve >> >> > >> your >> >> > >> last 3200 messages even if you had 40,000. >> >> > >> >> > >> On Sep 30, 4:20 pm, "M. Edward (Ed) Borasky" <zn...@borasky- >> >> > >> research.net> wrote: >> >> > >> >> > >>> I also posted a request a long time ago that an authenticated >> >> > >>> user >> >> > >>> be >> >> > >>> able to retrieve all of his own tweets, back beyond the 16-page >> >> > >>> limit. >> >> > >>> In retrospect, now that I'm within shooting distance of 40,000 >> >> > >>> I'm >> >> > >>> not >> >> > >>> sure how good an idea that is. ;-) >> >> > >> >> > >>> -- >> >> > >>> M. Edward (Ed) >> >> > >>> Boraskyhttp://borasky-research.nethttp://twitter.com/znmeb >> >> > >> >> > >>> "A mathematician is a device for turning coffee into theorems." - >> >> > >>> Paul >> >> > >>> Erdos >> >> > >> >> > >>> Quoting Abraham Williams <4bra...@gmail.com>: >> >> > >> >> > >>> > Here is a closed feature request from forever ago to return the >> >> > >>> > status_id of >> >> > >>> > all statuses for a user. Maybe Twitter wil reconsider the >> >> > >>> > request. >> >> > >> >> > >>> >http://code.google.com/p/twitter-api/issues/detail?id=379 >> >> > >> >> > >>> > Abraham >> >> > >>> > ------------- >> >> > >>> > Abraham Williams | Hacker Advocate |http://abrah.am >> >> > >>> > @abraham |http://projects.abrah.am|http://blog.abrah.am >> >> > >>> > This email is: [ ] shareable [x] ask first [ ] private. >> >> > >> >> > >>> > On Thu, Sep 30, 2010 at 11:32, linuslive >> >> > >>> > <michael.c....@gmail.com> >> >> > >>> > wrote: >> >> > >> >> > >>> >> Hmmm...but there is no way to know the ids of the tweets >> >> > >>> >> without >> >> > >>> >> having a list of all of the tweets, which would kind of defeat >> >> > >>> >> the >> >> > >>> >> purpose of the limit. >> >> > >> >> > >>> >> Oh well... >> >> > >> >> > >>> >> On Sep 30, 12:39 pm, Tom van der Woerdt <i...@tvdw.eu> wrote: >> >> > >>> >> > Only the 3200 most recent ones, unless you know the IDs of >> >> > >>> >> > the >> >> > >>> >> > tweets. >> >> > >> >> > >>> >> > Tom >> >> > >> >> > >>> >> > On 9/30/10 5:09 PM, linuslive wrote: >> >> > >> >> > >>> >> > > Good morning all! >> >> > >> >> > >>> >> > > Before the OAuth change, I wrote a twitter archiver that >> >> > >>> >> > > would >> >> > >>> >> > > grab my >> >> > >>> >> > > tweets and dump them into a database. Back then I only >> >> > >>> >> > > had >> >> > >>> >> > > about >> >> > >>> >> > > 1K >> >> > >>> >> > > tweets so I was able to grab all my tweets without hitting >> >> > >>> >> > > the >> >> > >>> >> > > limit. >> >> > >>> >> > > And of course since then, I don't post more than 20 tweets >> >> > >>> >> > > a >> >> > >>> >> > > day. >> >> > >> >> > >>> >> > > But today, I was curious to see if I could use the new >> >> > >>> >> > > OAuth >> >> > >>> >> > > to >> >> > >>> >> > > retrieve my timeline, starting from my first tweet. I >> >> > >>> >> > > can't >> >> > >>> >> > > seem >> >> > >>> >> > > to >> >> > >>> >> > > get it. >> >> > >> >> > >>> >> > > Here are my parameters: >> >> > >> >> > >>> >> > > >> >> > >>> >> > > > >> >> > >>> >> > > > > >http://api.twitter.com/1/statuses/user_timeline.json?screen_name=linu. >> >> > >>> >> .. >> >> > >> >> > >>> >> > > When I do this, it only gives me the top 200 tweets. It >> >> > >>> >> > > doesn't >> >> > >>> >> > > start >> >> > >>> >> > > from the beginning and give me my 200 oldest tweets, as I >> >> > >>> >> > > would >> >> > >>> >> > > expect. >> >> > >> >> > >>> >> > > I've tried various since_id's, even retrieving my first >> >> > >>> >> > > tweet: >> >> > >>> >> > >http://twitter.com/linuslive/statuses/872090728 >> >> > >> >> > >>> >> > > Which equals >> >> > >> >> > >>> >> > > >> >> > >>> >> > > > >> >> > >>> >> > > > > >http://api.twitter.com/1/statuses/user_timeline.json?screen_name=linu. >> >> > >>> >> .. >> >> > >> >> > >>> >> > > And I get the same response. >> >> > >> >> > >>> >> > > According to twitter, I have 3682 tweets, just over the >> >> > >>> >> > > 3200 >> >> > >>> >> > > limit >> >> > >>> >> > > according >> >> > >>> >> > > tohttp://dev.twitter.com/doc/get/statuses/user_timeline. >> >> > >>> >> > > The confusing thing is that the documentation does not >> >> > >>> >> > > make >> >> > >>> >> > > it >> >> > >>> >> > > clear >> >> > >>> >> > > that it's 3200 of the most recent tweets, or just 3200 per >> >> > >>> >> > > session. >> >> > >> >> > >>> >> > > Could I get a clarification on this? >> >> > >> >> > >>> >> > > Thanks!- Hide quoted text - >> >> > >> >> > >>> >> > - Show quoted text - >> >> > >> >> > >>> >> -- >> >> > >>> >> Twitter developer documentation and >> >> > >>> >> resources:http://dev.twitter.com/doc >> >> > >>> >> API updates via Twitter:http://twitter.com/twitterapi >> >> > >>> >> Issues/Enhancements Tracker: >> >> > >>> >>http://code.google.com/p/twitter-api/issues/list >> >> > >>> >> Change your membership to this group: >> >> > >>> >>http://groups.google.com/group/twitter-development-talk >> >> > >> >> > >>> > -- >> >> > >>> > Twitter developer documentation and >> >> > >>> > resources:http://dev.twitter.com/doc >> >> > >>> > API updates via Twitter:http://twitter.com/twitterapi >> >> > >>> > Issues/Enhancements >> >> > >>> > Tracker:http://code.google.com/p/twitter-api/issues/list >> >> > >>> > Change your membership to this group: >> >> > >>> >http://groups.google.com/group/twitter-development-talk >> >> > >> >> > >> -- >> >> > >> Twitter developer documentation and >> >> > >> resources:http://dev.twitter.com/doc >> >> > >> API updates via Twitter:http://twitter.com/twitterapi >> >> > >> Issues/Enhancements Tracker: >> >> > >>http://code.google.com/p/twitter-api/issues/list >> >> > >> Change your membership to this group: >> >> > >> http://groups.google.com/group/twitter-development-talk >> >> > >> >> > > -- >> >> > > Twitter developer documentation and >> >> > > resources:http://dev.twitter.com/doc >> >> > > API updates via Twitter:http://twitter.com/twitterapi >> >> > > Issues/Enhancements Tracker: >> >> > >http://code.google.com/p/twitter-api/issues/list >> >> > > Change your membership to this group: >> >> > >http://groups.google.com/group/twitter-development-talk >> >> >> >> -- >> >> Twitter developer documentation and resources: >> >> http://dev.twitter.com/doc >> >> API updates via Twitter: http://twitter.com/twitterapi >> >> Issues/Enhancements Tracker: >> >> http://code.google.com/p/twitter-api/issues/list >> >> Change your membership to this group: >> >> http://groups.google.com/group/twitter-development-talk >> > >> > -- >> > Twitter developer documentation and resources: >> > http://dev.twitter.com/doc >> > API updates via Twitter: http://twitter.com/twitterapi >> > Issues/Enhancements Tracker: >> > http://code.google.com/p/twitter-api/issues/list >> > Change your membership to this group: >> > http://groups.google.com/group/twitter-development-talk >> > >> >> -- >> Twitter developer documentation and resources: http://dev.twitter.com/doc >> API updates via Twitter: http://twitter.com/twitterapi >> Issues/Enhancements Tracker: >> http://code.google.com/p/twitter-api/issues/list >> Change your membership to this group: >> http://groups.google.com/group/twitter-development-talk > > -- > Twitter developer documentation and resources: http://dev.twitter.com/doc > API updates via Twitter: http://twitter.com/twitterapi > Issues/Enhancements Tracker: > http://code.google.com/p/twitter-api/issues/list > Change your membership to this group: > http://groups.google.com/group/twitter-development-talk > -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
