I migrated my code to cursors and successfully pulled large graphs in method #2.
My code handles each and every Twitter errors (at least those I got in the last few months)... which means that sometimes I wait for a short while and then resume, sometimes I wait for longer time, depends on the error code, In order not to get into an tight loop overloading the already overloaded Twitter server the wait time grows on the next attempts... Oh - and I use paralellism for multiple users, where each user has its own thread, so the code was synchronous to begin with... maybe that's the reason it was so easy to migrate the code to cursors. = Oren On Oct 15, 3:14 am, Tim Haines <[email protected]> wrote: > No - I pulled down the 7000 followers using the cursor calls - not > just the ids. > > Tim. > > On Oct 15, 1:48 pm, Michael Steuer <[email protected]> wrote: > > > > > So now that you pulled down 7000 IDs, are you making 7000 user/show calls to > > get the rest of the details? How's that working out? > > > On 10/14/09 5:03 PM, "Tim Haines" <[email protected]> wrote: > > > > I'm migrating my code now. I just pulled down 7000 users. If I get a > > > bad response to a call I'll retry it up to 5 times. > > > > It took > 20 mins and < 1 hour, which is going to be troublesome. > > > > Tim. > > > > On Oct 7, 6:59 pm, jmathai <[email protected]> wrote: > > >> So a user comes to the site and I need to build their social graph. I > > >> have two options. > > > >> 1) Use followers/ids and get ids of all their followers 5,000 at a > > >> time > > >> 2) Use statuses/followers and get profiles of all followers 100 at a > > >> time > > > >> Ids alone don't really do me much good. So option 2 is more efficient > > >> for me (unless there's a batch user fetch api I'm unaware of). > > > >> That being said, if a user has 10,000 followers (not uncommon) then I > > >> have to make 100 API calls to fetch profiles for all the followers. > > >> Not a big deal. Except, Twitter gives me random errors. Sometimes > > >> it's a 502 and other times a 400. I'm not confident that I won't > > >> randomly receive a different 4xx or 5xx response. > > > >> I tried to put code in place so that on 5xx responses that I would > > >> *continue* and retry the request. That's when I got a 400 response. > > >> I have yet to fetch someone's entire graph using a sample account with > > >> 13,000 followers. > > > >> Has anyone successfully migrated tocursorsand consistently pulled > > >> down a large (>10k) graph?
