I think, you should implement some sort of pagination in your app to deal with this issue. We had similar problems for the project http://whofollowswhom.com and we settled with the approach #1 followed by querying profile info for the each profile. Though it is a costlier approach, we had to do this because of the easiness in pagination.
We still have a problem with big profiles and are in the process of migrating to cursors for followers/ids. On Oct 7, 10:59 am, jmathai <[email protected]> wrote: > So a user comes to the site and I need to build their social graph. I > have two options. > > 1) Use followers/ids and get ids of all their followers 5,000 at a > time > 2) Use statuses/followers and get profiles of all followers 100 at a > time > > Ids alone don't really do me much good. So option 2 is more efficient > for me (unless there's a batch user fetch api I'm unaware of). > > That being said, if a user has 10,000 followers (not uncommon) then I > have to make 100 API calls to fetch profiles for all the followers. > Not a big deal. Except, Twitter gives me random errors. Sometimes > it's a 502 and other times a 400. I'm not confident that I won't > randomly receive a different 4xx or 5xx response. > > I tried to put code in place so that on 5xx responses that I would > *continue* and retry the request. That's when I got a 400 response. > I have yet to fetch someone's entire graph using a sample account with > 13,000 followers. > > Has anyone successfully migrated to cursors and consistently pulled > down a large (>10k) graph?
