I think, you should implement some sort of pagination in your app to
deal with this issue. We had similar problems for the project
http://whofollowswhom.com and we settled with the approach #1 followed
by querying profile info for the each  profile. Though it is a
costlier approach, we had to do this because of the easiness in
pagination.

We still have a problem with big profiles and are in the process of
migrating to cursors for followers/ids.

On Oct 7, 10:59 am, jmathai <jmat...@gmail.com> wrote:
> So a user comes to the site and I need to build their social graph.  I
> have two options.
>
> 1) Use followers/ids and get ids of all their followers 5,000 at a
> time
> 2) Use statuses/followers and get profiles of all followers 100 at a
> time
>
> Ids alone don't really do me much good.  So option 2 is more efficient
> for me (unless there's a batch user fetch api I'm unaware of).
>
> That being said, if a user has 10,000 followers (not uncommon) then I
> have to make 100 API calls to fetch profiles for all the followers.
> Not a big deal.  Except, Twitter gives me random errors.  Sometimes
> it's a 502 and other times a 400.  I'm not confident that I won't
> randomly receive a different 4xx or 5xx response.
>
> I tried to put code in place so that on 5xx responses that I would
> *continue* and retry the request.  That's when I got a 400 response.
> I have yet to fetch someone's entire graph using a sample account with
> 13,000 followers.
>
> Has anyone successfully migrated to cursors and consistently pulled
> down a large (>10k) graph?

Reply via email to