As far as retrieving the large graphs from a DB, flat files are one way -
another is to just store the full graph (of ids) in a single column in the
database and parse on retrieval.  This is what FriendFeed is doing
currently, so they've said.  Dewald and I are both talking about this
because we're also having to duplicate this on our own servers, so we too
have to deal with the pains of the social graph.  (and oh the pain it is!)

On Sun, Sep 6, 2009 at 8:44 PM, Dewald Pretorius <> wrote:

> If I worked for Twitter, here's what I would have done.
> I would have grabbed the follower id list of the large accounts (those
> that usually kicked back 502s) and written them to flat files once
> every 5 or so minutes.
> When an API request comes in for that list, I'd just grab it from the
> flat file, instead of asking the DB to select 2+ million ids from
> amongst a few billion records, while it's trying to do a few thousand
> other selects at the same time.
> That's one way of getting rid of 502s on large social graph lists.
> Okay, the data is going to be 5 minutes out-dated. To that I say, so
> bloody what?
> Dewald

Reply via email to