[twitter-dev] Re: How do you store the social graph locally?
We're using MongoDB and store them as arrays of integers as provided by the API. That makes for some simple yet powerful queries: http://www.mongodb.org/display/DOCS/Advanced+Queries#AdvancedQueries-ValueinanArray Wynn Netherland twitter: pengwynn http://floxee.com http://tweetcongress.org On Sat, Aug 15, 2009 at 9:24 AM, Arik Fraimovich arik...@gmail.com wrote: I was wondering how you store a local cache of the social graph methods results locally in your applications. One obvious solution is to create a two column table of the relations, but in such case how do you update it? Just prune everything of the user you're updating and inserting from the beginning? The other solution is to store the results of the API calls as blobs to the DB and parse them everytime in code instead of by SQL queries. The problem I can see with that is duplication of data, less ability to do smart stuff with the data and other issues. Would love to hear how you implemented it in your apps and other ideas related. Thanks, -- Arik Fraimovich follow me on twitter: http://twitter.com/arikfr -- Wynn Netherland twitter: pengwynn
[twitter-dev] Re: How do you store the social graph locally?
Your implementation here has much to do with how you intend to use the social graph. Are you simply caching, or do you intend to identify metrics by analyzing the shape of the relationships over time? If you're simply creating a local cache, blowing away the existing store and serializing the response from the api call is sufficient, since you cannot divide the results except by page. If you intend to get a little fancier, there are plenty of algorithms for diff'ing the results. Your saving grace is that the result set is a list of integers. If you take a little time to sort results, you will be able to perform your diffs swiftly. my two cents, not adjusted for inflation. On Sat, Aug 15, 2009 at 10:24 AM, Arik Fraimovicharik...@gmail.com wrote: On Aug 15, 10:56 pm, Kevin Mesiab ke...@mesiablabs.com wrote: If you store them as blobs, we're going to revoke your compiler privileges. Good thing that lately I was mainly doing PHP or Python, so no compiler privileges were needed - only parser Any other comments on the question in hand? Thanks, -- Arik Fraimovich follow me on twitter: http://twitter.com/arikfr -- Kevin Mesiab CEO, Mesiab Labs L.L.C. http://twitter.com/kmesiab http://mesiablabs.com http://retweet.com
[twitter-dev] Re: How do you store the social graph locally?
If you store them as blobs, we're going to revoke your compiler privileges. :P On Sat, Aug 15, 2009 at 4:24 AM, Arik Fraimovicharik...@gmail.com wrote: I was wondering how you store a local cache of the social graph methods results locally in your applications. One obvious solution is to create a two column table of the relations, but in such case how do you update it? Just prune everything of the user you're updating and inserting from the beginning? The other solution is to store the results of the API calls as blobs to the DB and parse them everytime in code instead of by SQL queries. The problem I can see with that is duplication of data, less ability to do smart stuff with the data and other issues. Would love to hear how you implemented it in your apps and other ideas related. Thanks, -- Arik Fraimovich follow me on twitter: http://twitter.com/arikfr -- Kevin Mesiab CEO, Mesiab Labs L.L.C. http://twitter.com/kmesiab http://mesiablabs.com http://retweet.com
