[twitter-dev] Re: How do you store the social graph locally?

2009-08-16 Thread Wynn Netherland
We're using MongoDB and store them as arrays of integers as provided by the
API.
That makes for some simple yet powerful queries:

http://www.mongodb.org/display/DOCS/Advanced+Queries#AdvancedQueries-ValueinanArray

Wynn Netherland
twitter: pengwynn
http://floxee.com
http://tweetcongress.org

On Sat, Aug 15, 2009 at 9:24 AM, Arik Fraimovich  wrote:

>
> I was wondering how you store a local cache of the social graph
> methods results locally in your applications.
>
> One obvious solution is to create a two column table of the relations,
> but in such case how do you update it? Just prune everything of the
> user you're updating and inserting from the beginning? The other
> solution is to store the results of the API calls as blobs to the DB
> and parse them everytime in code instead of by SQL queries. The
> problem I can see with that is duplication of data, less ability to do
> smart stuff with the data and other issues.
>
> Would love to hear how you implemented it in your apps and other ideas
> related.
>
> Thanks,
>
> --
> Arik Fraimovich
> follow me on twitter: http://twitter.com/arikfr




-- 
Wynn Netherland
twitter: pengwynn


[twitter-dev] Re: How do you store the social graph locally?

2009-08-16 Thread mcfnord

I'm storing integer arrays, a feature of PostgreSQL.

On Aug 15, 1:36 pm, Kevin Mesiab  wrote:
> Your implementation here has much to do with how you intend to use the
> social graph.
>
> Are you simply caching, or do you intend to identify metrics by
> analyzing the shape of the relationships over time?
>
> If you're simply creating a local cache, blowing away the existing
> store and serializing the response from the api call is sufficient,
> since you cannot divide the results except by page.
>
> If you intend to get a little fancier, there are plenty of algorithms
> for diff'ing the results.  Your saving grace is that the result set is
> a list of integers.  If you take a little time to sort results, you
> will be able to perform your diffs swiftly.
>
> my two cents, not adjusted for inflation.
>
> On Sat, Aug 15, 2009 at 10:24 AM, Arik Fraimovich wrote:
>
> > On Aug 15, 10:56 pm, Kevin Mesiab  wrote:
> >> If you store them as blobs, we're going to revoke your compiler privileges.
>
> > Good thing that lately I was mainly doing PHP or Python, so no
> > compiler privileges were needed - only parser
>
> > Any other comments on the question in hand?
>
> > Thanks,
> > --
> > Arik Fraimovich
> > follow me on twitter:http://twitter.com/arikfr
>
> --
> Kevin Mesiab
> CEO, Mesiab Labs 
> L.L.C.http://twitter.com/kmesiabhttp://mesiablabs.comhttp://retweet.com


[twitter-dev] Re: How do you store the social graph locally?

2009-08-15 Thread Kevin Mesiab

If you store them as blobs, we're going to revoke your compiler privileges.

:P

On Sat, Aug 15, 2009 at 4:24 AM, Arik Fraimovich wrote:
>
> I was wondering how you store a local cache of the social graph
> methods results locally in your applications.
>
> One obvious solution is to create a two column table of the relations,
> but in such case how do you update it? Just prune everything of the
> user you're updating and inserting from the beginning? The other
> solution is to store the results of the API calls as blobs to the DB
> and parse them everytime in code instead of by SQL queries. The
> problem I can see with that is duplication of data, less ability to do
> smart stuff with the data and other issues.
>
> Would love to hear how you implemented it in your apps and other ideas
> related.
>
> Thanks,
>
> --
> Arik Fraimovich
> follow me on twitter: http://twitter.com/arikfr
>



-- 
Kevin Mesiab
CEO, Mesiab Labs L.L.C.
http://twitter.com/kmesiab
http://mesiablabs.com
http://retweet.com


[twitter-dev] Re: How do you store the social graph locally?

2009-08-15 Thread Kevin Mesiab

Your implementation here has much to do with how you intend to use the
social graph.

Are you simply caching, or do you intend to identify metrics by
analyzing the shape of the relationships over time?

If you're simply creating a local cache, blowing away the existing
store and serializing the response from the api call is sufficient,
since you cannot divide the results except by page.

If you intend to get a little fancier, there are plenty of algorithms
for diff'ing the results.  Your saving grace is that the result set is
a list of integers.  If you take a little time to sort results, you
will be able to perform your diffs swiftly.

my two cents, not adjusted for inflation.


On Sat, Aug 15, 2009 at 10:24 AM, Arik Fraimovich wrote:
>
> On Aug 15, 10:56 pm, Kevin Mesiab  wrote:
>> If you store them as blobs, we're going to revoke your compiler privileges.
>
> Good thing that lately I was mainly doing PHP or Python, so no
> compiler privileges were needed - only parser
>
> Any other comments on the question in hand?
>
> Thanks,
> --
> Arik Fraimovich
> follow me on twitter: http://twitter.com/arikfr
>



-- 
Kevin Mesiab
CEO, Mesiab Labs L.L.C.
http://twitter.com/kmesiab
http://mesiablabs.com
http://retweet.com


[twitter-dev] Re: How do you store the social graph locally?

2009-08-15 Thread Arik Fraimovich

On Aug 15, 10:56 pm, Kevin Mesiab  wrote:
> If you store them as blobs, we're going to revoke your compiler privileges.

Good thing that lately I was mainly doing PHP or Python, so no
compiler privileges were needed - only parser

Any other comments on the question in hand?

Thanks,
--
Arik Fraimovich
follow me on twitter: http://twitter.com/arikfr