If you're around Brisbane next week, some of the neo4j guys will be at BroSQL [1]. They're also ding YOW in Melbourne from what I recall.
I used to work (2000-2004) for a company that built a graph DB, this kind of performance was pretty standard back then, actually from memory we were loading ~4-8K records (well triples) per second linearly with DB sizes in the hundreds of millions of records [2]. It's really about having the right kind of structure for the data, sounds like your data just isn't right for an SQL DB. For nostalgia, I did some more digging & found this old whitepaper [3] which kind of explains it (~p10), may be helpful. Tom [1] http://www.brosql.org/events/40496652/ [2] http://www.w3.org/2004/04/13-swdd/#rdfstore [3] http://itee.uq.edu.au/~dwood/docs/new_type_of_data_management.pdf On Thu, Dec 1, 2011 at 1:18 PM, Dmytrii Nagirniak <[email protected]> wrote: > Yeah, really interesting. > > Another figure. > > Inserting ~1000 relationships (batched and transactional!) already having > the DB with ~4 million entries. > It took ~450 ms. And it's over HTTP, not native bindings. > > I am really, really now considering neo4j. > Even if I'll have to give up all the ActiveRecord goodness and all the > current code. > > Unfortunately the choice of tooling is a little bit disappointing: > http://stackoverflow.com/questions/8335136/which-ruby-client-for-neo4j-rest-api > > > Cheers. > > On 01/12/2011, at 7:36 AM, Tim Uckun wrote: > > Wow those are impressive figures. I'll have to give neo4j a go. > > > > On Thu, Dec 1, 2011 at 2:42 AM, Dmytrii Nagirniak <[email protected]> wrote: > > Thanks Jason. > > > Yes, ancestry is definitely useful. > > > But I decided to play with neo4j for a bit. > > > Would be interesting to hear some experience from others with neo4j. > > > The challenge is the REST API. I can't find "decent" ActiveModel-ish gem for > > it. > > I am not ready (scared?) to embed the database and use JRuby even though it > > seems to be the simplest and easiest way to go. > > > It's pretty interesting so far. The database size is ~400000 "entries" and > > is under constant "write" load from ~100 Ruby processes. > > Most of the queries take ~3 ms, a bit more complex ones - ~12 ms. > > > I honestly can't imagine doing anything similar with SQL. > > > > Pretty imposed so far if not taking into account that I'll have to pretty > > much throw away all the ActiveRecord code and tests :( > > > Cheers, > > Dima > > http://www.ApproachE.com > > > > On 30/11/2011, at 9:39 PM, Jason Nah wrote: > > > Not sure if this is still useful... > > > We've used the ancestry gem (https://github.com/stefankroes/ancestry) with > > great success. It wasn't used in the matter that you described but might > > also be a suitable alternative. > > > Cheers, > > Jason > > > > On 29 November 2011 08:26, Dmytrii Nagirniak <[email protected]> wrote: > > > > A node is basically a permission (loaded with the information, such as > > subject, user, company etc) that is propagated through to the other nodes. > > > Is "propagating changes to all child node recursively" a strategy to > > improve query performance? > > > > Pretty much. So instead of traversing the path "upwards", I can update > > it's attributes and query on it. It is not always possible/easy to do > > though. > > For example: > > Company1 includes User1.1, User1.2 > > Company2 includes User2 > > Company includes User 3 > > > And given the permissions: > > User1.1 -- Allow read X--> User 3 > > User1.2 -- Allow write X--> User 2 > > User2 -- Allow write X --> User3 > > > In this case User3 should be able to read and write, but only when she is > > a part of Company 2 or 3 (both allow write), not Company 1 (only allowed > > reading). > > > > > If so, you might look at an ActiveRecord extension such as my "Arboreal" > > (shameless plug), or DHH's "acts_as_nested_set", to allow efficient queries > > on a tree structure. > > > https://github.com/mdub/arboreal > > https://github.com/rails/acts_as_nested_set > > > > I did presentation last year, that included an example of using Arboreal > > to query down a hierarchy (e.g. of roles or groups) while joining with > > another table (e.g. permissions). See page 10 of: > > > http://www.slideshare.net/mdubya/one-project-3-gems > > > > I think I remember your talk. I did like it a lot. > > I currently do something similar to fetch the subtrees using SQL LIKE and > > paths. > > The problem there is that I need to query all the parents for a condition. > > So that even if I retrieve the subtree, I will have to walk upchain on > > each node to verify the permissions. > > > If I will be able to propagate all the changes down, then it I can just > > query on the nodes themselves. > > Although it gets much more complicated as a lot of denormilised data will > > be stored... Leading me back to graph DBs :) > > > > In any case, thanks a lot to everybody for the suggestions. I am a little > > bit better aware of my options now. > > I'll actually go and try neo4j... (and will report back if somebody is > > interested). > > > Cheers, > > Dima. > > > > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Ruby or Rails Oceania" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group at > > http://groups.google.com/group/rails-oceania?hl=en. > > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Ruby or Rails Oceania" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group at > > http://groups.google.com/group/rails-oceania?hl=en. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Ruby or Rails Oceania" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group at > > http://groups.google.com/group/rails-oceania?hl=en. > > > -- > You received this message because you are subscribed to the Google Groups > "Ruby or Rails Oceania" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/rails-oceania?hl=en. > > > -- > You received this message because you are subscribed to the Google Groups > "Ruby or Rails Oceania" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/rails-oceania?hl=en. -- tom adams e:tomjadams<at>gmail.com -- You received this message because you are subscribed to the Google Groups "Ruby or Rails Oceania" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/rails-oceania?hl=en.
