Re: [Neo] Dispell the myth? FlockDB vs. Neo4j
Hmm...it's trivial to implement just user-followers in a relational db directly. Sounds like what they're mainly doing is just adding an easy partitioning layer. But they just released partitioning as a separate project, Gizzard: http://engineering.twitter.com/2010/04/introducing-gizzard-framework-for.html On that page they mention that FlockDb is built on top of Gizzard. I'm having trouble figuring out just what FlockDb really adds. On Tue, Apr 13, 2010 at 7:20 AM, Peter Neubauer peter.neuba...@neotechnology.com wrote: Mmh, FlockDB seems to be focused on making the graph very flat (just user-followers) in order to be able to partition it. In that respect, it almost implements a document model. I think the most interesting to start with would be to implement it under Gremlin and run some algos on in in order to test. Emil will write up more on FlockDB, but I imagine the characteristics are more these of a Document DB than of a Graph DB. We will need to get a better grip on it before jumping to conclusions though. Cheers, /peter neubauer COO and Sales, Neo Technology GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://nosqleu.org- The biggest NOSQL event. Ever. http://www.thoughtmade.com - Scandinavias coolest Bring-a-Thing party. On Tue, Apr 13, 2010 at 1:11 PM, Jeremy Day jeremy@gmail.com wrote: All, I haven't looked at FlockDB at all, but would it be possible to perhaps implement the Neo API on top of it? Jeremy On Tue, Apr 13, 2010 at 5:26 AM, Laurent Laborde kerdez...@gmail.com wrote: I just heavily twitted about it :) I'd call FlockDB a Key/Value/Relationship Store (KVRstore), but not a graphdb :) -- Laurent ker2x Laborde Sysadmin DBA at http://www.over-blog.com/ ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Heresy
Maybe something like Google's Protocol Buffers? They use it internally, efficient binary protocol, java bindings from google and a third-party c# library. http://code.google.com/apis/protocolbuffers/docs/encoding.html On Fri, Feb 5, 2010 at 12:12 PM, Rick Bullotta rick.bullo...@burningskysoftware.com wrote: There are big performance and scalability challenges with most REST-ful interfaces when compared to local invocations, largely due to things like connection establishment, two-ended serialization, limits on # of connections/sockets, and so on. Don't get me wrong: It works well, I'm a big fan of the approach and have implemented REST-ful interfaces to every product I've ever done. But there is typically a 1-2X order of magnitude performance gap, depending on how you implement it. Interestingly, by eliminating/reducing serialization overhead (avoid most frameworks' built-in serialization, which usually sucks), you can eliminate a big piece of that performance hit. What I've found works well is supporting multiple wire formats - XML/JSON for the loosely coupled consumers, and binary and/or streaming formats for close friends or non-browser clients. -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Taylor Cowan Sent: Friday, February 05, 2010 11:51 AM To: Neo user discussions Subject: Re: [Neo] Heresy Rick is correct in highlighting this. A lot of people (java developers) will be looking for something to pivot on. Perhaps ready to role client libs for the REST interface under work. Nolios did some kind of integration with .net and their restlets engine. Taylor - Original Message From: Rick Bullotta rick.bullo...@burningskysoftware.com To: Neo user discussions user@lists.neo4j.org Sent: Fri, February 5, 2010 10:41:23 AM Subject: Re: [Neo] Heresy Hi, Peter. Having ported big applications and libraries to and from .NET and Java, in general, a rewrite is the best approach. The code looks surprisingly similar, but exploiting platform and framework-specific capabilities for file I/O, threading, memory management, transactioning, and so on are essential. The so-called interpreters/translators, at the bytecode or source level, generally don't work well except in a few edge cases. Rick -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Peter Neubauer Sent: Friday, February 05, 2010 11:35 AM To: Neo user discussions Subject: Re: [Neo] Heresy Hi there, I know of some minor attempts to do that, and some major that might be underway, but never heard back from the persons involved. Not sure on the status of such thinking. We would love to do such a bridge, but it seems it is still after all these years not trivial to either run the Java code directly in .NET with something like IKVM, or to properly bridge things. Very strange since the platforms are very close to each other in thinking. Cheers, /peter neubauer COO and Sales, Neo Technology GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org- Your high performance graph database. http://gremlin.tinkerpop.com- The terminal to the Giant Global Graph. On Fri, Feb 5, 2010 at 5:23 PM, Rick Bullotta rick.bullo...@burningskysoftware.com wrote: Yes, yes, I know Microsoft is pure evil and all (though not as evil as Apple or Google these days), but has anyone considered the feasibility of a C#/.NET port of Neo? Just a curiosity more than anything else. ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] neo4j with terracotta?
Just curious, is there anything else people are doing with Neo4J to meet high-availability requirements? On Tue, Aug 18, 2009 at 2:23 AM, Peter Neubauer neubauer.pe...@gmail.comwrote: Ok, no, we have not looked into that, if not someone else had a try. If you do, please let us others know your findings! /peter GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org- New Energy for Data - The Graph Database. http://www.ops4j.org- New Energy for OSS Communities - Open Participation Software. http://www.linkedprocess.org - Distributed computing on LinkedData scale On Mon, Aug 17, 2009 at 2:06 PM, Marc Preddiempred...@gmail.com wrote: Hi Peter, Initially simply looking to replicate for high availability. Regards, Marc On Sun, Aug 16, 2009 at 2:01 PM, Peter Neubauer neubauer.pe...@gmail.comwrote: Mmh, have not heard anyone doing that, what kind of scaling are you imagining? Replicating a nodespace in a high availability scenario, sharding or partitioning? /peter GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org- New Energy for Data - The Graph Database. http://www.ops4j.org- New Energy for OSS Communities - Open Participation Software. http://www.linkedprocess.org - Distributed computing on LinkedData scale On Fri, Aug 14, 2009 at 7:54 PM, Marc Preddiempred...@gmail.com wrote: Hi All, I was just wondering if any users of neo4j has had to opportunity to integrate it with terracotta? Regards, Marc Preddie ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Write performance? Contention? How's this design?
Sounding better all the time, thanks! I think I'm sold on giving it a shot. util.map solves another problem I had, I thought I'd need something external for that. I'm more-or-less new to the java platform and planning to use scala or clojure, so I've got a lot of juicy work in front of me. On Wed, Jun 17, 2009 at 9:57 AM, Johan Svensson jo...@neotechnology.comwrote: On Wed, Jun 17, 2009 at 5:22 AM, Dennis Petersondennisbpeter...@gmail.com wrote: Sounds good...for your (b) I see that a traverser would easily do it, or I may do it client-side to show stubs for hidden comments. I'll also want (a), a global list of high-rated comments, which is why I'm thinking I'd maintain a list of the top 1000 or so. Comment ratings will degrade over time so nodes should drop out of this list. Yes, traverser for (b) and some type of list for (a). Have a look at the SortedTree and Timeline utilities in index-util component, they may help you to do (a). I'll want to do something like give a user a sum of the points he's accrued across his comments...assuming worst case of a couple thousand comments, would I likely be better off updating his sum whenever a comment of his is rated, or simply traversing his comments and summing them up when I need to? I think traversing will work well when you only have a couple of thousands of comments. In the event that I'm very lucky and need more than 200 transactions/s, would it work better to do updates in batches every few seconds? (Of course I guess then I'd need to temporarily save the update instructions somewhere outside the system.) Yes. Having a single thread running a transaction that batches work together will speed things up. Getting ahead of myself a bit: anything noteworthy about ram requirements? There is some information about it on the wiki (http://wiki.neo4j.org/content/FAQ). Amount of RAM needed depends on how much data you have. On 6/16/09, Johan Svensson jo...@neotechnology.com wrote: ... Does getting the highest rated comments mean: a) of all the latest comments since time X, give me the ones that had the highest rating or b) for this topic / thread, only show comments with rating higher then X ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo] Write performance? Contention? How's this design?
Hi everybody, I just came across Neo4J, and while the read performance sounds great, I haven't seen anything about write performance. I suspect it's good, but wanted to ask. I'm considering using Neo for a threaded discussion forum. It definitely sounds like it would retrieve a discussion tree very quickly. Would it also be fast in posting new messages? Root comments would be attached to a topic, so by starting at the topic's node you can get a list of the latest posts. It seems the way to do this would essentially be to make a linked list of posts, and with each new one, splice the new node in between the topic node and the previous head of the list. Would there likely be undue contention at that point? I'll also be rating comments, and would like to get a list of highest-rated comments. For that, I would likely keep a list of them. Keep the current low score somewhere, refresh it periodically with a scan of the list, anything above the low score just add it to the list, and periodically purge the excess. In a relational database of course it's easy, just sort by the scores...but that score index is going to be constantly re-sorting itself, so there's plenty going on, you just don't see it. I suspect Neo would actually be more performant since you don't have to keep *all* the comments sorted by score. Am I on the right track? ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Write performance? Contention? How's this design?
Sounds good...for your (b) I see that a traverser would easily do it, or I may do it client-side to show stubs for hidden comments. I'll also want (a), a global list of high-rated comments, which is why I'm thinking I'd maintain a list of the top 1000 or so. Comment ratings will degrade over time so nodes should drop out of this list. I'll want to do something like give a user a sum of the points he's accrued across his comments...assuming worst case of a couple thousand comments, would I likely be better off updating his sum whenever a comment of his is rated, or simply traversing his comments and summing them up when I need to? In the event that I'm very lucky and need more than 200 transactions/s, would it work better to do updates in batches every few seconds? (Of course I guess then I'd need to temporarily save the update instructions somewhere outside the system.) Getting ahead of myself a bit: anything noteworthy about ram requirements? On 6/16/09, Johan Svensson jo...@neotechnology.com wrote: Hi Dennis, As Neil mentioned a well configured batch inserter typically gives you 5x or higher write speed compared to normal writes running in a transaction. Batch inserter is there to speed up initial import of data but judging from the use-cases you presented you will be running in normal transactional mode. In a large transaction (thousands of node, relationship and/or property creates/updates/deletes) you do about 10-100 writes per ms. Running small write transactions with just a few updates in each transaction the bottleneck will be the flush operations and then you typically get around 200 transactions/s (numbers are very hardware dependent). Regarding contention, by default a write will not block a read but writes have to wait for each other. Modifying a node's or relationship's properties will acquire a write lock on that node or relationship holding it until the transaction commits or gets rolled back. Deleting or creating a relationship will acquire a write lock on the relationship, the start node and the end node. I think you are on the right track considering using Neo4j for this problem :) If you could give us some more information about how exactly the rating system is going to work we could advice you on how to design your graph. For example: Does getting the highest rated comments mean: a) of all the latest comments since time X, give me the ones that had the highest rating or b) for this topic / thread, only show comments with rating higher then X Regards, Johan ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
