[Neo4j] 20 billion primitives machine
Folks, I've got an application that has (will have) about 2 billion vertexes and maybe 8 billion edges (?). Maybe an avg of 4 properties per vertex -- with maybe an avg of 32 bytes/value. So I guess that's 16 billion primitives. Let's round to 20 billion. My edges estimate is a relatively uninformed guess. Just starting to dig into the data. Traversals will be relatively shallow. Concurrent access. Throughput is more important than latency. But latency should be better than maybe 50ms 99% of the time (allowing for some cache warming and some GC). I don't know much yet about locality. I'm not sure yet how sensitive the app will be to long GCs. We will need to do a big batch load, and writes will need to be fast in that phase. After that, we'll see more reads that writes. So I imagine a config for the batch load and another config for production. I understand cache sharding, application-level partitioning, and so forth. I'm wondering what I can do on a single machine -- and what that machine should look like. http://docs.neo4j.org/chunked/stable/configuration-jvm.html and http://wiki.neo4j.org/content/Neo4j_Performance_Guide are encouraging. And having knobs as documented at http://wiki.neo4j.org/content/Configuration_Settings is great. Nice work! I'm hoping I might be able to get away with 128GB RAM on 12 cores with data striped over a handful of disks (SSDs if required). We'll probably also need a cluster for both traffic and availability, but that's another topic. Does anybody have experience with a data set like this on a similar machine? How much RAM and how much disk -- and what kinds and in what configuration? Latency, throughput, general experience? Any production deployments? I'd appreciate any guidance or feedback. I'm happy to summarize later if that'd be helpful. BTW, my testbed uses Clojure with clojure.contrib.server-socket and https://github.com/wagjo/borneo. Very convenient! --Jamie ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] 20 billion primitives machine
Hi Jamie, Very interesting use case you have there. If you could just provide a few more bits of information about your data, I'd be able to provide a better analysis. * Is the distribution of relationships uniform across the nodes. If not, how much does it vary? * What kind of operation you want to do in the 50ms latency that you've specified. You just mentioned relatively shallow traversals, approximately how deep is that? The key figure to get at is how many relationships you need to traverse, and that is a function of the depth of the traversal and the distribution of the relationships. * Is the 50ms latency for each single request? What is the estimated number of concurrent requests? * What kind of properties are those 32 byte values? ASCII strings of length 32? * The bulk loading phase you describe, is it a one time import of initial data, or a regularly recurring thing? * In the bulk load phase, what form does the data have? Would it be possible to have that data in a format where each node is uniquely identified by a number (in both the node data listing, and the edge-list). Finally, do you have test data for these sizes? If possible I'd love to work with you on this, to get an good use case to work on for improving the large data story in Neo4j. Cheers, Tobias On Thu, Apr 14, 2011 at 3:12 PM, Jamie Stephens j...@morphism.com wrote: Folks, I've got an application that has (will have) about 2 billion vertexes and maybe 8 billion edges (?). Maybe an avg of 4 properties per vertex -- with maybe an avg of 32 bytes/value. So I guess that's 16 billion primitives. Let's round to 20 billion. My edges estimate is a relatively uninformed guess. Just starting to dig into the data. Traversals will be relatively shallow. Concurrent access. Throughput is more important than latency. But latency should be better than maybe 50ms 99% of the time (allowing for some cache warming and some GC). I don't know much yet about locality. I'm not sure yet how sensitive the app will be to long GCs. We will need to do a big batch load, and writes will need to be fast in that phase. After that, we'll see more reads that writes. So I imagine a config for the batch load and another config for production. I understand cache sharding, application-level partitioning, and so forth. I'm wondering what I can do on a single machine -- and what that machine should look like. http://docs.neo4j.org/chunked/stable/configuration-jvm.html and http://wiki.neo4j.org/content/Neo4j_Performance_Guide are encouraging. And having knobs as documented at http://wiki.neo4j.org/content/Configuration_Settings is great. Nice work! I'm hoping I might be able to get away with 128GB RAM on 12 cores with data striped over a handful of disks (SSDs if required). We'll probably also need a cluster for both traffic and availability, but that's another topic. Does anybody have experience with a data set like this on a similar machine? How much RAM and how much disk -- and what kinds and in what configuration? Latency, throughput, general experience? Any production deployments? I'd appreciate any guidance or feedback. I'm happy to summarize later if that'd be helpful. BTW, my testbed uses Clojure with clojure.contrib.server-socket and https://github.com/wagjo/borneo. Very convenient! --Jamie ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Tobias Ivarsson tobias.ivars...@neotechnology.com Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] 20 billion primitives machine
Tobias, Thanks very much for the response. I'll follow up with you privately, and then I can summarize here. --Jamie On Thu, Apr 14, 2011 at 6:24 PM, Tobias Ivarsson tobias.ivars...@neotechnology.com wrote: Hi Jamie, Very interesting use case you have there. If you could just provide a few more bits of information about your data, I'd be able to provide a better analysis. * Is the distribution of relationships uniform across the nodes. If not, how much does it vary? * What kind of operation you want to do in the 50ms latency that you've specified. You just mentioned relatively shallow traversals, approximately how deep is that? The key figure to get at is how many relationships you need to traverse, and that is a function of the depth of the traversal and the distribution of the relationships. * Is the 50ms latency for each single request? What is the estimated number of concurrent requests? * What kind of properties are those 32 byte values? ASCII strings of length 32? * The bulk loading phase you describe, is it a one time import of initial data, or a regularly recurring thing? * In the bulk load phase, what form does the data have? Would it be possible to have that data in a format where each node is uniquely identified by a number (in both the node data listing, and the edge-list). Finally, do you have test data for these sizes? If possible I'd love to work with you on this, to get an good use case to work on for improving the large data story in Neo4j. Cheers, Tobias On Thu, Apr 14, 2011 at 3:12 PM, Jamie Stephens j...@morphism.com wrote: Folks, I've got an application that has (will have) about 2 billion vertexes and maybe 8 billion edges (?). Maybe an avg of 4 properties per vertex -- with maybe an avg of 32 bytes/value. So I guess that's 16 billion primitives. Let's round to 20 billion. My edges estimate is a relatively uninformed guess. Just starting to dig into the data. Traversals will be relatively shallow. Concurrent access. Throughput is more important than latency. But latency should be better than maybe 50ms 99% of the time (allowing for some cache warming and some GC). I don't know much yet about locality. I'm not sure yet how sensitive the app will be to long GCs. We will need to do a big batch load, and writes will need to be fast in that phase. After that, we'll see more reads that writes. So I imagine a config for the batch load and another config for production. I understand cache sharding, application-level partitioning, and so forth. I'm wondering what I can do on a single machine -- and what that machine should look like. http://docs.neo4j.org/chunked/stable/configuration-jvm.html and http://wiki.neo4j.org/content/Neo4j_Performance_Guide are encouraging. And having knobs as documented at http://wiki.neo4j.org/content/Configuration_Settings is great. Nice work! I'm hoping I might be able to get away with 128GB RAM on 12 cores with data striped over a handful of disks (SSDs if required). We'll probably also need a cluster for both traffic and availability, but that's another topic. Does anybody have experience with a data set like this on a similar machine? How much RAM and how much disk -- and what kinds and in what configuration? Latency, throughput, general experience? Any production deployments? I'd appreciate any guidance or feedback. I'm happy to summarize later if that'd be helpful. BTW, my testbed uses Clojure with clojure.contrib.server-socket and https://github.com/wagjo/borneo. Very convenient! --Jamie ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Tobias Ivarsson tobias.ivars...@neotechnology.com Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
