In that case, what are the ways to estimate storage capacity numbers? Basic formula of nodes*9 + edges*33 doesn't seem like a practical one.
On Wed, Jun 2, 2010 at 11:26 PM, Mattias Persson <[email protected]>wrote: > String properties are stored in blocks so even if you have tiny string > values each property value will occupy a full block (30 or 60 bytes, > can someone correct me here?). That's what taking most of your space > IMHO > > 2010/6/3, Biren Gandhi <[email protected]>: > > Here is some content from neostore.propertystore.db.strings - another > huge > > file. What are the max number of nodes/relationships that people have > tried > > with Neo4j so far? Can someone share disk space usage characteristics? > > > > od -N 1000 -x -c neostore.propertystore.db.strings > > > > 0000000 0000 8500 0000 0000 0000 0000 0000 0000 > > \0 \0 \0 205 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 > > 0000020 0000 0000 0000 0000 0000 0000 0000 0000 > > \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 > > * > > 0000200 0000 0000 0100 ffff ffff 0000 0c00 ffff > > \0 \0 \0 \0 \0 001 377 377 377 377 \0 \0 \0 \f 377 377 > > 0000220 ffff 4e00 6f00 6400 6500 2d00 3000 0000 > > 377 377 \0 N \0 o \0 d \0 e \0 - \0 0 \0 \0 > > 0000240 0000 0000 0000 0000 0000 0000 0000 0000 > > \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 > > * > > 0000400 0000 0000 0000 0000 0000 ff01 ffff 00ff > > \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 001 377 377 377 377 \0 > > 0000420 0000 ff0c ffff 00ff 004e 006f 0064 0065 > > \0 \0 \f 377 377 377 377 \0 N \0 o \0 d \0 e \0 > > 0000440 002d 0031 0000 0000 0000 0000 0000 0000 > > - \0 1 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 > > 0000460 0000 0000 0000 0000 0000 0000 0000 0000 > > \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 > > * > > 0000600 0000 0000 0000 0000 0000 0000 0000 0100 > > \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 001 > > 0000620 ffff ffff 0000 0c00 ffff ffff 4e00 6f00 > > 377 377 377 377 \0 \0 \0 \f 377 377 377 377 \0 N \0 o > > 0000640 6400 6500 2d00 3200 0000 0000 0000 0000 > > \0 d \0 e \0 - \0 2 \0 \0 \0 \0 \0 \0 \0 \0 > > 0000660 0000 0000 0000 0000 0000 0000 0000 0000 > > \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 > > * > > 0001020 0000 0000 ff01 ffff 00ff 0000 ff0c ffff > > \0 \0 \0 \0 001 377 377 377 377 \0 \0 \0 \f 377 377 377 > > 0001040 00ff 004e 006f 0064 0065 002d 0033 0000 > > 377 \0 N \0 o \0 d \0 e \0 - \0 3 \0 \0 \0 > > 0001060 0000 0000 0000 0000 0000 0000 0000 0000 > > \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 > > * > > 0001220 0000 0000 0000 0000 0100 ffff ffff 0000 > > \0 \0 \0 \0 \0 \0 \0 \0 \0 001 377 377 377 377 \0 \0 > > 0001240 0c00 ffff ffff 4e00 6f00 6400 6500 2d00 > > \0 \f 377 377 377 377 \0 N \0 o \0 d \0 e \0 - > > 0001260 3400 0000 0000 0000 0000 0000 0000 0000 > > \0 4 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 > > 0001300 0000 0000 0000 0000 0000 0000 0000 0000 > > \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 > > * > > 0001420 0000 0000 0000 0000 0000 0000 0000 ff01 > > \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 001 377 > > 0001440 ffff 00ff 0000 ff0c ffff 00ff 004e 006f > > 377 377 377 \0 \0 \0 \f 377 377 377 377 \0 N \0 o \0 > > 0001460 0064 0065 002d 0035 0000 0000 0000 0000 > > d \0 e \0 - \0 5 \0 \0 \0 \0 \0 \0 \0 \0 \0 > > 0001500 0000 0000 0000 0000 0000 0000 0000 0000 > > \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 > > * > > > > > > On Wed, Jun 2, 2010 at 3:50 PM, Biren Gandhi <[email protected]> > wrote: > > > >> There is only 1 property - "n" (to store name of the node) - used as > >> follows: > >> > >> Node node = graphDb.createNode(); > >> node.setProperty( NAME_KEY, username ); > >> > >> And the values of username are "Node-1", "Node-2" etc. > >> > >> On Wed, Jun 2, 2010 at 3:14 PM, Mattias Persson < > [email protected] > >> > wrote: > >> > >>> Only 4,4mb out of those 80 is consumed by nodes so you must be storing > >>> some properties somewhere. Would you mind sharing your code so that it > >>> would be easier to get a better insight into your problem? > >>> > >>> 2010/6/2, Biren Gandhi <[email protected]>: > >>> > Thanks. Big transactions were indeed problematic. Splitting them down > >>> into > >>> > smaller chunks did the trick. > >>> > > >>> > I'm still disappointed by the on-disk size of a minimal node without > >>> > any > >>> > relationships or attributes. For 500K nodes, it is taking 80MB space > >>> (160 > >>> > byes/node) and for 1M objects it is consuming 160MB (again 160 > >>> byes/node). > >>> > Is this normal? > >>> > > >>> > 4.0K active_tx_log > >>> > 12K lucene > >>> > 12K lucene-fulltext > >>> > 4.0K neostore > >>> > 4.0K neostore.id > >>> > 4.4M neostore.nodestore.db > >>> > 4.0K neostore.nodestore.db.id > >>> > 12M neostore.propertystore.db > >>> > 4.0K neostore.propertystore.db.arrays > >>> > 4.0K neostore.propertystore.db.arrays.id > >>> > 4.0K neostore.propertystore.db.id > >>> > 4.0K neostore.propertystore.db.index > >>> > 4.0K neostore.propertystore.db.index.id > >>> > 4.0K neostore.propertystore.db.index.keys > >>> > 4.0K neostore.propertystore.db.index.keys.id > >>> > 64M neostore.propertystore.db.strings > >>> > 4.0K neostore.propertystore.db.strings.id > >>> > 4.0K neostore.relationshipstore.db > >>> > 4.0K neostore.relationshipstore.db.id > >>> > 4.0K neostore.relationshiptypestore.db > >>> > 4.0K neostore.relationshiptypestore.db.id > >>> > 4.0K neostore.relationshiptypestore.db.names > >>> > 4.0K neostore.relationshiptypestore.db.names.id > >>> > 4.0K nioneo_logical.log.active > >>> > 4.0K tm_tx_log.1 > >>> > 80M total > >>> > > >>> > > >>> > On Wed, Jun 2, 2010 at 12:17 AM, Mattias Persson > >>> > <[email protected]>wrote: > >>> > > >>> >> Exactly, the problem is most likely that you try to insert all your > >>> >> stuff in one transaction. All data for a transaction is kept in > memory > >>> >> until committed so for really big transactions it can fill your > entire > >>> >> heap. Try to group 10k operations or so for big insertions or use > the > >>> >> batch inserter. > >>> >> > >>> >> Links: > >>> >> http://wiki.neo4j.org/content/Transactions#Big_transactions > >>> >> http://wiki.neo4j.org/content/Batch_Insert > >>> >> > >>> >> 2010/6/2, Laurent Laborde <[email protected]>: > >>> >> > On Wed, Jun 2, 2010 at 3:50 AM, Biren Gandhi < > [email protected] > >>> > > >>> >> wrote: > >>> >> >> > >>> >> >> Is there any limit on number of nodes that can be created in a > >>> >> >> neo4j > >>> >> >> instance? Any other tips? > >>> >> > > >>> >> > I created hundreds of millions of nodes without problems, but it > was > >>> >> > splitted into many transaction. > >>> >> > > >>> >> > -- > >>> >> > Laurent "ker2x" Laborde > >>> >> > Sysadmin & DBA at http://www.over-blog.com/ > >>> >> > _______________________________________________ > >>> >> > Neo4j mailing list > >>> >> > [email protected] > >>> >> > https://lists.neo4j.org/mailman/listinfo/user > >>> >> > > >>> >> > >>> >> > >>> >> -- > >>> >> Mattias Persson, [[email protected]] > >>> >> Hacker, Neo Technology > >>> >> www.neotechnology.com > >>> >> _______________________________________________ > >>> >> Neo4j mailing list > >>> >> [email protected] > >>> >> https://lists.neo4j.org/mailman/listinfo/user > >>> >> > >>> > _______________________________________________ > >>> > Neo4j mailing list > >>> > [email protected] > >>> > https://lists.neo4j.org/mailman/listinfo/user > >>> > > >>> > >>> > >>> -- > >>> Mattias Persson, [[email protected]] > >>> Hacker, Neo Technology > >>> www.neotechnology.com > >>> _______________________________________________ > >>> Neo4j mailing list > >>> [email protected] > >>> https://lists.neo4j.org/mailman/listinfo/user > >>> > >> > >> > > _______________________________________________ > > Neo4j mailing list > > [email protected] > > https://lists.neo4j.org/mailman/listinfo/user > > > > > -- > Mattias Persson, [[email protected]] > Hacker, Neo Technology > www.neotechnology.com > _______________________________________________ > Neo4j mailing list > [email protected] > https://lists.neo4j.org/mailman/listinfo/user > _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
