String properties are stored in blocks so even if you have tiny string values each property value will occupy a full block (30 or 60 bytes, can someone correct me here?). That's what taking most of your space IMHO
2010/6/3, Biren Gandhi <[email protected]>: > Here is some content from neostore.propertystore.db.strings - another huge > file. What are the max number of nodes/relationships that people have tried > with Neo4j so far? Can someone share disk space usage characteristics? > > od -N 1000 -x -c neostore.propertystore.db.strings > > 0000000 0000 8500 0000 0000 0000 0000 0000 0000 > \0 \0 \0 205 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 > 0000020 0000 0000 0000 0000 0000 0000 0000 0000 > \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 > * > 0000200 0000 0000 0100 ffff ffff 0000 0c00 ffff > \0 \0 \0 \0 \0 001 377 377 377 377 \0 \0 \0 \f 377 377 > 0000220 ffff 4e00 6f00 6400 6500 2d00 3000 0000 > 377 377 \0 N \0 o \0 d \0 e \0 - \0 0 \0 \0 > 0000240 0000 0000 0000 0000 0000 0000 0000 0000 > \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 > * > 0000400 0000 0000 0000 0000 0000 ff01 ffff 00ff > \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 001 377 377 377 377 \0 > 0000420 0000 ff0c ffff 00ff 004e 006f 0064 0065 > \0 \0 \f 377 377 377 377 \0 N \0 o \0 d \0 e \0 > 0000440 002d 0031 0000 0000 0000 0000 0000 0000 > - \0 1 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 > 0000460 0000 0000 0000 0000 0000 0000 0000 0000 > \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 > * > 0000600 0000 0000 0000 0000 0000 0000 0000 0100 > \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 001 > 0000620 ffff ffff 0000 0c00 ffff ffff 4e00 6f00 > 377 377 377 377 \0 \0 \0 \f 377 377 377 377 \0 N \0 o > 0000640 6400 6500 2d00 3200 0000 0000 0000 0000 > \0 d \0 e \0 - \0 2 \0 \0 \0 \0 \0 \0 \0 \0 > 0000660 0000 0000 0000 0000 0000 0000 0000 0000 > \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 > * > 0001020 0000 0000 ff01 ffff 00ff 0000 ff0c ffff > \0 \0 \0 \0 001 377 377 377 377 \0 \0 \0 \f 377 377 377 > 0001040 00ff 004e 006f 0064 0065 002d 0033 0000 > 377 \0 N \0 o \0 d \0 e \0 - \0 3 \0 \0 \0 > 0001060 0000 0000 0000 0000 0000 0000 0000 0000 > \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 > * > 0001220 0000 0000 0000 0000 0100 ffff ffff 0000 > \0 \0 \0 \0 \0 \0 \0 \0 \0 001 377 377 377 377 \0 \0 > 0001240 0c00 ffff ffff 4e00 6f00 6400 6500 2d00 > \0 \f 377 377 377 377 \0 N \0 o \0 d \0 e \0 - > 0001260 3400 0000 0000 0000 0000 0000 0000 0000 > \0 4 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 > 0001300 0000 0000 0000 0000 0000 0000 0000 0000 > \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 > * > 0001420 0000 0000 0000 0000 0000 0000 0000 ff01 > \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 001 377 > 0001440 ffff 00ff 0000 ff0c ffff 00ff 004e 006f > 377 377 377 \0 \0 \0 \f 377 377 377 377 \0 N \0 o \0 > 0001460 0064 0065 002d 0035 0000 0000 0000 0000 > d \0 e \0 - \0 5 \0 \0 \0 \0 \0 \0 \0 \0 \0 > 0001500 0000 0000 0000 0000 0000 0000 0000 0000 > \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 > * > > > On Wed, Jun 2, 2010 at 3:50 PM, Biren Gandhi <[email protected]> wrote: > >> There is only 1 property - "n" (to store name of the node) - used as >> follows: >> >> Node node = graphDb.createNode(); >> node.setProperty( NAME_KEY, username ); >> >> And the values of username are "Node-1", "Node-2" etc. >> >> On Wed, Jun 2, 2010 at 3:14 PM, Mattias Persson <[email protected] >> > wrote: >> >>> Only 4,4mb out of those 80 is consumed by nodes so you must be storing >>> some properties somewhere. Would you mind sharing your code so that it >>> would be easier to get a better insight into your problem? >>> >>> 2010/6/2, Biren Gandhi <[email protected]>: >>> > Thanks. Big transactions were indeed problematic. Splitting them down >>> into >>> > smaller chunks did the trick. >>> > >>> > I'm still disappointed by the on-disk size of a minimal node without >>> > any >>> > relationships or attributes. For 500K nodes, it is taking 80MB space >>> (160 >>> > byes/node) and for 1M objects it is consuming 160MB (again 160 >>> byes/node). >>> > Is this normal? >>> > >>> > 4.0K active_tx_log >>> > 12K lucene >>> > 12K lucene-fulltext >>> > 4.0K neostore >>> > 4.0K neostore.id >>> > 4.4M neostore.nodestore.db >>> > 4.0K neostore.nodestore.db.id >>> > 12M neostore.propertystore.db >>> > 4.0K neostore.propertystore.db.arrays >>> > 4.0K neostore.propertystore.db.arrays.id >>> > 4.0K neostore.propertystore.db.id >>> > 4.0K neostore.propertystore.db.index >>> > 4.0K neostore.propertystore.db.index.id >>> > 4.0K neostore.propertystore.db.index.keys >>> > 4.0K neostore.propertystore.db.index.keys.id >>> > 64M neostore.propertystore.db.strings >>> > 4.0K neostore.propertystore.db.strings.id >>> > 4.0K neostore.relationshipstore.db >>> > 4.0K neostore.relationshipstore.db.id >>> > 4.0K neostore.relationshiptypestore.db >>> > 4.0K neostore.relationshiptypestore.db.id >>> > 4.0K neostore.relationshiptypestore.db.names >>> > 4.0K neostore.relationshiptypestore.db.names.id >>> > 4.0K nioneo_logical.log.active >>> > 4.0K tm_tx_log.1 >>> > 80M total >>> > >>> > >>> > On Wed, Jun 2, 2010 at 12:17 AM, Mattias Persson >>> > <[email protected]>wrote: >>> > >>> >> Exactly, the problem is most likely that you try to insert all your >>> >> stuff in one transaction. All data for a transaction is kept in memory >>> >> until committed so for really big transactions it can fill your entire >>> >> heap. Try to group 10k operations or so for big insertions or use the >>> >> batch inserter. >>> >> >>> >> Links: >>> >> http://wiki.neo4j.org/content/Transactions#Big_transactions >>> >> http://wiki.neo4j.org/content/Batch_Insert >>> >> >>> >> 2010/6/2, Laurent Laborde <[email protected]>: >>> >> > On Wed, Jun 2, 2010 at 3:50 AM, Biren Gandhi <[email protected] >>> > >>> >> wrote: >>> >> >> >>> >> >> Is there any limit on number of nodes that can be created in a >>> >> >> neo4j >>> >> >> instance? Any other tips? >>> >> > >>> >> > I created hundreds of millions of nodes without problems, but it was >>> >> > splitted into many transaction. >>> >> > >>> >> > -- >>> >> > Laurent "ker2x" Laborde >>> >> > Sysadmin & DBA at http://www.over-blog.com/ >>> >> > _______________________________________________ >>> >> > Neo4j mailing list >>> >> > [email protected] >>> >> > https://lists.neo4j.org/mailman/listinfo/user >>> >> > >>> >> >>> >> >>> >> -- >>> >> Mattias Persson, [[email protected]] >>> >> Hacker, Neo Technology >>> >> www.neotechnology.com >>> >> _______________________________________________ >>> >> Neo4j mailing list >>> >> [email protected] >>> >> https://lists.neo4j.org/mailman/listinfo/user >>> >> >>> > _______________________________________________ >>> > Neo4j mailing list >>> > [email protected] >>> > https://lists.neo4j.org/mailman/listinfo/user >>> > >>> >>> >>> -- >>> Mattias Persson, [[email protected]] >>> Hacker, Neo Technology >>> www.neotechnology.com >>> _______________________________________________ >>> Neo4j mailing list >>> [email protected] >>> https://lists.neo4j.org/mailman/listinfo/user >>> >> >> > _______________________________________________ > Neo4j mailing list > [email protected] > https://lists.neo4j.org/mailman/listinfo/user > -- Mattias Persson, [[email protected]] Hacker, Neo Technology www.neotechnology.com _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
