Hi I am using bacth importer to load my inital data. I have two type of node.(e.g. Animal, Holding) Thats why i have two type of csv structure. So i have to load data in two step. first I load Animals nodes and relations into graph.db than the holdings info and their relations with Animals into graph db.
My Machine is (Linux Centos 64bit. 24G Memory 16 CPU ) Here is my batc.properties file output. [root@localhost conf]# cd /u01/neo4j/batch_importer_20/ [root@localhost batch_importer_20]# more batch.properties batch_import.mapdb_cache.disable=true cache_type=none batch_import.keep_db=true use_memory_mapped_buffers=true -Dfile.encoding=UTF-8 neostore.nodestore.db.mapped_memory=1G neostore.relationshipstore.db.mapped_memory=12G neostore.propertystore.db.mapped_memory=3G neostore.propertystore.db.strings.mapped_memory=3G neostore.propertystore.db.arrays.mapped_memory=10M neostore.propertystore.db.index.keys.mapped_memory=200M neostore.propertystore.db.index.mapped_memory=200M And here is Animal load console output. Using: Importer batch.properties /u01/neo4j/data/graph.db /u01/neo4j-data/GRAPH_NODES_TUM_TRANSLATED_MERGED.t xt /u01/neo4j-data/GRAPH_EDGES_ALL.txt Using Existing Configuration File .................................................................................................... 151947 ms for 10000000 .................................................................................................... 116597 ms for 10000000 .................................................................................................... 100087 ms for 10000000 .................................................................................................... 215509 ms for 10000000 .................................................................................................... 224894 ms for 10000000 .................................................................................................... 210753 ms for 10000000 .................................................................................................... 217124 ms for 10000000 .................................................................................................... 197660 ms for 10000000 .................................................................................................... 204950 ms for 10000000 .................................................................................................... 245782 ms for 10000000 .................................................................................................... 313628 ms for 10000000 Importing 110045737 Nodes took 2206 seconds .................................................................................................... 99707 ms for 10000000 .................................................................................................... 212396 ms for 10000000 .................................................................................................... 69720 ms for 10000000 .................................................................................................... 650714 ms for 10000000 .................................................................................................... 966947 ms for 10000000 .................................................................................................... 1125123 ms for 10000000 .................................................................................................... 1148547 ms for 10000000 .................................................................................................... 1291582 ms for 10000000 .................................................................................................... 1417579 ms for 10000000 .................................................................................................... 1560844 ms for 10000000 .................................................................................................... 1411114 ms for 10000000 .................................................................................................... 1792981 ms for 10000000 .................................................................................................... 2015767 ms for 10000000 .................................................................................................... 1777708 ms for 10000000 .................................................................................................... 1702763 ms for 10000000 .................................................................................................... 1690538 ms for 10000000 .................................................................................................... 1673459 ms for 10000000 .................................................................................................... 1708809 ms for 10000000 .................................................................................................... 1894000 ms for 10000000 .................................................................................................... 1794012 ms for 10000000 .................................................................................................... 2077387 ms for 10000000 .................................................................................................... 2036990 ms for 10000000 .................................................................................................... 2211784 ms for 10000000 .................................................................................................... 2191097 ms for 10000000 .................................................................................................... 1773622 ms for 10000000 .................................................................................................... 2053962 ms for 10000000 .................................................................................................... 2110870 ms for 10000000 .................................................................................................... 2028949 ms for 10000000 .................................................................................................... 1972037 ms for 10000000 .................................................................................................... 2401527 ms for 10000000 .................................................................................................... 2574030 ms for 10000000 .................................................................................................... 2559779 ms for 10000000 .................................................................................................... 2752332 ms for 10000000 .................................................................................................... 3290351 ms for 10000000 .................................................................................................... 3421647 ms for 10000000 .................................................................................................... 3427667 ms for 10000000 .................................................................................................... 3359197 ms for 10000000 .................................................................................................... 3444381 ms for 10000000 .................................................................................................... 3250415 ms for 10000000 .................................................................................................... 3164657 ms for 10000000 .................................................................................................... 3230017 ms for 10000000 .................................................................................................... 3233463 ms for 10000000 .................................................................................................... 3323545 ms for 10000000 .................................................................................................... 3401135 ms for 10000000 .................................................................................................... 4652460 ms for 10000000 ....................................................................... Importing 457188710 Relationships took 99720 seconds But importing Holding nodes and their relations take too long time. The difference is Holding nodes have a few more propertie than Animal node. But edges file connects an Animal to a Hoding. (e.g. Animal-[:BELONGS]->Holding) It's console output is like below Using Existing Configuration File .................................................................................................... 2864079 ms for 10000000 .................................................................................................... 2504607 ms for 10000000 .................................................................................................... 2491906 ms for 10000000 .................................................................................................... 2518935 ms for 10000000 ....................................... Importing 43987553 Nodes took 11449 seconds .................................................................................................... 13467026 ms for 10000000 .................................................................................................... 8369045 ms for 10000000 .................................................................................................... 8463398 ms for 10000000 .................................................................................................... 8470881 ms for 10000000 .................................................................................................... 8424531 ms for 10000000 .................................................................................................... 8536233 ms for 10000000 .................................................................................................... 8853984 ms for 10000000 .................................................................................................... 10341061 ms for 10000000 .................................................................................................... 10265369 ms for 10000000 .................................................................................................... 13575782 ms for 10000000 .................................................................................................... 10690479 ms for 10000000 .................................................................................................... 9707492 ms for 10000000 .................................................................................................... 9853132 ms for 10000000 As you see, it takes 3 times longer than Animal nodes and edges import process. Do you have any idea about the reason of low performance? Do I miss something about memory config? Thanks. Ilhan -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
