It is quite slow to begin with. how do you run it? Do you provide enough heap to the importer? I.e. 20GB ?
Also can you check your disk performance? http://structr.org/blog/neo4j-performance-on-ext4 So you have 110M Animals and 460M rels and 42M Holdings and ??? rels there? More RAM would help to import the data more quickly. You should also make sure that your node-store fits in your memory mapping if possible: try to increase your node-store mapping to 2GB Property encoding takes time, so more properties will be slower. Do you use any kind of indexing? This will also reduce performance. Or do you control the node-id's externally? Perhaps you can share the header and first few lines for all your 4 files? Michael Am 06.01.2014 um 15:05 schrieb ilhan <[email protected]>: > Hi I am using bacth importer to load my inital data. I have two type of > node.(e.g. Animal, Holding) Thats why i have two type of csv structure. So i > have to load data in two step. first I load Animals nodes and relations into > graph.db than the holdings info and their relations with Animals into graph > db. > > My Machine is (Linux Centos 64bit. 24G Memory 16 CPU ) > > Here is my batc.properties file output. > > [root@localhost conf]# cd /u01/neo4j/batch_importer_20/ > [root@localhost batch_importer_20]# more batch.properties > batch_import.mapdb_cache.disable=true > cache_type=none > batch_import.keep_db=true > use_memory_mapped_buffers=true > -Dfile.encoding=UTF-8 > neostore.nodestore.db.mapped_memory=1G > neostore.relationshipstore.db.mapped_memory=12G > neostore.propertystore.db.mapped_memory=3G > neostore.propertystore.db.strings.mapped_memory=3G > neostore.propertystore.db.arrays.mapped_memory=10M > neostore.propertystore.db.index.keys.mapped_memory=200M > neostore.propertystore.db.index.mapped_memory=200M > > And here is Animal load console output. > > Using: Importer batch.properties /u01/neo4j/data/graph.db > /u01/neo4j-data/GRAPH_NODES_TUM_TRANSLATED_MERGED.t > xt /u01/neo4j-data/GRAPH_EDGES_ALL.txt > > Using Existing Configuration File > .................................................................................................... > 151947 ms for 10000000 > .................................................................................................... > 116597 ms for 10000000 > .................................................................................................... > 100087 ms for 10000000 > .................................................................................................... > 215509 ms for 10000000 > .................................................................................................... > 224894 ms for 10000000 > .................................................................................................... > 210753 ms for 10000000 > .................................................................................................... > 217124 ms for 10000000 > .................................................................................................... > 197660 ms for 10000000 > .................................................................................................... > 204950 ms for 10000000 > .................................................................................................... > 245782 ms for 10000000 > .................................................................................................... > 313628 ms for 10000000 > > Importing 110045737 Nodes took 2206 seconds > .................................................................................................... > 99707 ms for 10000000 > .................................................................................................... > 212396 ms for 10000000 > .................................................................................................... > 69720 ms for 10000000 > .................................................................................................... > 650714 ms for 10000000 > .................................................................................................... > 966947 ms for 10000000 > .................................................................................................... > 1125123 ms for 10000000 > .................................................................................................... > 1148547 ms for 10000000 > .................................................................................................... > 1291582 ms for 10000000 > .................................................................................................... > 1417579 ms for 10000000 > .................................................................................................... > 1560844 ms for 10000000 > .................................................................................................... > 1411114 ms for 10000000 > .................................................................................................... > 1792981 ms for 10000000 > .................................................................................................... > 2015767 ms for 10000000 > .................................................................................................... > 1777708 ms for 10000000 > .................................................................................................... > 1702763 ms for 10000000 > .................................................................................................... > 1690538 ms for 10000000 > .................................................................................................... > 1673459 ms for 10000000 > .................................................................................................... > 1708809 ms for 10000000 > .................................................................................................... > 1894000 ms for 10000000 > .................................................................................................... > 1794012 ms for 10000000 > .................................................................................................... > 2077387 ms for 10000000 > .................................................................................................... > 2036990 ms for 10000000 > .................................................................................................... > 2211784 ms for 10000000 > .................................................................................................... > 2191097 ms for 10000000 > .................................................................................................... > 1773622 ms for 10000000 > .................................................................................................... > 2053962 ms for 10000000 > .................................................................................................... > 2110870 ms for 10000000 > .................................................................................................... > 2028949 ms for 10000000 > .................................................................................................... > 1972037 ms for 10000000 > .................................................................................................... > 2401527 ms for 10000000 > .................................................................................................... > 2574030 ms for 10000000 > .................................................................................................... > 2559779 ms for 10000000 > .................................................................................................... > 2752332 ms for 10000000 > .................................................................................................... > 3290351 ms for 10000000 > .................................................................................................... > 3421647 ms for 10000000 > .................................................................................................... > 3427667 ms for 10000000 > .................................................................................................... > 3359197 ms for 10000000 > .................................................................................................... > 3444381 ms for 10000000 > .................................................................................................... > 3250415 ms for 10000000 > .................................................................................................... > 3164657 ms for 10000000 > .................................................................................................... > 3230017 ms for 10000000 > .................................................................................................... > 3233463 ms for 10000000 > .................................................................................................... > 3323545 ms for 10000000 > .................................................................................................... > 3401135 ms for 10000000 > .................................................................................................... > 4652460 ms for 10000000 > ....................................................................... > Importing 457188710 Relationships took 99720 seconds > > But importing Holding nodes and their relations take too long time. The > difference is Holding nodes have a few more propertie than Animal node. But > edges file connects an Animal to a Hoding. (e.g. Animal-[:BELONGS]->Holding) > > It's console output is like below > > > > Using Existing Configuration File > .................................................................................................... > 2864079 ms for 10000000 > .................................................................................................... > 2504607 ms for 10000000 > .................................................................................................... > 2491906 ms for 10000000 > .................................................................................................... > 2518935 ms for 10000000 > ....................................... > Importing 43987553 Nodes took 11449 seconds > .................................................................................................... > 13467026 ms for 10000000 > .................................................................................................... > 8369045 ms for 10000000 > .................................................................................................... > 8463398 ms for 10000000 > .................................................................................................... > 8470881 ms for 10000000 > .................................................................................................... > 8424531 ms for 10000000 > .................................................................................................... > 8536233 ms for 10000000 > .................................................................................................... > 8853984 ms for 10000000 > .................................................................................................... > 10341061 ms for 10000000 > .................................................................................................... > 10265369 ms for 10000000 > .................................................................................................... > 13575782 ms for 10000000 > .................................................................................................... > 10690479 ms for 10000000 > .................................................................................................... > 9707492 ms for 10000000 > .................................................................................................... > 9853132 ms for 10000000 > > As you see, it takes 3 times longer than Animal nodes and edges import > process. > > Do you have any idea about the reason of low performance? Do I miss something > about memory config? > > Thanks. > > Ilhan > > -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
