Q.how do you run it? Do you provide enough heap to the importer? I.e. 20GB ? A.i didnt unterstand exactly but i am running it import.sh utility. I am new in linux so i dont know how to provide heap to the importer? could you please tell me how?
Q. So you have 110M Animals and 460M rels and 42M Holdings and ??? rels there? A. 155M rels of holding and animals. It means that 110M Animal Node has 155M relations with holdings and 450M rels between with other Animals Q.Do you use any kind of indexing? This will also reduce performance. A. I dont use indexing Q. do you control the node-id's externally? A. Yes I do. It is important for me give ids externally in order to create relations properly. Animal nodes have 4 properties. Holding nodes have 18 properties. Are these import durations normal? or what is the first step to improve the performance? could you please guide me in configuring memory options of batch importer or java heap size properly? Thanks Ilhan 6 Ocak 2014 Pazartesi 16:05:37 UTC+2 tarihinde ilhan yazdı: > > Hi I am using bacth importer to load my inital data. I have two type of > node.(e.g. Animal, Holding) Thats why i have two type of csv structure. So > i have to load data in two step. first I load Animals nodes and relations > into graph.db than the holdings info and their relations with Animals into > graph db. > > My Machine is (Linux Centos 64bit. 24G Memory 16 CPU ) > > Here is my batc.properties file output. > > [root@localhost conf]# cd /u01/neo4j/batch_importer_20/ > [root@localhost batch_importer_20]# more batch.properties > batch_import.mapdb_cache.disable=true > cache_type=none > batch_import.keep_db=true > use_memory_mapped_buffers=true > -Dfile.encoding=UTF-8 > neostore.nodestore.db.mapped_memory=1G > neostore.relationshipstore.db.mapped_memory=12G > neostore.propertystore.db.mapped_memory=3G > neostore.propertystore.db.strings.mapped_memory=3G > neostore.propertystore.db.arrays.mapped_memory=10M > neostore.propertystore.db.index.keys.mapped_memory=200M > neostore.propertystore.db.index.mapped_memory=200M > > And here is Animal load console output. > > Using: Importer batch.properties /u01/neo4j/data/graph.db > /u01/neo4j-data/GRAPH_NODES_TUM_TRANSLATED_MERGED.t > xt /u01/neo4j-data/GRAPH_EDGES_ALL.txt > > Using Existing Configuration File > .................................................................................................... > > 151947 ms for 10000000 > .................................................................................................... > > 116597 ms for 10000000 > .................................................................................................... > > 100087 ms for 10000000 > .................................................................................................... > > 215509 ms for 10000000 > .................................................................................................... > > 224894 ms for 10000000 > .................................................................................................... > > 210753 ms for 10000000 > .................................................................................................... > > 217124 ms for 10000000 > .................................................................................................... > > 197660 ms for 10000000 > .................................................................................................... > > 204950 ms for 10000000 > .................................................................................................... > > 245782 ms for 10000000 > .................................................................................................... > > 313628 ms for 10000000 > > Importing 110045737 Nodes took 2206 seconds > .................................................................................................... > > 99707 ms for 10000000 > .................................................................................................... > > 212396 ms for 10000000 > .................................................................................................... > > 69720 ms for 10000000 > .................................................................................................... > > 650714 ms for 10000000 > .................................................................................................... > > 966947 ms for 10000000 > .................................................................................................... > > 1125123 ms for 10000000 > .................................................................................................... > > 1148547 ms for 10000000 > .................................................................................................... > > 1291582 ms for 10000000 > .................................................................................................... > > 1417579 ms for 10000000 > .................................................................................................... > > 1560844 ms for 10000000 > .................................................................................................... > > 1411114 ms for 10000000 > .................................................................................................... > > 1792981 ms for 10000000 > .................................................................................................... > > 2015767 ms for 10000000 > .................................................................................................... > > 1777708 ms for 10000000 > .................................................................................................... > > 1702763 ms for 10000000 > .................................................................................................... > > 1690538 ms for 10000000 > .................................................................................................... > > 1673459 ms for 10000000 > .................................................................................................... > > 1708809 ms for 10000000 > .................................................................................................... > > 1894000 ms for 10000000 > .................................................................................................... > > 1794012 ms for 10000000 > .................................................................................................... > > 2077387 ms for 10000000 > .................................................................................................... > > 2036990 ms for 10000000 > .................................................................................................... > > 2211784 ms for 10000000 > .................................................................................................... > > 2191097 ms for 10000000 > .................................................................................................... > > 1773622 ms for 10000000 > .................................................................................................... > > 2053962 ms for 10000000 > .................................................................................................... > > 2110870 ms for 10000000 > .................................................................................................... > > 2028949 ms for 10000000 > .................................................................................................... > > 1972037 ms for 10000000 > .................................................................................................... > > 2401527 ms for 10000000 > .................................................................................................... > > 2574030 ms for 10000000 > .................................................................................................... > > 2559779 ms for 10000000 > .................................................................................................... > > 2752332 ms for 10000000 > .................................................................................................... > > 3290351 ms for 10000000 > .................................................................................................... > > 3421647 ms for 10000000 > .................................................................................................... > > 3427667 ms for 10000000 > .................................................................................................... > > 3359197 ms for 10000000 > .................................................................................................... > > 3444381 ms for 10000000 > .................................................................................................... > > 3250415 ms for 10000000 > .................................................................................................... > > 3164657 ms for 10000000 > .................................................................................................... > > 3230017 ms for 10000000 > .................................................................................................... > > 3233463 ms for 10000000 > .................................................................................................... > > 3323545 ms for 10000000 > .................................................................................................... > > 3401135 ms for 10000000 > .................................................................................................... > > 4652460 ms for 10000000 > ....................................................................... > Importing 457188710 Relationships took 99720 seconds > > But importing Holding nodes and their relations take too long time. The > difference is Holding nodes have a few more propertie than Animal node. But > edges file connects an Animal to a Hoding. (e.g. > Animal-[:BELONGS]->Holding) > > It's console output is like below > > > > Using Existing Configuration File > .................................................................................................... > > 2864079 ms for 10000000 > .................................................................................................... > > 2504607 ms for 10000000 > .................................................................................................... > > 2491906 ms for 10000000 > .................................................................................................... > > 2518935 ms for 10000000 > ....................................... > Importing 43987553 Nodes took 11449 seconds > .................................................................................................... > > 13467026 ms for 10000000 > .................................................................................................... > > 8369045 ms for 10000000 > .................................................................................................... > > 8463398 ms for 10000000 > .................................................................................................... > > 8470881 ms for 10000000 > .................................................................................................... > > 8424531 ms for 10000000 > .................................................................................................... > > 8536233 ms for 10000000 > .................................................................................................... > > 8853984 ms for 10000000 > .................................................................................................... > > 10341061 ms for 10000000 > .................................................................................................... > > 10265369 ms for 10000000 > .................................................................................................... > > 13575782 ms for 10000000 > .................................................................................................... > > 10690479 ms for 10000000 > .................................................................................................... > > 9707492 ms for 10000000 > .................................................................................................... > > 9853132 ms for 10000000 > > As you see, it takes 3 times longer than Animal nodes and edges import > process. > > Do you have any idea about the reason of low performance? Do I miss > something about memory config? > > Thanks. > > Ilhan > -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
