Please check the import.sh script at the top it says HEAP=4G change that to HEAP=20G
Other than that it still can be a disk issue, do you have an SSD or a spinning disk? Perhaps you can have a look at the blog post I mentioned. HTH Michael Am 06.01.2014 um 23:38 schrieb ilhan <[email protected]>: > Q.how do you run it? Do you provide enough heap to the importer? I.e. 20GB ? > A.i didnt unterstand exactly but i am running it import.sh utility. I am new > in linux so i dont know how to provide heap to the importer? could you please > tell me how? > > Q. So you have 110M Animals and 460M rels and 42M Holdings and ??? rels > there? > A. 155M rels of holding and animals. It means that 110M Animal Node has 155M > relations with holdings and 450M rels between with other Animals > > Q.Do you use any kind of indexing? This will also reduce performance. > A. I dont use indexing > > Q. do you control the node-id's externally? > A. Yes I do. It is important for me give ids externally in order to create > relations properly. > > Animal nodes have 4 properties. Holding nodes have 18 properties. > > Are these import durations normal? or what is the first step to improve the > performance? could you please guide me in configuring memory options of batch > importer or java heap size properly? > > Thanks > > Ilhan > > 6 Ocak 2014 Pazartesi 16:05:37 UTC+2 tarihinde ilhan yazdı: > Hi I am using bacth importer to load my inital data. I have two type of > node.(e.g. Animal, Holding) Thats why i have two type of csv structure. So i > have to load data in two step. first I load Animals nodes and relations into > graph.db than the holdings info and their relations with Animals into graph > db. > > My Machine is (Linux Centos 64bit. 24G Memory 16 CPU ) > > Here is my batc.properties file output. > > [root@localhost conf]# cd /u01/neo4j/batch_importer_20/ > [root@localhost batch_importer_20]# more batch.properties > batch_import.mapdb_cache.disable=true > cache_type=none > batch_import.keep_db=true > use_memory_mapped_buffers=true > -Dfile.encoding=UTF-8 > neostore.nodestore.db.mapped_memory=1G > neostore.relationshipstore.db.mapped_memory=12G > neostore.propertystore.db.mapped_memory=3G > neostore.propertystore.db.strings.mapped_memory=3G > neostore.propertystore.db.arrays.mapped_memory=10M > neostore.propertystore.db.index.keys.mapped_memory=200M > neostore.propertystore.db.index.mapped_memory=200M > > And here is Animal load console output. > > Using: Importer batch.properties /u01/neo4j/data/graph.db > /u01/neo4j-data/GRAPH_NODES_TUM_TRANSLATED_MERGED.t > xt /u01/neo4j-data/GRAPH_EDGES_ALL.txt > > Using Existing Configuration File > .................................................................................................... > 151947 ms for 10000000 > .................................................................................................... > 116597 ms for 10000000 > .................................................................................................... > 100087 ms for 10000000 > .................................................................................................... > 215509 ms for 10000000 > .................................................................................................... > 224894 ms for 10000000 > .................................................................................................... > 210753 ms for 10000000 > .................................................................................................... > 217124 ms for 10000000 > .................................................................................................... > 197660 ms for 10000000 > .................................................................................................... > 204950 ms for 10000000 > .................................................................................................... > 245782 ms for 10000000 > .................................................................................................... > 313628 ms for 10000000 > > Importing 110045737 Nodes took 2206 seconds > .................................................................................................... > 99707 ms for 10000000 > .................................................................................................... > 212396 ms for 10000000 > .................................................................................................... > 69720 ms for 10000000 > .................................................................................................... > 650714 ms for 10000000 > .................................................................................................... > 966947 ms for 10000000 > .................................................................................................... > 1125123 ms for 10000000 > .................................................................................................... > 1148547 ms for 10000000 > .................................................................................................... > 1291582 ms for 10000000 > .................................................................................................... > 1417579 ms for 10000000 > .................................................................................................... > 1560844 ms for 10000000 > .................................................................................................... > 1411114 ms for 10000000 > .................................................................................................... > 1792981 ms for 10000000 > .................................................................................................... > 2015767 ms for 10000000 > .................................................................................................... > 1777708 ms for 10000000 > .................................................................................................... > 1702763 ms for 10000000 > .................................................................................................... > 1690538 ms for 10000000 > .................................................................................................... > 1673459 ms for 10000000 > .................................................................................................... > 1708809 ms for 10000000 > .................................................................................................... > 1894000 ms for 10000000 > .................................................................................................... > 1794012 ms for 10000000 > .................................................................................................... > 2077387 ms for 10000000 > .................................................................................................... > 2036990 ms for 10000000 > .................................................................................................... > 2211784 ms for 10000000 > .................................................................................................... > 2191097 ms for 10000000 > .................................................................................................... > 1773622 ms for 10000000 > .................................................................................................... > 2053962 ms for 10000000 > .................................................................................................... > 2110870 ms for 10000000 > .................................................................................................... > 2028949 ms for 10000000 > .................................................................................................... > 1972037 ms for 10000000 > .................................................................................................... > 2401527 ms for 10000000 > .................................................................................................... > 2574030 ms for 10000000 > .................................................................................................... > 2559779 ms for 10000000 > .................................................................................................... > 2752332 ms for 10000000 > .................................................................................................... > 3290351 ms for 10000000 > .................................................................................................... > 3421647 ms for 10000000 > .................................................................................................... > 3427667 ms for 10000000 > .................................................................................................... > 3359197 ms for 10000000 > .................................................................................................... > 3444381 ms for 10000000 > .................................................................................................... > 3250415 ms for 10000000 > .................................................................................................... > 3164657 ms for 10000000 > .................................................................................................... > 3230017 ms for 10000000 > .................................................................................................... > 3233463 ms for 10000000 > .................................................................................................... > 3323545 ms for 10000000 > .................................................................................................... > 3401135 ms for 10000000 > .................................................................................................... > 4652460 ms for 10000000 > ....................................................................... > Importing 457188710 Relationships took 99720 seconds > > But importing Holding nodes and their relations take too long time. The > difference is Holding nodes have a few more propertie than Animal node. But > edges file connects an Animal to a Hoding. (e.g. Animal-[:BELONGS]->Holding) > > It's console output is like below > > > > Using Existing Configuration File > .................................................................................................... > 2864079 ms for 10000000 > .................................................................................................... > 2504607 ms for 10000000 > .................................................................................................... > 2491906 ms for 10000000 > .................................................................................................... > 2518935 ms for 10000000 > ....................................... > Importing 43987553 Nodes took 11449 seconds > .................................................................................................... > 13467026 ms for 10000000 > .................................................................................................... > 8369045 ms for 10000000 > .................................................................................................... > 8463398 ms for 10000000 > .................................................................................................... > 8470881 ms for 10000000 > .................................................................................................... > 8424531 ms for 10000000 > .................................................................................................... > 8536233 ms for 10000000 > .................................................................................................... > 8853984 ms for 10000000 > .................................................................................................... > 10341061 ms for 10000000 > .................................................................................................... > 10265369 ms for 10000000 > .................................................................................................... > 13575782 ms for 10000000 > .................................................................................................... > 10690479 ms for 10000000 > .................................................................................................... > 9707492 ms for 10000000 > .................................................................................................... > 9853132 ms for 10000000 > > As you see, it takes 3 times longer than Animal nodes and edges import > process. > > Do you have any idea about the reason of low performance? Do I miss something > about memory config? > > Thanks. > > Ilhan > > -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
