Re: [Neo4j] Big Traverser perfomance : are '23 rels per milliseconds' good ?
Hi Peter, Now It seems ok (though I do not know why). I even tried to increase the value to : neostore.nodestore.db.mapped_memory=400M neostore.propertystore.db.arrays.mapped_memory=200M neostore.propertystore.db.index.keys.mapped_memory=10M neostore.propertystore.db.index.mapped_memory=10M neostore.propertystore.db.mapped_memory=2000M neostore.propertystore.db.strings.mapped_memory=400M neostore.relationshipstore.db.mapped_memory=1600M However, the performance increased a bit (20%) but not that much : I am traversing 130 rels per millisecond. I would be cool to have 500 rels per millisec ;-) I did check vmstat during the run and it does not seem to swap. BR. Guillaume De : Peter Neubauer [via Neo4j Community Discussions] [mailto:ml-node+s438527n3411947...@n3.nabble.com] Envoyé : mardi 11 octobre 2011 10:16 À : Allee, Guillaume Objet : Re: [Neo4j] Big Traverser perfomance : are '23 rels per milliseconds' good ? Guillaume, the memory mapping happens outside the JVM. As such there is no guarantee that you actually have that memory and you need to make sure no other proceses are competing on that, e.g. you browser which might take a large chunk of the memory you have on the machine (5GB-2.2GB for the JVM) Could you check that you have memory available? Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org ">http://www.neo4j.org - Your high performance graph database. http://startupbootcamp.org/- Öresund - Innovation happens HERE. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Mon, Oct 10, 2011 at 1:54 PM, Guillaume ALLEE <[hidden email]> wrote: > Hi Michael, > > I tried to lower the values by 30% => same and then by 50% => same again. > (MapMemException) > Here is the message.log http://pastebin.com/bvhhZfjZ > > My neo4j.properties looks like that : > > neostore.nodestore.db.mapped_memory=100M > neostore.propertystore.db.arrays.mapped_memory=50M > neostore.propertystore.db.index.keys.mapped_memory=1M > neostore.propertystore.db.index.mapped_memory=1M > neostore.propertystore.db.mapped_memory=500M > neostore.propertystore.db.strings.mapped_memory=200M > neostore.relationshipstore.db.mapped_memory=400M > > Any idea ? > > -- > View this message in context: > http://neo4j-community-discussions.438527.n3.nabble.com/Big-Traverser-perfomance-are-23-rels-per-milliseconds-good-tp3400711p3409259.html > Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. > ___ > Neo4j mailing list > [hidden email] > https://lists.neo4j.org/mailman/listinfo/user > ___ Neo4j mailing list [hidden email] https://lists.neo4j.org/mailman/listinfo/user If you reply to this email, your message will be added to the discussion below: http://neo4j-community-discussions.438527.n3.nabble.com/Big-Traverser-perfomance-are-23-rels-per-milliseconds-good-tp3400711p3411947.html To unsubscribe from Big Traverser perfomance : are '23 rels per milliseconds' good ?, click here<http://neo4j-community-discussions.438527.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3400711&code=Z3VpbGxhdW1lLmFsbGVlQGNhcGdlbWluaS5jb218MzQwMDcxMXwtNDE2OTAwMDI0>. This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message. -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/Big-Traverser-perfomance-are-23-rels-per-milliseconds-good-tp3400711p3412391.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Big Traverser perfomance : are '23 rels per milliseconds' good ?
Guillaume, the memory mapping happens outside the JVM. As such there is no guarantee that you actually have that memory and you need to make sure no other proceses are competing on that, e.g. you browser which might take a large chunk of the memory you have on the machine (5GB-2.2GB for the JVM) Could you check that you have memory available? Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://startupbootcamp.org/ - Öresund - Innovation happens HERE. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Mon, Oct 10, 2011 at 1:54 PM, Guillaume ALLEE wrote: > Hi Michael, > > I tried to lower the values by 30% => same and then by 50% => same again. > (MapMemException) > Here is the message.log http://pastebin.com/bvhhZfjZ > > My neo4j.properties looks like that : > > neostore.nodestore.db.mapped_memory=100M > neostore.propertystore.db.arrays.mapped_memory=50M > neostore.propertystore.db.index.keys.mapped_memory=1M > neostore.propertystore.db.index.mapped_memory=1M > neostore.propertystore.db.mapped_memory=500M > neostore.propertystore.db.strings.mapped_memory=200M > neostore.relationshipstore.db.mapped_memory=400M > > Any idea ? > > -- > View this message in context: > http://neo4j-community-discussions.438527.n3.nabble.com/Big-Traverser-perfomance-are-23-rels-per-milliseconds-good-tp3400711p3409259.html > Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Big Traverser perfomance : are '23 rels per milliseconds' good ?
Guillaume, you still having problems or is the performance ok now? I think with tweaking the 110 hops/ms can be upped to at least 500, since you are not touching any indexes except in the beginning. However, that would require to look at the underlying system and do some inspection. Let me know if you are interested in that? Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://startupbootcamp.org/ - Öresund - Innovation happens HERE. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Mon, Oct 10, 2011 at 1:54 PM, Guillaume ALLEE wrote: > Hi Michael, > > I tried to lower the values by 30% => same and then by 50% => same again. > (MapMemException) > Here is the message.log http://pastebin.com/bvhhZfjZ > > My neo4j.properties looks like that : > > neostore.nodestore.db.mapped_memory=100M > neostore.propertystore.db.arrays.mapped_memory=50M > neostore.propertystore.db.index.keys.mapped_memory=1M > neostore.propertystore.db.index.mapped_memory=1M > neostore.propertystore.db.mapped_memory=500M > neostore.propertystore.db.strings.mapped_memory=200M > neostore.relationshipstore.db.mapped_memory=400M > > Any idea ? > > -- > View this message in context: > http://neo4j-community-discussions.438527.n3.nabble.com/Big-Traverser-perfomance-are-23-rels-per-milliseconds-good-tp3400711p3409259.html > Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Big Traverser perfomance : are '23 rels per milliseconds' good ?
Hi Michael, I tried to lower the values by 30% => same and then by 50% => same again. (MapMemException) Here is the message.log http://pastebin.com/bvhhZfjZ My neo4j.properties looks like that : neostore.nodestore.db.mapped_memory=100M neostore.propertystore.db.arrays.mapped_memory=50M neostore.propertystore.db.index.keys.mapped_memory=1M neostore.propertystore.db.index.mapped_memory=1M neostore.propertystore.db.mapped_memory=500M neostore.propertystore.db.strings.mapped_memory=200M neostore.relationshipstore.db.mapped_memory=400M Any idea ? -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/Big-Traverser-perfomance-are-23-rels-per-milliseconds-good-tp3400711p3409259.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Big Traverser perfomance : are '23 rels per milliseconds' good ?
Guillaume, please try to reduce the memory mapped settings I provided, e.g by 30%. You can also inspect your system how much "free" memory is reported when you're running the neo4j test. This amount of memory can be added to the memory mapped file caches. Have you looked at the linux transaction test? I see the only problem with your use-case if you traverse the graph in a way that constantly churns all the nodes + relationships through the caches so that they get removed before the next cache hit can occur. Michael Am 09.10.2011 um 14:50 schrieb Guillaume ALLEE: > Hi, > > 1) I had a problem with my cache warm up code which was not working. > Now with the default cache properties, I am traversing 110 relations per > millesecond. > > 2) When I load my db with your cache properties, I have MappedMemException. > See here the full stack. http://pastebin.com/XwV41iJB > I tried to put lower values, but I get the same. > FYI My heap size is 2500M. > > I do not get the algorithm to put correct values in the cache properties. > > Is 110 rels per milliseconds a good value or can I do much better ? > > About the graphDB generator : my code only imports 'real' data that are > extracted from an RDBMS. Since those data are confidential, I cannot send > them to you. > > Also FYI I get a 404 on > http://docs.neo4j.org/chunked/snapshot/configuration.html > > Cheers. > > -- > View this message in context: > http://neo4j-community-discussions.438527.n3.nabble.com/Big-Traverser-perfomance-are-23-rels-per-milliseconds-good-tp3400711p3407300.html > Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Big Traverser perfomance : are '23 rels per milliseconds' good ?
Hi, 1) I had a problem with my cache warm up code which was not working. Now with the default cache properties, I am traversing 110 relations per millesecond. 2) When I load my db with your cache properties, I have MappedMemException. See here the full stack. http://pastebin.com/XwV41iJB I tried to put lower values, but I get the same. FYI My heap size is 2500M. I do not get the algorithm to put correct values in the cache properties. Is 110 rels per milliseconds a good value or can I do much better ? About the graphDB generator : my code only imports 'real' data that are extracted from an RDBMS. Since those data are confidential, I cannot send them to you. Also FYI I get a 404 on http://docs.neo4j.org/chunked/snapshot/configuration.html Cheers. -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/Big-Traverser-perfomance-are-23-rels-per-milliseconds-good-tp3400711p3407300.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Big Traverser perfomance : are '23 rels per milliseconds' good ?
It should, but in Guillaume's example they didn't differ by much, both were slow. It seems to be running with the default soft references cache which might turn over if every node and property is just visited once and then only a second time in the second run. Michael Am 07.10.2011 um 09:19 schrieb Mattias Persson: > Also, is that the first run directly after a JVM start? The first time you > encounter a Node or Relationship it is loaded from disk into memory so that > the next time it's read from memory instead. The difference between two runs > can be order of magnitudes in difference. > > 2011/10/6 Michael Hunger > >> Hmm virtual machines might be difficult, esp. with the io indirection. >> >> Your memory settings for the db are: >> • Fri Oct 07 00:10:01 IST 2011: >> neostore.nodestore.db.mapped_memory=20M >> • Fri Oct 07 00:10:01 IST 2011: >> neostore.propertystore.db.arrays.mapped_memory=130M >> • Fri Oct 07 00:10:01 IST 2011: >> neostore.propertystore.db.index.keys.mapped_memory=1M >> • Fri Oct 07 00:10:01 IST 2011: >> neostore.propertystore.db.index.mapped_memory=1M >> • Fri Oct 07 00:10:01 IST 2011: >> neostore.propertystore.db.mapped_memory=90M >> • Fri Oct 07 00:10:01 IST 2011: >> neostore.propertystore.db.strings.mapped_memory=130M >> • Fri Oct 07 00:10:01 IST 2011: >> neostore.relationshipstore.db.mapped_memory=100M >> >> Can you change that so that it uses the rest of your memory (say 2GB). >> >> • Fri Oct 07 00:10:01 IST 2011: >> neostore.nodestore.db.mapped_memory=200M >> • Fri Oct 07 00:10:01 IST 2011: >> neostore.propertystore.db.arrays.mapped_memory=100M >> • Fri Oct 07 00:10:01 IST 2011: >> neostore.propertystore.db.index.keys.mapped_memory=1M >> • Fri Oct 07 00:10:01 IST 2011: >> neostore.propertystore.db.index.mapped_memory=1M >> • Fri Oct 07 00:10:01 IST 2011: >> neostore.propertystore.db.mapped_memory=1000M >> • Fri Oct 07 00:10:01 IST 2011: >> neostore.propertystore.db.strings.mapped_memory=200M >> • Fri Oct 07 00:10:01 IST 2011: >> neostore.relationshipstore.db.mapped_memory=800M >> >> See also: >> http://docs.neo4j.org/chunked/snapshot/configuration.html >> >> Could you also have a look at this: >> http://wiki.neo4j.org/content/Linux_Performance_Guide >> >> Is it possible to share your code/graphdb(generator) - also off-list. I'd >> like to have a look. >> >> Thanks >> >> Michael >> >> Am 06.10.2011 um 22:14 schrieb Guillaume ALLEE: >> >>> Hi, >>> >>> Here are the answers I can give you quickly: >>> >>> How much memory does your machine have? 5066 MB (from free -m) >>> >>> What kind of disk is in there? I do not know, the machine is a VM provided by another department of my >>> company. What I can tell you is that on my i5 laptop the same was taking >> 6-8 >>> minutes. >>> >>> Have you looked at the memory config for the neo4j db? No I did only change the head size. >>> >>> What kind of scheduler do you use (please try deadline or as)? it seems to be as. >>> >>> Can you please share the config and JVM info that is output at the head >> of >>> graphdb/messages.log ? see here http://pastebin.com/0geHdbNu >>> >>> >>> >>> -- >>> View this message in context: >> http://neo4j-community-discussions.438527.n3.nabble.com/Big-Traverser-perfomance-are-23-rels-per-milliseconds-good-tp3400711p3400925.html >>> Sent from the Neo4j Community Discussions mailing list archive at >> Nabble.com. >>> ___ >>> Neo4j mailing list >>> User@lists.neo4j.org >>> https://lists.neo4j.org/mailman/listinfo/user >> >> ___ >> Neo4j mailing list >> User@lists.neo4j.org >> https://lists.neo4j.org/mailman/listinfo/user >> > > > > -- > Mattias Persson, [matt...@neotechnology.com] > Hacker, Neo Technology > www.neotechnology.com > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Big Traverser perfomance : are '23 rels per milliseconds' good ?
Also, is that the first run directly after a JVM start? The first time you encounter a Node or Relationship it is loaded from disk into memory so that the next time it's read from memory instead. The difference between two runs can be order of magnitudes in difference. 2011/10/6 Michael Hunger > Hmm virtual machines might be difficult, esp. with the io indirection. > > Your memory settings for the db are: >• Fri Oct 07 00:10:01 IST 2011: > neostore.nodestore.db.mapped_memory=20M >• Fri Oct 07 00:10:01 IST 2011: > neostore.propertystore.db.arrays.mapped_memory=130M >• Fri Oct 07 00:10:01 IST 2011: > neostore.propertystore.db.index.keys.mapped_memory=1M >• Fri Oct 07 00:10:01 IST 2011: > neostore.propertystore.db.index.mapped_memory=1M >• Fri Oct 07 00:10:01 IST 2011: > neostore.propertystore.db.mapped_memory=90M >• Fri Oct 07 00:10:01 IST 2011: > neostore.propertystore.db.strings.mapped_memory=130M >• Fri Oct 07 00:10:01 IST 2011: > neostore.relationshipstore.db.mapped_memory=100M > > Can you change that so that it uses the rest of your memory (say 2GB). > >• Fri Oct 07 00:10:01 IST 2011: > neostore.nodestore.db.mapped_memory=200M >• Fri Oct 07 00:10:01 IST 2011: > neostore.propertystore.db.arrays.mapped_memory=100M >• Fri Oct 07 00:10:01 IST 2011: > neostore.propertystore.db.index.keys.mapped_memory=1M >• Fri Oct 07 00:10:01 IST 2011: > neostore.propertystore.db.index.mapped_memory=1M >• Fri Oct 07 00:10:01 IST 2011: > neostore.propertystore.db.mapped_memory=1000M >• Fri Oct 07 00:10:01 IST 2011: > neostore.propertystore.db.strings.mapped_memory=200M >• Fri Oct 07 00:10:01 IST 2011: > neostore.relationshipstore.db.mapped_memory=800M > > See also: > http://docs.neo4j.org/chunked/snapshot/configuration.html > > Could you also have a look at this: > http://wiki.neo4j.org/content/Linux_Performance_Guide > > Is it possible to share your code/graphdb(generator) - also off-list. I'd > like to have a look. > > Thanks > > Michael > > Am 06.10.2011 um 22:14 schrieb Guillaume ALLEE: > > > Hi, > > > > Here are the answers I can give you quickly: > > > > How much memory does your machine have? > >> 5066 MB (from free -m) > > > > What kind of disk is in there? > >> I do not know, the machine is a VM provided by another department of my > > company. What I can tell you is that on my i5 laptop the same was taking > 6-8 > > minutes. > > > > Have you looked at the memory config for the neo4j db? > >> No I did only change the head size. > > > > What kind of scheduler do you use (please try deadline or as)? > >> it seems to be as. > > > > Can you please share the config and JVM info that is output at the head > of > > graphdb/messages.log ? > >> see here http://pastebin.com/0geHdbNu > > > > > > > > -- > > View this message in context: > http://neo4j-community-discussions.438527.n3.nabble.com/Big-Traverser-perfomance-are-23-rels-per-milliseconds-good-tp3400711p3400925.html > > Sent from the Neo4j Community Discussions mailing list archive at > Nabble.com. > > ___ > > Neo4j mailing list > > User@lists.neo4j.org > > https://lists.neo4j.org/mailman/listinfo/user > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Big Traverser perfomance : are '23 rels per milliseconds' good ?
Hmm virtual machines might be difficult, esp. with the io indirection. Your memory settings for the db are: • Fri Oct 07 00:10:01 IST 2011: neostore.nodestore.db.mapped_memory=20M • Fri Oct 07 00:10:01 IST 2011: neostore.propertystore.db.arrays.mapped_memory=130M • Fri Oct 07 00:10:01 IST 2011: neostore.propertystore.db.index.keys.mapped_memory=1M • Fri Oct 07 00:10:01 IST 2011: neostore.propertystore.db.index.mapped_memory=1M • Fri Oct 07 00:10:01 IST 2011: neostore.propertystore.db.mapped_memory=90M • Fri Oct 07 00:10:01 IST 2011: neostore.propertystore.db.strings.mapped_memory=130M • Fri Oct 07 00:10:01 IST 2011: neostore.relationshipstore.db.mapped_memory=100M Can you change that so that it uses the rest of your memory (say 2GB). • Fri Oct 07 00:10:01 IST 2011: neostore.nodestore.db.mapped_memory=200M • Fri Oct 07 00:10:01 IST 2011: neostore.propertystore.db.arrays.mapped_memory=100M • Fri Oct 07 00:10:01 IST 2011: neostore.propertystore.db.index.keys.mapped_memory=1M • Fri Oct 07 00:10:01 IST 2011: neostore.propertystore.db.index.mapped_memory=1M • Fri Oct 07 00:10:01 IST 2011: neostore.propertystore.db.mapped_memory=1000M • Fri Oct 07 00:10:01 IST 2011: neostore.propertystore.db.strings.mapped_memory=200M • Fri Oct 07 00:10:01 IST 2011: neostore.relationshipstore.db.mapped_memory=800M See also: http://docs.neo4j.org/chunked/snapshot/configuration.html Could you also have a look at this: http://wiki.neo4j.org/content/Linux_Performance_Guide Is it possible to share your code/graphdb(generator) - also off-list. I'd like to have a look. Thanks Michael Am 06.10.2011 um 22:14 schrieb Guillaume ALLEE: > Hi, > > Here are the answers I can give you quickly: > > How much memory does your machine have? >> 5066 MB (from free -m) > > What kind of disk is in there? >> I do not know, the machine is a VM provided by another department of my > company. What I can tell you is that on my i5 laptop the same was taking 6-8 > minutes. > > Have you looked at the memory config for the neo4j db? >> No I did only change the head size. > > What kind of scheduler do you use (please try deadline or as)? >> it seems to be as. > > Can you please share the config and JVM info that is output at the head of > graphdb/messages.log ? >> see here http://pastebin.com/0geHdbNu > > > > -- > View this message in context: > http://neo4j-community-discussions.438527.n3.nabble.com/Big-Traverser-perfomance-are-23-rels-per-milliseconds-good-tp3400711p3400925.html > Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Big Traverser perfomance : are '23 rels per milliseconds' good ?
Hi, Here are the answers I can give you quickly: How much memory does your machine have? >5066 MB (from free -m) What kind of disk is in there? >I do not know, the machine is a VM provided by another department of my company. What I can tell you is that on my i5 laptop the same was taking 6-8 minutes. Have you looked at the memory config for the neo4j db? > No I did only change the head size. What kind of scheduler do you use (please try deadline or as)? > it seems to be as. Can you please share the config and JVM info that is output at the head of graphdb/messages.log ? > see here http://pastebin.com/0geHdbNu -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/Big-Traverser-perfomance-are-23-rels-per-milliseconds-good-tp3400711p3400925.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Big Traverser perfomance : are '23 rels per milliseconds' good ?
How much memory does your machine have? What kind of disk is in there? Have you looked at the memory config for the neo4j db? What kind of scheduler do you use (please try deadline or as)? Can you please share the config and JVM info that is output at the head of graphdb/messages.log ? I'll attach some things we do to improve performance: • Linux: scheduler used (cat /sys/block/*/queue/scheduler) • Disk • Output from bonnie++ or other disk performance tool • Please also run the write-test suite from here: http://wiki.neo4j.org/content/Linux_Performance_Guide • Limits: • Memory • Disk space, Used Swap • ulimit -a (most probably open files should be >> 10k) sysctl fs.file-max http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files • encrypted fs hurts df -T /path/to/database/directory Am 06.10.2011 um 21:03 schrieb Guillaume ALLEE: > Hi all, > > > I am using neo4j 1.5 java embedded. My traverser is a main java program > http://pastebin.com/1ynVESbc which takes a db path as input. > > What it does is pretty basic : it follows my graph (which is a tree) and > stores the ending leaves in a file. However it is quite big : I goes through > 800.000 relationships and outputs 590.000 nodes in a txt file (11 MB) > > The global properties of my DB are the following : > 962.008 nodes, 5.697.359 properties, 4.301.267 relationships, 2 relationship > types. > > > I do have the best results with the following JVM parameters : "-server > -Xmx2500m -d32 -XX:+UseNUMA" > JVM is 32 bits SUN JVM 1.6.0_24 > CPU is single core Intel(R) Xeon(R) CPU X5660 @ 2.80GHz > OS: RHEL 5.5. > > As you can see in my code I did warm up the cache by making 2 traversers. > First pass : 44 seconds > 2nd pass : 34.5 seconds > > which make 23 relationships traversed per milliseconds. > > > What do you think of that figures ? > Do you see obvious things I should do to improve the speed of the traversal > ? > > Thanks, > > Guillaume > > > > -- > View this message in context: > http://neo4j-community-discussions.438527.n3.nabble.com/Big-Traverser-perfomance-are-23-rels-per-milliseconds-good-tp3400711p3400711.html > Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Big Traverser perfomance : are '23 rels per milliseconds' good ?
Hi all, I am using neo4j 1.5 java embedded. My traverser is a main java program http://pastebin.com/1ynVESbc which takes a db path as input. What it does is pretty basic : it follows my graph (which is a tree) and stores the ending leaves in a file. However it is quite big : I goes through 800.000 relationships and outputs 590.000 nodes in a txt file (11 MB) The global properties of my DB are the following : 962.008 nodes, 5.697.359 properties, 4.301.267 relationships, 2 relationship types. I do have the best results with the following JVM parameters : "-server -Xmx2500m -d32 -XX:+UseNUMA" JVM is 32 bits SUN JVM 1.6.0_24 CPU is single core Intel(R) Xeon(R) CPU X5660 @ 2.80GHz OS: RHEL 5.5. As you can see in my code I did warm up the cache by making 2 traversers. First pass : 44 seconds 2nd pass : 34.5 seconds which make 23 relationships traversed per milliseconds. What do you think of that figures ? Do you see obvious things I should do to improve the speed of the traversal ? Thanks, Guillaume -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/Big-Traverser-perfomance-are-23-rels-per-milliseconds-good-tp3400711p3400711.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
