Re: [Neo4j] Slow down on insertion as db grow
Hi Massimo, I'm terribly sorry, I've had a lot of things on my plate, and this sadly fell off. Thanks for the reminder! Could you please try lowering the heap size (to somewhere between 512M-1G with your available RAM). The Neo4j memory mapping is managed outside of the Java heap. What you want to achieve for ideal performance is to get the entire nodestore and relationshipstore to fit in the memory mapped file cache. You might have to tweak the memory mapping settings to make sure that they are configured to prefer the nodestore and relationship, and have them fully memory mapped. See documentation here: http://docs.neo4j.org/chunked/snapshot/configuration-caches.html#_configuration I didn't see any information on which operating system you are using, so I assumed linux or some other unix. If you are running on Windows the memory mapping IS managed in the Java heap, then you should keep your heap settings at the same size (1.5G) and instead just focus on configuring the memory mapping so that the nodestore and relationship store are fully memory mapped. Cheers, Tobias On Thu, Mar 3, 2011 at 11:18 AM, Massimo Lusetti wrote: > On Fri, Feb 18, 2011 at 2:31 PM, Tobias Ivarsson > wrote: > > > WRT the slowdown you've seen, I'll have to investigate that further. > > Didn't mean to bother anyone but do you have any news regarding the > slowdown I'm still experiencing... !? > > Cheers > -- > Massimo > http://meridio.blogspot.com > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Slow down on insertion as db grow
On Fri, Feb 18, 2011 at 2:31 PM, Tobias Ivarsson wrote: > WRT the slowdown you've seen, I'll have to investigate that further. Didn't mean to bother anyone but do you have any news regarding the slowdown I'm still experiencing... !? Cheers -- Massimo http://meridio.blogspot.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Slow down on insertion as db grow
On Fri, Feb 18, 2011 at 2:31 PM, Tobias Ivarsson wrote: > I recently (yesterday) committed a new feature for Neo4j that will store > these kinds of short strings, making Neo4j store them without having to > involve the DynamicStringStore at all. You should see a substantial speedup > from using that. IPv4 addresses will always be storeable as a short string, > so if all you store as properties are IPv4 addresses the DynamicStringStore > wouldn't be used at all in your use case. I'll give it a go in next days and report back... > WRT the slowdown you've seen, I'll have to investigate that further. Really appreciated, if you want feedback, let me know. Cheers -- Massimo http://meridio.blogspot.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Slow down on insertion as db grow
On Thu, Feb 17, 2011 at 3:32 PM, Massimo Lusetti wrote: > I'm trying to populate the DB with Nodes with only one String > property, an IPv4 address. This is just a test case for my typical use > case and it shows that when the process starts the first 440744 nodes > are going in at a speed of 1.05ms each Node but this speed drops to > 8.75ms after having inserted 12545155 nodes. > The final DB size is: 2.9G since i tweaked the sintrg_block_size at > graphdb creation time to 60bytes instead of 120... > I recently (yesterday) committed a new feature for Neo4j that will store these kinds of short strings, making Neo4j store them without having to involve the DynamicStringStore at all. You should see a substantial speedup from using that. IPv4 addresses will always be storeable as a short string, so if all you store as properties are IPv4 addresses the DynamicStringStore wouldn't be used at all in your use case. I've written a bit about how it works on my blog: http://journal.thobe.org/2011/02/better-support-for-short-strings-in.html The feature is available in the latest snapshot build: http://neo4j.org/get?file=neo4j-1.3-SNAPSHOT-unix.tar.gz And will be released in the next milestone. WRT the slowdown you've seen, I'll have to investigate that further. Cheers, -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Slow down on insertion as db grow
On Thu, Feb 17, 2011 at 3:10 PM, Johan Svensson wrote: > Massimo, you had problems with injection that created duplicates due > to a synchronization issue. That issue has been resolved and now you > are experiencing a slowdown during batch inserter injection? Yep, as I said I basically had saw big slow downs during Node's insertion using the "Big Transaction" pattern as soon as the DB grow in size o in Node's number. I'm trying to populate the DB with Nodes with only one String property, an IPv4 address. This is just a test case for my typical use case and it shows that when the process starts the first 440744 nodes are going in at a speed of 1.05ms each Node but this speed drops to 8.75ms after having inserted 12545155 nodes. The final DB size is: 2.9G since i tweaked the sintrg_block_size at graphdb creation time to 60bytes instead of 120... The box is a CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz (2993.02-MHz 686-class CPU) with 2G RAM of which 1.5 are dedicated to the JVM process: -Djava.awt.headless=true -Xms40m -Xmx1536m -XX:+UseConcMarkSweepGC Have you any clue? Thanks for your interest -- Massimo http://meridio.blogspot.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
