Hi Johan, Thank you very much for the advices. Simply turning off the logical log rotation fixed my problem.
Just to let you know (as a feeback) that only modifying the vm.dirty_background_ratio and vm.dirty_ratio configuration parameters as you proposed, did not reduce the time values of writing. However, as this is an immediate feedback, maybe I will notice the difference after extensive testing :) Thanks a lot again! Best regards, Alisa -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Johan Svensson Sent: Tuesday, May 25, 2010 5:04 PM To: Neo user discussions Subject: Re: [Neo4j] [Neo] Question regarding insertRelationship performance - memory issue? Hi, Could you try to turn off logical log rotation or change the default target log size to see if that changes anything. You can do this by accessing the data source for the native store: GraphDatabaseService graphDb = new EmbeddedGraphDatabase( "dbdir", ... ); TxModule txModule = ((EmbeddedGraphDatabase) graphDb).getConfig().getTxModule(); XaDataSourceManager xaDsMgr = txModule.getXaDataSourceManager(); XaDataSource xaDs = xaDsMgr.getXaDataSource( "nioneodb" ); xaDs.setAutoRotate( false ); // or increase log target size to 100MB (default 10MB) like this // xaDs.setLogicalLogTargetSize( 100 * 1024 * 1024l ); The page out problem could also be a Linux kernel tweaking issue were it is configured to start writing out dirty pages to early. If you type the following in a console: #watch grep -A 1 dirty /proc/vmstat then observe "nr_writeback". If that value starts to increase more and more when performing writes you need to tweak your virtual memory settings. You can control when the linux kernel will start writing out dirty pages by modifying the "vm.dirty_background_ratio" and "vm.dirty_ratio" settings. The "dirty_background_ratio" tells how many percent of memory/pages can be dirty before a background task is started to write them out to disk. When that happens performance will drop and write performance will be heavily affected since it will result in more random writes instead of only sequential writes to the log. To change these setting edit "/etc/sysctl.conf": vm.dirty_background_ratio = 80 vm.dirty_ratio = 90 Then to load the new settings type the following in a console: #sysctl -p The "80" means when 80% of pages are dirty start the background "write out pages" task. The other setting (at 90%) is when all writes will have to wait until it has been written to the underlying device (will kill write performance even more). Hopefully you can tweak away the "page out" problem by modifying these settings. Regards, Johan On Mon, May 24, 2010 at 1:28 PM, Alisa Devlic <[email protected]> wrote: > Hello, > > > > My name is Alisa Devlic and I am a new neo4j mailing list subscriber. > > I have the following problem. > > > > I performed the following tests in Neo4j (with and without indexing): > > - Inserting nodes with 2 properties > > - Inserting 2 relationship between nodes > > - Get nodes by Id/name > > - Traversing > > > > In these tests I have increased the number of nodes I am inserting, > inserting 2 relationships between each 2 nodes, doing the traversing of all > these nodes and obtaining the nodes by their Id/property value. > > I measured the time needed to perform these operations, the resulted graph I > have put in the attachment. > > > > Before performing the tests, I have looked into the performance guide and > configuration settings, see 2 following links : > > http://wiki.neo4j.org/content/Configuration_Settings > > http://wiki.neo4j.org/content/Neo4j_Performance_Guide > > > > I ran the java in server mode with 512 MB heap size on Ubuntu 8.04, with the > default neo4j configuration. > > The RAM memory of my machine is 3550 MB and the amount of memory reserved > for linux and other programs is 805 MB. > > > > I performed tests with different number of nodes, ranging from 1 to 80000 (I > will continue with more nodes later). > > The problem that I found is with insert relationship performance (see > attachment), that it "jumps" at 39000 nodes to 12 ms from 3.5 ms (at 38000 > nodes) and after increasing the number of nodes further (40000, 50000) the > time decreases, then for 70000 and 80000 increases again. The interesting > thing is if you look at the graph, you can see that the difference between > the time of 38000 and 39000 nodes is around 8seconds, then starts decreasing > when increasing number of nodes further. My assumption is that here the > page faults problem appears, but I don't understand why the time after 40000 > nodes decreases. > > > > I checked the page size at my machine and it is 4096 bytes, meaning that the > number of pages for 512 MB is 128. > > I looked with vmstat command the memory statistics (at 39000 and 40000 > nodes) and I noticed that the number of page-outs (the event when pages are > written to disk) increases each time I run the insert relationship command. > However, there were no swap ins or swap outs. > > > > I also did not try bulk inserter yet, which I read about that to have the > better performance. > > > > I did not look into previous posts of this emailing list, if you encounter > such an issue before, please let me know where to look for the > explanation/solution. > > In any case I am looking forward to your response and suggestions, > > > > Best regards, > > > > Alisa > > > > > > > > > > > > > _______________________________________________ > Neo mailing list > [email protected] > https://lists.neo4j.org/mailman/listinfo/user > > -- Johan Svensson [[email protected]] Chief Technology Officer, Neo Technology www.neotechnology.com _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
