Hi Peter, I'd be glad to share the code, I'll commit soon and share with the users list.
I've run some more load/concurrency tests and am seeing some strange results. Maybe someone can help explain this to me: I run a load test where I fire off 100K "create empty node" REST requests to Neo as quickly as possible. With my code updates to allow configuration of the Jetty thread pool size, I can effectively reduce or increase the maximum concurrent transaction limit on the server. If I limit the thread pool so that there is only 1 thread available for requests, I see (as expected) the PeakNumberOfConcurrentTransactions reported by the Neo4j Transactions MBean is 1. If I scale the thread pool up so that there are 800 available request threads, I can throw enough load at the server to cause 800 concurrent transactions. From what I have read, node creation causes a node-local lock, not a global node lock, so there shouldn't be a lock-imposed concurrency bottleneck. The strange thing is, no matter whether I have 1 or 800 concurrent transactions, my total node creation throughput is always the same (~1600 nodes/sec). Even with 800 concurrent transactions, my server is only using ~15% CPU and ~25% memory (JVM Xmm/Xmx = 1024m/2048m), so server load wouldn't appear to be an issue. I've followed all the recommendations I could find including sysctl limits and JVM settings, but the rate doesn't change. I have also tried running the load test from multiple clients simultaneously (just to be sure I'm not running into any limits on the client machine), and indeed as soon as I add a second load test client, the throughput on each client gets cut in half. If I'm talking to Neo in a way that is unrestricted by things like thread pool size and concurrency limits, I'd expect to be able to scale up my load tests and see at least some level of throughput improvement until I start to saturate/overload the box. The fact that increasing concurrency doesn't increase throughput makes me think that there's some internal bottleneck or synchronization point that's limiting. Any thoughts? I'm glad to look through the code and investigate, any ideas you have would be a big help. Thanks, and sorry for the long question! Stephen -----Original Message----- From: Peter Neubauer [mailto:peter.neuba...@neotechnology.com] Sent: Monday, April 18, 2011 12:50 AM To: Neo4j user discussions Subject: Re: [Neo4j] Question about REST interface concurrency Stephen, did you fork the code? Would be good to merge in the changes or at least take a look at them! Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://startupbootcamp.org/ - Öresund - Innovation happens HERE. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Mon, Apr 18, 2011 at 4:08 AM, Stephen Roos <sr...@careerarcgroup.com> wrote: > Hi Jim, > > Thanks for the quick reply. I tried the configuration mentioned here > ("rest_max_jetty_threads"): > > https://trac.neo4j.org/changeset/6157/laboratory/components/rest > > But it doesn't seem to have changed anything. I took a look through the code > and didn't see any configuration settings exposed in Jetty6WebServer. I > added the changes myself and am starting to see some good results (I've > exposed settings for min/max threadpool size, # acceptor threads, acceptor > queue size, and request buffer size). Is there anything else that you'd > recommend tweaking to improve throughput? > > Thanks again for your help! > > > > -----Original Message----- > From: Jim Webber [mailto:j...@neotechnology.com] > Sent: Friday, April 15, 2011 1:57 AM > To: Neo4j user discussions > Subject: Re: [Neo4j] Question about REST interface concurrency > > Hi Stephen, > > The same Jetty tweaks that worked in previous versions will work with 1.3. We > haven't changed any of the Jetty stuff under the covers. > > Jim > _______________________________________________ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
