Re: [Neo4j] neo4j REST server configuration
Brock, I've written a patch that allows setting the max threads via the neo4j configuration file. Applying it fixes the problem you mention, but leads to some kind of resource starvation higher up. This is not as critical, at 400 concurrent requests it will make the server alternate between 12ms read response times (which should be the norm) and 300ms. It's still bad, so before this patch is applied I'd like to dig into where that comes from. If you want, though, you can apply the patch yourself. To do that, you would download the REST component and the REST standalone project, avaliable here: svn co https://svn.neo4j.org/laboratory/components/rest/ svn co https://svn.neo4j.org/assemblies/rest-standalone/trunk/ Apply the patch (attached to this email) to the REST component. Run mvn install on the REST component, and then mvn package on the standalone project. This will give you a patched version with a default max thread count of 128 (compared to 5 before). To increase it more, go to the neo4j-rest-db-folder in the deployed system, open (or create) the file neo4j.properties, and use the following two settings to modify the thread count: rest_min_grizzly_threads=16 rest_max_grizzly_threads=48 I can't give you a perfect number, but it makes a big difference in response times, in both directions, to change these settings. /Jacob On Mon, Aug 9, 2010 at 8:51 PM, Brock Rousseau bro...@gmail.com wrote: Hey Jacob, Thanks for the quick response! We saw your post on the grizzly mailing list about the transaction limit fix. Is that something we'd be able to implement on our end today? We've had to throttle back the traffic significantly and are eager to see it in action at full volume. -Brock ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Jacob Hansson Phone: +46 (0) 763503395 Twitter: @jakewins ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] neo4j REST server configuration
Hey Jacob, The patch sounds perfect, but I didn't see an attachement. Did i miss it somehow? The standalone project built fine for me, but we're getting these test failures for mvn install on the rest component: Failed tests: shouldBeAbleToDescribeTraverser(org.neo4j.rest.web.JsonWebServiceTest) Tests in error: shouldBeAbleToTraverseEverything(org.neo4j.rest.domain.StorageActionsTest) shouldBeAbleToUseCustomReturnFilter(org.neo4j.rest.domain.StorageActionsTest) shouldBeAbleToTraverseWithMaxDepthAndPruneEvaluatorCombined(org.neo4j.rest.domain.StorageActionsTest) shouldGetExpectedHitsWhenTraversingWithDescription(org.neo4j.rest.functional.TraverserFunctionalTest) Should we be running the install by ignoring tests or should those be passing? Thanks, Brock On Tue, Aug 10, 2010 at 12:33 AM, Jacob Hansson ja...@voltvoodoo.comwrote: Brock, I've written a patch that allows setting the max threads via the neo4j configuration file. Applying it fixes the problem you mention, but leads to some kind of resource starvation higher up. This is not as critical, at 400 concurrent requests it will make the server alternate between 12ms read response times (which should be the norm) and 300ms. It's still bad, so before this patch is applied I'd like to dig into where that comes from. If you want, though, you can apply the patch yourself. To do that, you would download the REST component and the REST standalone project, avaliable here: svn co https://svn.neo4j.org/laboratory/components/rest/ svn co https://svn.neo4j.org/assemblies/rest-standalone/trunk/ Apply the patch (attached to this email) to the REST component. Run mvn install on the REST component, and then mvn package on the standalone project. This will give you a patched version with a default max thread count of 128 (compared to 5 before). To increase it more, go to the neo4j-rest-db-folder in the deployed system, open (or create) the file neo4j.properties, and use the following two settings to modify the thread count: rest_min_grizzly_threads=16 rest_max_grizzly_threads=48 I can't give you a perfect number, but it makes a big difference in response times, in both directions, to change these settings. /Jacob On Mon, Aug 9, 2010 at 8:51 PM, Brock Rousseau bro...@gmail.com wrote: Hey Jacob, Thanks for the quick response! We saw your post on the grizzly mailing list about the transaction limit fix. Is that something we'd be able to implement on our end today? We've had to throttle back the traffic significantly and are eager to see it in action at full volume. -Brock ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Jacob Hansson Phone: +46 (0) 763503395 Twitter: @jakewins ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] neo4j REST server configuration
Apparently the email list deletes attachments it does not recognize, I'll send it directly to your email. As far as the test failures go, I don't know what causes that. I ran and passed all tests with the patch, but the trunk version should do that too. It might be something platform specific.. what platform are you running on? /Jacob On Tue, Aug 10, 2010 at 10:06 AM, Brock Rousseau bro...@gmail.com wrote: Hey Jacob, The patch sounds perfect, but I didn't see an attachement. Did i miss it somehow? The standalone project built fine for me, but we're getting these test failures for mvn install on the rest component: Failed tests: shouldBeAbleToDescribeTraverser(org.neo4j.rest.web.JsonWebServiceTest) Tests in error: shouldBeAbleToTraverseEverything(org.neo4j.rest.domain.StorageActionsTest) shouldBeAbleToUseCustomReturnFilter(org.neo4j.rest.domain.StorageActionsTest) shouldBeAbleToTraverseWithMaxDepthAndPruneEvaluatorCombined(org.neo4j.rest.domain.StorageActionsTest) shouldGetExpectedHitsWhenTraversingWithDescription(org.neo4j.rest.functional.TraverserFunctionalTest) Should we be running the install by ignoring tests or should those be passing? Thanks, Brock On Tue, Aug 10, 2010 at 12:33 AM, Jacob Hansson ja...@voltvoodoo.com wrote: Brock, I've written a patch that allows setting the max threads via the neo4j configuration file. Applying it fixes the problem you mention, but leads to some kind of resource starvation higher up. This is not as critical, at 400 concurrent requests it will make the server alternate between 12ms read response times (which should be the norm) and 300ms. It's still bad, so before this patch is applied I'd like to dig into where that comes from. If you want, though, you can apply the patch yourself. To do that, you would download the REST component and the REST standalone project, avaliable here: svn co https://svn.neo4j.org/laboratory/components/rest/ svn co https://svn.neo4j.org/assemblies/rest-standalone/trunk/ Apply the patch (attached to this email) to the REST component. Run mvn install on the REST component, and then mvn package on the standalone project. This will give you a patched version with a default max thread count of 128 (compared to 5 before). To increase it more, go to the neo4j-rest-db-folder in the deployed system, open (or create) the file neo4j.properties, and use the following two settings to modify the thread count: rest_min_grizzly_threads=16 rest_max_grizzly_threads=48 I can't give you a perfect number, but it makes a big difference in response times, in both directions, to change these settings. /Jacob On Mon, Aug 9, 2010 at 8:51 PM, Brock Rousseau bro...@gmail.com wrote: Hey Jacob, Thanks for the quick response! We saw your post on the grizzly mailing list about the transaction limit fix. Is that something we'd be able to implement on our end today? We've had to throttle back the traffic significantly and are eager to see it in action at full volume. -Brock ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Jacob Hansson Phone: +46 (0) 763503395 Twitter: @jakewins ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Jacob Hansson Phone: +46 (0) 763503395 Twitter: @jakewins ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] neo4j REST server configuration
I got the patch Jacob, thanks! We're running Centos 5.5, Java 1.6.0, and Maven 2.2.1 If it's fine for you guys then as you said, likely some configuration difference with our system, so i'll just run it with-Dmaven.test.skip=true Applying the patch now, i'll let you know how it goes. Thanks again for all the support. -Brock On Tue, Aug 10, 2010 at 1:37 AM, Anders Nawroth and...@neotechnology.comwrote: hi! On 08/10/2010 10:06 AM, Brock Rousseau wrote: The standalone project built fine for me, but we're getting these test failures for mvn install on the rest component: Try using Java 6 or skip the tests. From the pom.xml: !-- NOTICE: Tests will not run on standard Java 5! This is due to the fact that there is no easily available implementation of the javax.script API for Java 5 in any maven repositories. For building on Java 5, either skip running the tests, or include the javax.script js reference implementation from http://jcp.org/en/jsr/detail?id=223 in the classpath of the JVM. -- /anders ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] neo4j REST server configuration
The patched trunk code is working fine after dropping it over our existing deployment. I'm going to wait until we have the support of our site-speed engineers in the morning before testing the transaction limit under full production load, but I'll post the results as soon as we have them. Thanks, Brock On Tue, Aug 10, 2010 at 1:47 AM, Brock Rousseau bro...@gmail.com wrote: I got the patch Jacob, thanks! We're running Centos 5.5, Java 1.6.0, and Maven 2.2.1 If it's fine for you guys then as you said, likely some configuration difference with our system, so i'll just run it with-Dmaven.test.skip=true Applying the patch now, i'll let you know how it goes. Thanks again for all the support. -Brock On Tue, Aug 10, 2010 at 1:37 AM, Anders Nawroth and...@neotechnology.com wrote: hi! On 08/10/2010 10:06 AM, Brock Rousseau wrote: The standalone project built fine for me, but we're getting these test failures for mvn install on the rest component: Try using Java 6 or skip the tests. From the pom.xml: !-- NOTICE: Tests will not run on standard Java 5! This is due to the fact that there is no easily available implementation of the javax.script API for Java 5 in any maven repositories. For building on Java 5, either skip running the tests, or include the javax.script js reference implementation from http://jcp.org/en/jsr/detail?id=223 in the classpath of the JVM. -- /anders ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] neo4j REST server configuration
The problem with the resource starvation turned out to be fairly simple, and was due to a missing connection:close header in the neo4j REST client I was using. It would be interesting to look into the connection:keep-alive behaviour of Grizzly, using HTTP 1.1 pipelining would be an excellent way to increase performance in the clients, but the few things I've read so far suggests Grizzly does not yet support that. /Jacob On Tue, Aug 10, 2010 at 11:19 AM, Brock Rousseau bro...@gmail.com wrote: The patched trunk code is working fine after dropping it over our existing deployment. I'm going to wait until we have the support of our site-speed engineers in the morning before testing the transaction limit under full production load, but I'll post the results as soon as we have them. Thanks, Brock On Tue, Aug 10, 2010 at 1:47 AM, Brock Rousseau bro...@gmail.com wrote: I got the patch Jacob, thanks! We're running Centos 5.5, Java 1.6.0, and Maven 2.2.1 If it's fine for you guys then as you said, likely some configuration difference with our system, so i'll just run it with-Dmaven.test.skip=true Applying the patch now, i'll let you know how it goes. Thanks again for all the support. -Brock On Tue, Aug 10, 2010 at 1:37 AM, Anders Nawroth and...@neotechnology.com wrote: hi! On 08/10/2010 10:06 AM, Brock Rousseau wrote: The standalone project built fine for me, but we're getting these test failures for mvn install on the rest component: Try using Java 6 or skip the tests. From the pom.xml: !-- NOTICE: Tests will not run on standard Java 5! This is due to the fact that there is no easily available implementation of the javax.script API for Java 5 in any maven repositories. For building on Java 5, either skip running the tests, or include the javax.script js reference implementation from http://jcp.org/en/jsr/detail?id=223 in the classpath of the JVM. -- /anders ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Jacob Hansson Phone: +46 (0) 763503395 Twitter: @jakewins ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] neo4j REST server configuration
The patch worked perfectly for increasing the concurrent transaction cap, but unfortunately exposed another issue. After increasing the load hitting our rest server, it performs smoothly for 10-15 minutes then begins issuing 500 responses on all transactions. When it happens, the number of open transactions freezes in JMX and the heap size essentially remains static. Below are the two stack traces we see in the wrapper.log. Here are what i think to be the relevant configuration lines: wrapper.conf: wrapper.java.additional.1=-d64 wrapper.java.additional.2=-server wrapper.java.additional.4=-Xmx8192m wrapper.java.additional.3=-XX:+UseConcMarkSweepGC wrapper.java.additional.4=-Dcom.sun.management.jmxremote neo4j.properties: rest_min_grizzly_threads=4 rest_max_grizzly_threads=128 neostore.nodestore.db.mapped_memory=4000M neostore.relationshipstore.db.mapped_memory=4M neostore.propertystore.db.mapped_memory=1800M neostore.propertystore.db.index.mapped_memory=100M neostore.propertystore.db.index.keys.mapped_memory=100M neostore.propertystore.db.strings.mapped_memory=3G neostore.propertystore.db.arrays.mapped_memory=0M The server has 64Gb of total RAM so there should be a little over 6 left for the system. At the initial time of failure there are several of this error: INFO | jvm 1| 2010/08/10 13:00:33 | Aug 10, 2010 1:00:33 PM com.sun.grizzly.http.KeepAliveThreadAttachment timedOut INFO | jvm 1| 2010/08/10 13:00:33 | WARNING: Interrupting idle Thread: Grizzly-9555-WorkerThread(1) INFO | jvm 1| 2010/08/10 13:00:33 | Aug 10, 2010 1:00:33 PM com.sun.jersey.spi.container.ContainerResponse mapMappableContainerException INFO | jvm 1| 2010/08/10 13:00:33 | SEVERE: The RuntimeException could not be mapped to a response, re-throwing to the HTTP container INFO | jvm 1| 2010/08/10 13:00:33 | org.neo4j.kernel.impl.nioneo.store.UnderlyingStorageException: Unable to load position[7280476] @[968303308] INFO | jvm 1| 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.nioneo.store.PersistenceRow.readPosition(PersistenceRow.java:101) INFO | jvm 1| 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:152) INFO | jvm 1| 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:474) INFO | jvm 1| 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.nioneo.store.AbstractDynamicStore.getLightRecords(AbstractDynamicStore.java:375) INFO | jvm 1| 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.nioneo.store.PropertyStore.getRecord(PropertyStore.java:324) INFO | jvm 1| 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.propertyGetValue(ReadTransaction.java:237) INFO | jvm 1| 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.nioneo.xa.NioNeoDbPersistenceSource$ReadOnlyResourceConnection.loadPropertyValue(NioNeoDbPersistenceSource.java:216) INFO | jvm 1| 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.persistence.PersistenceManager.loadPropertyValue(PersistenceManager.java:79) INFO | jvm 1| 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.core.NodeManager.loadPropertyValue(NodeManager.java:579) INFO | jvm 1| 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.core.Primitive.getPropertyValue(Primitive.java:546) INFO | jvm 1| 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.core.Primitive.getProperty(Primitive.java:167) INFO | jvm 1| 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.core.NodeProxy.getProperty(NodeProxy.java:134) INFO | jvm 1| 2010/08/10 13:00:33 | at org.neo4j.rest.domain.PropertiesMap.init(PropertiesMap.java:20) INFO | jvm 1| 2010/08/10 13:00:33 | at org.neo4j.rest.domain.NodeRepresentation.init(NodeRepresentation.java:20) INFO | jvm 1| 2010/08/10 13:00:33 | at org.neo4j.rest.domain.StorageActions$TraverserReturnType$1.toRepresentation(StorageActions.java:421) INFO | jvm 1| 2010/08/10 13:00:33 | at org.neo4j.rest.domain.StorageActions.traverseAndCollect(StorageActions.java:403) INFO | jvm 1| 2010/08/10 13:00:33 | at org.neo4j.rest.web.GenericWebService.traverse(GenericWebService.java:725) INFO | jvm 1| 2010/08/10 13:00:33 | at org.neo4j.rest.web.JsonAndHtmlWebService.jsonTraverse(JsonAndHtmlWebService.java:324) INFO | jvm 1| 2010/08/10 13:00:33 | at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) INFO | jvm 1| 2010/08/10 13:00:33 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) INFO | jvm 1| 2010/08/10 13:00:33 | at java.lang.reflect.Method.invoke(Method.java:616) INFO | jvm 1| 2010/08/10 13:00:33 | at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:184) INFO | jvm 1| 2010/08/10 13:00:33 | at
