The patch worked perfectly for increasing the concurrent transaction cap, but unfortunately exposed another issue.
After increasing the load hitting our rest server, it performs smoothly for 10-15 minutes then begins issuing 500 responses on all transactions. When it happens, the number of open transactions freezes in JMX and the heap size essentially remains static. Below are the two stack traces we see in the wrapper.log. Here are what i think to be the relevant configuration lines: wrapper.conf: wrapper.java.additional.1=-d64 wrapper.java.additional.2=-server wrapper.java.additional.4=-Xmx8192m wrapper.java.additional.3=-XX:+UseConcMarkSweepGC wrapper.java.additional.4=-Dcom.sun.management.jmxremote neo4j.properties: rest_min_grizzly_threads=4 rest_max_grizzly_threads=128 neostore.nodestore.db.mapped_memory=4000M neostore.relationshipstore.db.mapped_memory=40000M neostore.propertystore.db.mapped_memory=1800M neostore.propertystore.db.index.mapped_memory=100M neostore.propertystore.db.index.keys.mapped_memory=100M neostore.propertystore.db.strings.mapped_memory=3G neostore.propertystore.db.arrays.mapped_memory=0M The server has 64Gb of total RAM so there should be a little over 6 left for the system. At the initial time of failure there are several of this error: INFO | jvm 1 | 2010/08/10 13:00:33 | Aug 10, 2010 1:00:33 PM com.sun.grizzly.http.KeepAliveThreadAttachment timedOut INFO | jvm 1 | 2010/08/10 13:00:33 | WARNING: Interrupting idle Thread: Grizzly-9555-WorkerThread(1) INFO | jvm 1 | 2010/08/10 13:00:33 | Aug 10, 2010 1:00:33 PM com.sun.jersey.spi.container.ContainerResponse mapMappableContainerException INFO | jvm 1 | 2010/08/10 13:00:33 | SEVERE: The RuntimeException could not be mapped to a response, re-throwing to the HTTP container INFO | jvm 1 | 2010/08/10 13:00:33 | org.neo4j.kernel.impl.nioneo.store.UnderlyingStorageException: Unable to load position[7280476] @[968303308] INFO | jvm 1 | 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.nioneo.store.PersistenceRow.readPosition(PersistenceRow.java:101) INFO | jvm 1 | 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:152) INFO | jvm 1 | 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:474) INFO | jvm 1 | 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.nioneo.store.AbstractDynamicStore.getLightRecords(AbstractDynamicStore.java:375) INFO | jvm 1 | 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.nioneo.store.PropertyStore.getRecord(PropertyStore.java:324) INFO | jvm 1 | 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.propertyGetValue(ReadTransaction.java:237) INFO | jvm 1 | 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.nioneo.xa.NioNeoDbPersistenceSource$ReadOnlyResourceConnection.loadPropertyValue(NioNeoDbPersistenceSource.java:216) INFO | jvm 1 | 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.persistence.PersistenceManager.loadPropertyValue(PersistenceManager.java:79) INFO | jvm 1 | 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.core.NodeManager.loadPropertyValue(NodeManager.java:579) INFO | jvm 1 | 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.core.Primitive.getPropertyValue(Primitive.java:546) INFO | jvm 1 | 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.core.Primitive.getProperty(Primitive.java:167) INFO | jvm 1 | 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.core.NodeProxy.getProperty(NodeProxy.java:134) INFO | jvm 1 | 2010/08/10 13:00:33 | at org.neo4j.rest.domain.PropertiesMap.<init>(PropertiesMap.java:20) INFO | jvm 1 | 2010/08/10 13:00:33 | at org.neo4j.rest.domain.NodeRepresentation.<init>(NodeRepresentation.java:20) INFO | jvm 1 | 2010/08/10 13:00:33 | at org.neo4j.rest.domain.StorageActions$TraverserReturnType$1.toRepresentation(StorageActions.java:421) INFO | jvm 1 | 2010/08/10 13:00:33 | at org.neo4j.rest.domain.StorageActions.traverseAndCollect(StorageActions.java:403) INFO | jvm 1 | 2010/08/10 13:00:33 | at org.neo4j.rest.web.GenericWebService.traverse(GenericWebService.java:725) INFO | jvm 1 | 2010/08/10 13:00:33 | at org.neo4j.rest.web.JsonAndHtmlWebService.jsonTraverse(JsonAndHtmlWebService.java:324) INFO | jvm 1 | 2010/08/10 13:00:33 | at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) INFO | jvm 1 | 2010/08/10 13:00:33 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) INFO | jvm 1 | 2010/08/10 13:00:33 | at java.lang.reflect.Method.invoke(Method.java:616) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:184) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:67) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:276) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:133) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:83) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:133) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:71) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1171) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1103) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1053) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1043) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:406) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:477) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:662) INFO | jvm 1 | 2010/08/10 13:00:33 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.grizzly.http.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:188) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.grizzly.http.servlet.FilterChainImpl.invokeFilterChain(FilterChainImpl.java:137) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.grizzly.http.servlet.ServletAdapter.doService(ServletAdapter.java:376) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.grizzly.http.servlet.ServletAdapter.service(ServletAdapter.java:324) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:166) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.grizzly.tcp.http11.GrizzlyAdapterChain.service(GrizzlyAdapterChain.java:180) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:166) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:753) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:661) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:914) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:166) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:166) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:135) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:102) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:88) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.grizzly.ContextTask.run(ContextTask.java:69) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.grizzly.util.FixedThreadPool$BasicWorker.dowork(FixedThreadPool.java:379) INFO | jvm 1 | 2010/08/10 13:00:33 | at com.sun.grizzly.util.FixedThreadPool$BasicWorker.run(FixedThreadPool.java:360) INFO | jvm 1 | 2010/08/10 13:00:33 | at java.lang.Thread.run(Thread.java:636) INFO | jvm 1 | 2010/08/10 13:00:33 | Caused by: java.nio.channels.ClosedByInterruptException INFO | jvm 1 | 2010/08/10 13:00:33 | at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:201) INFO | jvm 1 | 2010/08/10 13:00:33 | at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:335) INFO | jvm 1 | 2010/08/10 13:00:33 | at org.neo4j.kernel.impl.nioneo.store.PersistenceRow.readPosition(PersistenceRow.java:80) INFO | jvm 1 | 2010/08/10 13:00:33 | ... 56 more Then this to this for every subsequent request: INFO | jvm 1 | 2010/08/10 13:02:13 | Aug 10, 2010 1:02:13 PM com.sun.jersey.spi.container.ContainerResponse mapMappableContainerException INFO | jvm 1 | 2010/08/10 13:02:13 | SEVERE: The RuntimeException could not be mapped to a response, re-throwing to the HTTP container INFO | jvm 1 | 2010/08/10 13:02:13 | org.neo4j.graphdb.TransactionFailureException: Unable to begin transaction INFO | jvm 1 | 2010/08/10 13:02:13 | at org.neo4j.kernel.EmbeddedGraphDbImpl.beginTx(EmbeddedGraphDbImpl.java:334) INFO | jvm 1 | 2010/08/10 13:02:13 | at org.neo4j.kernel.EmbeddedGraphDatabase.beginTx(EmbeddedGraphDatabase.java:142) INFO | jvm 1 | 2010/08/10 13:02:13 | at org.neo4j.rest.domain.StorageActions.createNode(StorageActions.java:46) INFO | jvm 1 | 2010/08/10 13:02:13 | at org.neo4j.rest.web.GenericWebService.createNode(GenericWebService.java:147) INFO | jvm 1 | 2010/08/10 13:02:13 | at org.neo4j.rest.web.JsonAndHtmlWebService.jsonCreateNode(JsonAndHtmlWebService.java:90) INFO | jvm 1 | 2010/08/10 13:02:13 | at sun.reflect.GeneratedMethodAccessor96.invoke(Unknown Source) INFO | jvm 1 | 2010/08/10 13:02:13 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) INFO | jvm 1 | 2010/08/10 13:02:13 | at java.lang.reflect.Method.invoke(Method.java:616) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:184) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:67) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:276) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:133) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:83) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:133) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:71) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1171) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1103) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1053) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1043) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:406) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:477) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:662) INFO | jvm 1 | 2010/08/10 13:02:13 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.grizzly.http.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:188) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.grizzly.http.servlet.FilterChainImpl.invokeFilterChain(FilterChainImpl.java:137) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.grizzly.http.servlet.ServletAdapter.doService(ServletAdapter.java:376) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.grizzly.http.servlet.ServletAdapter.service(ServletAdapter.java:324) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:166) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.grizzly.tcp.http11.GrizzlyAdapterChain.service(GrizzlyAdapterChain.java:180) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:166) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:753) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:661) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:914) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:166) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:135) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:102) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:88) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.grizzly.ContextTask.run(ContextTask.java:69) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.grizzly.util.FixedThreadPool$BasicWorker.dowork(FixedThreadPool.java:379) INFO | jvm 1 | 2010/08/10 13:02:13 | at com.sun.grizzly.util.FixedThreadPool$BasicWorker.run(FixedThreadPool.java:360) INFO | jvm 1 | 2010/08/10 13:02:13 | at java.lang.Thread.run(Thread.java:636) INFO | jvm 1 | 2010/08/10 13:02:13 | Caused by: javax.transaction.SystemException: TM has encountered some problem, please perform neccesary action (tx recovery/restart) INFO | jvm 1 | 2010/08/10 13:02:13 | at org.neo4j.kernel.impl.transaction.TxManager.begin(TxManager.java:497) INFO | jvm 1 | 2010/08/10 13:02:13 | at org.neo4j.kernel.EmbeddedGraphDbImpl.beginTx(EmbeddedGraphDbImpl.java:329) INFO | jvm 1 | 2010/08/10 13:02:13 | ... 43 more Do those look like they point to anything in particular? As always your insight is greatly appreciated! Thanks, Brock On Tue, Aug 10, 2010 at 7:53 AM, Jacob Hansson <ja...@voltvoodoo.com> wrote: > The problem with the resource starvation turned out to be fairly simple, > and > was due to a missing "connection:close" header in the neo4j REST client I > was using. > > It would be interesting to look into the "connection:keep-alive" behaviour > of Grizzly, using HTTP 1.1 pipelining would be an excellent way to increase > performance in the clients, but the few things I've read so far suggests > Grizzly does not yet support that. > > /Jacob > > On Tue, Aug 10, 2010 at 11:19 AM, Brock Rousseau <bro...@gmail.com> wrote: > > > The patched trunk code is working fine after dropping it over our > existing > > deployment. I'm going to wait until we have the support of our site-speed > > engineers in the morning before testing the transaction limit under > > full production load, but I'll post the results as soon as we have them. > > > > Thanks, > > Brock > > > > > > On Tue, Aug 10, 2010 at 1:47 AM, Brock Rousseau <bro...@gmail.com> > wrote: > > > > > I got the patch Jacob, thanks! > > > > > > We're running Centos 5.5, Java 1.6.0, and Maven 2.2.1 > > > > > > If it's fine for you guys then as you said, likely some configuration > > > difference with our system, so i'll just run it > > with"-Dmaven.test.skip=true" > > > > > > Applying the patch now, i'll let you know how it goes. Thanks again for > > all > > > the support. > > > > > > -Brock > > > > > > On Tue, Aug 10, 2010 at 1:37 AM, Anders Nawroth < > > > and...@neotechnology.com> wrote: > > > > > >> hi! > > >> > > >> On 08/10/2010 10:06 AM, Brock Rousseau wrote: > > >> > The standalone project built fine for me, but we're getting these > test > > >> > failures for mvn install on the rest component: > > >> > > >> Try using Java 6 or skip the tests. > > >> > > >> From the pom.xml: > > >> <!-- NOTICE: Tests will not run on standard Java 5! > > >> This is due to the fact that there is no easily > > available > > >> implementation of the javax.script API for Java 5 in > > any > > >> maven repositories. For building on Java 5, either > skip > > >> running the tests, or include the javax.script js > > >> reference > > >> implementation from > > http://jcp.org/en/jsr/detail?id=223 > > >> in the classpath of the JVM. > > >> --> > > >> > > >> /anders > > >> _______________________________________________ > > >> Neo4j mailing list > > >> User@lists.neo4j.org > > >> https://lists.neo4j.org/mailman/listinfo/user > > >> > > > > > > > > _______________________________________________ > > Neo4j mailing list > > User@lists.neo4j.org > > https://lists.neo4j.org/mailman/listinfo/user > > > > > > -- > Jacob Hansson > Phone: +46 (0) 763503395 > Twitter: @jakewins > _______________________________________________ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
