Hi Brock, Ok, that should have been taken care of by Maven, let me have a look at that. It should of course work to just mvn install:install-file them yourself into your repository. But I'll have a look at that.
I'm free for gchat any time today if you want. David On Thu, Aug 12, 2010 at 12:29 AM, Brock Rousseau <bro...@gmail.com> wrote: > Hey David, > > No worries about the disclaimer. I am getting a runtime error on startup > though due to the lack of the Jetty libraries. Any special instructions > there or should I just grab them from Jetty's website? > > Also, would any of you be available via gchat some time in the next 24 > hours > so I can relay the results of load testing? I can adjust my schedule since > you guys are CEST if I'm not mistaken, just let me know. > > Thanks, > Brock > > On Wed, Aug 11, 2010 at 2:46 PM, David Montag < > david.mon...@neotechnology.com> wrote: > > > Hi Brock, > > > > If you svn update to the latest version of the REST component, apply the > > patch I'll send to you, and rebuild it as per Jacob's previous > > instructions, > > then it should use Jetty instead. > > > > Keep in mind that this was a quick fix done today, so it might break down > > for the same or other reasons, especially as we haven't been able to > > reproduce the error you're seeing, and hence test that it actually fixes > > anything. Just a disclaimer. > > > > David > > > > On Wed, Aug 11, 2010 at 7:30 PM, Brock Rousseau <bro...@gmail.com> > wrote: > > > > > Hi Jacob, > > > > > > Would you be able to email me that patch? It's probably easier for me > to > > > throw it on our server and let you know how it goes rather than you > guys > > > having to try and reproduce it. > > > > > > Rough data for our server: > > > ~1.5 billion relationships > > > ~400 million nodes > > > ~1,200 transactions per minute > > > ~90% are lookups, 10% inserts > > > > > > Not sure if you're still around due to the time difference, but if you > > > could > > > provide that patch today I can test it right away. > > > > > > Thanks, > > > Brock > > > > > > On Wed, Aug 11, 2010 at 9:22 AM, Jacob Hansson <ja...@voltvoodoo.com> > > > wrote: > > > > > > > So the current status is that David has got neo4j REST running on > Jetty > > > > with > > > > all tests passing. We've also searched through the code, and found > that > > > > there are no interrupt() calls in the jersey source, while there are > a > > > few > > > > on the grizzly side. There is one in particular that we have been > > looking > > > > at, related to keep-alive timeouts, that may be the culprit. If that > > was > > > > the > > > > problem, we've got a fix for it. > > > > > > > > We have, however, been unable to recreate the problem so far, so we > > can't > > > > tell if we've solved it or not :) Brock: could you give us an idea of > > > what > > > > types of requests you were throwing at the server, and a rough > estimate > > > of > > > > how many? > > > > > > > > /Jacob > > > > > > > > On Wed, Aug 11, 2010 at 2:35 PM, Jacob Hansson <ja...@voltvoodoo.com > > > > > > wrote: > > > > > > > > > Hi all! > > > > > > > > > > Johan took a look at the stack trace, and explained the problem. > What > > > > > happens is that something, either the Grizzly server or the jersey > > > > wrapper > > > > > calls Thread.interrupt() on one of the neo4j threads (which should > be > > > > > considered a bug in whichever one of them does that). This triggers > > an > > > > > IOError deep down in neo4j, which in turn causes the rest of the > > > > problems. > > > > > > > > > > I'm working on recreating the situation, and David is working on > > > > switching > > > > > the REST system over to run on Jetty instead of Grizzly. We'll keep > > you > > > > > posted on the progress. > > > > > > > > > > /Jacob > > > > > > > > > > > > > > > On Wed, Aug 11, 2010 at 1:51 PM, Peter Neubauer < > > > > > peter.neuba...@neotechnology.com> wrote: > > > > > > > > > >> Nice, > > > > >> will try that out Jim! Grinder seems cool. > > > > >> > > > > >> Cheers, > > > > >> > > > > >> /peter neubauer > > > > >> > > > > >> COO and Sales, Neo Technology > > > > >> > > > > >> GTalk: neubauer.peter > > > > >> Skype peter.neubauer > > > > >> Phone +46 704 106975 > > > > >> LinkedIn http://www.linkedin.com/in/neubauer > > > > >> Twitter http://twitter.com/peterneubauer > > > > >> > > > > >> http://www.neo4j.org - Your high performance graph > > > > >> database. > > > > >> http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing > > > party. > > > > >> > > > > >> > > > > >> > > > > >> On Wed, Aug 11, 2010 at 12:52 PM, Jim Webber <j...@webber.name> > > wrote: > > > > >> > Perhaps something as simple as a Grinder script might help? > > > > >> > > > > > >> > Jim > > > > >> > > > > > >> > > > > > >> > On 11 Aug 2010, at 17:57, Brock Rousseau wrote: > > > > >> > > > > > >> >> Thanks Peter. Let us know if there is anything else we can > > provide > > > in > > > > >> the > > > > >> >> way of logs or diagnosis from our server. > > > > >> >> > > > > >> >> -Brock > > > > >> >> > > > > >> >> On Tue, Aug 10, 2010 at 11:51 PM, Peter Neubauer < > > > > >> >> peter.neuba...@neotechnology.com> wrote: > > > > >> >> > > > > >> >>> Mmh, > > > > >> >>> seems we should stresstest the server and Grizzly with e.g. > > > > >> >>> http://www.soapui.org and see if we can reproduce the > scenario, > > > if > > > > >> >>> there is no obvious hint to this. Will try to set it up ... > > > > >> >>> > > > > >> >>> Cheers, > > > > >> >>> > > > > >> >>> /peter neubauer > > > > >> >>> > > > > >> >>> COO and Sales, Neo Technology > > > > >> >>> > > > > >> >>> GTalk: neubauer.peter > > > > >> >>> Skype peter.neubauer > > > > >> >>> Phone +46 704 106975 > > > > >> >>> LinkedIn http://www.linkedin.com/in/neubauer > > > > >> >>> Twitter http://twitter.com/peterneubauer > > > > >> >>> > > > > >> >>> http://www.neo4j.org - Your high performance > > graph > > > > >> database. > > > > >> >>> http://www.thoughtmade.com - Scandinavia's coolest > > Bring-a-Thing > > > > >> party. > > > > >> >>> > > > > >> >>> > > > > >> >>> > > > > >> >>> On Wed, Aug 11, 2010 at 4:14 AM, Brock Rousseau < > > bro...@gmail.com > > > > > > > > >> wrote: > > > > >> >>>> The patch worked perfectly for increasing the concurrent > > > > transaction > > > > >> cap, > > > > >> >>>> but unfortunately exposed another issue. > > > > >> >>>> > > > > >> >>>> After increasing the load hitting our rest server, it > performs > > > > >> smoothly > > > > >> >>> for > > > > >> >>>> 10-15 minutes then begins issuing 500 responses on all > > > > transactions. > > > > >> When > > > > >> >>> it > > > > >> >>>> happens, the number of open transactions freezes in JMX and > the > > > > heap > > > > >> size > > > > >> >>>> essentially remains static. Below are the two stack traces we > > see > > > > in > > > > >> the > > > > >> >>>> wrapper.log. Here are what i think to be the relevant > > > configuration > > > > >> >>> lines: > > > > >> >>>> > > > > >> >>>> wrapper.conf: > > > > >> >>>> wrapper.java.additional.1=-d64 > > > > >> >>>> wrapper.java.additional.2=-server > > > > >> >>>> wrapper.java.additional.4=-Xmx8192m > > > > >> >>>> wrapper.java.additional.3=-XX:+UseConcMarkSweepGC > > > > >> >>>> wrapper.java.additional.4=-Dcom.sun.management.jmxremote > > > > >> >>>> > > > > >> >>>> neo4j.properties: > > > > >> >>>> > > > > >> >>>> rest_min_grizzly_threads=4 > > > > >> >>>> rest_max_grizzly_threads=128 > > > > >> >>>> > > > > >> >>>> neostore.nodestore.db.mapped_memory=4000M > > > > >> >>>> neostore.relationshipstore.db.mapped_memory=40000M > > > > >> >>>> neostore.propertystore.db.mapped_memory=1800M > > > > >> >>>> neostore.propertystore.db.index.mapped_memory=100M > > > > >> >>>> neostore.propertystore.db.index.keys.mapped_memory=100M > > > > >> >>>> neostore.propertystore.db.strings.mapped_memory=3G > > > > >> >>>> neostore.propertystore.db.arrays.mapped_memory=0M > > > > >> >>>> > > > > >> >>>> The server has 64Gb of total RAM so there should be a little > > over > > > 6 > > > > >> left > > > > >> >>> for > > > > >> >>>> the system. > > > > >> >>>> > > > > >> >>>> > > > > >> >>>> At the initial time of failure there are several of this > error: > > > > >> >>>> > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | Aug 10, 2010 > 1:00:33 > > PM > > > > >> >>>> com.sun.grizzly.http.KeepAliveThreadAttachment timedOut > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | WARNING: > Interrupting > > > > idle > > > > >> >>> Thread: > > > > >> >>>> Grizzly-9555-WorkerThread(1) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | Aug 10, 2010 > 1:00:33 > > PM > > > > >> >>>> com.sun.jersey.spi.container.ContainerResponse > > > > >> >>> mapMappableContainerException > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | SEVERE: The > > > > >> RuntimeException > > > > >> >>> could > > > > >> >>>> not be mapped to a response, re-throwing to the HTTP > container > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | > > > > >> >>>> > org.neo4j.kernel.impl.nioneo.store.UnderlyingStorageException: > > > > Unable > > > > >> to > > > > >> >>>> load position[7280476] @[968303308] > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > org.neo4j.kernel.impl.nioneo.store.PersistenceRow.readPosition(PersistenceRow.java:101) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:152) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:474) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > org.neo4j.kernel.impl.nioneo.store.AbstractDynamicStore.getLightRecords(AbstractDynamicStore.java:375) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > org.neo4j.kernel.impl.nioneo.store.PropertyStore.getRecord(PropertyStore.java:324) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.propertyGetValue(ReadTransaction.java:237) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > org.neo4j.kernel.impl.nioneo.xa.NioNeoDbPersistenceSource$ReadOnlyResourceConnection.loadPropertyValue(NioNeoDbPersistenceSource.java:216) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > org.neo4j.kernel.impl.persistence.PersistenceManager.loadPropertyValue(PersistenceManager.java:79) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > org.neo4j.kernel.impl.core.NodeManager.loadPropertyValue(NodeManager.java:579) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> > > > > > > org.neo4j.kernel.impl.core.Primitive.getPropertyValue(Primitive.java:546) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > org.neo4j.kernel.impl.core.Primitive.getProperty(Primitive.java:167) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > org.neo4j.kernel.impl.core.NodeProxy.getProperty(NodeProxy.java:134) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > org.neo4j.rest.domain.PropertiesMap.<init>(PropertiesMap.java:20) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > org.neo4j.rest.domain.NodeRepresentation.<init>(NodeRepresentation.java:20) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > org.neo4j.rest.domain.StorageActions$TraverserReturnType$1.toRepresentation(StorageActions.java:421) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > org.neo4j.rest.domain.StorageActions.traverseAndCollect(StorageActions.java:403) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> > > > > > > org.neo4j.rest.web.GenericWebService.traverse(GenericWebService.java:725) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > org.neo4j.rest.web.JsonAndHtmlWebService.jsonTraverse(JsonAndHtmlWebService.java:324) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> java.lang.reflect.Method.invoke(Method.java:616) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:184) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:67) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:276) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:133) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:83) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:133) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:71) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1171) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1103) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1053) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1043) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:406) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:477) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:662) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.http.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:188) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.http.servlet.FilterChainImpl.invokeFilterChain(FilterChainImpl.java:137) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.http.servlet.ServletAdapter.doService(ServletAdapter.java:376) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.http.servlet.ServletAdapter.service(ServletAdapter.java:324) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:166) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.tcp.http11.GrizzlyAdapterChain.service(GrizzlyAdapterChain.java:180) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:166) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> > > > > com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:753) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:661) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:914) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:166) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:166) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:135) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:102) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:88) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> > > > > > > com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> com.sun.grizzly.ContextTask.run(ContextTask.java:69) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.util.FixedThreadPool$BasicWorker.dowork(FixedThreadPool.java:379) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.util.FixedThreadPool$BasicWorker.run(FixedThreadPool.java:360) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> java.lang.Thread.run(Thread.java:636) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | Caused by: > > > > >> >>>> java.nio.channels.ClosedByInterruptException > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:201) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:335) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > org.neo4j.kernel.impl.nioneo.store.PersistenceRow.readPosition(PersistenceRow.java:80) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | ... 56 more > > > > >> >>>> > > > > >> >>>> Then this to this for every subsequent request: > > > > >> >>>> > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | Aug 10, 2010 > 1:02:13 > > PM > > > > >> >>>> com.sun.jersey.spi.container.ContainerResponse > > > > >> >>> mapMappableContainerException > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | SEVERE: The > > > > >> RuntimeException > > > > >> >>> could > > > > >> >>>> not be mapped to a response, re-throwing to the HTTP > container > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | > > > > >> >>>> org.neo4j.graphdb.TransactionFailureException: Unable to > begin > > > > >> >>> transaction > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > org.neo4j.kernel.EmbeddedGraphDbImpl.beginTx(EmbeddedGraphDbImpl.java:334) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > org.neo4j.kernel.EmbeddedGraphDatabase.beginTx(EmbeddedGraphDatabase.java:142) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> > > > org.neo4j.rest.domain.StorageActions.createNode(StorageActions.java:46) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > org.neo4j.rest.web.GenericWebService.createNode(GenericWebService.java:147) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > org.neo4j.rest.web.JsonAndHtmlWebService.jsonCreateNode(JsonAndHtmlWebService.java:90) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> sun.reflect.GeneratedMethodAccessor96.invoke(Unknown Source) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> java.lang.reflect.Method.invoke(Method.java:616) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:184) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:67) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:276) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:133) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:83) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:133) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:71) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1171) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1103) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1053) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1043) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:406) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:477) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:662) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.http.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:188) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.http.servlet.FilterChainImpl.invokeFilterChain(FilterChainImpl.java:137) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.http.servlet.ServletAdapter.doService(ServletAdapter.java:376) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.http.servlet.ServletAdapter.service(ServletAdapter.java:324) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:166) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.tcp.http11.GrizzlyAdapterChain.service(GrizzlyAdapterChain.java:180) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:166) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> > > > > com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:753) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:661) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:914) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:166) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:135) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:102) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:88) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> > > > > > > com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> com.sun.grizzly.ContextTask.run(ContextTask.java:69) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.util.FixedThreadPool$BasicWorker.dowork(FixedThreadPool.java:379) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > com.sun.grizzly.util.FixedThreadPool$BasicWorker.run(FixedThreadPool.java:360) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> java.lang.Thread.run(Thread.java:636) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | Caused by: > > > > >> >>>> javax.transaction.SystemException: TM has encountered some > > > problem, > > > > >> >>> please > > > > >> >>>> perform neccesary action (tx recovery/restart) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > org.neo4j.kernel.impl.transaction.TxManager.begin(TxManager.java:497) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > > > > > > org.neo4j.kernel.EmbeddedGraphDbImpl.beginTx(EmbeddedGraphDbImpl.java:329) > > > > >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | ... 43 more > > > > >> >>>> > > > > >> >>>> Do those look like they point to anything in particular? As > > > always > > > > >> your > > > > >> >>>> insight is greatly appreciated! > > > > >> >>>> > > > > >> >>>> Thanks, > > > > >> >>>> Brock > > > > >> >>>> > > > > >> >>>> On Tue, Aug 10, 2010 at 7:53 AM, Jacob Hansson < > > > > ja...@voltvoodoo.com > > > > >> > > > > > >> >>> wrote: > > > > >> >>>> > > > > >> >>>>> The problem with the resource starvation turned out to be > > fairly > > > > >> simple, > > > > >> >>>>> and > > > > >> >>>>> was due to a missing "connection:close" header in the neo4j > > REST > > > > >> client > > > > >> >>> I > > > > >> >>>>> was using. > > > > >> >>>>> > > > > >> >>>>> It would be interesting to look into the > > "connection:keep-alive" > > > > >> >>> behaviour > > > > >> >>>>> of Grizzly, using HTTP 1.1 pipelining would be an excellent > > way > > > to > > > > >> >>> increase > > > > >> >>>>> performance in the clients, but the few things I've read so > > far > > > > >> suggests > > > > >> >>>>> Grizzly does not yet support that. > > > > >> >>>>> > > > > >> >>>>> /Jacob > > > > >> >>>>> > > > > >> >>>>> On Tue, Aug 10, 2010 at 11:19 AM, Brock Rousseau < > > > > bro...@gmail.com> > > > > >> >>> wrote: > > > > >> >>>>> > > > > >> >>>>>> The patched trunk code is working fine after dropping it > over > > > our > > > > >> >>>>> existing > > > > >> >>>>>> deployment. I'm going to wait until we have the support of > > our > > > > >> >>> site-speed > > > > >> >>>>>> engineers in the morning before testing the transaction > limit > > > > under > > > > >> >>>>>> full production load, but I'll post the results as soon as > we > > > > have > > > > >> >>> them. > > > > >> >>>>>> > > > > >> >>>>>> Thanks, > > > > >> >>>>>> Brock > > > > >> >>>>>> > > > > >> >>>>>> > > > > >> >>>>>> On Tue, Aug 10, 2010 at 1:47 AM, Brock Rousseau < > > > > bro...@gmail.com> > > > > >> >>>>> wrote: > > > > >> >>>>>> > > > > >> >>>>>>> I got the patch Jacob, thanks! > > > > >> >>>>>>> > > > > >> >>>>>>> We're running Centos 5.5, Java 1.6.0, and Maven 2.2.1 > > > > >> >>>>>>> > > > > >> >>>>>>> If it's fine for you guys then as you said, likely some > > > > >> >>> configuration > > > > >> >>>>>>> difference with our system, so i'll just run it > > > > >> >>>>>> with"-Dmaven.test.skip=true" > > > > >> >>>>>>> > > > > >> >>>>>>> Applying the patch now, i'll let you know how it goes. > > Thanks > > > > >> again > > > > >> >>> for > > > > >> >>>>>> all > > > > >> >>>>>>> the support. > > > > >> >>>>>>> > > > > >> >>>>>>> -Brock > > > > >> >>>>>>> > > > > >> >>>>>>> On Tue, Aug 10, 2010 at 1:37 AM, Anders Nawroth < > > > > >> >>>>>>> and...@neotechnology.com> wrote: > > > > >> >>>>>>> > > > > >> >>>>>>>> hi! > > > > >> >>>>>>>> > > > > >> >>>>>>>> On 08/10/2010 10:06 AM, Brock Rousseau wrote: > > > > >> >>>>>>>>> The standalone project built fine for me, but we're > > getting > > > > >> these > > > > >> >>>>> test > > > > >> >>>>>>>>> failures for mvn install on the rest component: > > > > >> >>>>>>>> > > > > >> >>>>>>>> Try using Java 6 or skip the tests. > > > > >> >>>>>>>> > > > > >> >>>>>>>> From the pom.xml: > > > > >> >>>>>>>> <!-- NOTICE: Tests will not run on standard Java 5! > > > > >> >>>>>>>> This is due to the fact that there is no > > > easily > > > > >> >>>>>> available > > > > >> >>>>>>>> implementation of the javax.script API > for > > > Java > > > > 5 > > > > >> >>> in > > > > >> >>>>>> any > > > > >> >>>>>>>> maven repositories. For building on Java > 5, > > > > >> either > > > > >> >>>>> skip > > > > >> >>>>>>>> running the tests, or include the > > > javax.script > > > > js > > > > >> >>>>>>>> reference > > > > >> >>>>>>>> implementation from > > > > >> >>>>>> http://jcp.org/en/jsr/detail?id=223 > > > > >> >>>>>>>> in the classpath of the JVM. > > > > >> >>>>>>>> --> > > > > >> >>>>>>>> > > > > >> >>>>>>>> /anders > > > > >> >>>>>>>> _______________________________________________ > > > > >> >>>>>>>> Neo4j mailing list > > > > >> >>>>>>>> User@lists.neo4j.org > > > > >> >>>>>>>> https://lists.neo4j.org/mailman/listinfo/user > > > > >> >>>>>>>> > > > > >> >>>>>>> > > > > >> >>>>>>> > > > > >> >>>>>> _______________________________________________ > > > > >> >>>>>> Neo4j mailing list > > > > >> >>>>>> User@lists.neo4j.org > > > > >> >>>>>> https://lists.neo4j.org/mailman/listinfo/user > > > > >> >>>>>> > > > > >> >>>>> > > > > >> >>>>> > > > > >> >>>>> > > > > >> >>>>> -- > > > > >> >>>>> Jacob Hansson > > > > >> >>>>> Phone: +46 (0) 763503395 > > > > >> >>>>> Twitter: @jakewins > > > > >> >>>>> _______________________________________________ > > > > >> >>>>> Neo4j mailing list > > > > >> >>>>> User@lists.neo4j.org > > > > >> >>>>> https://lists.neo4j.org/mailman/listinfo/user > > > > >> >>>>> > > > > >> >>>> _______________________________________________ > > > > >> >>>> Neo4j mailing list > > > > >> >>>> User@lists.neo4j.org > > > > >> >>>> https://lists.neo4j.org/mailman/listinfo/user > > > > >> >>>> > > > > >> >>> _______________________________________________ > > > > >> >>> Neo4j mailing list > > > > >> >>> User@lists.neo4j.org > > > > >> >>> https://lists.neo4j.org/mailman/listinfo/user > > > > >> >>> > > > > >> >> _______________________________________________ > > > > >> >> Neo4j mailing list > > > > >> >> User@lists.neo4j.org > > > > >> >> https://lists.neo4j.org/mailman/listinfo/user > > > > >> > > > > > >> > _______________________________________________ > > > > >> > Neo4j mailing list > > > > >> > User@lists.neo4j.org > > > > >> > https://lists.neo4j.org/mailman/listinfo/user > > > > >> > > > > > >> _______________________________________________ > > > > >> Neo4j mailing list > > > > >> User@lists.neo4j.org > > > > >> https://lists.neo4j.org/mailman/listinfo/user > > > > >> > > > > > > > > > > > > > > > > > > > > -- > > > > > Jacob Hansson > > > > > Phone: +46 (0) 763503395 > > > > > Twitter: @jakewins > > > > > > > > > > > > > > > > > > > > > -- > > > > Jacob Hansson > > > > Phone: +46 (0) 763503395 > > > > Twitter: @jakewins > > > > _______________________________________________ > > > > Neo4j mailing list > > > > User@lists.neo4j.org > > > > https://lists.neo4j.org/mailman/listinfo/user > > > > > > > _______________________________________________ > > > Neo4j mailing list > > > User@lists.neo4j.org > > > https://lists.neo4j.org/mailman/listinfo/user > > > > > _______________________________________________ > > Neo4j mailing list > > User@lists.neo4j.org > > https://lists.neo4j.org/mailman/listinfo/user > > > _______________________________________________ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
