Re: [Neo4j] REST batch completes with error code
Damn, this is a really good catch. I think it should be rather simple to wait with committing the transaction until the full response is generated. I'll look into it. In general, and as has been mentioned several times before, we obviously need to look at how we could stream the batch operations, rather than keeping it all in memory.. /Jake On Fri, Sep 2, 2011 at 5:39 AM, jadell josh.ad...@gmail.com wrote: I'm sending a batch REST request, with 15000 operations (1 create nodes, and 5000 create relationships between those nodes.) In webadmin, I can see all the nodes and relationships have been created, and I can browse them using the data browser. So the all my entities were successfully created. But the HTTP response is a 500 Java heap space error. It looks from the stack trace in the log like the failure occurred when mapping the results into the HTTP response. This could be a pretty nasty surprise for someone who thinks their batch failed when all the entities were actually created. Is there anyway to not commit the transaction on the server until the response is generated and sent successfully? Here is the stack trace from the log: Sep 1, 2011 11:30:01 PM com.sun.jersey.spi.container.ContainerResponse mapMappableContainerException SEVERE: The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390) at java.lang.StringBuffer.append(StringBuffer.java:219) at org.apache.commons.lang.StringUtils.join(StringUtils.java:3088) at org.apache.commons.lang.StringUtils.join(StringUtils.java:3133) at org.neo4j.server.rest.repr.BatchOperationResults.toJSON(BatchOperationResults.java:99) at org.neo4j.server.rest.web.BatchOperationService.performBatchOperations(BatchOperationService.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:184) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:67) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:276) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:83) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:133) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:71) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1171) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1103) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1053) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1043) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:406) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:477) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:662) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) Sep 1, 2011 11:30:01 PM org.mortbay.log.Slf4jLog warn SEVERE: /db/data/batch java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390) at java.lang.StringBuffer.append(StringBuffer.java:219) at
[Neo4j] REST batch completes with error code
I'm sending a batch REST request, with 15000 operations (1 create nodes, and 5000 create relationships between those nodes.) In webadmin, I can see all the nodes and relationships have been created, and I can browse them using the data browser. So the all my entities were successfully created. But the HTTP response is a 500 Java heap space error. It looks from the stack trace in the log like the failure occurred when mapping the results into the HTTP response. This could be a pretty nasty surprise for someone who thinks their batch failed when all the entities were actually created. Is there anyway to not commit the transaction on the server until the response is generated and sent successfully? Here is the stack trace from the log: Sep 1, 2011 11:30:01 PM com.sun.jersey.spi.container.ContainerResponse mapMappableContainerException SEVERE: The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390) at java.lang.StringBuffer.append(StringBuffer.java:219) at org.apache.commons.lang.StringUtils.join(StringUtils.java:3088) at org.apache.commons.lang.StringUtils.join(StringUtils.java:3133) at org.neo4j.server.rest.repr.BatchOperationResults.toJSON(BatchOperationResults.java:99) at org.neo4j.server.rest.web.BatchOperationService.performBatchOperations(BatchOperationService.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:184) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:67) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:276) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:83) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:133) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:71) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1171) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1103) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1053) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1043) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:406) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:477) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:662) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) Sep 1, 2011 11:30:01 PM org.mortbay.log.Slf4jLog warn SEVERE: /db/data/batch java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390) at java.lang.StringBuffer.append(StringBuffer.java:219) at org.apache.commons.lang.StringUtils.join(StringUtils.java:3088) at org.apache.commons.lang.StringUtils.join(StringUtils.java:3133) at org.neo4j.server.rest.repr.BatchOperationResults.toJSON(BatchOperationResults.java:99) at org.neo4j.server.rest.web.BatchOperationService.performBatchOperations(BatchOperationService.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
