Nevermind... Playing around with the options, I got it to work setting: --batch_size=5
On Dec 15, 9:09 pm, stephan <[email protected]> wrote: > Thanks Mathew -- > > The --dump --restoreis exactly what I've been looking for. However... > > While I was able to export some of my entities (mostly those with only > a handful of items), for entities with a LOT of items I'm seeing the > following: > > on the command line from bulkloader.py: > > HTTPError: HTTP Error 500: Internal Server Error > [INFO ] Backing off due to errors: 256.0 seconds > > and in my error logs: > > /remote_api > java.nio.BufferOverflowException > at java.nio.HeapByteBuffer.put(Unknown Source) > at java.nio.ByteBuffer.put(Unknown Source) > at com.google.apphosting.runtime.jetty.RpcResponseGenerator.addContent > (RpcResponseGenerator.java:68) > at org.mortbay.jetty.AbstractGenerator$Output.write > (AbstractGenerator.java:624) > at org.mortbay.jetty.AbstractGenerator$Output.write > (AbstractGenerator.java:585) > at com.google.apphosting.utils.remoteapi.RemoteApiServlet.doPost > (RemoteApiServlet.java:139) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:713) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:806) > at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java: > 487) > at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter > (ServletHandler.java:1093) > at org.purplex.lbas.servlet.filter.ResponseHeaderFilter.doFilter > (ResponseHeaderFilter.java:49) > at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter > (ServletHandler.java:1084) > at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter > (ParseBlobUploadFilter.java:97) > at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter > (ServletHandler.java:1084) > at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter > (SaveSessionFilter.java:35) > at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter > (ServletHandler.java:1084) > at > com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter > (TransactionCleanupFilter.java:43) > at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter > (ServletHandler.java:1084) > at org.mortbay.jetty.servlet.ServletHandler.handle > (ServletHandler.java:360) > at org.mortbay.jetty.security.SecurityHandler.handle > (SecurityHandler.java:216) > at org.mortbay.jetty.servlet.SessionHandler.handle > (SessionHandler.java:181) > at org.mortbay.jetty.handler.ContextHandler.handle > (ContextHandler.java:712) > at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java: > 405) > at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle > (AppVersionHandlerMap.java:238) > at org.mortbay.jetty.handler.HandlerWrapper.handle > (HandlerWrapper.java:139) > at org.mortbay.jetty.Server.handle(Server.java:313) > at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java: > 506) > at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete > (HttpConnection.java:830) > at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable > (RpcRequestParser.java:76) > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:381) > at > com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest > (JettyServletEngineAdapter.java:135) > at com.google.apphosting.runtime.JavaRuntime.handleRequest > (JavaRuntime.java:235) > at com.google.apphosting.base.RuntimePb$EvaluationRuntime > $6.handleBlockingRequest(RuntimePb.java:5235) > at com.google.apphosting.base.RuntimePb$EvaluationRuntime > $6.handleBlockingRequest(RuntimePb.java:5233) > at com.google.net.rpc.impl.BlockingApplicationHandler.handleRequest > (BlockingApplicationHandler.java:24) > at com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java: > 363) > at com.google.net.rpc.impl.Server$2.run(Server.java:838) > at com.google.tracing.LocalTraceSpanRunnable.run > (LocalTraceSpanRunnable.java:56) > at com.google.tracing.LocalTraceSpanBuilder.internalContinueSpan > (LocalTraceSpanBuilder.java:536) > at com.google.net.rpc.impl.Server.startRpc(Server.java:793) > at com.google.net.rpc.impl.Server.processRequest(Server.java:368) > at com.google.net.rpc.impl.ServerConnection.messageReceived > (ServerConnection.java:448) > at com.google.net.rpc.impl.RpcConnection.parseMessages > (RpcConnection.java:319) > at com.google.net.rpc.impl.RpcConnection.dataReceived > (RpcConnection.java:290) > at com.google.net.async.Connection.handleReadEvent(Connection.java: > 466) > at com.google.net.async.EventDispatcher.processNetworkEvents > (EventDispatcher.java:759) > at com.google.net.async.EventDispatcher.internalLoop > (EventDispatcher.java:205) > at com.google.net.async.EventDispatcher.loop(EventDispatcher.java: > 101) > at com.google.net.rpc.RpcService.runUntilServerShutdown > (RpcService.java:251) > at com.google.apphosting.runtime.JavaRuntime$RpcRunnable.run > (JavaRuntime.java:394) > at java.lang.Thread.run(Unknown Source) > > Any ideas? > > ...stephan > > On Nov 18, 2:51 pm, Matthew Blain <[email protected]> wrote: > > > Hi everyone, > > Thanks for the response so far--we'll be getting back to you over the next > > few days. > > I'd like to point out an existing feature which will be helpful to many of > > you: --dump and --restore. This will download or upload all of > > the entities for a particular Kind and save them in a local sqlite database. > > It's useful for general backup (e.g. a weekly backup), and also useful for > > moving data across applications, such as between your app running on the > > dev_appserver and on App Engine, or to load a staging instance with a known > > set of test data. > > > You can find more information > > here:http://code.google.com/appengine/docs/python/tools/uploadingdata.html... > > > Also, for Java developers, there is a remote API handler available; adding > > following to your web xml file should work (disclaimer: I have not yet > > personally tested this.) > > > <servlet> > > <servlet-name>remoteapi</servlet-name> > > > > <servlet-class>com.google.apphosting.utils.remoteapi.RemoteApiServlet</servlet-class> > > </servlet> > > <servlet-mapping> > > <servlet-name>remoteapi</servlet-name> > > <url-pattern>/remote_api</url-pattern> > > </servlet-mapping> > > <security-constraint> > > <web-resource-collection> > > <web-resource-name>remoteapi</web-resource-name> > > <url-pattern>/remote_api</url-pattern> > > </web-resource-collection> > > <auth-constraint> > > <role-name>admin</role-name> > > </auth-constraint> > > </security-constraint> > > > --Matthew > > > On Tue, Nov 17, 2009 at 12:53 PM, Matthew Blain > > <[email protected]>wrote: > > > > Hi App Engine developers, > > > We're working on some improvements and additions to the bulk loader to > > > make it easier to move data between the App Engine Datastore and other > > > data files you may have. If you're doing this right now, we're looking > > > for some beta testers. > > > Two specific issues we're addressing at this time are: > > > * More language-neutral: Java developers, this will help you! > > > * More format-neutral: If you use some format other than CSV, this > > > will help you! > > > > If you're interested, fill out the form at > > > >https://spreadsheets.google.com/viewform?formkey=dC15V2hwczhpZ1VVWFhP... > > > to get started. > > > > --Matthew -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
