Yes. Currently, persisting a set of entities using JDO or JPA persists each entity in the set serially, one after the other, which would explain the long runtime. The low-level datastore API supports batch puts, meaning all of the entities are written in parallel, saving a huge chunk of time. I know that JDO and JPA will support batch puts in the not-too-distant future, but for now, you should see a huge difference using the low-level API. - Jason
On Wed, Aug 19, 2009 at 3:52 PM, PLX <[email protected]> wrote: > > Thanks for reply > > All right, I've got next obstacle because of the differences in > production and development implementations. > I have Node, Product and Attribute models: > > public class Product { > @PrimaryKey > @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY) > private Key id; > > @Persistent(mappedBy = "product") > private Set<Attribute> attributes = new HashSet<Attribute>(); > > private Key nodeKey; > ... > } > > public class Attribute { > @PrimaryKey > @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY) > private Key id; > > @Persistent > private String name; > > @Persistent > private Text value; > > @Persistent > private Product product; > ... > } > > So, when I call pm.makePersistent(product); on my local machine it > takes ~200ms to persist 1 Product and a large(~100 name/values pairs) > collection of related Attributes. But on the production server the > same build runs ~7sec (from 5 to 10sec) > Commenting this line makes response timeout less than 1 sec on > production. That is why I'm sure the delay is in the persisting > operation. > > BTW, is it true that low-level api and regular one(JDO) operate with > datastore in a different way? > I mean, is regular JDO api just a wrapper for low-level datastore api > or they utilize different mechanisms to send data to datastore? > Can I speed up my application by using low-level api? > > Thank you > > On Aug 17, 2:45 pm, "Jason (Google)" <[email protected]> wrote: > > Naturally, there will be some distinctions between the > services/components > > used in the production environment and those included with the local SDK. > > We're working hard to keep this distinction as transparent as possible, > but > > we do not currently have a document listing the implementation > differences > > between the local and production components. That said, if you have any > > specific questions on how the local SDK works in relation to the > production > > environment, please ask. > > - Jason > > > > On Fri, Aug 14, 2009 at 5:09 PM, PLX <[email protected]> wrote: > > > > > I appreciate > > > > > This is very useful information. Can you provide me with a link to > > > dev.server documentation, please? I want to review all differences > > > between production and development servers. > > > > > Thank you > > > > > On Aug 12, 3:26 pm, PLX <[email protected]> wrote: > > > > Hi, > > > > > > I'm trying to understand, how is it possible to insert a large amount > > > > of data to datastore with JDO. But I have a memory leak problem. > > > > > > PersistenceManager pm = null; > > > > for (int i = 0; i < 100; i++) { > > > > try { > > > > pm = PMF.get().getPersistenceManager(); > > > > List<Product> products = new Vector<Product>(); > > > > for (int j = 0; j < 1000; j++) { > > > > Product p = new Product(); > > > > products.add(p); > > > > } > > > > pm.makePersistentAll(products); > > > > products.clear(); > > > > System.out.println("committed " + i); > > > > } catch (Exception e) { > > > > e.printStackTrace(); > > > > } > > > > finally { > > > > pm.close(); > > > > } > > > > > > } > > > > > > I've tried: > > > > - to use the same PM instance all the time without recreating it > after > > > > every 1000 inserts. > > > > - pm.flush(); > > > > - pm.evictAll(); > > > > - pm.makePersistent(p); instead of creating the list and calling > > > > pm.makePersistentAll() for it > > > > > > Nothing helps. After approximately 55000 inserts I get the same: > > > > > > java.lang.OutOfMemoryError: Java heap space > > > > at java.lang.reflect.Method.copy(Method.java:143) > > > > at java.lang.reflect.ReflectAccess.copyMethod > > > > (ReflectAccess.java:118) > > > > at sun.reflect.ReflectionFactory.copyMethod > > > > (ReflectionFactory.java:282) > > > > at java.lang.Class.copyMethods(Class.java:2748) > > > > at java.lang.Class.getMethods(Class.java:1410) > > > > at > > > > > > com.google.appengine.tools.development.ApiProxyLocalImpl.getDispatchMethod > > > > (ApiProxyLocalImpl.java:149) > > > > at > > > > com.google.appengine.tools.development.ApiProxyLocalImpl.makeSyncCall > > > > (ApiProxyLocalImpl.java:88) > > > > at com.google.apphosting.api.ApiProxy.makeSyncCall > > > > (ApiProxy.java:79) > > > > at > > > > com.google.appengine.api.datastore.DatastoreApiHelper.makeSyncCall > > > > (DatastoreApiHelper.java:48) > > > > at com.google.appengine.api.datastore.DatastoreServiceImpl > > > > $2.run(DatastoreServiceImpl.java:169) > > > > at > > > > com.google.appengine.api.datastore.TransactionRunner.runInTransaction > > > > (TransactionRunner.java:30) > > > > at > com.google.appengine.api.datastore.DatastoreServiceImpl.put > > > > (DatastoreServiceImpl.java:157) > > > > at > com.google.appengine.api.datastore.DatastoreServiceImpl.put > > > > (DatastoreServiceImpl.java:137) > > > > at > com.google.appengine.api.datastore.DatastoreServiceImpl.put > > > > (DatastoreServiceImpl.java:129) > > > > at > > > > > > org.datanucleus.store.appengine.RuntimeExceptionWrappingDatastoreService.put > > > > (RuntimeExceptionWrappingDatastoreService.java:92) > > > > at > > > > org.datanucleus.store.appengine.DatastorePersistenceHandler.put > > > > (DatastorePersistenceHandler.java:108) > > > > at > > > > org.datanucleus.store.appengine.DatastorePersistenceHandler.put > > > > (DatastorePersistenceHandler.java:94) > > > > at > > > > > org.datanucleus.store.appengine.DatastorePersistenceHandler.insertObject > > > > (DatastorePersistenceHandler.java:195) > > > > at > > > > org.datanucleus.state.JDOStateManagerImpl.internalMakePersistent > > > > (JDOStateManagerImpl.java:3185) > > > > at org.datanucleus.state.JDOStateManagerImpl.makePersistent > > > > (JDOStateManagerImpl.java:3161) > > > > at org.datanucleus.ObjectManagerImpl.persistObjectInternal > > > > (ObjectManagerImpl.java:1298) > > > > at org.datanucleus.ObjectManagerImpl.persistObject > > > > (ObjectManagerImpl.java:1175) > > > > at > org.datanucleus.jdo.JDOPersistenceManager.jdoMakePersistent > > > > (JDOPersistenceManager.java:669) > > > > at > org.datanucleus.jdo.JDOPersistenceManager.makePersistentAll > > > > (JDOPersistenceManager.java:723) > > > > at test.servlet.RunServlet.doGet(RunServlet.java:46) > > > > at javax.servlet.http.HttpServlet.service(HttpServlet.java: > > > > 693) > > > > at javax.servlet.http.HttpServlet.service(HttpServlet.java: > > > > 806) > > > > at org.mortbay.jetty.servlet.ServletHolder.handle > > > > (ServletHolder.java:487) > > > > at org.mortbay.jetty.servlet.ServletHandler > > > > $CachedChain.doFilter(ServletHandler.java:1093) > > > > at > > > > com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter > > > > (TransactionCleanupFilter.java:43) > > > > at org.mortbay.jetty.servlet.ServletHandler > > > > $CachedChain.doFilter(ServletHandler.java:1084) > > > > at > > > > com.google.appengine.tools.development.StaticFileFilter.doFilter > > > > (StaticFileFilter.java:124) > > > > > > Simply saying: does JDO have something like JPA's session.clear() ? > > > > > > Thanks > > > > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~----------~----~----~----~------~----~------~--~---
