We're always looking at ways to improve datastore performance, but we really try to push developers towards optimizing the performance from the User's perspective. That is - if you have to delete or create 150 persistent, indexed objects, you may want to rethink what problems you are trying to solve. Users have come to expect a certain level of quality of service when it comes to certain types of page serves, and you can fake this out by using Memcache and task queues for writes. True, the amount of computation still does not go down, but there's a net gain here from an improved experience and increased user retention.
I'm not surprised that you're seeing the numbers you're seeing based on the data you've provided. To really scale up, think about how to make operations fast. There are a few tips that go a long way: - make as much as you can asynchronous. For some operations this may not make sense - cache heavily. Use a write-through cache when saving and task queues for big saves - write to as few indexes and entities as possible if you need to do anything synchronously This is one of my favorite articles about this subject: http://highscalability.com/blog/2010/1/22/how-buddypoke-scales-on-facebook-using-google-app-engine.html <http://highscalability.com/blog/2010/1/22/how-buddypoke-scales-on-facebook-using-google-app-engine.html>BuddyPoke realized early on they weren't working with a relational database and made full use of the fact that App Engine's datastore is build on a distributed key-value store with custom indexing. On Mon, Feb 8, 2010 at 11:37 AM, RSN <[email protected]> wrote: > Hi, > > I am kind of new to GAE, and I was wondering if someone has statistics > or data that shows what are the most optimal (efficient) ways for > deleting and creating objects in the Datastore. > > I did some small experiments in order to identify a more efficient > mechanism to delete and create objects in the datastore. In these > tests, I have a single persistent object that contains 8 persistent > fields. > > In the first test, I have a loop of 150 instances, and at each > iteration I create the persistent object, and I call the > makePersistent function to add the single instance to the datastore. > This should take much longer, and it does. It takes on average 16-17 > seconds. > > In the second test, the loop only creates the 150 objects, and then > outside the loop, I call the makePersistentAll function, passing all > my collection of objects. As it is mentioned in the documentation, > this is supposed to be faster, and it is. It takes on average 1-2 > seconds. > > I have experimented only with one way for deleting objects. > Basically, I make a query to retrieve "ONLY" the keys of my > objects. > Query query = pm.newQuery("select objectKey from " + > objectClass.getName()); > keys = (List<Key>) query.execute(); > > Then, once all keys have been retrieved, I call the following service > method: > DatastoreService ds = DatastoreServiceFactory.getDatastoreService(); > ds.delete(keys); > > To delete the objects. It takes on average 2-4 seconds to delete 150 > instances. > > I was just wondering if I am missing something, and if there is a more > efficient way for deleting (or creating) objects in the datastore. I > see also that there is a higher variance in deleting objects than in > creating them. For example, the fastest run in my test took 1.6 > seconds to delete the 150 instances, while the slowest run took almost > 8 seconds. I do not see this variability when I create the objects > (i.e., when I add them to GAE). > > Any advice or suggestions on alternative methods for doing these kinds > of operations in GAE is GREATLY APPRECIATED. Or, If I am in the right > track, please let me know. > > THANKS A LOT. > > > > > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine-java%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- Ikai Lan Developer Programs Engineer, Google App Engine http://googleappengine.blogspot.com | http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
