For the time being, not without using task queues. We're looking at providing a map/reduce interface, which would likely fit your use case. It's on our roadmap:
http://code.google.com/appengine/docs/roadmap.html On Tue, Feb 9, 2010 at 3:43 PM, RSN <[email protected]> wrote: > Ikai, > > First, I really would like to thank you for your response. I > appreciate the details and links you gave me on improving datastore > performance. I work on research, and somehow I need to able to update > and/or delete significant amounts of data. This is part of a setup > cost our application has to undergo in order to function properly. Our > users really do not expect to see an immediate response from the > application, but we also would like to reduce the amount of delay when > processing data. I think your links about using cache, batch > processing, and AJAX operations could help us to alleviate the setup > cost of our application. Those may be the alternative mechanisms that > we need in our data intensive operations. > > Are you aware of the possibility of running computer simulations in > App Engine? Is it possible? Is there a way to keep running a process > longer than the timeout deadlines? > > Thanks for your help. > > > > > > On Feb 9, 3:20 pm, "Ikai L (Google)" <[email protected]> wrote: > > We're always looking at ways to improve datastore performance, but we > really > > try to push developers towards optimizing the performance from the User's > > perspective. That is - if you have to delete or create 150 persistent, > > indexed objects, you may want to rethink what problems you are trying to > > solve. Users have come to expect a certain level of quality of service > when > > it comes to certain types of page serves, and you can fake this out by > using > > Memcache and task queues for writes. True, the amount of computation > still > > does not go down, but there's a net gain here from an improved experience > > and increased user retention. > > > > I'm not surprised that you're seeing the numbers you're seeing based on > the > > data you've provided. To really scale up, think about how to make > operations > > fast. There are a few tips that go a long way: > > > > - make as much as you can asynchronous. For some operations this may not > > make sense > > - cache heavily. Use a write-through cache when saving and task queues > for > > big saves > > - write to as few indexes and entities as possible if you need to do > > anything synchronously > > > > This is one of my favorite articles about this subject: > > > > http://highscalability.com/blog/2010/1/22/how-buddypoke-scales-on-fac... > > > > <http://highscalability.com/blog/2010/1/22/how-buddypoke-scales-on-fac.. > .>BuddyPoke > > realized early on they weren't working with a relational database and > made > > full use of the fact that App Engine's datastore is build on a > distributed > > key-value store with custom indexing. > > > > > > > > > > > > On Mon, Feb 8, 2010 at 11:37 AM, RSN <[email protected]> wrote: > > > Hi, > > > > > I am kind of new to GAE, and I was wondering if someone has statistics > > > or data that shows what are the most optimal (efficient) ways for > > > deleting and creating objects in the Datastore. > > > > > I did some small experiments in order to identify a more efficient > > > mechanism to delete and create objects in the datastore. In these > > > tests, I have a single persistent object that contains 8 persistent > > > fields. > > > > > In the first test, I have a loop of 150 instances, and at each > > > iteration I create the persistent object, and I call the > > > makePersistent function to add the single instance to the datastore. > > > This should take much longer, and it does. It takes on average 16-17 > > > seconds. > > > > > In the second test, the loop only creates the 150 objects, and then > > > outside the loop, I call the makePersistentAll function, passing all > > > my collection of objects. As it is mentioned in the documentation, > > > this is supposed to be faster, and it is. It takes on average 1-2 > > > seconds. > > > > > I have experimented only with one way for deleting objects. > > > Basically, I make a query to retrieve "ONLY" the keys of my > > > objects. > > > Query query = pm.newQuery("select objectKey from " + > > > objectClass.getName()); > > > keys = (List<Key>) query.execute(); > > > > > Then, once all keys have been retrieved, I call the following service > > > method: > > > DatastoreService ds = DatastoreServiceFactory.getDatastoreService(); > > > ds.delete(keys); > > > > > To delete the objects. It takes on average 2-4 seconds to delete 150 > > > instances. > > > > > I was just wondering if I am missing something, and if there is a more > > > efficient way for deleting (or creating) objects in the datastore. I > > > see also that there is a higher variance in deleting objects than in > > > creating them. For example, the fastest run in my test took 1.6 > > > seconds to delete the 150 instances, while the slowest run took almost > > > 8 seconds. I do not see this variability when I create the objects > > > (i.e., when I add them to GAE). > > > > > Any advice or suggestions on alternative methods for doing these kinds > > > of operations in GAE is GREATLY APPRECIATED. Or, If I am in the right > > > track, please let me know. > > > > > THANKS A LOT. > > > > > -- > > > You received this message because you are subscribed to the Google > Groups > > > "Google App Engine for Java" group. > > > To post to this group, send email to> > [email protected]. > > > To unsubscribe from this group, send email to> > [email protected]<google-appengine-java%[email protected]> > <google-appengine-java%[email protected]<google-appengine-java%[email protected]> > > > > > . > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine-java?hl=en. > > > > -- > > Ikai Lan > > Developer Programs Engineer, Google App Enginehttp:// > googleappengine.blogspot.com|http://twitter.com/app_engine > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine-java%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- Ikai Lan Developer Programs Engineer, Google App Engine http://googleappengine.blogspot.com | http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
