Thanks again! On Feb 9, 3:46 pm, "Ikai L (Google)" <[email protected]> wrote: > For the time being, not without using task queues. We're looking at > providing a map/reduce interface, which would likely fit your use case. It's > on our roadmap: > > http://code.google.com/appengine/docs/roadmap.html > > > > > > On Tue, Feb 9, 2010 at 3:43 PM, RSN <[email protected]> wrote: > > Ikai, > > > First, I really would like to thank you for your response. I > > appreciate the details and links you gave me on improving datastore > > performance. I work on research, and somehow I need to able to update > > and/or delete significant amounts of data. This is part of a setup > > cost our application has to undergo in order to function properly. Our > > users really do not expect to see an immediate response from the > > application, but we also would like to reduce the amount of delay when > > processing data. I think your links about using cache, batch > > processing, and AJAX operations could help us to alleviate the setup > > cost of our application. Those may be the alternative mechanisms that > > we need in our data intensive operations. > > > Are you aware of the possibility of running computer simulations in > > App Engine? Is it possible? Is there a way to keep running a process > > longer than the timeout deadlines? > > > Thanks for your help. > > > On Feb 9, 3:20 pm, "Ikai L (Google)" <[email protected]> wrote: > > > We're always looking at ways to improve datastore performance, but we > > really > > > try to push developers towards optimizing the performance from the User's > > > perspective. That is - if you have to delete or create 150 persistent, > > > indexed objects, you may want to rethink what problems you are trying to > > > solve. Users have come to expect a certain level of quality of service > > when > > > it comes to certain types of page serves, and you can fake this out by > > using > > > Memcache and task queues for writes. True, the amount of computation > > still > > > does not go down, but there's a net gain here from an improved experience > > > and increased user retention. > > > > I'm not surprised that you're seeing the numbers you're seeing based on > > the > > > data you've provided. To really scale up, think about how to make > > operations > > > fast. There are a few tips that go a long way: > > > > - make as much as you can asynchronous. For some operations this may not > > > make sense > > > - cache heavily. Use a write-through cache when saving and task queues > > for > > > big saves > > > - write to as few indexes and entities as possible if you need to do > > > anything synchronously > > > > This is one of my favorite articles about this subject: > > > >http://highscalability.com/blog/2010/1/22/how-buddypoke-scales-on-fac... > > > > <http://highscalability.com/blog/2010/1/22/how-buddypoke-scales-on-fac.. > > .>BuddyPoke > > > realized early on they weren't working with a relational database and > > made > > > full use of the fact that App Engine's datastore is build on a > > distributed > > > key-value store with custom indexing. > > > > On Mon, Feb 8, 2010 at 11:37 AM, RSN <[email protected]> wrote: > > > > Hi, > > > > > I am kind of new to GAE, and I was wondering if someone has statistics > > > > or data that shows what are the most optimal (efficient) ways for > > > > deleting and creating objects in the Datastore. > > > > > I did some small experiments in order to identify a more efficient > > > > mechanism to delete and create objects in the datastore. In these > > > > tests, I have a single persistent object that contains 8 persistent > > > > fields. > > > > > In the first test, I have a loop of 150 instances, and at each > > > > iteration I create the persistent object, and I call the > > > > makePersistent function to add the single instance to the datastore. > > > > This should take much longer, and it does. It takes on average 16-17 > > > > seconds. > > > > > In the second test, the loop only creates the 150 objects, and then > > > > outside the loop, I call the makePersistentAll function, passing all > > > > my collection of objects. As it is mentioned in the documentation, > > > > this is supposed to be faster, and it is. It takes on average 1-2 > > > > seconds. > > > > > I have experimented only with one way for deleting objects. > > > > Basically, I make a query to retrieve "ONLY" the keys of my > > > > objects. > > > > Query query = pm.newQuery("select objectKey from " + > > > > objectClass.getName()); > > > > keys = (List<Key>) query.execute(); > > > > > Then, once all keys have been retrieved, I call the following service > > > > method: > > > > DatastoreService ds = DatastoreServiceFactory.getDatastoreService(); > > > > ds.delete(keys); > > > > > To delete the objects. It takes on average 2-4 seconds to delete 150 > > > > instances. > > > > > I was just wondering if I am missing something, and if there is a more > > > > efficient way for deleting (or creating) objects in the datastore. I > > > > see also that there is a higher variance in deleting objects than in > > > > creating them. For example, the fastest run in my test took 1.6 > > > > seconds to delete the 150 instances, while the slowest run took almost > > > > 8 seconds. I do not see this variability when I create the objects > > > > (i.e., when I add them to GAE). > > > > > Any advice or suggestions on alternative methods for doing these kinds > > > > of operations in GAE is GREATLY APPRECIATED. Or, If I am in the right > > > > track, please let me know. > > > > > THANKS A LOT. > > > > > -- > > > > You received this message because you are subscribed to the Google > > Groups > > > > "Google App Engine for Java" group. > > > > To post to this group, send email > > > > to>>[email protected]. > > > > To unsubscribe from this group, send email > > > > to>>[email protected]<google-appengine-java%[email protected]>> > > > > > > > > <google-appengine-java%[email protected]<google-appengine-java%[email protected]> > > > > > . > > > > For more options, visit this group at > > > >http://groups.google.com/group/google-appengine-java?hl=en. > > > > -- > > > Ikai Lan > > > Developer Programs Engineer, Google App Enginehttp:// > > googleappengine.blogspot.com|http://twitter.com/app_engine > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine for Java" group. > > To post to this group, send email to>[email protected]. > > To unsubscribe from this group, send email > > to>[email protected]<google-appengine-java%[email protected]> > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine-java?hl=en. > > -- > Ikai Lan > Developer Programs Engineer, Google App > Enginehttp://googleappengine.blogspot.com|http://twitter.com/app_engine
-- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
