For the record, I regularly pull 100 entities, that are much smaller than yours, for my application in order to page through them. Basically to meet a paging requirement I pull 100, cache that result, then page within it. I do think they size of your entities are part of the problem
Though, I did find with the same models I pull 100 of for viewing, I had to drop down to 75 for deletion. Even pulling 75 to delete I tend to run into high CPU and datastore timeouts on many requests. Overall, bulk management within the datastore is very difficult. On Jan 9, 1:22 pm, Tzakie <[email protected]> wrote: > > The datastore is > > designed to maintain its performance characteristics over very large data > > sets and heavy traffic, so performance with small data sets and low traffic > > may not always compare to a single-server SQL database under similar > > conditions. in the same sense that it won't compare to a RAM cache of the > > same data. > > I not expecting it to work the same as sql. But I do expect to build a > list > of a couple hundred records. I'm not crazy this is a totally > reasonable thing > for an app to do. Sorry guys the serial return speed of these entity > loaded > queries is an issue. It's not me it's you. And it's just a couple > changes away > from working. Can't change it myself so I'm at your mercy google. > > Can we allow your libraries to spawn threads? SimpleDB solves this by > allowing you to bang it with many threads at once. Even though the > fetch > is slow for the entities since you can do so many in parallel it works > out. > > That way you could: > > Keys=QueryObject.getKeys() > NeededEntities=[] > for Key in Keys: > #if key is not in memcache add to NeededEntities > > DictonaryOfReturnedEntities=Model.fetchMassEntities > (NeededEntities,Threads=30) > > Now I want .fetchMassEntities(NeededEntities,Threads=30) to kick 30 > threads off > and get my entities. Which I will cache. This should be totally > workable for lists > up to 1,000 I would think. It's not a sql vs BigTable think it's a > architecture issue > with the current setup. > > > Part of scaling data is > > determining how quickly updates need to be available, so you can > > pre-calculate and cache results to optimize queries, i.e. make data that > > doesn't need to be instantly dynamic less dynamic. > > If I make a degenerate table model just the info I want to query and > the > key stored is it not just the same number of entity fetches right? I > am I getting > punished by the number or returned entities? or the number of > properties > returned? > > > Incidentally, returning just keys for a query has been on our radar for a > > while, we just haven't gotten to it. I can't promise anything, but it's on > > our list. Feel free to file a feature request in the issue tracker to > > promote it. > > Will do. Thanks for your time Dan. :) --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
