Hi i posed the question to optimize this tool i was working on http://github.com/matrixownsyou/MultiTaskBob
it "works" by serially segmenting records in buckets of X records, by the keys: segemnt1, startkey1, endkey2 segemnt2, startkey3, endkey4 <....> this should be fast since it only selects keys and it uses the memcached cursor for the result set, but getting the last key of a fetch() is taking some time : http://github.com/matrixownsyou/MultiTaskBob/blob/master/MultiTaskBob/base.py#L188 the idea is that after the records are segmented, each segment (and its contained records) can be processed concurrently, for example in an htttp threading tool. the reason i didnt use something like httpmr was because the data model (wich needed to be updated due to new properties ) didnt had an unique property other than the key or anything else i tought manageable to map upon. 2010/4/6 Wesley C (Google) <[email protected]>: > agreeing with ryan here that you'll need to do a pair of questions, > regardless of whether you do it via a Query or GqlQuery. > > however, a better question would be: what are you trying to do, and > why do you need the first and last keys? non-relational datastores are > generally not geared up for doing counting (thus fetching the "last > key"). perhaps there is another way to accomplish your task without > doing exactly what you've described. > > for example, an alternative to querying is to create another type of > entity in the datastore that stores the first key and the (current) > last key, and update that data structure as you continue to add more > applications entities into your datastore. this way you'll have a much > shorter query to perform (only one object that contains exactly what > you need). > > just curious... > -- wesley > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > "Core Python Programming", Prentice Hall, (c)2007,2001 > "Python Fundamentals", Prentice Hall, (c)2009 > http://corepython.com > > wesley.j.chun :: wesc+api-at-google-dot-com > developer relations :: google app engine > @app_engine :: googleappengine.blogspot.com > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- [email protected] http://beta.icodedhere.com http://pt.linkedin.com/in/josemoreira http://djangopeople.net/josemoreira http://www.crunchbase.com/user/josemoreira -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
