http://code.google.com/p/googleappengine/issues/detail?id=3973
On Oct 30, 2010, at 7:49 AM, Tim Hoffman wrote: > Hi Joshua > > Yep very good suggestion. I will star it > > T > > On Oct 30, 7:31 pm, Joshua Smith <[email protected]> wrote: >> Does it strike anyone else as bizarre that we'd have to deal with this above >> the API level? Why doesn't db.put work in batches under the covers? >> >> Should I create an issue for this? >> >> -Joshua >> >> On Oct 30, 2010, at 12:34 AM, Robert Kluin wrote: >> >> >> >> >> >> >> >>> I think Tim is right. It sounds like it is already going to be an >>> expensive put operation, why add extra time to serialize twice? >> >>> Robert >> >>> On Fri, Oct 29, 2010 at 20:53, 风笑雪 <[email protected]> wrote: >>>> You can use model_to_protobuf(): >>>> http://code.google.com/intl/en/appengine/docs/python/datastore/functi... >> >>>> eg: >>>> def size(entity): >>>> return len(db.model_to_protobuf(entity).Encode()) >> >>>> But I'm not sure it's exactly the same size calculated by GAE. >> >>>> ---------- >>>> keakon >> >>>> On Sat, Oct 30, 2010 at 7:28 AM, Tim Hoffman <[email protected]> wrote: >>>>> Hi >> >>>>> repr won't be very accurate. In many cases you will just get <MyModel >>>>> hashvalue> which is no where near the size of the entities. >> >>>>> An expensive will be to serialize the entity with proto buffer as >>>>> thats what will happen to them when they are put into the datastore, >>>>> this approach is also redundant as it means the system will be doing >>>>> it twice. >> >>>>> I suggest a better strategy might be to have a look at the statistics >>>>> in your control and and set a value based on the average entity size. >>>>> This might need reviewing on a regular basis, but it will be a lot >>>>> quicker. Then put batch sizes in a config file for each entity type. >> >>>>> Rgds >> >>>>> Tim >> >>>>> On Oct 30, 1:53 am, Rafael Sierra <[email protected]> wrote: >>>>>> On Fri, Oct 29, 2010 at 3:24 PM, Joshua Smith <[email protected]> >>>>>> wrote: >>>>>>> I'm running into a too-large exception when I bulk put a bunch of >>>>>>> entities. So obviously, I need to break up my puts into batches. I >>>>>>> want to do something like this pseudo code: >> >>>>>>> size = 0 >>>>>>> for o in objects: >>>>>>> if size + o.size() > 1MB: >>>>>>> db.put(list) >>>>>>> size = 0 >>>>>>> list = [] >>>>>>> list.append(o) >> >>>>>> If you are talking about db entities, you can try repr(o), it's used >>>>>> to send objects to cache, I saw somewhere in the documentation: >> >>>>>> `obj = eval(repr(obj))` >> >>>>>> So, you can do "if size + len(repr(o)) > 1MB:" (pseudo code) >> >>>>>>> Any idea what I could use for the "o.size()" method? I could crawl >>>>>>> through all the fields and build up an estimate, but it seems likely to >>>>>>> me that there is a way to get the API-size of an entity more elegantly. >> >>>>>>> Thoughts? >> >>>>>>> -Joshua >> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Google App Engine" group. >>>>>>> To post to this group, send email to [email protected]. >>>>>>> To unsubscribe from this group, send email to >>>>>>> [email protected]. >>>>>>> For more options, visit this group >>>>>>> athttp://groups.google.com/group/google-appengine?hl=en. >> >>>>>> -- >>>>>> Rafael Sierrahttp://blog.rafaelsdm.com >> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "Google App Engine" group. >>>>> To post to this group, send email to [email protected]. >>>>> To unsubscribe from this group, send email to >>>>> [email protected]. >>>>> For more options, visit this group >>>>> athttp://groups.google.com/group/google-appengine?hl=en. >> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Google App Engine" group. >>>> To post to this group, send email to [email protected]. >>>> To unsubscribe from this group, send email to >>>> [email protected]. >>>> For more options, visit this group >>>> athttp://groups.google.com/group/google-appengine?hl=en. >> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group >>> athttp://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
