Does it strike anyone else as bizarre that we'd have to deal with this above the API level? Why doesn't db.put work in batches under the covers?
Should I create an issue for this? -Joshua On Oct 30, 2010, at 12:34 AM, Robert Kluin wrote: > I think Tim is right. It sounds like it is already going to be an > expensive put operation, why add extra time to serialize twice? > > > Robert > > > > > > > On Fri, Oct 29, 2010 at 20:53, 风笑雪 <[email protected]> wrote: >> You can use model_to_protobuf(): >> http://code.google.com/intl/en/appengine/docs/python/datastore/functions.html >> >> eg: >> def size(entity): >> return len(db.model_to_protobuf(entity).Encode()) >> >> But I'm not sure it's exactly the same size calculated by GAE. >> >> ---------- >> keakon >> >> >> >> On Sat, Oct 30, 2010 at 7:28 AM, Tim Hoffman <[email protected]> wrote: >>> Hi >>> >>> repr won't be very accurate. In many cases you will just get <MyModel >>> hashvalue> which is no where near the size of the entities. >>> >>> An expensive will be to serialize the entity with proto buffer as >>> thats what will happen to them when they are put into the datastore, >>> this approach is also redundant as it means the system will be doing >>> it twice. >>> >>> I suggest a better strategy might be to have a look at the statistics >>> in your control and and set a value based on the average entity size. >>> This might need reviewing on a regular basis, but it will be a lot >>> quicker. Then put batch sizes in a config file for each entity type. >>> >>> Rgds >>> >>> Tim >>> >>> On Oct 30, 1:53 am, Rafael Sierra <[email protected]> wrote: >>>> On Fri, Oct 29, 2010 at 3:24 PM, Joshua Smith <[email protected]> >>>> wrote: >>>>> I'm running into a too-large exception when I bulk put a bunch of >>>>> entities. So obviously, I need to break up my puts into batches. I want >>>>> to do something like this pseudo code: >>>> >>>>> size = 0 >>>>> for o in objects: >>>>> if size + o.size() > 1MB: >>>>> db.put(list) >>>>> size = 0 >>>>> list = [] >>>>> list.append(o) >>>> >>>> If you are talking about db entities, you can try repr(o), it's used >>>> to send objects to cache, I saw somewhere in the documentation: >>>> >>>> `obj = eval(repr(obj))` >>>> >>>> So, you can do "if size + len(repr(o)) > 1MB:" (pseudo code) >>>> >>>> >>>> >>>>> Any idea what I could use for the "o.size()" method? I could crawl >>>>> through all the fields and build up an estimate, but it seems likely to >>>>> me that there is a way to get the API-size of an entity more elegantly. >>>> >>>>> Thoughts? >>>> >>>>> -Joshua >>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "Google App Engine" group. >>>>> To post to this group, send email to [email protected]. >>>>> To unsubscribe from this group, send email to >>>>> [email protected]. >>>>> For more options, visit this group >>>>> athttp://groups.google.com/group/google-appengine?hl=en. >>>> >>>> -- >>>> Rafael Sierrahttp://blog.rafaelsdm.com >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
