Hi Joshua Yep very good suggestion. I will star it
T On Oct 30, 7:31 pm, Joshua Smith <[email protected]> wrote: > Does it strike anyone else as bizarre that we'd have to deal with this above > the API level? Why doesn't db.put work in batches under the covers? > > Should I create an issue for this? > > -Joshua > > On Oct 30, 2010, at 12:34 AM, Robert Kluin wrote: > > > > > > > > > I think Tim is right. It sounds like it is already going to be an > > expensive put operation, why add extra time to serialize twice? > > > Robert > > > On Fri, Oct 29, 2010 at 20:53, 风笑雪 <[email protected]> wrote: > >> You can use model_to_protobuf(): > >>http://code.google.com/intl/en/appengine/docs/python/datastore/functi... > > >> eg: > >> def size(entity): > >> return len(db.model_to_protobuf(entity).Encode()) > > >> But I'm not sure it's exactly the same size calculated by GAE. > > >> ---------- > >> keakon > > >> On Sat, Oct 30, 2010 at 7:28 AM, Tim Hoffman <[email protected]> wrote: > >>> Hi > > >>> repr won't be very accurate. In many cases you will just get <MyModel > >>> hashvalue> which is no where near the size of the entities. > > >>> An expensive will be to serialize the entity with proto buffer as > >>> thats what will happen to them when they are put into the datastore, > >>> this approach is also redundant as it means the system will be doing > >>> it twice. > > >>> I suggest a better strategy might be to have a look at the statistics > >>> in your control and and set a value based on the average entity size. > >>> This might need reviewing on a regular basis, but it will be a lot > >>> quicker. Then put batch sizes in a config file for each entity type. > > >>> Rgds > > >>> Tim > > >>> On Oct 30, 1:53 am, Rafael Sierra <[email protected]> wrote: > >>>> On Fri, Oct 29, 2010 at 3:24 PM, Joshua Smith <[email protected]> > >>>> wrote: > >>>>> I'm running into a too-large exception when I bulk put a bunch of > >>>>> entities. So obviously, I need to break up my puts into batches. I > >>>>> want to do something like this pseudo code: > > >>>>> size = 0 > >>>>> for o in objects: > >>>>> if size + o.size() > 1MB: > >>>>> db.put(list) > >>>>> size = 0 > >>>>> list = [] > >>>>> list.append(o) > > >>>> If you are talking about db entities, you can try repr(o), it's used > >>>> to send objects to cache, I saw somewhere in the documentation: > > >>>> `obj = eval(repr(obj))` > > >>>> So, you can do "if size + len(repr(o)) > 1MB:" (pseudo code) > > >>>>> Any idea what I could use for the "o.size()" method? I could crawl > >>>>> through all the fields and build up an estimate, but it seems likely to > >>>>> me that there is a way to get the API-size of an entity more elegantly. > > >>>>> Thoughts? > > >>>>> -Joshua > > >>>>> -- > >>>>> You received this message because you are subscribed to the Google > >>>>> Groups "Google App Engine" group. > >>>>> To post to this group, send email to [email protected]. > >>>>> To unsubscribe from this group, send email to > >>>>> [email protected]. > >>>>> For more options, visit this group > >>>>> athttp://groups.google.com/group/google-appengine?hl=en. > > >>>> -- > >>>> Rafael Sierrahttp://blog.rafaelsdm.com > > >>> -- > >>> You received this message because you are subscribed to the Google Groups > >>> "Google App Engine" group. > >>> To post to this group, send email to [email protected]. > >>> To unsubscribe from this group, send email to > >>> [email protected]. > >>> For more options, visit this group > >>> athttp://groups.google.com/group/google-appengine?hl=en. > > >> -- > >> You received this message because you are subscribed to the Google Groups > >> "Google App Engine" group. > >> To post to this group, send email to [email protected]. > >> To unsubscribe from this group, send email to > >> [email protected]. > >> For more options, visit this group > >> athttp://groups.google.com/group/google-appengine?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
