On Fri, Jun 26, 2009 at 4:42 PM, Andy Freeman <[email protected]> wrote:
> > > the 1MB limit applies only to single API calls > > Does that mean that db.put((e1, e2, e3,)) where all of the entities > are 500kb will fail? Yes. > > > Where are limits on the total size per call documented? > > http://code.google.com/appengine/docs/python/datastore/overview.html#Quotas_and_Limits > only mentions a limit on the size of individual entities and the total > number of entities for batch methods. The batch method documentation > (http://code.google.com/appengine/docs/python/datastore/functions.html > and http://code.google.com/appengine/docs/python/memcache/functions.html) > does not mention any limits. You're right - we need to improve our documentation in that area. The 1MB limit applies to _all_ API calls. > > Is there a documented limit on the number of entities per memcache > call? No. > > > BTW - There is a typo in > http://code.google.com/appengine/docs/python/memcache/overview.html#Quotas_and_Limits > . > It says "In addition to quotas, the following limits apply to the use > of the Mail service:" instead of "Memcache service" Thanks for the heads-up. -Nick Johnson > > > On Jun 26, 7:28 am, "Nick Johnson (Google)" <[email protected]> > wrote: > > Hi tav, > > > > Batch puts aren't transactional unless all the entities are in the > > same entity group. Transactions, however, _are_ transactional, and the > > 1MB limit applies only to single API calls, so you can make multiple > > puts to the same entity group in a transaction. > > > > -Nick Johnson > > > > > > > > > > > > On Fri, Jun 26, 2009 at 8:53 AM, tav<[email protected]> wrote: > > > > > Hey guys and girls, > > > > > I've got a situation where I'd have to "transactionally" update > > > multiple entities which would cumulatively be greater than the 1MB > > > datastore API limit... is there a decent solution for this? > > > > > For example, let's say that I start off with entities E1, E2, E3 which > > > are all about 400kb each. All the entities are specific to a given > > > User. I grab them all on a "remote node" and do some calculations on > > > them to yield new "computed" entities E1', E2', and E3'. > > > > > Any failure of the remote node or the datastore is recoverable except > > > when the remote node tries to *update* the datastore... in that > > > situation, it'd have to batch the update into 2 separate .put() calls > > > to overcome the 1MB limit. And should the remote node die after the > > > first put(), we have a messy situation =) > > > > > My solution at the moment is to: > > > > > 1. Create a UserRecord entity which has a 'version' attribute > > > corresponding to the "latest" versions of the related entities for any > > > given User. > > > > > 2. Add a 'version' attribute to all the entities. > > > > > 3. Whenever the remote node creates the "computed" new set of > > > entities, it creates them all with a new version number -- applying > > > the same version for all the entities in the same "transaction". > > > > > 4. These new entities are actually .put() as totally separate and new > > > entities, i.e. they do not overwrite the old entities. > > > > > 5. Once a remote node successfully writes new versions of all the > > > entities relating to a User, it updates the UserRecord with the latest > > > version number. > > > > > 6. From the remote node, delete all Entities related to a User which > > > don't have the latest version number. > > > > > 7. Have a background thread check and do deletions of invalid versions > > > in case a remote node had died whilst doing step 4, 5 or 6... > > > > > I've skipped out the complications caused by multiple remote nodes > > > working on data relating to the same User -- but, overall, the > > > approach is pretty much the same. > > > > > Now, the advantage of this approach (as far as I can see) is that data > > > relating to a User is never *lost*. That is, data is never lost before > > > there is valid data to replace it. > > > > > However, the disadvantage is that for (unknown) periods of time, there > > > would be duplicate data sets for a given User... All of which is > > > caused by the fact that the datastore calls cannot exceed 1MB. =( > > > > > So queries will yield duplicate data -- gah!! > > > > > Is there a better approach to try at all? Thanks! > > > > > -- > > > love, tav > > > > > plex:espians/tav | [email protected] | +44 (0) 7809 569 369 > > >http://tav.espians.com|http://twitter.com/tav|<http://twitter.com/tav%7C>skype:tavespian > > > > -- > > Nick Johnson, App Engine Developer Programs Engineer > > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration > > Number: 368047- Hide quoted text - > > > > - Show quoted text - > > > -- Nick Johnson, App Engine Developer Programs Engineer Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
