> There's no need to use a new model name: You can simply create new entities > to replace the old ones, under the current model name. If you're using key > names, you can construct a new entity with the same values as the old ones, > and store that.
Note the precise wording. You can't just put() the instance that you read from the datastore, the instance that doesn't have the properties that you've deleted, you have to get(), make a new db.Model instance with the same key, populate its properties from the instance that you got, and put the new instance. If you're not using key names, you can't create that new db.Model instance (as of 1.2.5) because you can't create an instance with a specified id. The problem is in db.Model._to_entity() (and maybe db.Expando._to_entity()). If the instance was created from a protocol buffer, put() tries to reuse said protocol buffer, and it still contains values for properties that you've deleted. These values are not deleted by _to_entity() so they end up being sent back to the datastore. I've filed http://code.google.com/p/googleappengine/issues/detail?id=2251 . On Oct 10, 1:29 pm, "Nick Johnson (Google)" <[email protected]> wrote: > On Sat, Oct 10, 2009 at 6:27 PM, Jason Smith > <[email protected]>wrote: > > > > > > > > > Thanks for the help guys. I think this is an important matter to have > > cleared up. > > > It's bedtime here (GMT+7) however tomorrow I think I will do some > > benchmarks along the lines of the example I wrote up in the SO > > question. > > > At this point I would think the safest thing would be to completely > > change the model name, thereby guaranteeing that you will be writing > > entities with fresh keys. However I suspect it's not necessary to go > > that far. I'm thinking that on the production datastore, changing the > > model definition and then re-put()ing the entity will be what's > > required to realize a speed benefit when reducing the number of > > properties on a model. But the facts will speak for themselves. > > There's no need to use a new model name: You can simply create new entities > to replace the old ones, under the current model name. If you're using key > names, you can construct a new entity with the same values as the old ones, > and store that. > > You can also use the low-level API in google.appengine.api.datastore; this > provides a dict-like interface from which you can delete unwanted fields. > > -Nick Johnson > > > > > > > On Oct 11, 12:17 am, Andy Freeman <[email protected]> wrote: > > > > In other words: if I want to reduce the size of my entities, is > > > > it necessary to migrate the old entities to ones with the new > > > > definition? > > > > I'm pretty sure that the answer to that is yes. > > > > > If so, is it sufficient to re-put() the entity, or must I > > > > save under a wholly new key? > > > > I think that it should be sufficient re-put() but decided to test that > > > hypothesis. > > > > It isn't sufficient in the SDK - the SDK admin console continues to > > > show values for properties that you've deleted from the model > > > definition after the re-put(). Yes, I checked to make sure that those > > > properties didn't have values before the re-put(). > > > > I did the get and re-put() in a transaction, namely: > > > > def txn(key): > > > obj = Model.get(key) > > > obj.put() > > > assert db.run_in_transaction(txn, key) > > > > I tried two things to get around this problem. The first was to add > > > db.delete(obj.key()) right before obj.put(). (You can't do obj.delete > > > because that trashes the obj.) > > > > The second was to add "obj.old_property = None" right before the > > > obj.put() (old_property is the name of the property that I deleted > > > from Model's definition.) > > > > Neither one worked. According to the SDK's datastore viewer, existing > > > instances of Model continued to have values for old_property after I > > > updated them with that transaction even with the two changes, together > > > or separately. > > > > If this is also true of the production datastore, this is a big deal. > > > > On Oct 10, 4:44 am, Jason Smith <[email protected]> wrote: > > > > > Hi, group. My app's main cost (in dollars and response time) is in the > > > > db.get([list, of, keys, here]) call in some very high-trafficked code. > > > > I want to pare down the size of that model to the bare minimum with > > > > the hope of reducing the time and CPU fee for this very common > > > > activity. Many users who are experiencing growth in the app popularity > > > > probably have this objective as well. > > > > > I have two questions that hopefully others are thinking about too. > > > > > 1. Can I expect the API time of a db.get() with several hundred keys > > > > to reduce roughly linearly as I reduce the size of the entity? > > > > Currently the entity has the following data attached: 9 String, 9 > > > > Boolean, 8 Integer, 1 GeoPt, 2 DateTime, 1 Text (avg size ~100 bytes > > > > FWIW), 1 Reference, 1 StringList (avg size 500 bytes). The goal is to > > > > move the vast majority of this data to related classes so that the > > > > core fetch of the main model will be quick. > > > > > 2. If I do not change the name of the entity (i.e. just delete all the > > > > db.*Property definitions in the model), will I still incur the same > > > > high cost fetching existing entities? The documentation says that all > > > > properties of a model are fetched simultaneously. Will the old > > > > unneeded properties still transfer over RPC on my dime and while users > > > > wait? In other words: if I want to reduce the size of my entities, is > > > > it necessary to migrate the old entities to ones with the new > > > > definition? If so, is it sufficient to re-put() the entity, or must I > > > > save under a wholly new key? > > > > > Thanks very much to anyone who knows about this matter! > > -- > Nick Johnson, Developer Programs Engineer, App Engine > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: > 368047- Hide quoted text - > > - Show quoted text -- Hide quoted text - > > - Show quoted text - --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
