There is lots of thought about how to migrate to HR for the promised increase in reliability. But how do I resolve the following situation. I have millions of entities with a key_name that is a compressed hash of 4 other entity key-strings. This ensures that there are no duplicate records. Also, lots of queries are optimized by pre-building key_names hashes and performing a get.
In other words, handling a migration currently appears to require a significant rewrite of key parts of the application, rebuilding millions of entities after the transition, high cpu costs, tons of hackery, headaches, and downtime. The question is (and I've seen multiple attempts from various vectors to ask a similar question): will there be improved tools for migration? Will these improved tools handle my use-case? Will Appengine eventually implement an ability to simply switch a domain from MS to HR? Or simply, Black and White, "John - your application is stuck in a dead-end part of our system. It's your problem to get it out". Is my assumption that Master-Slave is a dead-end, correct? I am hopeful that this question will receive attention because a solid answer, one way or another, is required for planning our resources and development. In other words, instead of further optimizing by moving further in a key_name_hash direction, we'd roll back a bunch of work, perform the transition, then continue. johnP On Feb 1, 10:26 pm, Albert <[email protected]> wrote: > Thank you! > > On Feb 2, 1:30 pm, "Nick Johnson (Google)" <[email protected]> > wrote: > > > > > > > > > Hi Albert, > > > On Wed, Feb 2, 2011 at 12:44 PM, Albert <[email protected]> wrote: > > > Hi! > > > > I just migrated to theHigh Replicationdatastore. > > > > Some of my entities contain stringified keys of other entities. > > > > For example: > > > > Entity(db.Model): > > > owner = db.StringProperty() # the str(Key) of the owner. > > > > Owner(db.Model): > > > name = db.StringProperty() > > > > Now that I've migrated to thehigh-replicationdatastore, all the > > > stringified keys in the Entity Model contain bad keys because the > > > Owner Entities have new keys now. > > > The migration tool rewrites the App ID of keys, but only if they're stored > > as keys (or reference properties). This is one reason why you shouldn't > > store stringified keys in the datastore! > > > > 2 questions: > > > > 1. Would this have been prevented by using db.ReferenceProperty() > > > instead? > > > Yes. > > > > 2. How can I resolve this issue? I need to update all entities to > > > contain the new keys instead. > > > You should probably use the mapreduce library. For your mapper function, > > deserialize the key, and construct a new one with the correct App ID (you > > can simply use db.Key.from_path to do this, as it will fill in the App ID > > for you) then store it again. You might want to move to using > > ReferenceProperties when you do this. > > > -Nick Johnson > > > > Thanks! > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "Google App Engine" group. > > > To post to this group, send email to [email protected]. > > > To unsubscribe from this group, send email to > > > [email protected]<google-appengine%2Bunsubscrib > > > [email protected]> > > > . > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine?hl=en. > > > -- > > Nick Johnson, Developer Programs Engineer, App Engine > > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: > > 368047 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
