We are unlikely to create a tool that will let you switch from a master/slave application to high replication. A data copy will be required.
As far as master-slave, we will be emphasizing high replication in the future and will be basing our uptime metrics on applications running on high replication. There will be very few cases, if any, where we will be recommending that applications start on master-slave datastore. With regards to keys: I don't believe the String form of a Key contains an application ID, so if you hash this, the migration should work fine. We weren't prepared for your use case, unfortunately, as the migration tool takes a key, understand the app ID, rewrite it, and save it to the new datastore. We can't do this to a computed hash. -- Ikai Lan Developer Programs Engineer, Google App Engine Blogger: http://googleappengine.blogspot.com Reddit: http://www.reddit.com/r/appengine Twitter: http://twitter.com/app_engine On Tue, Feb 22, 2011 at 9:44 PM, johnP <[email protected]> wrote: > There is lots of thought about how to migrate to HR for the promised > increase in reliability. But how do I resolve the following > situation. I have millions of entities with a key_name that is a > compressed hash of 4 other entity key-strings. This ensures that > there are no duplicate records. Also, lots of queries are optimized by > pre-building key_names hashes and performing a get. > > In other words, handling a migration currently appears to require a > significant rewrite of key parts of the application, rebuilding > millions of entities after the transition, high cpu costs, tons of > hackery, headaches, and downtime. > > The question is (and I've seen multiple attempts from various vectors > to ask a similar question): will there be improved tools for > migration? Will these improved tools handle my use-case? Will > Appengine eventually implement an ability to simply switch a domain > from MS to HR? Or simply, Black and White, "John - your application > is stuck in a dead-end part of our system. It's your problem to get > it out". > > Is my assumption that Master-Slave is a dead-end, correct? > > I am hopeful that this question will receive attention because a solid > answer, one way or another, is required for planning our resources and > development. In other words, instead of further optimizing by moving > further in a key_name_hash direction, we'd roll back a bunch of work, > perform the transition, then continue. > > johnP > > > On Feb 1, 10:26 pm, Albert <[email protected]> wrote: > > Thank you! > > > > On Feb 2, 1:30 pm, "Nick Johnson (Google)" <[email protected]> > > wrote: > > > > > > > > > > > > > > > > > Hi Albert, > > > > > On Wed, Feb 2, 2011 at 12:44 PM, Albert <[email protected]> wrote: > > > > Hi! > > > > > > I just migrated to theHigh Replicationdatastore. > > > > > > Some of my entities contain stringified keys of other entities. > > > > > > For example: > > > > > > Entity(db.Model): > > > > owner = db.StringProperty() # the str(Key) of the owner. > > > > > > Owner(db.Model): > > > > name = db.StringProperty() > > > > > > Now that I've migrated to thehigh-replicationdatastore, all the > > > > stringified keys in the Entity Model contain bad keys because the > > > > Owner Entities have new keys now. > > > > > The migration tool rewrites the App ID of keys, but only if they're > stored > > > as keys (or reference properties). This is one reason why you shouldn't > > > store stringified keys in the datastore! > > > > > > 2 questions: > > > > > > 1. Would this have been prevented by using db.ReferenceProperty() > > > > instead? > > > > > Yes. > > > > > > 2. How can I resolve this issue? I need to update all entities to > > > > contain the new keys instead. > > > > > You should probably use the mapreduce library. For your mapper > function, > > > deserialize the key, and construct a new one with the correct App ID > (you > > > can simply use db.Key.from_path to do this, as it will fill in the App > ID > > > for you) then store it again. You might want to move to using > > > ReferenceProperties when you do this. > > > > > -Nick Johnson > > > > > > Thanks! > > > > > > -- > > > > You received this message because you are subscribed to the Google > Groups > > > > "Google App Engine" group. > > > > To post to this group, send email to > [email protected]. > > > > To unsubscribe from this group, send email to > > > > [email protected]<google-appengine%2Bunsubscrib > [email protected]> > > > > . > > > > For more options, visit this group at > > > >http://groups.google.com/group/google-appengine?hl=en. > > > > > -- > > > Nick Johnson, Developer Programs Engineer, App Engine > > > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration > Number: > > > 368047 > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
