A couple thoughts. Maybe the GAE team should borrow the idea of spot prices from Amazon. That's a great way to have lower-priority jobs that can run when there are instances available. We set the price we're willing to pay, if the spot cost drops below that, we get the resources. It creates a market where more urgent jobs get done sooner and Google makes better use of quiet periods.
On your issue: Do you need to update every entity when you do this? How many items on the listproperty need to be changed? Could you tell us a bit more of what the data looks like? I'm thinking that 14 million entities x 18 items each is the amount of entries you really have, each distributed across at least 3 servers and then indexed. That seems like a lot of writes if you're re-writing everything. It's likely a bad idea to rely on an infrastructure change to fix this (recurring) issue, but there is hopefully a way to reduce the amount of writes you have to do. Also, could you maybe run your mapreduce on smaller sets of the data to spread it out over multiple days and avoid adding too many instances? Has anyone done anything like this? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/bQiOhsYFmv4J. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
