On Fri, Jan 6, 2012 at 8:00 AM, Amy Unruh <[email protected]> wrote:
> Iván, > > 2012/1/6 Iván Rodríguez <[email protected]> > >> I think your problem is similar to the mine. >> >> >> http://groups.google.com/group/google-appengine-java/browse_thread/thread/1ace5bd8658d89d/a62d0b3f2b3c4e74#a62d0b3f2b3c4e74 >> >> Ikai, please, can explain us how many cost in terms of write ops, should >> us expect for updating indexed list property adding X items to the list? > > > This page can help you work out the costs for your particular entities and > indexes: > > http://code.google.com/appengine/docs/billing.html#Billable_Resource_Unit_Cost > E.g., it details the costs for the different datastore operations given an > entity's properties and indexes. > See this as well: http://code.google.com/appengine/docs/python/datastore/entities.html#Understanding_Write_Costs, which discusses multi-value properties. These can lead to expensive indexes. -Amy > > > > >> >> For example >> >> Modeling (Objectify annotations) >> >> @Entity >> class RelationIndex () { >> @Parent >> Key<User> ownerKey; >> @Indexed >> List<Key> receiverKeyList; >> } >> >> Define >> >> X = nº New items for add to the list. >> Y = nº Entities to update (same entity group), 1 list property indexed >> per entity >> Z = nº Items before updating list properties. >> >> >> Magic calculator >> >> Total write ops = Y * ???? >> >> >> >> >> 2012/1/5 Ikai Lan (Google) <[email protected]> >> >> Brian (apologies if that is not your name), >>> >>> How much of the costs are instance hours versus datastore writes? >>> There's probably something going on here. The largest costs are to update >>> indexes, not entities. Assuming $6500 is the cost of datastore writes >>> alone, that breaks down to: >>> >>> ~$0.0004 a write >>> >>> Pricing is $0.10 per 100k operations, so that means using this equation: >>> >>> (6500.00 / 14000000) / (0.10 / 100000) >>> >>> You're doing about 464 write operations per put, which roughly >>> translates to 6.5 billion writes. >>> >>> I'm trying to extrapolate what you are doing, and it sounds like you are >>> doing full text indexing or something similar ... and having to update all >>> the indexes. When you update a property, it takes a certain amount of >>> writes. Assuming you are changing String properties, each property you >>> update takes this many writes: >>> >>> - 2 indexes deleted (ascending and descending) >>> - 2 indexes update (ascending and descending) >>> >>> So if you were only updating all the list properties, that means you are >>> updating 100 list properties. >>> >>> Given that this is a regular thing you need to do, perhaps there is an >>> engineering solution for what you are trying to do that will be more cost >>> effective. Can you describe why you're running this job? What features does >>> this support in your product? >>> >>> -- >>> Ikai Lan >>> Developer Programs Engineer, Google App Engine >>> plus.ikailan.com | twitter.com/ikai >>> >>> >>> >>> On Thu, Jan 5, 2012 at 10:08 AM, Petey <[email protected]> wrote: >>> >>>> In this one case we had to change all of the items in the >>>> listproperty. In our most common case we might have to add and delete >>>> a couple items to the list property every once in a while. That would >>>> still cost us well over $1,000 each time. >>>> >>>> Most of the reasons for this type of data in our product is to >>>> compensate for the fact that there isn't full text search yet. I know >>>> they are beta testing full text, but I'm still worried that that also >>>> might be too expensive per write. >>>> >>>> On Jan 5, 6:54 am, Richard Watson <[email protected]> wrote: >>>> > A couple thoughts. >>>> > >>>> > Maybe the GAE team should borrow the idea of spot prices from Amazon. >>>> > That's a great way to have lower-priority jobs that can run when >>>> there are >>>> > instances available. We set the price we're willing to pay, if the >>>> spot >>>> > cost drops below that, we get the resources. It creates a market >>>> where more >>>> > urgent jobs get done sooner and Google makes better use of quiet >>>> periods. >>>> > >>>> > On your issue: >>>> > Do you need to update every entity when you do this? How many items >>>> on the >>>> > listproperty need to be changed? Could you tell us a bit more of what >>>> the >>>> > data looks like? >>>> > >>>> > I'm thinking that 14 million entities x 18 items each is the amount of >>>> > entries you really have, each distributed across at least 3 servers >>>> and >>>> > then indexed. That seems like a lot of writes if you're re-writing >>>> > everything. It's likely a bad idea to rely on an infrastructure >>>> change to >>>> > fix this (recurring) issue, but there is hopefully a way to reduce the >>>> > amount of writes you have to do. >>>> > >>>> > Also, could you maybe run your mapreduce on smaller sets of the data >>>> to >>>> > spread it out over multiple days and avoid adding too many instances? >>>> Has >>>> > anyone done anything like this? >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Google App Engine" group. >>>> To post to this group, send email to [email protected]. >>>> To unsubscribe from this group, send email to >>>> [email protected]. >>>> For more options, visit this group at >>>> http://groups.google.com/group/google-appengine?hl=en. >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
