How about buffering the index changes, so only one index update needs to be written?
On Jan 5, 2:58 pm, "Ikai Lan (Google)" <[email protected]> wrote: > Brian (apologies if that is not your name), > > How much of the costs are instance hours versus datastore writes? There's > probably something going on here. The largest costs are to update indexes, > not entities. Assuming $6500 is the cost of datastore writes alone, that > breaks down to: > > ~$0.0004 a write > > Pricing is $0.10 per 100k operations, so that means using this equation: > > (6500.00 / 14000000) / (0.10 / 100000) > > You're doing about 464 write operations per put, which roughly translates > to 6.5 billion writes. > > I'm trying to extrapolate what you are doing, and it sounds like you are > doing full text indexing or something similar ... and having to update all > the indexes. When you update a property, it takes a certain amount of > writes. Assuming you are changing String properties, each property you > update takes this many writes: > > - 2 indexes deleted (ascending and descending) > - 2 indexes update (ascending and descending) > > So if you were only updating all the list properties, that means you are > updating 100 list properties. > > Given that this is a regular thing you need to do, perhaps there is an > engineering solution for what you are trying to do that will be more cost > effective. Can you describe why you're running this job? What features does > this support in your product? > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > plus.ikailan.com | twitter.com/ikai > > > > > > > > On Thu, Jan 5, 2012 at 10:08 AM, Petey <[email protected]> wrote: > > In this one case we had to change all of the items in the > > listproperty. In our most common case we might have to add and delete > > a couple items to the list property every once in a while. That would > > still cost us well over $1,000 each time. > > > Most of the reasons for this type of data in our product is to > > compensate for the fact that there isn't full text search yet. I know > > they are beta testing full text, but I'm still worried that that also > > might be too expensive per write. > > > On Jan 5, 6:54 am, Richard Watson <[email protected]> wrote: > > > A couple thoughts. > > > > Maybe the GAE team should borrow the idea of spot prices from Amazon. > > > That's a great way to have lower-priority jobs that can run when there > > are > > > instances available. We set the price we're willing to pay, if the spot > > > cost drops below that, we get the resources. It creates a market where > > more > > > urgent jobs get done sooner and Google makes better use of quiet periods. > > > > On your issue: > > > Do you need to update every entity when you do this? How many items on > > the > > > listproperty need to be changed? Could you tell us a bit more of what the > > > data looks like? > > > > I'm thinking that 14 million entities x 18 items each is the amount of > > > entries you really have, each distributed across at least 3 servers and > > > then indexed. That seems like a lot of writes if you're re-writing > > > everything. It's likely a bad idea to rely on an infrastructure change > > to > > > fix this (recurring) issue, but there is hopefully a way to reduce the > > > amount of writes you have to do. > > > > Also, could you maybe run your mapreduce on smaller sets of the data to > > > spread it out over multiple days and avoid adding too many instances? Has > > > anyone done anything like this? > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
