I think your problem is similar to the mine.
http://groups.google.com/group/google-appengine-java/browse_thread/thread/1ace5bd8658d89d/a62d0b3f2b3c4e74#a62d0b3f2b3c4e74
Ikai, please, can explain us how many cost in terms of write ops, should us
expect for updating indexed list property adding X items to the list?
For example
Modeling (Objectify annotations)
@Entity
class RelationIndex () {
@Parent
Key<User> ownerKey;
@Indexed
List<Key> receiverKeyList;
}
Define
X = nº New items for add to the list.
Y = nº Entities to update (same entity group), 1 list property indexed per
entity
Z = nº Items before updating list properties.
Magic calculator
Total write ops = Y * ????
2012/1/5 Ikai Lan (Google) <[email protected]>
> Brian (apologies if that is not your name),
>
> How much of the costs are instance hours versus datastore writes? There's
> probably something going on here. The largest costs are to update indexes,
> not entities. Assuming $6500 is the cost of datastore writes alone, that
> breaks down to:
>
> ~$0.0004 a write
>
> Pricing is $0.10 per 100k operations, so that means using this equation:
>
> (6500.00 / 14000000) / (0.10 / 100000)
>
> You're doing about 464 write operations per put, which roughly translates
> to 6.5 billion writes.
>
> I'm trying to extrapolate what you are doing, and it sounds like you are
> doing full text indexing or something similar ... and having to update all
> the indexes. When you update a property, it takes a certain amount of
> writes. Assuming you are changing String properties, each property you
> update takes this many writes:
>
> - 2 indexes deleted (ascending and descending)
> - 2 indexes update (ascending and descending)
>
> So if you were only updating all the list properties, that means you are
> updating 100 list properties.
>
> Given that this is a regular thing you need to do, perhaps there is an
> engineering solution for what you are trying to do that will be more cost
> effective. Can you describe why you're running this job? What features does
> this support in your product?
>
> --
> Ikai Lan
> Developer Programs Engineer, Google App Engine
> plus.ikailan.com | twitter.com/ikai
>
>
>
> On Thu, Jan 5, 2012 at 10:08 AM, Petey <[email protected]> wrote:
>
>> In this one case we had to change all of the items in the
>> listproperty. In our most common case we might have to add and delete
>> a couple items to the list property every once in a while. That would
>> still cost us well over $1,000 each time.
>>
>> Most of the reasons for this type of data in our product is to
>> compensate for the fact that there isn't full text search yet. I know
>> they are beta testing full text, but I'm still worried that that also
>> might be too expensive per write.
>>
>> On Jan 5, 6:54 am, Richard Watson <[email protected]> wrote:
>> > A couple thoughts.
>> >
>> > Maybe the GAE team should borrow the idea of spot prices from Amazon.
>> > That's a great way to have lower-priority jobs that can run when there
>> are
>> > instances available. We set the price we're willing to pay, if the spot
>> > cost drops below that, we get the resources. It creates a market where
>> more
>> > urgent jobs get done sooner and Google makes better use of quiet
>> periods.
>> >
>> > On your issue:
>> > Do you need to update every entity when you do this? How many items on
>> the
>> > listproperty need to be changed? Could you tell us a bit more of what
>> the
>> > data looks like?
>> >
>> > I'm thinking that 14 million entities x 18 items each is the amount of
>> > entries you really have, each distributed across at least 3 servers and
>> > then indexed. That seems like a lot of writes if you're re-writing
>> > everything. It's likely a bad idea to rely on an infrastructure change
>> to
>> > fix this (recurring) issue, but there is hopefully a way to reduce the
>> > amount of writes you have to do.
>> >
>> > Also, could you maybe run your mapreduce on smaller sets of the data to
>> > spread it out over multiple days and avoid adding too many instances?
>> Has
>> > anyone done anything like this?
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to [email protected].
>> To unsubscribe from this group, send email to
>> [email protected].
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
