Hi Richard Arrano,
You might want to check out Brett Slatikin's talk on High-throughput
Data Pipelines with App Engine from IO 2010. He's covers some
techniques for batching writes to an entity, which is a really good
match for what you need.
http://www.google.com/events/io/2010/sessions/high-throughput-data-pipelines-appengine.html
Robert
On Fri, Jan 14, 2011 at 20:05, Richard Arrano <[email protected]> wrote:
> Thanks for the response! I was thinking it over and I have a question
> - so if a timestamp with its monotonically increasing index causes a
> performance hit at a high write rate, would updating the high bid do
> so as well ? The high bid itself will be monotonically increasing - it
> will never go down, but perhaps I misunderstood something about how
> indices work.
>
> And I guess a broader question that I have is(and I assume the answer
> is yes) - does the size of what we're writing affect the performance?
> As in, writing a simple update to an integer property as opposed to
> creating new complex objects and writing them.
>
> Thanks,
> Richard
>
> On Jan 14, 10:05 am, "Ikai Lan (Google)" <[email protected]>
> wrote:
>> You can certainly write to Memcache, but I don't think your application can
>> tolerate any kind of volatility. Persistence is the price you have to pay.
>> Fortunately, I think this can be done pretty cheaply. Just be aware of
>> monotonically increasing indexes like timestamps: if you have an application
>> with a high write rate that has a timestamp, this will cause the persistence
>> unit storing the indexes to be unable to be autosplit easily across multiple
>> hardware instances and you will take a performance hit. The solution here
>> is, again, to shard the timestamp by prefixing a value to distribute the
>> writes.
>>
>> --
>> Ikai Lan
>> Developer Programs Engineer, Google App Engine
>> Blogger:http://googleappengine.blogspot.com
>> Reddit:http://www.reddit.com/r/appengine
>> Twitter:http://twitter.com/app_engine
>>
>> On Fri, Jan 14, 2011 at 6:06 AM, Richard Arrano <[email protected]>wrote:
>>
>> > I'm looking to make a silent-auction type of application where you
>> > have 20-30 users bidding on an item at a time, with potentially
>> > hundreds or thousands of auctions happening simultaneously. As soon as
>> > a high bid is made, it updates this information and sends it via the
>> > Channel API to the other users in the auction. I see two potential
>> > difficulties:
>>
>> > 1. The limit on updating an entity group about once per second - I
>> > believe this can be solved with sharding bids amongst users in the
>> > auction and querying all shards to find the maximum bid at any given
>> > time, correct?
>>
>> > 2. The nature of the auction lends itself to a heavy amount of
>> > writing to the datastore - this itself eats up CPU and I’m trying to
>> > figure out if it can be avoided. Is this just inevitable in this type
>> > of application? Does it matter that I would only be only updating a
>> > single IntegerProperty() in any given write? Is there some clever
>> > solution that we can apply that avoids the hammering of the datastore?
>>
>> > Any tips or suggestions would be appreciated.
>>
>> > Thank you,
>> > Richard
>>
>> > --
>> > You received this message because you are subscribed to the Google Groups
>> > "Google App Engine" group.
>> > To post to this group, send email to [email protected].
>> > To unsubscribe from this group, send email to
>> > [email protected]<google-appengine%[email protected]>
>> > .
>> > For more options, visit this group at
>> >http://groups.google.com/group/google-appengine?hl=en.
>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
