On Thu, Aug 2, 2012 at 3:24 AM, Richard <[email protected]> wrote: > Ok, based on Takashi's suggestion, I now do the following: > > new_user_score = Score() > new_user_score.member = thing > new_user_score.put() > > # next part is new based on Takashi's suggestion > k = new_user_score.key() > db.get(k) > > WOW... I mean... HOLY CRIPES! Time to process each request went from > ~150msec to 4.5-7 SECONDS. > > This means I effectively need a one instance PER REQUEST.
Sorry it didn't work like a magic, but a single datastore get shouldn't take that long, so there's another issue with datastore latency. It might be tablet splitting issue which I mentioned in another thread: http://ikaisays.com/2011/01/25/app-engine-datastore-tip-monotonically-increasing-values-are-bad/ I'd suggest generating somewhat ramdom(well-distributed) keys for those small entities instead of using automatic key assignment. Besides using datastore, maybe I'll experiment with an implementation of a memory db with some backends instances tomorrow. -- Takashi > > > > On Wednesday, August 1, 2012 2:01:43 PM UTC-4, Richard wrote: >> >> Yep... got that. However, a query() will still return stale data between >> the put() and the (internal) commit(). Which is where I believe I am >> sitting... >> > > On Wednesday, August 1, 2012 2:01:43 PM UTC-4, Richard wrote: >> >> Yep... got that. However, a query() will still return stale data between >> the put() and the (internal) commit(). Which is where I believe I am >> sitting... >> >> On Wednesday, August 1, 2012 12:39:41 PM UTC-4, Jeff Schnitzer wrote: >>> >>> I presume the essential line is this: >>> >>> "In the (now standard) High Replication Datastore, the transaction >>> typically is completely applied within a few hundred milliseconds >>> after the commit returns. However, even if it is not completely >>> applied, subsequent reads, writes, and ancestor queries will always >>> reflect the results of the commit, because these operations apply any >>> outstanding modifications before executing." >>> >>> ...which is interesting and not something I ever thought of. I don't >>> think this intermediate state was discussed in Alfred's "Under The >>> Covers" I/O talk last year. Can someone describe it in more detail? >>> >>> Jeff >>> >>> On Wed, Aug 1, 2012 at 8:45 AM, Richard <[email protected]> wrote: >>> > Ok, so I read that web page, and I understand that the put() has >>> > completed, >>> > but not the commit(). >>> > >>> > However, I don't really see the potential solution you are suggesting >>> > :( >>> > Can you (or someone else) please explain in a bit more detail ? >>> > >>> > >>> > On Wednesday, August 1, 2012 10:55:35 AM UTC-4, Takashi Matsuo (Google) >>> > wrote: >>> >> >>> >> I forgot to mention about one possible workaround which could mitigate >>> >> your pain. In the frontend instances, maybe you can get those small >>> >> entities with a newly created keys just after putting them in order to >>> >> apply the change locally. >>> >> >>> >> This article describes this behavior well: >>> >> https://developers.google.com/appengine/articles/transaction_isolation >>> >> >>> >> However, please note that >>> >> * It doesn't guarantee 100% 'strong consistency'. >>> >> * The latency of submitting scores will become larger. >>> >> * Definitely it will cost you more. >>> >> >>> >> Sorry if this doesn't work well for you, but I think it's worth >>> >> trying. >>> >> >>> >> -- Takashi >>> >> >>> >> On Wed, Aug 1, 2012 at 11:30 PM, Takashi Matsuo <[email protected]> >>> >> wrote: >>> >> > Hi Richard, >>> >> > >>> >> > I actually played your game and probably encountered the exact >>> >> > issue. >>> >> > BTW, it's a cool addictive game. >>> >> > >>> >> > I agree that eventually consistent window should ideally be more >>> >> > steady than now. I have passed your issue to the engineering team. >>> >> > >>> >> > I'll get back to you when I have some updates. >>> >> > >>> >> > -- Takashi >>> >> > >>> >> > On Wed, Aug 1, 2012 at 10:37 PM, Richard <[email protected]> >>> >> > wrote: >>> >> >> The thing is.... GAE does a wonderful job. It has been fine for a >>> >> >> year >>> >> >> or >>> >> >> so. Now suddenly, we are getting bitten by 'eventual consistency'. >>> >> >> And not >>> >> >> at peak load either! This is hitting us at the lowest load time >>> >> >> and at >>> >> >> the >>> >> >> same time each day. >>> >> >> >>> >> >> So, maybe we were just lucky before this..... but it still sucks to >>> >> >> have >>> >> >> something work for a year or so, and then suddenly find out you >>> >> >> might >>> >> >> need >>> >> >> to consider moving to a different stack. >>> >> >> >>> >> >> >>> >> >> On Wednesday, August 1, 2012 7:14:19 AM UTC-4, Richard Watson >>> >> >> wrote: >>> >> >>> >>> >> >>> On Wed, Aug 1, 2012 at 12:13 PM, Jeff Schnitzer >>> >> >>> <[email protected]> >>> >> >>> wrote: >>> >> >>> >>> >> >>> > Anything that involves batching in a frontend risks orphaning >>> >> >>> > data >>> >> >>> > in >>> >> >>> > the frontend... there's just no efective way to ensure that >>> >> >>> > batching >>> >> >>> > happens and that the queue is purged when "done". >>> >> >>> >>> >> >>> It's definitely not bulletproof, but I'd investigate whether I >>> >> >>> could >>> >> >>> achieve "good enough" for 99% and be happy that for the rest the >>> >> >>> client could re-request the score if it didn't reflect their own >>> >> >>> correct value. >>> >> >>> >>> >> >>> As you say, if clients can connect directly to some other more >>> >> >>> reliable service, maybe that's the way to go. Mixing platforms >>> >> >>> (GAE >>> >> >>> url fetch to Amazon SQS, or whatever) would leave me worrying >>> >> >>> about >>> >> >>> extra points of failure - you'll now fail if either GAE or >>> >> >>> Amazon/Heroku go down. >>> >> >>> >>> >> >>> Pity. I would imaging a back-end for a game with hundreds of >>> >> >>> users is >>> >> >>> something GAE should eat up. >>> >> >>> >>> >> >>> Richard >>> >> >> >>> >> >> -- >>> >> >> You received this message because you are subscribed to the Google >>> >> >> Groups >>> >> >> "Google App Engine" group. >>> >> >> To view this discussion on the web visit >>> >> >> https://groups.google.com/d/msg/google-appengine/-/MpsNezxGjpkJ. >>> >> >> >>> >> >> To post to this group, send email to >>> >> >> [email protected]. >>> >> >> To unsubscribe from this group, send email to >>> >> >> [email protected]. >>> >> >> For more options, visit this group at >>> >> >> http://groups.google.com/group/google-appengine?hl=en. >>> >> > >>> >> > >>> >> > >>> >> > -- >>> >> > Takashi Matsuo >>> >> >>> >> >>> >> >>> >> -- >>> >> Takashi Matsuo > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/v9XP85rmNM8J. > > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. -- Takashi Matsuo -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
