Would this really help? There's still going to be a splitting issue with the index tablet, and that one you will have a hard time distributing the keyspace for.
Jeff On Wed, Aug 1, 2012 at 12:52 PM, Richard <[email protected]> wrote: > Hi Takashi, > Yeah, the first few characters would probably be very similar. I am not > sure if that would be a problem ? > > A single user has only one score in the leaderboard for the game. So, > overwriting the score the next time is not a problem, provided we know when > to NOT include a user in the leaderboards (timestamp... if they are no > longer playing). > > We could also use the user's name as that is guaranteed unique too. Would > that work ? > > -R > > > On Wednesday, August 1, 2012 3:32:27 PM UTC-4, Takashi Matsuo (Google) > wrote: >> >> On Thu, Aug 2, 2012 at 4:11 AM, Richard <[email protected]> wrote: >> > Hi Takashi & Joshua, >> > >> > How about using my user's id as the key ? >> > >> > eg: >> > new_score = >> > Score(key_name='ahFkZXZ-c3Zlbi13b3JkaGVyb3ILCxIEVXNlchjFAww') >> > >> > I assume this would not count as monotonically increasing ? >> >> I don't think it's ok. Probably the first few characters would be >> always the same. It's not good. >> Additionally, from your explanation bellow, a single user has multiple >> scores, right? If you use userid as key_name, the latest score always >> override the previous score. >> >> What about something like user_name + timestamp? >> >> > >> > @ Joshua: >> > At the moment, I create all new Score objects for each player of that >> > game >> > round and then have the backend reap them and afterwards fire a task to >> > delete them. HOWEVER, what is I did not delete them, but instead marked >> > them with the user's account key_name as the Score's key_name (see >> > above) >> > and also included the following: >> > timestamp = db.DateTimeProperty(auto_now = True) >> > >> > Then I could figure out which ones were from the last round and which >> > were >> > from players that had last played a while ago. The assumption is that >> > at >> > this point, players from the previous have would have their results >> > stored >> > using a put() with a key name, which would be strongly consistent. >> > Players >> > whose first game it is, would have the chance that the DB was slow and >> > did >> > not yet have an entry for them ... in which case, they would just miss >> > out >> > on one round (unless the internal commit() takes > 3 minutes!) ... and I >> > could update the game client to add in a user's score to the results if >> > they >> > were not found to 'fudge' things. >> > >> > Comments/thoughts ? >> > >> > -R >> > >> > >> >> Sorry it didn't work like a magic, but a single datastore get >> >> shouldn't take that long, so there's another issue with datastore >> >> latency. It might be tablet splitting issue which I mentioned in >> >> another thread: >> >> >> >> >> >> http://ikaisays.com/2011/01/25/app-engine-datastore-tip-monotonically-increasing-values-are-bad/ >> >> >> >> I'd suggest generating somewhat ramdom(well-distributed) keys for >> >> those small entities instead of using automatic key assignment. >> >> Besides using datastore, maybe I'll experiment with an implementation >> >> of a memory db with some backends instances tomorrow. >> >> >> >> -- Takashi >> >> >> >> > >> >> > >> >> > >> >> > On Wednesday, August 1, 2012 2:01:43 PM UTC-4, Richard wrote: >> >> >> >> >> >> Yep... got that. However, a query() will still return stale data >> >> >> between >> >> >> the put() and the (internal) commit(). Which is where I believe I >> >> >> am >> >> >> sitting... >> >> >> >> >> > >> >> > On Wednesday, August 1, 2012 2:01:43 PM UTC-4, Richard wrote: >> >> >> >> >> >> Yep... got that. However, a query() will still return stale data >> >> >> between >> >> >> the put() and the (internal) commit(). Which is where I believe I >> >> >> am >> >> >> sitting... >> >> >> >> >> >> On Wednesday, August 1, 2012 12:39:41 PM UTC-4, Jeff Schnitzer >> >> >> wrote: >> >> >>> >> >> >>> I presume the essential line is this: >> >> >>> >> >> >>> "In the (now standard) High Replication Datastore, the transaction >> >> >>> typically is completely applied within a few hundred milliseconds >> >> >>> after the commit returns. However, even if it is not completely >> >> >>> applied, subsequent reads, writes, and ancestor queries will always >> >> >>> reflect the results of the commit, because these operations apply >> >> >>> any >> >> >>> outstanding modifications before executing." >> >> >>> >> >> >>> ...which is interesting and not something I ever thought of. I >> >> >>> don't >> >> >>> think this intermediate state was discussed in Alfred's "Under The >> >> >>> Covers" I/O talk last year. Can someone describe it in more >> >> >>> detail? >> >> >>> >> >> >>> Jeff >> >> >>> >> >> >>> On Wed, Aug 1, 2012 at 8:45 AM, Richard <[email protected]> >> >> >>> wrote: >> >> >>> > Ok, so I read that web page, and I understand that the put() has >> >> >>> > completed, >> >> >>> > but not the commit(). >> >> >>> > >> >> >>> > However, I don't really see the potential solution you are >> >> >>> > suggesting >> >> >>> > :( >> >> >>> > Can you (or someone else) please explain in a bit more detail ? >> >> >>> > >> >> >>> > >> >> >>> > On Wednesday, August 1, 2012 10:55:35 AM UTC-4, Takashi Matsuo >> >> >>> > (Google) >> >> >>> > wrote: >> >> >>> >> >> >> >>> >> I forgot to mention about one possible workaround which could >> >> >>> >> mitigate >> >> >>> >> your pain. In the frontend instances, maybe you can get those >> >> >>> >> small >> >> >>> >> entities with a newly created keys just after putting them in >> >> >>> >> order >> >> >>> >> to >> >> >>> >> apply the change locally. >> >> >>> >> >> >> >>> >> This article describes this behavior well: >> >> >>> >> >> >> >>> >> >> >> >>> >> https://developers.google.com/appengine/articles/transaction_isolation >> >> >>> >> >> >> >>> >> However, please note that >> >> >>> >> * It doesn't guarantee 100% 'strong consistency'. >> >> >>> >> * The latency of submitting scores will become larger. >> >> >>> >> * Definitely it will cost you more. >> >> >>> >> >> >> >>> >> Sorry if this doesn't work well for you, but I think it's worth >> >> >>> >> trying. >> >> >>> >> >> >> >>> >> -- Takashi >> >> >>> >> >> >> >>> >> On Wed, Aug 1, 2012 at 11:30 PM, Takashi Matsuo >> >> >>> >> <[email protected]> >> >> >>> >> wrote: >> >> >>> >> > Hi Richard, >> >> >>> >> > >> >> >>> >> > I actually played your game and probably encountered the exact >> >> >>> >> > issue. >> >> >>> >> > BTW, it's a cool addictive game. >> >> >>> >> > >> >> >>> >> > I agree that eventually consistent window should ideally be >> >> >>> >> > more >> >> >>> >> > steady than now. I have passed your issue to the engineering >> >> >>> >> > team. >> >> >>> >> > >> >> >>> >> > I'll get back to you when I have some updates. >> >> >>> >> > >> >> >>> >> > -- Takashi >> >> >>> >> > >> >> >>> >> > On Wed, Aug 1, 2012 at 10:37 PM, Richard <[email protected]> >> >> >>> >> > wrote: >> >> >>> >> >> The thing is.... GAE does a wonderful job. It has been fine >> >> >>> >> >> for >> >> >>> >> >> a >> >> >>> >> >> year >> >> >>> >> >> or >> >> >>> >> >> so. Now suddenly, we are getting bitten by 'eventual >> >> >>> >> >> consistency'. >> >> >>> >> >> And not >> >> >>> >> >> at peak load either! This is hitting us at the lowest load >> >> >>> >> >> time >> >> >>> >> >> and at >> >> >>> >> >> the >> >> >>> >> >> same time each day. >> >> >>> >> >> >> >> >>> >> >> So, maybe we were just lucky before this..... but it still >> >> >>> >> >> sucks >> >> >>> >> >> to >> >> >>> >> >> have >> >> >>> >> >> something work for a year or so, and then suddenly find out >> >> >>> >> >> you >> >> >>> >> >> might >> >> >>> >> >> need >> >> >>> >> >> to consider moving to a different stack. >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> On Wednesday, August 1, 2012 7:14:19 AM UTC-4, Richard Watson >> >> >>> >> >> wrote: >> >> >>> >> >>> >> >> >>> >> >>> On Wed, Aug 1, 2012 at 12:13 PM, Jeff Schnitzer >> >> >>> >> >>> <[email protected]> >> >> >>> >> >>> wrote: >> >> >>> >> >>> >> >> >>> >> >>> > Anything that involves batching in a frontend risks >> >> >>> >> >>> > orphaning >> >> >>> >> >>> > data >> >> >>> >> >>> > in >> >> >>> >> >>> > the frontend... there's just no efective way to ensure >> >> >>> >> >>> > that >> >> >>> >> >>> > batching >> >> >>> >> >>> > happens and that the queue is purged when "done". >> >> >>> >> >>> >> >> >>> >> >>> It's definitely not bulletproof, but I'd investigate whether >> >> >>> >> >>> I >> >> >>> >> >>> could >> >> >>> >> >>> achieve "good enough" for 99% and be happy that for the rest >> >> >>> >> >>> the >> >> >>> >> >>> client could re-request the score if it didn't reflect their >> >> >>> >> >>> own >> >> >>> >> >>> correct value. >> >> >>> >> >>> >> >> >>> >> >>> As you say, if clients can connect directly to some other >> >> >>> >> >>> more >> >> >>> >> >>> reliable service, maybe that's the way to go. Mixing >> >> >>> >> >>> platforms >> >> >>> >> >>> (GAE >> >> >>> >> >>> url fetch to Amazon SQS, or whatever) would leave me >> >> >>> >> >>> worrying >> >> >>> >> >>> about >> >> >>> >> >>> extra points of failure - you'll now fail if either GAE or >> >> >>> >> >>> Amazon/Heroku go down. >> >> >>> >> >>> >> >> >>> >> >>> Pity. I would imaging a back-end for a game with hundreds >> >> >>> >> >>> of >> >> >>> >> >>> users is >> >> >>> >> >>> something GAE should eat up. >> >> >>> >> >>> >> >> >>> >> >>> Richard >> >> >>> >> >> >> >> >>> >> >> -- >> >> >>> >> >> You received this message because you are subscribed to the >> >> >>> >> >> Google >> >> >>> >> >> Groups >> >> >>> >> >> "Google App Engine" group. >> >> >>> >> >> To view this discussion on the web visit >> >> >>> >> >> >> >> >>> >> >> https://groups.google.com/d/msg/google-appengine/-/MpsNezxGjpkJ. >> >> >>> >> >> >> >> >>> >> >> To post to this group, send email to >> >> >>> >> >> [email protected]. >> >> >>> >> >> To unsubscribe from this group, send email to >> >> >>> >> >> [email protected]. >> >> >>> >> >> For more options, visit this group at >> >> >>> >> >> http://groups.google.com/group/google-appengine?hl=en. >> >> >>> >> > >> >> >>> >> > >> >> >>> >> > >> >> >>> >> > -- >> >> >>> >> > Takashi Matsuo >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> -- >> >> >>> >> Takashi Matsuo >> >> > >> >> > -- >> >> > You received this message because you are subscribed to the Google >> >> > Groups >> >> > "Google App Engine" group. >> >> > To view this discussion on the web visit >> >> > https://groups.google.com/d/msg/google-appengine/-/v9XP85rmNM8J. >> >> > >> >> > To post to this group, send email to >> >> > [email protected]. >> >> > To unsubscribe from this group, send email to >> >> > [email protected]. >> >> > For more options, visit this group at >> >> > http://groups.google.com/group/google-appengine?hl=en. >> >> >> >> >> >> >> >> -- >> >> Takashi Matsuo >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "Google App Engine" group. >> > To view this discussion on the web visit >> > https://groups.google.com/d/msg/google-appengine/-/iJA7RSzgG0oJ. >> > >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group at >> > http://groups.google.com/group/google-appengine?hl=en. >> >> >> >> -- >> Takashi Matsuo -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
