Hi Matthew, Thanks for these great ideas!
Ben On Apr 15, 8:30 pm, Matthew Blain <[email protected]> wrote: > Hi Ben, > One possible way to handle this is to retrieve the top 10 items in > your 'queue', then loop through them in a transaction until you find > one which is not recent. > > Better yet, you can cache this query--query a bit more, say 100 items, > save the keys into some simple structure in memcache, and consider > using memcache.incr to track where you are in those 100 items as you > test for the ability to pull off the queue--I assume you don't need to > assign these in exact order to your users. > > something like this pseudocode: > > get top item from the queue(): > possiblekeys = memcache.get("queuekeys") > index = memcache.incr("queueindex") > if index is None or index > len(possiblekeys): > fallback to current code which does the query of the top 1 item > else: > key = possiblekeys[index] > in transaction: > check that date is not recent > update date > if transaction succeeded: > return db.get(key) > else: > return gettopitemfromqueue() > > then some code to fill the queue: > fillqueue(): > possibleEntities query(500 most recent items) > possibleKeys = [keys of possibleEntities] > memcache.set_multi({"queuekeys": possibleKeys, "queueindex": 0}) > > then you need to fill the queue somehow--either some sort of mechanism > when the cached queue is almost empty (which has mild locking issues), > or some external mechanism like cron. > > --Matthew > > On Apr 15, 1:31 pm, BenNevile<[email protected]> wrote: > > > > > This is happening frequently today. I think it must expose a weakness > > in the transaction code. Hey, I know that what I'm doing with this > > code is totally lame -- basically causing many many transaction > > collisions - but clearly there's a problem here with the transaction > > code that creates the possibility of corrupting an entity. Not not > > not not cool. > > > I'm working under a harsh time constraint right now (start of hockey > > playoffs) and my users need this crappy queue I've set up, so I'm not > > currently at liberty to leave one of these entities in its corrupted > > state for very long. But I would like to work with Google to get this > > sorted out. > > > Ben > > > On Apr 15, 8:04 am, BenNevile<[email protected]> wrote: > > > > To add another twist, now I have an entity at the top of the stack > > > that is found using > > > > q = FacebookUser.all() > > > q.filter('authorized = ',True) > > > q.order('last_nudged_at') > > > u = q.get() > > > > but, if you try to load this entity by key, as you need to do for a > > > transaction, you get None back. ???? Trying to access that entity's > > > record in the Data Viewer also threw an exception. I was, however, > > > able to delete it from the top of that index listing using the Data > > > Viewer's GQL option. > > > > Ben > > > > On Apr 14, 3:38 pm, BenNevile<[email protected]> wrote: > > > > > Hi Jeff, > > > > > To clarify, the last_nudged_at property *is* changed, but the entity > > > > stays at the top of the stack. > > > > > Ben > > > > > On Apr 14, 12:26 pm, Jeff S <[email protected]> wrote: > > > > > > Hi Ben, > > > > > > If you are experiencing write contention on some of these updates, > > > > > then that may explain why the last_nudged_at property is not changed > > > > > and an entity tends to stay at the top of the stack. Do you see > > > > > datastore timeout errors in these cases (I imagine the exception may > > > > > be hidden somewhere in the JavaScript <--> server exchange)? > > > > > > Thank you, > > > > > > Jeff > > > > > > On Apr 14, 11:48 am, BenNevile<[email protected]> wrote: > > > > > > > bump. I have observed this many times over the past few days. > > > > > > Maybe > > > > > > has to do with transactions and/or write contention, since there is > > > > > > a > > > > > > lot of that in my app. > > > > > > > Ben > > > > > > > On Apr 1, 8:23 pm, BenNevile<[email protected]> wrote: > > > > > > > > Hi GAE team and devotees, > > > > > > > > I have a entity that looks like > > > > > > > > class FacebookUser(db.Model): > > > > > > > expired_fotm = db.BooleanProperty() > > > > > > > last_nudged_at = db.DateTimeProperty() > > > > > > > > and an index that looks like > > > > > > > > - kind: FacebookUser > > > > > > > properties: > > > > > > > - name: expired_fotm > > > > > > > - name: last_nudged_at > > > > > > > > Unbeknownst to the users of my application, some javascript is > > > > > > > constantly polling a worker handler that finds the first entity > > > > > > > in the > > > > > > > index, operates on it, then sets "last_nudged_at" to the current > > > > > > > time. There are more than a million of these entities so it takes > > > > > > > some time to go through them all. > > > > > > > > This afternoon using the Data Viewer and the GQL query "SELECT * > > > > > > > FROM > > > > > > > FacebookUser where expired_fotm=True order by last_nudged_at" I > > > > > > > observed several times that an entity would remain "stuck" in the > > > > > > > first position in the index after it was processed. Usually it > > > > > > > would > > > > > > > remain there only for 10-30 seconds. However this evening one > > > > > > > entity > > > > > > > stayed in an incorrect index position for several hours. I put() > > > > > > > it > > > > > > > again and it properly indexed itself. > > > > > > > > Has anyone ever observed anything like this? Let me know if I'm > > > > > > > not > > > > > > > making myself clear. > > > > > > > > Ben <3 GAE --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
