Hi Ben,
One possible way to handle this is to retrieve the top 10 items in
your 'queue', then loop through them in a transaction until you find
one which is not recent.
Better yet, you can cache this query--query a bit more, say 100 items,
save the keys into some simple structure in memcache, and consider
using memcache.incr to track where you are in those 100 items as you
test for the ability to pull off the queue--I assume you don't need to
assign these in exact order to your users.
something like this pseudocode:
get top item from the queue():
possiblekeys = memcache.get("queuekeys")
index = memcache.incr("queueindex")
if index is None or index > len(possiblekeys):
fallback to current code which does the query of the top 1 item
else:
key = possiblekeys[index]
in transaction:
check that date is not recent
update date
if transaction succeeded:
return db.get(key)
else:
return gettopitemfromqueue()
then some code to fill the queue:
fillqueue():
possibleEntities query(500 most recent items)
possibleKeys = [keys of possibleEntities]
memcache.set_multi({"queuekeys": possibleKeys, "queueindex": 0})
then you need to fill the queue somehow--either some sort of mechanism
when the cached queue is almost empty (which has mild locking issues),
or some external mechanism like cron.
--Matthew
On Apr 15, 1:31 pm, Ben Nevile <[email protected]> wrote:
> This is happening frequently today. I think it must expose a weakness
> in the transaction code. Hey, I know that what I'm doing with this
> code is totally lame -- basically causing many many transaction
> collisions - but clearly there's a problem here with the transaction
> code that creates the possibility of corrupting an entity. Not not
> not not cool.
>
> I'm working under a harsh time constraint right now (start of hockey
> playoffs) and my users need this crappy queue I've set up, so I'm not
> currently at liberty to leave one of these entities in its corrupted
> state for very long. But I would like to work with Google to get this
> sorted out.
>
> Ben
>
> On Apr 15, 8:04 am, Ben Nevile <[email protected]> wrote:
>
> > To add another twist, now I have an entity at the top of the stack
> > that is found using
>
> > q = FacebookUser.all()
> > q.filter('authorized = ',True)
> > q.order('last_nudged_at')
> > u = q.get()
>
> > but, if you try to load this entity by key, as you need to do for a
> > transaction, you get None back. ???? Trying to access that entity's
> > record in the Data Viewer also threw an exception. I was, however,
> > able to delete it from the top of that index listing using the Data
> > Viewer's GQL option.
>
> > Ben
>
> > On Apr 14, 3:38 pm, BenNevile<[email protected]> wrote:
>
> > > Hi Jeff,
>
> > > To clarify, the last_nudged_at property *is* changed, but the entity
> > > stays at the top of the stack.
>
> > > Ben
>
> > > On Apr 14, 12:26 pm, Jeff S <[email protected]> wrote:
>
> > > > Hi Ben,
>
> > > > If you are experiencing write contention on some of these updates,
> > > > then that may explain why the last_nudged_at property is not changed
> > > > and an entity tends to stay at the top of the stack. Do you see
> > > > datastore timeout errors in these cases (I imagine the exception may
> > > > be hidden somewhere in the JavaScript <--> server exchange)?
>
> > > > Thank you,
>
> > > > Jeff
>
> > > > On Apr 14, 11:48 am, BenNevile<[email protected]> wrote:
>
> > > > > bump. I have observed this many times over the past few days. Maybe
> > > > > has to do with transactions and/or write contention, since there is a
> > > > > lot of that in my app.
>
> > > > > Ben
>
> > > > > On Apr 1, 8:23 pm, BenNevile<[email protected]> wrote:
>
> > > > > > Hi GAE team and devotees,
>
> > > > > > I have a entity that looks like
>
> > > > > > class FacebookUser(db.Model):
> > > > > > expired_fotm = db.BooleanProperty()
> > > > > > last_nudged_at = db.DateTimeProperty()
>
> > > > > > and an index that looks like
>
> > > > > > - kind: FacebookUser
> > > > > > properties:
> > > > > > - name: expired_fotm
> > > > > > - name: last_nudged_at
>
> > > > > > Unbeknownst to the users of my application, some javascript is
> > > > > > constantly polling a worker handler that finds the first entity in
> > > > > > the
> > > > > > index, operates on it, then sets "last_nudged_at" to the current
> > > > > > time. There are more than a million of these entities so it takes
> > > > > > some time to go through them all.
>
> > > > > > This afternoon using the Data Viewer and the GQL query "SELECT *
> > > > > > FROM
> > > > > > FacebookUser where expired_fotm=True order by last_nudged_at" I
> > > > > > observed several times that an entity would remain "stuck" in the
> > > > > > first position in the index after it was processed. Usually it
> > > > > > would
> > > > > > remain there only for 10-30 seconds. However this evening one
> > > > > > entity
> > > > > > stayed in an incorrect index position for several hours. I put() it
> > > > > > again and it properly indexed itself.
>
> > > > > > Has anyone ever observed anything like this? Let me know if I'm not
> > > > > > making myself clear.
>
> > > > > > Ben <3 GAE
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~----------~----~----~----~------~----~------~--~---
