We're using pybeanstalk 0.11.1, python 2.6.5, Ubuntu 10.04.3 LTS. The main
loop of our multi-threaded beanstalk client looks like:
while True:
m = mq.reserve()
jid = m['jid']
try:
# process job
except Exception:
logger.error("skipped job %d" % jid)
finally:
logger.info("deleted job %d" % jid)
mq.delete(jid)
Every once in a while (yesterday it happened once in 60k jobs processed),
the delete() call in the finally block raises a NotFound exception. I've
included a stack dump below.
2012-04-02T04:56:57+00:00 cluster1.songza.com 2012-04-02 04:56:57,497 -
scrobble_mill - Caught NotFound('Server returned: NOT_FOUND',) in
<Thread(Thread-9, started 139969929115392)>
Traceback (most recent call last):
File "/home/songza/deploy/current/scrobble/scrobble_mill.py", line 98, in
listen_wrapper
listen(host, port)
File "/home/songza/deploy/current/scrobble/scrobble_mill.py", line 162,
in listen
mq.delete(jid)
File "/usr/lib/pymodules/python2.6/beanstalk/serverconn.py", line 94, in
caller
*getattr(protohandler, 'process_%s' % attr)(*args, **kw))
File "/usr/lib/pymodules/python2.6/beanstalk/serverconn.py", line 68, in
_do_interaction
return self._get_response(handler)
File "/usr/lib/pymodules/python2.6/beanstalk/serverconn.py", line 58, in
_get_response
res = handler(recv)
File "/usr/lib/pymodules/python2.6/beanstalk/protohandler.py", line 56,
in __call__
return self.__h(val)
File "/usr/lib/pymodules/python2.6/beanstalk/protohandler.py", line 68,
in handler
checkError(response)
File "/usr/lib/pymodules/python2.6/beanstalk/errors.py", line 35, in
checkError
raise err
NotFound: Server returned: NOT_FOUND
Looking over the logs, it appears that two different threads got the same
job in their reserve() calls. Is this a bug? I was assuming that
beanstalk was thread safe, but maybe not?
--
You received this message because you are subscribed to the Google Groups
"beanstalk-talk" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/beanstalk-talk/-/GMkJoHdsoOgJ.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/beanstalk-talk?hl=en.
