Long time lurker here. First, thank you all for this incredible
product and community. I have learned a great deal in a very short
order of time thanks to web2py and this mailing list.
I may have the opportunity to finally contribute back meaningfully.
Regarding background message/task queuing I have had great success
with a very simple combination that has many extra side benefits. All
dependencies (BSON, redis-py, hotqueue) are available via pip for easy
installation. Some pros and cons listed below.
The short version:
Serialize your task/message into a dict, list of dicts, whatever via
BSON (Binary JSON = FAST),
push it to a queue kept in Redis,
background worker pulls from queue and processes it/inserts into db/
whatever you like.
Not only messages but I have successfully tested this as a queue for
tasks as well. Use a k,v pair in a dict describing your {'action':
'data'}. It's easy to get creative with it.
Cons:
- More dependencies if you don't already run Redis.
- Time to familiarize yourself with Redis. (15 - 20 minutes maybe)
- No scheduling built-in but uwsgi decorators can provide cron
capabilities for those running uwsgi. I haven't played with them yet
but intend to over the next month or two.
Benefits:
- Still fewer dependencies than celery. Redis becomes the task/
message queue as well as the back end storage and web2py workers
process them.
- Multiple dbs in Redis can be organized into different queues.
- Flexible - Can be adapted to your project fairly easily.
- Also can act as a cache. Redis has some really useful data types
which you can use to do a lot of your processing in memory.
- Has a pub/sub system which can also be incorporated into your
queue. Perhaps as a central logging server.
- All data in Redis is kept in memory but persisted to disk = blazing
fast performance with durability.
- Possible simplified application architecture if you use Redis for
your task queue, async message/db insert queue, pub/sub system
I will be happy to create a slice detailing the process with some
abstract examples if anyone has interest in my solution.
Thanks again for all your work. I intend to contribute back to web2py
as much as I can and all open source because I have learned and built
so much with it.
-David Bloom
