Re: Batch processing in parallel (celery)

2012-03-12 Thread Shawn Milochik
You can just use Celery. It's very simple if you use django-celery and MongoDB as the broker. Next, you could make sure you're using the ORM effectively. Use select_related where possible, avoid doing any querying in loops, pre-pulling data from the database and storing it in memory (in a

Batch processing in parallel (celery)

2012-03-12 Thread Thomas Guettler
Hi, there is a background job which uses the django ORM and takes too long. How to speed up the job? It processes files, and updates the database. It would be easy to cut the job into peaces, since the files can be processed in parallel random order. I know that I could use celery, but it is