>
> Using encode video example above, my server could handle encoding 8 videos
> at the same time. I'm assuming from the documentation and from my testing
> that that is not possible with the spooler. It looks like it's one process
> that pulls items from the spool one at a time - is that right?
Yes, it is right, but in 1.1 you can run multiple spoolers
--spooler spool1 --spooler spool2 --spooler spool3 --spooler spool4 ...
To choose in which spooler to enqueue the task, simply add the "spooler" key
mytask.spool(spooler='spool2',...)
>
> If that is the case, could I just run 8 mules and pass the work of
> encoding
> the video to them? If that is so, then if I got 10 requests to encode
> video, but had only 8 mules running, would the extra 2 be queued up until
> there was a free mule?
This is the current (1.0) approach. The internal mule message queue is
limited by the socket size (it is high but could not be enough). If you
want a more reliable queue system, use your db (where each record is a
task) and let the mules read from it (using uwsgi.lock to avoid two mules
getting the same task) or use something like rabbitmq or zeromq (i prefer
the second, even if it cannot store messages on disk, so if it crashes you
loose tasks).
Personally, i have a very simple model in my django apps defining a task
(it is only a text containing the shell command to run, and a lock flag)
bt = BackgroundTask()
bt.command = "ffmpeg -i ...."
bt.save()
then each mule run this code:
while True:
time.sleep(30)
uwsgi.lock()
# use the ordering you want
task = BackgroundTask.objects.filter(locked=0)[0]
task.locked = 1
task.save()
uwsgi.unlock()
...run the task...
This is an over-simplification, for example i avoid the time.sleep() and
relay on a "wakeup" message. But you should have enough infos to build a
customized system.
Once you have defined your mule brain you can scale it easily simply
adding more mules.
--
Roberto De Ioris
http://unbit.it
_______________________________________________
uWSGI mailing list
[email protected]
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
