Re: [pylons-discuss] Re: Creating sub-processes from Pyramid application

2021-01-06 Thread Adam Groszer
Hi, Yeah celery is the way to go. There's https://github.com/sontek/pyramid_celery to check out. Some things to keep in mind is, you'll need a shared DB, good-old ZODB with filestorage is not enough, because more processes will need to have access. Do not store valuable data in the celery queue,

Re: [pylons-discuss] Re: Creating sub-processes from Pyramid application

2021-01-05 Thread Thierry Florac
Hi Jonathan, Thank you for this description of Celery! I'll try to have a look at it if I can get a little time, it seems to be a good replacement (probably more reliable!) of my own developments... Best regards, Thierry -- https://www.ulthar.net -- http://pyams.readthedocs.io Le mar. 5

Re: [pylons-discuss] Re: Creating sub-processes from Pyramid application

2021-01-05 Thread 'Jonathan Vanasco' via pylons-discuss
Thierry, That is what I mostly use Celery for (it's also used for generating Reports, ACME SSL Certificates and a few other things). A Pyramid Request will defer a job(s) to Celery via the "Transport Backend" and receive a task/messaging ID. Subsequent requests will poll the Celery "Result

Re: [pylons-discuss] Re: Creating sub-processes from Pyramid application

2021-01-04 Thread Thierry Florac
Hi (again)! I think it's finally OK! I forgot ZCA hook is based on Pyramid's "current registry", which is bound to a thread-local variable, so that's why I have to push them on threadlocal manager in my threads "run" methods (and not in the constructor as I have tried before)!! Maybe there is

Re: [pylons-discuss] Re: Creating sub-processes from Pyramid application

2021-01-04 Thread Thierry Florac
Hi Jonathan, I didn't have a look at Celery yet, maybe it could work... My goal is to be able to start long-running tasks (medias files conversions for example) whose definition is stored in database and managed by end users from my main Pyramid application; for this, I create a dedicated

[pylons-discuss] Re: Creating sub-processes from Pyramid application

2021-01-04 Thread 'Jonathan Vanasco' via pylons-discuss
Have you considered using Celery for this? I started offloading everything related to subprocesses and message queues to it a while back, and have been much happier. On Monday, January 4, 2021 at 11:20:46 AM UTC-5 tfl...@gmail.com wrote: > Hi, > > I need to create custom sub-processes from my