Thierry, That is what I mostly use Celery for (it's also used for generating Reports, ACME SSL Certificates and a few other things). A Pyramid Request will defer a job(s) to Celery via the "Transport Backend" and receive a task/messaging ID. Subsequent requests will poll the Celery "Result Backend" for the status of that job, based on the ID. This way we show the "Still processing!" message to users.
Celery is run as a worker with multiple processes. The Celery task manager grabs a task off the queue, then does the resizing, uploads to S3, and notifies the Result Backend when complete. I use Redis for Result and Transport. I think there was once a ZeroMQ integration; there is definitely RabbitMQ integration, it's one of the more popular options. On Monday, January 4, 2021 at 12:25:44 PM UTC-5 tfl...@gmail.com wrote: > Hi Jonathan, > > I didn't have a look at Celery yet, maybe it could work... > My goal is to be able to start long-running tasks (medias files > conversions for example) whose definition is stored in database and managed > by end users from my main Pyramid application; for this, I create a > dedicated sub-process which is "managed" using ZeroMQ messages (I use this > because it's also used for other non-Python/Pyramid applications). > Tasks are then started using dedicated threads. > > Everything is OK until now... My only requirement now is to be able to get > access to main Pyramid's registry from my sub-process and it's threads, and > I don't really understand why I get a pointer to > ZCA global registry... :( > > Best regards, > Thierry > -- > https://www.ulthar.net -- http://pyams.readthedocs.io > > Le lun. 4 janv. 2021 à 17:51, 'Jonathan Vanasco' via pylons-discuss < > pylons-...@googlegroups.com> a écrit : > >> Have you considered using Celery for this? I started offloading >> everything related to subprocesses and message queues to it a while back, >> and have been much happier. >> >> On Monday, January 4, 2021 at 11:20:46 AM UTC-5 tfl...@gmail.com wrote: >> >>> Hi, >>> >>> I need to create custom sub-processes from my main Pyramid application; >>> these processes are used to handle "commands" received from ZeroMQ messages. >>> If I use the components registry to register adapters and utilities >>> which are also required in these processes, is there a way to start these >>> processes so that they can "inherit" from the main application registry ? >>> >>> I'm actually using Pyramid 1.10.5, and I generally do a >>> "config.hook_zca()" on application startup before doing >>> the "config.setup_registry()" and "config.scan()". >>> I tried to do the same operations in my sub-process "run" method, but I >>> always get the "global" registry instead of my "webapp" registry which was >>> configured in my main application... >>> >>> Best regards, >>> Thierry >>> -- >>> https://www.ulthar.net -- http://pyams.readthedocs.io >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "pylons-discuss" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to pylons-discus...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/5ac83ff1-747f-4df9-994c-ffa48f339013n%40googlegroups.com.
