I also have to be able to schedule tasks, so I actually include the APScheduler package in my projects to make this scheduling. Can pyramid-tasks be used also in this context?
Le mer. 6 janv. 2021 à 19:44, 'Jonathan Vanasco' via pylons-discuss < pylons-discuss@googlegroups.com> a écrit : > pyramid_tasks looks very promising! thanks for sharing this. i may use it > in the future. i'll have to see if it is compatible with our setup. > > our celery app is mostly tasks that have been migrated from a twisted > daemon. we already ran web spiders and social-media-account importers > through twisted, so all the resizing stuff was originally pyramid creating > database entries (messages) for twisted to process. although we still use > twisted for those tasks (and more!) migrating to celery for these simpler > functions has been really productive and timesaving. the code is cleaner > and easier to maintain, and troubleshooting/bugfixing is so much simpler. > On Wednesday, January 6, 2021 at 12:51:37 PM UTC-5 the...@luhn.com wrote: > >> Seems like a good time to plug my new project: >> https://pypi.org/project/pyramid-tasks/ >> >> Pyramid and Celery are so dramatically different it’s a pain to integrate >> them, and any code shared between them can’t access anything >> Pyramid-specific. Got tired of trying to make a square peg fit in a round >> hole, so I brought my entire Pyramid application over to my Celery worker. >> Having a request object available in a Celery task is incredibly >> convenient, and my code is much tidier not having to tiptoe around the >> differences in environments. >> >> On Jan 6, 2021, at 2:14 AM, Adam Groszer <agro...@gmail.com> wrote: >> >> Hi, >> >> Yeah celery is the way to go. >> There's https://github.com/sontek/pyramid_celery to check out. >> Some things to keep in mind is, you'll need a shared DB, good-old ZODB >> with filestorage is not enough, because more processes will need to have >> access. >> Do not store valuable data in the celery queue, consider it ephemeral, go >> through a real DB. >> >> On Tuesday, January 5, 2021 at 10:31:15 PM UTC+1 tfl...@gmail.com wrote: >> >>> Hi Jonathan, >>> >>> Thank you for this description of Celery! >>> I'll try to have a look at it if I can get a little time, it seems to be >>> a good replacement (probably more reliable!) of my own developments... >>> >>> Best regards, >>> Thierry >>> -- >>> https://www.ulthar.net -- http://pyams.readthedocs.io >>> >>> Le mar. 5 janv. 2021 à 22:18, 'Jonathan Vanasco' via pylons-discuss < >>> pylons-...@googlegroups.com> a écrit : >>> >>>> Thierry, >>>> >>>> That is what I mostly use Celery for (it's also used for generating >>>> Reports, ACME SSL Certificates and a few other things). A Pyramid Request >>>> will defer a job(s) to Celery via the "Transport Backend" and receive a >>>> task/messaging ID. Subsequent requests will poll the Celery "Result >>>> Backend" for the status of that job, based on the ID. This way we show >>>> the "Still processing!" message to users. >>>> >>>> Celery is run as a worker with multiple processes. The Celery task >>>> manager grabs a task off the queue, then does the resizing, uploads to S3, >>>> and notifies the Result Backend when complete. I use Redis for Result and >>>> Transport. I think there was once a ZeroMQ integration; there is >>>> definitely RabbitMQ integration, it's one of the more popular options. >>>> >>>> On Monday, January 4, 2021 at 12:25:44 PM UTC-5 tfl...@gmail.com wrote: >>>> >>>>> Hi Jonathan, >>>>> >>>>> I didn't have a look at Celery yet, maybe it could work... >>>>> My goal is to be able to start long-running tasks (medias files >>>>> conversions for example) whose definition is stored in database and >>>>> managed >>>>> by end users from my main Pyramid application; for this, I create a >>>>> dedicated sub-process which is "managed" using ZeroMQ messages (I use this >>>>> because it's also used for other non-Python/Pyramid applications). >>>>> Tasks are then started using dedicated threads. >>>>> >>>>> Everything is OK until now... My only requirement now is to be able to >>>>> get access to main Pyramid's registry from my sub-process and it's >>>>> threads, >>>>> and I don't really understand why I get a pointer to >>>>> ZCA global registry... :( >>>>> >>>>> Best regards, >>>>> Thierry >>>>> -- >>>>> https://www.ulthar.net -- http://pyams.readthedocs.io >>>>> >>>>> Le lun. 4 janv. 2021 à 17:51, 'Jonathan Vanasco' via pylons-discuss < >>>>> pylons-...@googlegroups.com> a écrit : >>>>> >>>>>> Have you considered using Celery for this? I started offloading >>>>>> everything related to subprocesses and message queues to it a while back, >>>>>> and have been much happier. >>>>>> >>>>>> On Monday, January 4, 2021 at 11:20:46 AM UTC-5 tfl...@gmail.com >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I need to create custom sub-processes from my main Pyramid >>>>>>> application; these processes are used to handle "commands" received from >>>>>>> ZeroMQ messages. >>>>>>> If I use the components registry to register adapters and utilities >>>>>>> which are also required in these processes, is there a way to start >>>>>>> these >>>>>>> processes so that they can "inherit" from the main application registry >>>>>>> ? >>>>>>> >>>>>>> I'm actually using Pyramid 1.10.5, and I generally do a >>>>>>> "config.hook_zca()" on application startup before doing >>>>>>> the "config.setup_registry()" and "config.scan()". >>>>>>> I tried to do the same operations in my sub-process "run" method, >>>>>>> but I always get the "global" registry instead of my "webapp" registry >>>>>>> which was configured in my main application... >>>>>>> >>>>>>> Best regards, >>>>>>> Thierry >>>>>>> -- >>>>>>> https://www.ulthar.net -- http://pyams.readthedocs.io >>>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "pylons-discuss" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to pylons-discus...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "pylons-discuss" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to pylons-discus...@googlegroups.com. >>>> >>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/pylons-discuss/5ac83ff1-747f-4df9-994c-ffa48f339013n%40googlegroups.com >>>> <https://groups.google.com/d/msgid/pylons-discuss/5ac83ff1-747f-4df9-994c-ffa48f339013n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "pylons-discuss" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to pylons-discus...@googlegroups.com. >> >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/pylons-discuss/0a0d6985-c5b5-48b6-88f9-cce5d837c583n%40googlegroups.com >> <https://groups.google.com/d/msgid/pylons-discuss/0a0d6985-c5b5-48b6-88f9-cce5d837c583n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> >> -- > You received this message because you are subscribed to the Google Groups > "pylons-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to pylons-discuss+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/pylons-discuss/aaea1ab5-6d86-475c-8499-9435cdba411bn%40googlegroups.com > <https://groups.google.com/d/msgid/pylons-discuss/aaea1ab5-6d86-475c-8499-9435cdba411bn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAPX_VWC5EhcCj%2BRmr_ABMObbgYtTUYskaSR8TL0Nexm_1fCZ-A%40mail.gmail.com.
