Fine! I'll have to take some time to have a look at Celery and at your package, to see how I can adapt my own scheduler framework to it... Best regards, Thierry -- https://www.ulthar.net -- http://pyams.readthedocs.io
Le dim. 10 janv. 2021 à 03:09, Theron Luhn <the...@luhn.com> a écrit : > Hi Thierry, > > I’ve added support for periodic tasks in Pyramid Tasks. > https://github.com/luhn/pyramid-tasks#periodic-tasks > > On Jan 6, 2021, at 11:46 PM, Adam Groszer <agros...@gmail.com> wrote: > > You can schedule any task with various options with celery: > > > https://docs.celeryproject.org/en/stable/userguide/calling.html#eta-and-countdown > > On Wednesday, January 6, 2021 at 8:22:28 PM UTC+1 tfl...@gmail.com wrote: > >> I also have to be able to schedule tasks, so I actually include the >> APScheduler package in my projects to make this scheduling. >> Can pyramid-tasks be used also in this context? >> >> >> Le mer. 6 janv. 2021 à 19:44, 'Jonathan Vanasco' via pylons-discuss < >> pylons-...@googlegroups.com> a écrit : >> >>> pyramid_tasks looks very promising! thanks for sharing this. i may use >>> it in the future. i'll have to see if it is compatible with our setup. >>> >>> our celery app is mostly tasks that have been migrated from a twisted >>> daemon. we already ran web spiders and social-media-account importers >>> through twisted, so all the resizing stuff was originally pyramid creating >>> database entries (messages) for twisted to process. although we still use >>> twisted for those tasks (and more!) migrating to celery for these simpler >>> functions has been really productive and timesaving. the code is cleaner >>> and easier to maintain, and troubleshooting/bugfixing is so much simpler. >>> On Wednesday, January 6, 2021 at 12:51:37 PM UTC-5 the...@luhn.com >>> wrote: >>> >>>> Seems like a good time to plug my new project: >>>> https://pypi.org/project/pyramid-tasks/ >>>> >>>> Pyramid and Celery are so dramatically different it’s a pain to >>>> integrate them, and any code shared between them can’t access anything >>>> Pyramid-specific. Got tired of trying to make a square peg fit in a round >>>> hole, so I brought my entire Pyramid application over to my Celery worker. >>>> Having a request object available in a Celery task is incredibly >>>> convenient, and my code is much tidier not having to tiptoe around the >>>> differences in environments. >>>> >>>> On Jan 6, 2021, at 2:14 AM, Adam Groszer <agro...@gmail.com> wrote: >>>> >>>> Hi, >>>> >>>> Yeah celery is the way to go. >>>> There's https://github.com/sontek/pyramid_celery to check out. >>>> Some things to keep in mind is, you'll need a shared DB, good-old ZODB >>>> with filestorage is not enough, because more processes will need to have >>>> access. >>>> Do not store valuable data in the celery queue, consider it ephemeral, >>>> go through a real DB. >>>> >>>> On Tuesday, January 5, 2021 at 10:31:15 PM UTC+1 tfl...@gmail.com >>>> wrote: >>>> >>>>> Hi Jonathan, >>>>> >>>>> Thank you for this description of Celery! >>>>> I'll try to have a look at it if I can get a little time, it seems to >>>>> be a good replacement (probably more reliable!) of my own developments... >>>>> >>>>> Best regards, >>>>> Thierry >>>>> -- >>>>> https://www.ulthar.net -- http://pyams.readthedocs.io >>>>> >>>>> Le mar. 5 janv. 2021 à 22:18, 'Jonathan Vanasco' via pylons-discuss < >>>>> pylons-...@googlegroups.com> a écrit : >>>>> >>>>>> Thierry, >>>>>> >>>>>> That is what I mostly use Celery for (it's also used for generating >>>>>> Reports, ACME SSL Certificates and a few other things). A Pyramid Request >>>>>> will defer a job(s) to Celery via the "Transport Backend" and receive a >>>>>> task/messaging ID. Subsequent requests will poll the Celery "Result >>>>>> Backend" for the status of that job, based on the ID. This way we show >>>>>> the "Still processing!" message to users. >>>>>> >>>>>> Celery is run as a worker with multiple processes. The Celery task >>>>>> manager grabs a task off the queue, then does the resizing, uploads to >>>>>> S3, >>>>>> and notifies the Result Backend when complete. I use Redis for Result >>>>>> and >>>>>> Transport. I think there was once a ZeroMQ integration; there is >>>>>> definitely RabbitMQ integration, it's one of the more popular options. >>>>>> >>>>>> On Monday, January 4, 2021 at 12:25:44 PM UTC-5 tfl...@gmail.com >>>>>> wrote: >>>>>> >>>>>>> Hi Jonathan, >>>>>>> >>>>>>> I didn't have a look at Celery yet, maybe it could work... >>>>>>> My goal is to be able to start long-running tasks (medias files >>>>>>> conversions for example) whose definition is stored in database and >>>>>>> managed >>>>>>> by end users from my main Pyramid application; for this, I create a >>>>>>> dedicated sub-process which is "managed" using ZeroMQ messages (I use >>>>>>> this >>>>>>> because it's also used for other non-Python/Pyramid applications). >>>>>>> Tasks are then started using dedicated threads. >>>>>>> >>>>>>> Everything is OK until now... My only requirement now is to be able >>>>>>> to get access to main Pyramid's registry from my sub-process and it's >>>>>>> threads, and I don't really understand why I get a pointer to >>>>>>> ZCA global registry... :( >>>>>>> >>>>>>> Best regards, >>>>>>> Thierry >>>>>>> -- >>>>>>> https://www.ulthar.net -- http://pyams.readthedocs.io >>>>>>> >>>>>>> Le lun. 4 janv. 2021 à 17:51, 'Jonathan Vanasco' via pylons-discuss < >>>>>>> pylons-...@googlegroups.com> a écrit : >>>>>>> >>>>>>>> Have you considered using Celery for this? I started offloading >>>>>>>> everything related to subprocesses and message queues to it a while >>>>>>>> back, >>>>>>>> and have been much happier. >>>>>>>> >>>>>>>> On Monday, January 4, 2021 at 11:20:46 AM UTC-5 tfl...@gmail.com >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I need to create custom sub-processes from my main Pyramid >>>>>>>>> application; these processes are used to handle "commands" received >>>>>>>>> from >>>>>>>>> ZeroMQ messages. >>>>>>>>> If I use the components registry to register adapters and >>>>>>>>> utilities which are also required in these processes, is there a way >>>>>>>>> to >>>>>>>>> start these processes so that they can "inherit" from the main >>>>>>>>> application >>>>>>>>> registry ? >>>>>>>>> >>>>>>>>> I'm actually using Pyramid 1.10.5, and I generally do a >>>>>>>>> "config.hook_zca()" on application startup before doing >>>>>>>>> the "config.setup_registry()" and "config.scan()". >>>>>>>>> I tried to do the same operations in my sub-process "run" method, >>>>>>>>> but I always get the "global" registry instead of my "webapp" registry >>>>>>>>> which was configured in my main application... >>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>> Thierry >>>>>>>>> -- >>>>>>>>> https://www.ulthar.net -- http://pyams.readthedocs.io >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "pylons-discuss" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to pylons-discus...@googlegroups.com. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com >>>>>>>> <https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "pylons-discuss" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to pylons-discus...@googlegroups.com. >>>>>> >>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/pylons-discuss/5ac83ff1-747f-4df9-994c-ffa48f339013n%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/pylons-discuss/5ac83ff1-747f-4df9-994c-ffa48f339013n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "pylons-discuss" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to pylons-discus...@googlegroups.com. >>>> >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/pylons-discuss/0a0d6985-c5b5-48b6-88f9-cce5d837c583n%40googlegroups.com >>>> <https://groups.google.com/d/msgid/pylons-discuss/0a0d6985-c5b5-48b6-88f9-cce5d837c583n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "pylons-discuss" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to pylons-discus...@googlegroups.com. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/pylons-discuss/aaea1ab5-6d86-475c-8499-9435cdba411bn%40googlegroups.com >>> <https://groups.google.com/d/msgid/pylons-discuss/aaea1ab5-6d86-475c-8499-9435cdba411bn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> > -- > You received this message because you are subscribed to the Google Groups > "pylons-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to pylons-discuss+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/pylons-discuss/627ab125-7fb3-429d-ab43-7340caf75b5en%40googlegroups.com > <https://groups.google.com/d/msgid/pylons-discuss/627ab125-7fb3-429d-ab43-7340caf75b5en%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > > -- > You received this message because you are subscribed to the Google Groups > "pylons-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to pylons-discuss+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/pylons-discuss/06818065-18BB-4203-B598-04962D58372F%40luhn.com > <https://groups.google.com/d/msgid/pylons-discuss/06818065-18BB-4203-B598-04962D58372F%40luhn.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAPX_VWC%2BVBvpxCok3Biw_D1q8rq5e8ApZ8CRYS1HWGsbD5cmEQ%40mail.gmail.com.
