Re: [pylons-discuss] Creating sub-processes from Pyramid application

[Thierry Florac]

Fine!
I'll have to take some time to have a look at Celery and at your package,
to see how I can adapt my own scheduler framework to it...
Best regards,
Thierry
-- 
  https://www.ulthar.net -- http://pyams.readthedocs.io


Le dim. 10 janv. 2021 à 03:09, Theron Luhn  a écrit :

> Hi Thierry,
>
> I’ve added support for periodic tasks in Pyramid Tasks.
> https://github.com/luhn/pyramid-tasks#periodic-tasks
>
> On Jan 6, 2021, at 11:46 PM, Adam Groszer  wrote:
>
> You can schedule any task with various options with celery:
>
>
> https://docs.celeryproject.org/en/stable/userguide/calling.html#eta-and-countdown
>
> On Wednesday, January 6, 2021 at 8:22:28 PM UTC+1 tfl...@gmail.com wrote:
>
>> I also have to be able to schedule tasks, so I actually include the
>> APScheduler package in my projects to make this scheduling.
>> Can pyramid-tasks be used also in this context?
>>
>>
>> Le mer. 6 janv. 2021 à 19:44, 'Jonathan Vanasco' via pylons-discuss <
>> pylons-...@googlegroups.com> a écrit :
>>
>>> pyramid_tasks looks very promising! thanks for sharing this.  i may use
>>> it in the future.  i'll have to see if it is compatible with our setup.
>>>
>>> our celery app is mostly tasks that have been migrated from a twisted
>>> daemon.  we already ran web spiders and social-media-account importers
>>> through twisted, so all the resizing stuff was originally pyramid creating
>>> database entries (messages) for twisted to process.  although we still use
>>> twisted for those tasks (and more!) migrating to celery for these simpler
>>> functions has been really productive and timesaving.  the code is cleaner
>>> and easier to maintain, and troubleshooting/bugfixing is so much simpler.
>>> On Wednesday, January 6, 2021 at 12:51:37 PM UTC-5 the...@luhn.com
>>> wrote:
>>>
 Seems like a good time to plug my new project:
 https://pypi.org/project/pyramid-tasks/

 Pyramid and Celery are so dramatically different it’s a pain to
 integrate them, and any code shared between them can’t access anything
 Pyramid-specific.  Got tired of trying to make a square peg fit in a round
 hole, so I brought my entire Pyramid application over to my Celery worker.
 Having a request object available in a Celery task is incredibly
 convenient, and my code is much tidier not having to tiptoe around the
 differences in environments.

 On Jan 6, 2021, at 2:14 AM, Adam Groszer  wrote:

 Hi,

 Yeah celery is the way to go.
 There's https://github.com/sontek/pyramid_celery to check out.
 Some things to keep in mind is, you'll need a shared DB, good-old ZODB
 with filestorage is not enough, because more processes will need to have
 access.
 Do not store valuable data in the celery queue, consider it ephemeral,
 go through a real DB.

 On Tuesday, January 5, 2021 at 10:31:15 PM UTC+1 tfl...@gmail.com
 wrote:

> Hi Jonathan,
>
> Thank you for this description of Celery!
> I'll try to have a look at it if I can get a little time, it seems to
> be a good replacement (probably more reliable!) of my own developments...
>
> Best regards,
> Thierry
> --
>   https://www.ulthar.net -- http://pyams.readthedocs.io
>
> Le mar. 5 janv. 2021 à 22:18, 'Jonathan Vanasco' via pylons-discuss <
> pylons-...@googlegroups.com> a écrit :
>
>> Thierry,
>>
>> That is what I mostly use Celery for (it's also used for generating
>> Reports, ACME SSL Certificates and a few other things). A Pyramid Request
>> will defer a job(s) to Celery via the "Transport Backend" and receive a
>> task/messaging ID.  Subsequent requests will poll the Celery "Result
>> Backend"  for the status of that job, based on the ID.  This way we show
>> the "Still processing!" message to users.
>>
>> Celery is run as a worker with multiple processes. The Celery task
>> manager grabs a task off the queue, then does the resizing, uploads to 
>> S3,
>> and notifies the Result Backend when complete.  I use Redis for Result 
>> and
>> Transport.  I think there was once a ZeroMQ integration; there is
>> definitely RabbitMQ integration, it's one of the more popular options.
>>
>> On Monday, January 4, 2021 at 12:25:44 PM UTC-5 tfl...@gmail.com
>> wrote:
>>
>>> Hi Jonathan,
>>>
>>> I didn't have a look at Celery yet, maybe it could work...
>>> My goal is to be able to start long-running tasks (medias files
>>> conversions for example) whose definition is stored in database and 
>>> managed
>>> by end users from my main Pyramid application; for this, I create a
>>> dedicated sub-process which is "managed" using ZeroMQ messages (I use 
>>> this
>>> because it's also used for other non-Python/Pyramid applications).
>>> Tasks are then started using dedicated threads.
>>>
>>> Everything is OK until now... My only requirement now is 

Re: [pylons-discuss] Creating sub-processes from Pyramid application

[Theron Luhn]

Hi Thierry,

I’ve added support for periodic tasks in Pyramid Tasks.  
https://github.com/luhn/pyramid-tasks#periodic-tasks

> On Jan 6, 2021, at 11:46 PM, Adam Groszer  wrote:
> 
> You can schedule any task with various options with celery:
> 
> https://docs.celeryproject.org/en/stable/userguide/calling.html#eta-and-countdown
> 
> On Wednesday, January 6, 2021 at 8:22:28 PM UTC+1 tfl...@gmail.com wrote:
> I also have to be able to schedule tasks, so I actually include the 
> APScheduler package in my projects to make this scheduling.
> Can pyramid-tasks be used also in this context?
> 
> 
> Le mer. 6 janv. 2021 à 19:44, 'Jonathan Vanasco' via pylons-discuss 
>  > a écrit :
> pyramid_tasks looks very promising! thanks for sharing this.  i may use it in 
> the future.  i'll have to see if it is compatible with our setup.
> 
> our celery app is mostly tasks that have been migrated from a twisted daemon. 
>  we already ran web spiders and social-media-account importers through 
> twisted, so all the resizing stuff was originally pyramid creating database 
> entries (messages) for twisted to process.  although we still use twisted for 
> those tasks (and more!) migrating to celery for these simpler functions has 
> been really productive and timesaving.  the code is cleaner and easier to 
> maintain, and troubleshooting/bugfixing is so much simpler.
> On Wednesday, January 6, 2021 at 12:51:37 PM UTC-5 the...@luhn.com 
>  wrote:
> Seems like a good time to plug my new project:  
> https://pypi.org/project/pyramid-tasks/ 
> 
> 
> Pyramid and Celery are so dramatically different it’s a pain to integrate 
> them, and any code shared between them can’t access anything 
> Pyramid-specific.  Got tired of trying to make a square peg fit in a round 
> hole, so I brought my entire Pyramid application over to my Celery worker.  
> Having a request object available in a Celery task is incredibly convenient, 
> and my code is much tidier not having to tiptoe around the differences in 
> environments.
> 
> 
>> On Jan 6, 2021, at 2:14 AM, Adam Groszer > wrote:
>> 
> 
>> Hi,
>> 
>> Yeah celery is the way to go.
>> There's https://github.com/sontek/pyramid_celery 
>>  to check out.
>> Some things to keep in mind is, you'll need a shared DB, good-old ZODB with 
>> filestorage is not enough, because more processes will need to have access.
>> Do not store valuable data in the celery queue, consider it ephemeral, go 
>> through a real DB.
>> 
>> On Tuesday, January 5, 2021 at 10:31:15 PM UTC+1 tfl...@gmail.com 
>>  wrote:
>> Hi Jonathan,
>> 
>> Thank you for this description of Celery!
>> I'll try to have a look at it if I can get a little time, it seems to be a 
>> good replacement (probably more reliable!) of my own developments...
>> 
>> Best regards,
>> Thierry
>> -- 
>>   https://www.ulthar.net  -- 
>> http://pyams.readthedocs.io 
>> Le mar. 5 janv. 2021 à 22:18, 'Jonathan Vanasco' via pylons-discuss 
>> > a écrit :
>> Thierry,
>> 
>> That is what I mostly use Celery for (it's also used for generating Reports, 
>> ACME SSL Certificates and a few other things). A Pyramid Request will defer 
>> a job(s) to Celery via the "Transport Backend" and receive a task/messaging 
>> ID.  Subsequent requests will poll the Celery "Result Backend"  for the 
>> status of that job, based on the ID.  This way we show the "Still 
>> processing!" message to users.
>> 
>> Celery is run as a worker with multiple processes. The Celery task manager 
>> grabs a task off the queue, then does the resizing, uploads to S3, and 
>> notifies the Result Backend when complete.  I use Redis for Result and 
>> Transport.  I think there was once a ZeroMQ integration; there is definitely 
>> RabbitMQ integration, it's one of the more popular options.
>> 
>> On Monday, January 4, 2021 at 12:25:44 PM UTC-5 tfl...@gmail.com <> wrote:
>> Hi Jonathan,
>> 
>> I didn't have a look at Celery yet, maybe it could work...
>> My goal is to be able to start long-running tasks (medias files conversions 
>> for example) whose definition is stored in database and managed by end users 
>> from my main Pyramid application; for this, I create a dedicated sub-process 
>> which is "managed" using ZeroMQ messages (I use this because it's also used 
>> for other non-Python/Pyramid applications).
>> Tasks are then started using dedicated threads.
>> 
>> Everything is OK until now... My only requirement now is to be able to get 
>> access to main Pyramid's registry from my sub-process and it's threads, and 
>> I don't really understand why I get a pointer to
>> ZCA global registry...  :(
>> 
>> Best regards,
>> Thierry
>> -- 
>>   https://www.ulthar.net  -- 
>> http://pyams.readthedocs.io 
>> Le lun. 4 janv. 2021 à 17:51, 'Jonathan Vanasco' via pylons-discuss 
>> > a écrit :

Re: [pylons-discuss] Creating sub-processes from Pyramid application

[Adam Groszer]

You can schedule any task with various options with celery:

https://docs.celeryproject.org/en/stable/userguide/calling.html#eta-and-countdown

On Wednesday, January 6, 2021 at 8:22:28 PM UTC+1 tfl...@gmail.com wrote:

> I also have to be able to schedule tasks, so I actually include the 
> APScheduler package in my projects to make this scheduling.
> Can pyramid-tasks be used also in this context?
>
>
> Le mer. 6 janv. 2021 à 19:44, 'Jonathan Vanasco' via pylons-discuss <
> pylons-...@googlegroups.com> a écrit :
>
>> pyramid_tasks looks very promising! thanks for sharing this.  i may use 
>> it in the future.  i'll have to see if it is compatible with our setup.
>>
>> our celery app is mostly tasks that have been migrated from a twisted 
>> daemon.  we already ran web spiders and social-media-account importers 
>> through twisted, so all the resizing stuff was originally pyramid creating 
>> database entries (messages) for twisted to process.  although we still use 
>> twisted for those tasks (and more!) migrating to celery for these simpler 
>> functions has been really productive and timesaving.  the code is cleaner 
>> and easier to maintain, and troubleshooting/bugfixing is so much simpler.
>> On Wednesday, January 6, 2021 at 12:51:37 PM UTC-5 the...@luhn.com wrote:
>>
>>> Seems like a good time to plug my new project:  
>>> https://pypi.org/project/pyramid-tasks/
>>>
>>> Pyramid and Celery are so dramatically different it’s a pain to 
>>> integrate them, and any code shared between them can’t access anything 
>>> Pyramid-specific.  Got tired of trying to make a square peg fit in a round 
>>> hole, so I brought my entire Pyramid application over to my Celery worker.  
>>> Having a request object available in a Celery task is incredibly 
>>> convenient, and my code is much tidier not having to tiptoe around the 
>>> differences in environments.
>>>
>>> On Jan 6, 2021, at 2:14 AM, Adam Groszer  wrote:
>>>
>>> Hi,
>>>
>>> Yeah celery is the way to go.
>>> There's https://github.com/sontek/pyramid_celery to check out.
>>> Some things to keep in mind is, you'll need a shared DB, good-old ZODB 
>>> with filestorage is not enough, because more processes will need to have 
>>> access.
>>> Do not store valuable data in the celery queue, consider it ephemeral, 
>>> go through a real DB.
>>>
>>> On Tuesday, January 5, 2021 at 10:31:15 PM UTC+1 tfl...@gmail.com wrote:
>>>
 Hi Jonathan,

 Thank you for this description of Celery!
 I'll try to have a look at it if I can get a little time, it seems to 
 be a good replacement (probably more reliable!) of my own developments...

 Best regards,
 Thierry
 -- 
   https://www.ulthar.net -- http://pyams.readthedocs.io

 Le mar. 5 janv. 2021 à 22:18, 'Jonathan Vanasco' via pylons-discuss <
 pylons-...@googlegroups.com> a écrit :

> Thierry,
>
> That is what I mostly use Celery for (it's also used for generating 
> Reports, ACME SSL Certificates and a few other things). A Pyramid Request 
> will defer a job(s) to Celery via the "Transport Backend" and receive a 
> task/messaging ID.  Subsequent requests will poll the Celery "Result 
> Backend"  for the status of that job, based on the ID.  This way we show 
> the "Still processing!" message to users.
>
> Celery is run as a worker with multiple processes. The Celery task 
> manager grabs a task off the queue, then does the resizing, uploads to 
> S3, 
> and notifies the Result Backend when complete.  I use Redis for Result 
> and 
> Transport.  I think there was once a ZeroMQ integration; there is 
> definitely RabbitMQ integration, it's one of the more popular options.
>
> On Monday, January 4, 2021 at 12:25:44 PM UTC-5 tfl...@gmail.com 
> wrote:
>
>> Hi Jonathan,
>>
>> I didn't have a look at Celery yet, maybe it could work...
>> My goal is to be able to start long-running tasks (medias files 
>> conversions for example) whose definition is stored in database and 
>> managed 
>> by end users from my main Pyramid application; for this, I create a 
>> dedicated sub-process which is "managed" using ZeroMQ messages (I use 
>> this 
>> because it's also used for other non-Python/Pyramid applications).
>> Tasks are then started using dedicated threads.
>>
>> Everything is OK until now... My only requirement now is to be able 
>> to get access to main Pyramid's registry from my sub-process and it's 
>> threads, and I don't really understand why I get a pointer to
>> ZCA global registry...  :(
>>
>> Best regards,
>> Thierry
>> -- 
>>   https://www.ulthar.net -- http://pyams.readthedocs.io
>>
>> Le lun. 4 janv. 2021 à 17:51, 'Jonathan Vanasco' via pylons-discuss <
>> pylons-...@googlegroups.com> a écrit :
>>
>>> Have you considered using Celery for this?  I started offloading 
>>> everything 

Re: [pylons-discuss] Creating sub-processes from Pyramid application

[Steve Piercy]

On 1/6/21 9:51 AM, Theron Luhn wrote:
> Seems like a good time to plug my new project:  
> https://pypi.org/project/pyramid-tasks/ 
> 
> 
> Pyramid and Celery are so dramatically different it’s a pain to integrate 
> them, and any code shared between them can’t access anything 
> Pyramid-specific.  Got tired of trying to make a square peg fit in a round 
> hole, so I brought my entire Pyramid application over to my Celery worker.  
> Having a request object available in a Celery task is incredibly convenient, 
> and my code is much tidier not having to tiptoe around the differences in 
> environments.

I think this would be a good add-on to add to Extending Pyramid.

https://trypyramid.com/extending-pyramid.html

If you submit a PR, we'll merge it and deploy.

Also if you Tweet to @PylonsProject, I'll retweet it.

--steve

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/9f348d0b-12b3-f029-7782-56be25ea0a92%40gmail.com.

Re: [pylons-discuss] Creating sub-processes from Pyramid application

[Theron Luhn]

I’m not familiar with APScheduler.  celery-beat seems to be the de facto way to 
run scheduled tasks in Celery.  
https://docs.celeryproject.org/en/stable/userguide/periodic-tasks.html 


celery-beat should be compatible with pyramid-tasks, but I haven’t tested or 
documented it yet.

I personally use AWS CloudWatch Events -> SNS -> HTTP to my app -> 
validate_aws_sns_message -> trigger Celery task.

> On Jan 6, 2021, at 11:22 AM, Thierry Florac  wrote:
> 
> I also have to be able to schedule tasks, so I actually include the 
> APScheduler package in my projects to make this scheduling.
> Can pyramid-tasks be used also in this context?
> 
> 
> Le mer. 6 janv. 2021 à 19:44, 'Jonathan Vanasco' via pylons-discuss 
> mailto:pylons-discuss@googlegroups.com>> a 
> écrit :
> pyramid_tasks looks very promising! thanks for sharing this.  i may use it in 
> the future.  i'll have to see if it is compatible with our setup.
> 
> our celery app is mostly tasks that have been migrated from a twisted daemon. 
>  we already ran web spiders and social-media-account importers through 
> twisted, so all the resizing stuff was originally pyramid creating database 
> entries (messages) for twisted to process.  although we still use twisted for 
> those tasks (and more!) migrating to celery for these simpler functions has 
> been really productive and timesaving.  the code is cleaner and easier to 
> maintain, and troubleshooting/bugfixing is so much simpler.
> On Wednesday, January 6, 2021 at 12:51:37 PM UTC-5 the...@luhn.com 
>  wrote:
> Seems like a good time to plug my new project:  
> https://pypi.org/project/pyramid-tasks/ 
> 
> 
> Pyramid and Celery are so dramatically different it’s a pain to integrate 
> them, and any code shared between them can’t access anything 
> Pyramid-specific.  Got tired of trying to make a square peg fit in a round 
> hole, so I brought my entire Pyramid application over to my Celery worker.  
> Having a request object available in a Celery task is incredibly convenient, 
> and my code is much tidier not having to tiptoe around the differences in 
> environments.
> 
> 
>> On Jan 6, 2021, at 2:14 AM, Adam Groszer > wrote:
>> 
> 
>> Hi,
>> 
>> Yeah celery is the way to go.
>> There's https://github.com/sontek/pyramid_celery 
>>  to check out.
>> Some things to keep in mind is, you'll need a shared DB, good-old ZODB with 
>> filestorage is not enough, because more processes will need to have access.
>> Do not store valuable data in the celery queue, consider it ephemeral, go 
>> through a real DB.
>> 
>> On Tuesday, January 5, 2021 at 10:31:15 PM UTC+1 tfl...@gmail.com 
>>  wrote:
>> Hi Jonathan,
>> 
>> Thank you for this description of Celery!
>> I'll try to have a look at it if I can get a little time, it seems to be a 
>> good replacement (probably more reliable!) of my own developments...
>> 
>> Best regards,
>> Thierry
>> -- 
>>   https://www.ulthar.net  -- 
>> http://pyams.readthedocs.io 
>> Le mar. 5 janv. 2021 à 22:18, 'Jonathan Vanasco' via pylons-discuss 
>> > a écrit :
>> Thierry,
>> 
>> That is what I mostly use Celery for (it's also used for generating Reports, 
>> ACME SSL Certificates and a few other things). A Pyramid Request will defer 
>> a job(s) to Celery via the "Transport Backend" and receive a task/messaging 
>> ID.  Subsequent requests will poll the Celery "Result Backend"  for the 
>> status of that job, based on the ID.  This way we show the "Still 
>> processing!" message to users.
>> 
>> Celery is run as a worker with multiple processes. The Celery task manager 
>> grabs a task off the queue, then does the resizing, uploads to S3, and 
>> notifies the Result Backend when complete.  I use Redis for Result and 
>> Transport.  I think there was once a ZeroMQ integration; there is definitely 
>> RabbitMQ integration, it's one of the more popular options.
>> 
>> On Monday, January 4, 2021 at 12:25:44 PM UTC-5 tfl...@gmail.com <> wrote:
>> Hi Jonathan,
>> 
>> I didn't have a look at Celery yet, maybe it could work...
>> My goal is to be able to start long-running tasks (medias files conversions 
>> for example) whose definition is stored in database and managed by end users 
>> from my main Pyramid application; for this, I create a dedicated sub-process 
>> which is "managed" using ZeroMQ messages (I use this because it's also used 
>> for other non-Python/Pyramid applications).
>> Tasks are then started using dedicated threads.
>> 
>> Everything is OK until now... My only requirement now is to be able to get 
>> access to main Pyramid's registry from my sub-process and it's threads, and 
>> I don't really understand why I get a pointer to
>> ZCA global registry...  :(
>> 
>> Best regards,
>> Thierry
>> -- 
>>   

Re: [pylons-discuss] Creating sub-processes from Pyramid application

[Thierry Florac]

I also have to be able to schedule tasks, so I actually include the
APScheduler package in my projects to make this scheduling.
Can pyramid-tasks be used also in this context?


Le mer. 6 janv. 2021 à 19:44, 'Jonathan Vanasco' via pylons-discuss <
pylons-discuss@googlegroups.com> a écrit :

> pyramid_tasks looks very promising! thanks for sharing this.  i may use it
> in the future.  i'll have to see if it is compatible with our setup.
>
> our celery app is mostly tasks that have been migrated from a twisted
> daemon.  we already ran web spiders and social-media-account importers
> through twisted, so all the resizing stuff was originally pyramid creating
> database entries (messages) for twisted to process.  although we still use
> twisted for those tasks (and more!) migrating to celery for these simpler
> functions has been really productive and timesaving.  the code is cleaner
> and easier to maintain, and troubleshooting/bugfixing is so much simpler.
> On Wednesday, January 6, 2021 at 12:51:37 PM UTC-5 the...@luhn.com wrote:
>
>> Seems like a good time to plug my new project:
>> https://pypi.org/project/pyramid-tasks/
>>
>> Pyramid and Celery are so dramatically different it’s a pain to integrate
>> them, and any code shared between them can’t access anything
>> Pyramid-specific.  Got tired of trying to make a square peg fit in a round
>> hole, so I brought my entire Pyramid application over to my Celery worker.
>> Having a request object available in a Celery task is incredibly
>> convenient, and my code is much tidier not having to tiptoe around the
>> differences in environments.
>>
>> On Jan 6, 2021, at 2:14 AM, Adam Groszer  wrote:
>>
>> Hi,
>>
>> Yeah celery is the way to go.
>> There's https://github.com/sontek/pyramid_celery to check out.
>> Some things to keep in mind is, you'll need a shared DB, good-old ZODB
>> with filestorage is not enough, because more processes will need to have
>> access.
>> Do not store valuable data in the celery queue, consider it ephemeral, go
>> through a real DB.
>>
>> On Tuesday, January 5, 2021 at 10:31:15 PM UTC+1 tfl...@gmail.com wrote:
>>
>>> Hi Jonathan,
>>>
>>> Thank you for this description of Celery!
>>> I'll try to have a look at it if I can get a little time, it seems to be
>>> a good replacement (probably more reliable!) of my own developments...
>>>
>>> Best regards,
>>> Thierry
>>> --
>>>   https://www.ulthar.net -- http://pyams.readthedocs.io
>>>
>>> Le mar. 5 janv. 2021 à 22:18, 'Jonathan Vanasco' via pylons-discuss <
>>> pylons-...@googlegroups.com> a écrit :
>>>
 Thierry,

 That is what I mostly use Celery for (it's also used for generating
 Reports, ACME SSL Certificates and a few other things). A Pyramid Request
 will defer a job(s) to Celery via the "Transport Backend" and receive a
 task/messaging ID.  Subsequent requests will poll the Celery "Result
 Backend"  for the status of that job, based on the ID.  This way we show
 the "Still processing!" message to users.

 Celery is run as a worker with multiple processes. The Celery task
 manager grabs a task off the queue, then does the resizing, uploads to S3,
 and notifies the Result Backend when complete.  I use Redis for Result and
 Transport.  I think there was once a ZeroMQ integration; there is
 definitely RabbitMQ integration, it's one of the more popular options.

 On Monday, January 4, 2021 at 12:25:44 PM UTC-5 tfl...@gmail.com wrote:

> Hi Jonathan,
>
> I didn't have a look at Celery yet, maybe it could work...
> My goal is to be able to start long-running tasks (medias files
> conversions for example) whose definition is stored in database and 
> managed
> by end users from my main Pyramid application; for this, I create a
> dedicated sub-process which is "managed" using ZeroMQ messages (I use this
> because it's also used for other non-Python/Pyramid applications).
> Tasks are then started using dedicated threads.
>
> Everything is OK until now... My only requirement now is to be able to
> get access to main Pyramid's registry from my sub-process and it's 
> threads,
> and I don't really understand why I get a pointer to
> ZCA global registry...  :(
>
> Best regards,
> Thierry
> --
>   https://www.ulthar.net -- http://pyams.readthedocs.io
>
> Le lun. 4 janv. 2021 à 17:51, 'Jonathan Vanasco' via pylons-discuss <
> pylons-...@googlegroups.com> a écrit :
>
>> Have you considered using Celery for this?  I started offloading
>> everything related to subprocesses and message queues to it a while back,
>> and have been much happier.
>>
>> On Monday, January 4, 2021 at 11:20:46 AM UTC-5 tfl...@gmail.com
>> wrote:
>>
>>> Hi,
>>>
>>> I need to create custom sub-processes from my main Pyramid
>>> application; these processes are used to handle "commands" received from
>>> ZeroMQ 

Re: [pylons-discuss] Creating sub-processes from Pyramid application

['Jonathan Vanasco' via pylons-discuss]

pyramid_tasks looks very promising! thanks for sharing this.  i may use it 
in the future.  i'll have to see if it is compatible with our setup.

our celery app is mostly tasks that have been migrated from a twisted 
daemon.  we already ran web spiders and social-media-account importers 
through twisted, so all the resizing stuff was originally pyramid creating 
database entries (messages) for twisted to process.  although we still use 
twisted for those tasks (and more!) migrating to celery for these simpler 
functions has been really productive and timesaving.  the code is cleaner 
and easier to maintain, and troubleshooting/bugfixing is so much simpler.
On Wednesday, January 6, 2021 at 12:51:37 PM UTC-5 the...@luhn.com wrote:

> Seems like a good time to plug my new project:  
> https://pypi.org/project/pyramid-tasks/
>
> Pyramid and Celery are so dramatically different it’s a pain to integrate 
> them, and any code shared between them can’t access anything 
> Pyramid-specific.  Got tired of trying to make a square peg fit in a round 
> hole, so I brought my entire Pyramid application over to my Celery worker. 
>  Having a request object available in a Celery task is incredibly 
> convenient, and my code is much tidier not having to tiptoe around the 
> differences in environments.
>
> On Jan 6, 2021, at 2:14 AM, Adam Groszer  wrote:
>
> Hi,
>
> Yeah celery is the way to go.
> There's https://github.com/sontek/pyramid_celery to check out.
> Some things to keep in mind is, you'll need a shared DB, good-old ZODB 
> with filestorage is not enough, because more processes will need to have 
> access.
> Do not store valuable data in the celery queue, consider it ephemeral, go 
> through a real DB.
>
> On Tuesday, January 5, 2021 at 10:31:15 PM UTC+1 tfl...@gmail.com wrote:
>
>> Hi Jonathan,
>>
>> Thank you for this description of Celery!
>> I'll try to have a look at it if I can get a little time, it seems to be 
>> a good replacement (probably more reliable!) of my own developments...
>>
>> Best regards,
>> Thierry
>> -- 
>>   https://www.ulthar.net -- http://pyams.readthedocs.io
>>
>> Le mar. 5 janv. 2021 à 22:18, 'Jonathan Vanasco' via pylons-discuss <
>> pylons-...@googlegroups.com> a écrit :
>>
>>> Thierry,
>>>
>>> That is what I mostly use Celery for (it's also used for generating 
>>> Reports, ACME SSL Certificates and a few other things). A Pyramid Request 
>>> will defer a job(s) to Celery via the "Transport Backend" and receive a 
>>> task/messaging ID.  Subsequent requests will poll the Celery "Result 
>>> Backend"  for the status of that job, based on the ID.  This way we show 
>>> the "Still processing!" message to users.
>>>
>>> Celery is run as a worker with multiple processes. The Celery task 
>>> manager grabs a task off the queue, then does the resizing, uploads to S3, 
>>> and notifies the Result Backend when complete.  I use Redis for Result and 
>>> Transport.  I think there was once a ZeroMQ integration; there is 
>>> definitely RabbitMQ integration, it's one of the more popular options.
>>>
>>> On Monday, January 4, 2021 at 12:25:44 PM UTC-5 tfl...@gmail.com wrote:
>>>
 Hi Jonathan,

 I didn't have a look at Celery yet, maybe it could work...
 My goal is to be able to start long-running tasks (medias files 
 conversions for example) whose definition is stored in database and 
 managed 
 by end users from my main Pyramid application; for this, I create a 
 dedicated sub-process which is "managed" using ZeroMQ messages (I use this 
 because it's also used for other non-Python/Pyramid applications).
 Tasks are then started using dedicated threads.

 Everything is OK until now... My only requirement now is to be able to 
 get access to main Pyramid's registry from my sub-process and it's 
 threads, 
 and I don't really understand why I get a pointer to
 ZCA global registry...  :(

 Best regards,
 Thierry
 -- 
   https://www.ulthar.net -- http://pyams.readthedocs.io

 Le lun. 4 janv. 2021 à 17:51, 'Jonathan Vanasco' via pylons-discuss <
 pylons-...@googlegroups.com> a écrit :

> Have you considered using Celery for this?  I started offloading 
> everything related to subprocesses and message queues to it a while back, 
> and have been much happier.
>
> On Monday, January 4, 2021 at 11:20:46 AM UTC-5 tfl...@gmail.com 
> wrote:
>
>> Hi,
>>
>> I need to create custom sub-processes from my main Pyramid 
>> application; these processes are used to handle "commands" received from 
>> ZeroMQ messages.
>> If I use the components registry to register adapters and utilities 
>> which are also required in these processes, is there a way to start 
>> these 
>> processes so that they can "inherit" from the main application registry ?
>>
>> I'm actually using Pyramid 1.10.5, and I generally do a 
>> "config.hook_zca()" on 

Re: [pylons-discuss] Creating sub-processes from Pyramid application

[Theron Luhn]

Seems like a good time to plug my new project:  
https://pypi.org/project/pyramid-tasks/ 


Pyramid and Celery are so dramatically different it’s a pain to integrate them, 
and any code shared between them can’t access anything Pyramid-specific.  Got 
tired of trying to make a square peg fit in a round hole, so I brought my 
entire Pyramid application over to my Celery worker.  Having a request object 
available in a Celery task is incredibly convenient, and my code is much tidier 
not having to tiptoe around the differences in environments.

> On Jan 6, 2021, at 2:14 AM, Adam Groszer  wrote:
> 
> Hi,
> 
> Yeah celery is the way to go.
> There's https://github.com/sontek/pyramid_celery to check out.
> Some things to keep in mind is, you'll need a shared DB, good-old ZODB with 
> filestorage is not enough, because more processes will need to have access.
> Do not store valuable data in the celery queue, consider it ephemeral, go 
> through a real DB.
> 
> On Tuesday, January 5, 2021 at 10:31:15 PM UTC+1 tfl...@gmail.com wrote:
> Hi Jonathan,
> 
> Thank you for this description of Celery!
> I'll try to have a look at it if I can get a little time, it seems to be a 
> good replacement (probably more reliable!) of my own developments...
> 
> Best regards,
> Thierry
> -- 
>   https://www.ulthar.net  -- 
> http://pyams.readthedocs.io 
> Le mar. 5 janv. 2021 à 22:18, 'Jonathan Vanasco' via pylons-discuss 
>  > a écrit :
> Thierry,
> 
> That is what I mostly use Celery for (it's also used for generating Reports, 
> ACME SSL Certificates and a few other things). A Pyramid Request will defer a 
> job(s) to Celery via the "Transport Backend" and receive a task/messaging ID. 
>  Subsequent requests will poll the Celery "Result Backend"  for the status of 
> that job, based on the ID.  This way we show the "Still processing!" message 
> to users.
> 
> Celery is run as a worker with multiple processes. The Celery task manager 
> grabs a task off the queue, then does the resizing, uploads to S3, and 
> notifies the Result Backend when complete.  I use Redis for Result and 
> Transport.  I think there was once a ZeroMQ integration; there is definitely 
> RabbitMQ integration, it's one of the more popular options.
> 
> On Monday, January 4, 2021 at 12:25:44 PM UTC-5 tfl...@gmail.com 
>  wrote:
> Hi Jonathan,
> 
> I didn't have a look at Celery yet, maybe it could work...
> My goal is to be able to start long-running tasks (medias files conversions 
> for example) whose definition is stored in database and managed by end users 
> from my main Pyramid application; for this, I create a dedicated sub-process 
> which is "managed" using ZeroMQ messages (I use this because it's also used 
> for other non-Python/Pyramid applications).
> Tasks are then started using dedicated threads.
> 
> Everything is OK until now... My only requirement now is to be able to get 
> access to main Pyramid's registry from my sub-process and it's threads, and I 
> don't really understand why I get a pointer to
> ZCA global registry...  :(
> 
> Best regards,
> Thierry
> -- 
>   https://www.ulthar.net  -- 
> http://pyams.readthedocs.io 
> Le lun. 4 janv. 2021 à 17:51, 'Jonathan Vanasco' via pylons-discuss 
> > a écrit :
> Have you considered using Celery for this?  I started offloading everything 
> related to subprocesses and message queues to it a while back, and have been 
> much happier.
> 
> On Monday, January 4, 2021 at 11:20:46 AM UTC-5 tfl...@gmail.com <> wrote:
> Hi,
> 
> I need to create custom sub-processes from my main Pyramid application; these 
> processes are used to handle "commands" received from ZeroMQ messages.
> If I use the components registry to register adapters and utilities which are 
> also required in these processes, is there a way to start these processes so 
> that they can "inherit" from the main application registry ?
> 
> I'm actually using Pyramid 1.10.5, and I generally do a "config.hook_zca()" 
> on application startup before doing the "config.setup_registry()" and 
> "config.scan()".
> I tried to do the same operations in my sub-process "run" method, but I 
> always get the "global" registry instead of my "webapp" registry which was 
> configured in my main application...
> 
> Best regards,
> Thierry
> -- 
>   https://www.ulthar.net  -- 
> http://pyams.readthedocs.io 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to pylons-discus...@googlegroups.com <>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com
>  
> 

[pylons-discuss] Creating sub-processes from Pyramid application

[Thierry Florac]

Hi,

I need to create custom sub-processes from my main Pyramid application;
these processes are used to handle "commands" received from ZeroMQ messages.
If I use the components registry to register adapters and utilities which
are also required in these processes, is there a way to start these
processes so that they can "inherit" from the main application registry ?

I'm actually using Pyramid 1.10.5, and I generally do a "config.hook_zca()"
on application startup before doing the "config.setup_registry()" and
"config.scan()".
I tried to do the same operations in my sub-process "run" method, but I
always get the "global" registry instead of my "webapp" registry which was
configured in my main application...

Best regards,
Thierry
-- 
  https://www.ulthar.net -- http://pyams.readthedocs.io

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAPX_VWD%2B65yeLUG2%3DSijVbTU5dJewnOJX%2BJ7bWSfW%3DjmNPCyLw%40mail.gmail.com.