Re: [pylons-discuss] Re: Creating sub-processes from Pyramid application

[Adam Groszer]

Hi,

Yeah celery is the way to go.
There's https://github.com/sontek/pyramid_celery to check out.
Some things to keep in mind is, you'll need a shared DB, good-old ZODB with 
filestorage is not enough, because more processes will need to have access.
Do not store valuable data in the celery queue, consider it ephemeral, go 
through a real DB.

On Tuesday, January 5, 2021 at 10:31:15 PM UTC+1 tfl...@gmail.com wrote:

> Hi Jonathan,
>
> Thank you for this description of Celery!
> I'll try to have a look at it if I can get a little time, it seems to be a 
> good replacement (probably more reliable!) of my own developments...
>
> Best regards,
> Thierry
> -- 
>   https://www.ulthar.net -- http://pyams.readthedocs.io
>
> Le mar. 5 janv. 2021 à 22:18, 'Jonathan Vanasco' via pylons-discuss <
> pylons-...@googlegroups.com> a écrit :
>
>> Thierry,
>>
>> That is what I mostly use Celery for (it's also used for generating 
>> Reports, ACME SSL Certificates and a few other things). A Pyramid Request 
>> will defer a job(s) to Celery via the "Transport Backend" and receive a 
>> task/messaging ID.  Subsequent requests will poll the Celery "Result 
>> Backend"  for the status of that job, based on the ID.  This way we show 
>> the "Still processing!" message to users.
>>
>> Celery is run as a worker with multiple processes. The Celery task 
>> manager grabs a task off the queue, then does the resizing, uploads to S3, 
>> and notifies the Result Backend when complete.  I use Redis for Result and 
>> Transport.  I think there was once a ZeroMQ integration; there is 
>> definitely RabbitMQ integration, it's one of the more popular options.
>>
>> On Monday, January 4, 2021 at 12:25:44 PM UTC-5 tfl...@gmail.com wrote:
>>
>>> Hi Jonathan,
>>>
>>> I didn't have a look at Celery yet, maybe it could work...
>>> My goal is to be able to start long-running tasks (medias files 
>>> conversions for example) whose definition is stored in database and managed 
>>> by end users from my main Pyramid application; for this, I create a 
>>> dedicated sub-process which is "managed" using ZeroMQ messages (I use this 
>>> because it's also used for other non-Python/Pyramid applications).
>>> Tasks are then started using dedicated threads.
>>>
>>> Everything is OK until now... My only requirement now is to be able to 
>>> get access to main Pyramid's registry from my sub-process and it's threads, 
>>> and I don't really understand why I get a pointer to
>>> ZCA global registry...  :(
>>>
>>> Best regards,
>>> Thierry
>>> -- 
>>>   https://www.ulthar.net -- http://pyams.readthedocs.io
>>>
>>> Le lun. 4 janv. 2021 à 17:51, 'Jonathan Vanasco' via pylons-discuss <
>>> pylons-...@googlegroups.com> a écrit :
>>>
 Have you considered using Celery for this?  I started offloading 
 everything related to subprocesses and message queues to it a while back, 
 and have been much happier.

 On Monday, January 4, 2021 at 11:20:46 AM UTC-5 tfl...@gmail.com wrote:

> Hi,
>
> I need to create custom sub-processes from my main Pyramid 
> application; these processes are used to handle "commands" received from 
> ZeroMQ messages.
> If I use the components registry to register adapters and utilities 
> which are also required in these processes, is there a way to start these 
> processes so that they can "inherit" from the main application registry ?
>
> I'm actually using Pyramid 1.10.5, and I generally do a 
> "config.hook_zca()" on application startup before doing 
> the "config.setup_registry()" and "config.scan()".
> I tried to do the same operations in my sub-process "run" method, but 
> I always get the "global" registry instead of my "webapp" registry which 
> was configured in my main application...
>
> Best regards,
> Thierry
> -- 
>   https://www.ulthar.net -- http://pyams.readthedocs.io
>
 -- 
 You received this message because you are subscribed to the Google 
 Groups "pylons-discuss" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to pylons-discus...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com
  
 
 .

>>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "pylons-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to pylons-discus...@googlegroups.com.
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/pylons-discuss/5ac83ff1-747f-4df9-994c-ffa48f339013n%40googlegroups.com
>>  
>> 

Re: [pylons-discuss] Re: Creating sub-processes from Pyramid application

[Thierry Florac]

Hi Jonathan,

Thank you for this description of Celery!
I'll try to have a look at it if I can get a little time, it seems to be a
good replacement (probably more reliable!) of my own developments...

Best regards,
Thierry
-- 
  https://www.ulthar.net -- http://pyams.readthedocs.io


Le mar. 5 janv. 2021 à 22:18, 'Jonathan Vanasco' via pylons-discuss <
pylons-discuss@googlegroups.com> a écrit :

> Thierry,
>
> That is what I mostly use Celery for (it's also used for generating
> Reports, ACME SSL Certificates and a few other things). A Pyramid Request
> will defer a job(s) to Celery via the "Transport Backend" and receive a
> task/messaging ID.  Subsequent requests will poll the Celery "Result
> Backend"  for the status of that job, based on the ID.  This way we show
> the "Still processing!" message to users.
>
> Celery is run as a worker with multiple processes. The Celery task manager
> grabs a task off the queue, then does the resizing, uploads to S3, and
> notifies the Result Backend when complete.  I use Redis for Result and
> Transport.  I think there was once a ZeroMQ integration; there is
> definitely RabbitMQ integration, it's one of the more popular options.
>
> On Monday, January 4, 2021 at 12:25:44 PM UTC-5 tfl...@gmail.com wrote:
>
>> Hi Jonathan,
>>
>> I didn't have a look at Celery yet, maybe it could work...
>> My goal is to be able to start long-running tasks (medias files
>> conversions for example) whose definition is stored in database and managed
>> by end users from my main Pyramid application; for this, I create a
>> dedicated sub-process which is "managed" using ZeroMQ messages (I use this
>> because it's also used for other non-Python/Pyramid applications).
>> Tasks are then started using dedicated threads.
>>
>> Everything is OK until now... My only requirement now is to be able to
>> get access to main Pyramid's registry from my sub-process and it's threads,
>> and I don't really understand why I get a pointer to
>> ZCA global registry...  :(
>>
>> Best regards,
>> Thierry
>> --
>>   https://www.ulthar.net -- http://pyams.readthedocs.io
>>
>> Le lun. 4 janv. 2021 à 17:51, 'Jonathan Vanasco' via pylons-discuss <
>> pylons-...@googlegroups.com> a écrit :
>>
>>> Have you considered using Celery for this?  I started offloading
>>> everything related to subprocesses and message queues to it a while back,
>>> and have been much happier.
>>>
>>> On Monday, January 4, 2021 at 11:20:46 AM UTC-5 tfl...@gmail.com wrote:
>>>
 Hi,

 I need to create custom sub-processes from my main Pyramid application;
 these processes are used to handle "commands" received from ZeroMQ 
 messages.
 If I use the components registry to register adapters and utilities
 which are also required in these processes, is there a way to start these
 processes so that they can "inherit" from the main application registry ?

 I'm actually using Pyramid 1.10.5, and I generally do a
 "config.hook_zca()" on application startup before doing
 the "config.setup_registry()" and "config.scan()".
 I tried to do the same operations in my sub-process "run" method, but I
 always get the "global" registry instead of my "webapp" registry which was
 configured in my main application...

 Best regards,
 Thierry
 --
   https://www.ulthar.net -- http://pyams.readthedocs.io

>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "pylons-discuss" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to pylons-discus...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pylons-discuss+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/5ac83ff1-747f-4df9-994c-ffa48f339013n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAPX_VWC46yjf55pcMEYKmjv3%3DpwBP8%2BvTs%2B_vdn5uBySpLHNsg%40mail.gmail.com.

Re: [pylons-discuss] Re: Creating sub-processes from Pyramid application

['Jonathan Vanasco' via pylons-discuss]

Thierry,

That is what I mostly use Celery for (it's also used for generating 
Reports, ACME SSL Certificates and a few other things). A Pyramid Request 
will defer a job(s) to Celery via the "Transport Backend" and receive a 
task/messaging ID.  Subsequent requests will poll the Celery "Result 
Backend"  for the status of that job, based on the ID.  This way we show 
the "Still processing!" message to users.

Celery is run as a worker with multiple processes. The Celery task manager 
grabs a task off the queue, then does the resizing, uploads to S3, and 
notifies the Result Backend when complete.  I use Redis for Result and 
Transport.  I think there was once a ZeroMQ integration; there is 
definitely RabbitMQ integration, it's one of the more popular options.

On Monday, January 4, 2021 at 12:25:44 PM UTC-5 tfl...@gmail.com wrote:

> Hi Jonathan,
>
> I didn't have a look at Celery yet, maybe it could work...
> My goal is to be able to start long-running tasks (medias files 
> conversions for example) whose definition is stored in database and managed 
> by end users from my main Pyramid application; for this, I create a 
> dedicated sub-process which is "managed" using ZeroMQ messages (I use this 
> because it's also used for other non-Python/Pyramid applications).
> Tasks are then started using dedicated threads.
>
> Everything is OK until now... My only requirement now is to be able to get 
> access to main Pyramid's registry from my sub-process and it's threads, and 
> I don't really understand why I get a pointer to
> ZCA global registry...  :(
>
> Best regards,
> Thierry
> -- 
>   https://www.ulthar.net -- http://pyams.readthedocs.io
>
> Le lun. 4 janv. 2021 à 17:51, 'Jonathan Vanasco' via pylons-discuss <
> pylons-...@googlegroups.com> a écrit :
>
>> Have you considered using Celery for this?  I started offloading 
>> everything related to subprocesses and message queues to it a while back, 
>> and have been much happier.
>>
>> On Monday, January 4, 2021 at 11:20:46 AM UTC-5 tfl...@gmail.com wrote:
>>
>>> Hi,
>>>
>>> I need to create custom sub-processes from my main Pyramid application; 
>>> these processes are used to handle "commands" received from ZeroMQ messages.
>>> If I use the components registry to register adapters and utilities 
>>> which are also required in these processes, is there a way to start these 
>>> processes so that they can "inherit" from the main application registry ?
>>>
>>> I'm actually using Pyramid 1.10.5, and I generally do a 
>>> "config.hook_zca()" on application startup before doing 
>>> the "config.setup_registry()" and "config.scan()".
>>> I tried to do the same operations in my sub-process "run" method, but I 
>>> always get the "global" registry instead of my "webapp" registry which was 
>>> configured in my main application...
>>>
>>> Best regards,
>>> Thierry
>>> -- 
>>>   https://www.ulthar.net -- http://pyams.readthedocs.io
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "pylons-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to pylons-discus...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/5ac83ff1-747f-4df9-994c-ffa48f339013n%40googlegroups.com.

Re: [pylons-discuss] Re: Creating sub-processes from Pyramid application

[Thierry Florac]

Hi (again)!

I think it's finally OK!
I forgot ZCA hook is based on Pyramid's "current registry", which is bound
to a thread-local variable, so that's why I have to push them on
threadlocal manager in my threads "run" methods (and not in the constructor
as I have tried before)!!
Maybe there is another better way to handle that anyway, so any advice is
always welcome!

Best regards,
Thierry
-- 
  https://www.ulthar.net -- http://pyams.readthedocs.io


Le lun. 4 janv. 2021 à 18:25, Thierry Florac  a écrit :

> Hi Jonathan,
>
> I didn't have a look at Celery yet, maybe it could work...
> My goal is to be able to start long-running tasks (medias files
> conversions for example) whose definition is stored in database and managed
> by end users from my main Pyramid application; for this, I create a
> dedicated sub-process which is "managed" using ZeroMQ messages (I use this
> because it's also used for other non-Python/Pyramid applications).
> Tasks are then started using dedicated threads.
>
> Everything is OK until now... My only requirement now is to be able to get
> access to main Pyramid's registry from my sub-process and it's threads, and
> I don't really understand why I get a pointer to
> ZCA global registry...  :(
>
> Best regards,
> Thierry
> --
>   https://www.ulthar.net -- http://pyams.readthedocs.io
>
>
> Le lun. 4 janv. 2021 à 17:51, 'Jonathan Vanasco' via pylons-discuss <
> pylons-discuss@googlegroups.com> a écrit :
>
>> Have you considered using Celery for this?  I started offloading
>> everything related to subprocesses and message queues to it a while back,
>> and have been much happier.
>>
>> On Monday, January 4, 2021 at 11:20:46 AM UTC-5 tfl...@gmail.com wrote:
>>
>>> Hi,
>>>
>>> I need to create custom sub-processes from my main Pyramid application;
>>> these processes are used to handle "commands" received from ZeroMQ messages.
>>> If I use the components registry to register adapters and utilities
>>> which are also required in these processes, is there a way to start these
>>> processes so that they can "inherit" from the main application registry ?
>>>
>>> I'm actually using Pyramid 1.10.5, and I generally do a
>>> "config.hook_zca()" on application startup before doing
>>> the "config.setup_registry()" and "config.scan()".
>>> I tried to do the same operations in my sub-process "run" method, but I
>>> always get the "global" registry instead of my "webapp" registry which was
>>> configured in my main application...
>>>
>>> Best regards,
>>> Thierry
>>> --
>>>   https://www.ulthar.net -- http://pyams.readthedocs.io
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "pylons-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to pylons-discuss+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAPX_VWCL7mmiSOr1nLUgN2mMpqvsnCKsXghTDON0YpJMXen1Jw%40mail.gmail.com.

Re: [pylons-discuss] Re: Creating sub-processes from Pyramid application

[Thierry Florac]

Hi Jonathan,

I didn't have a look at Celery yet, maybe it could work...
My goal is to be able to start long-running tasks (medias files conversions
for example) whose definition is stored in database and managed by end
users from my main Pyramid application; for this, I create a dedicated
sub-process which is "managed" using ZeroMQ messages (I use this
because it's also used for other non-Python/Pyramid applications).
Tasks are then started using dedicated threads.

Everything is OK until now... My only requirement now is to be able to get
access to main Pyramid's registry from my sub-process and it's threads, and
I don't really understand why I get a pointer to
ZCA global registry...  :(

Best regards,
Thierry
-- 
  https://www.ulthar.net -- http://pyams.readthedocs.io


Le lun. 4 janv. 2021 à 17:51, 'Jonathan Vanasco' via pylons-discuss <
pylons-discuss@googlegroups.com> a écrit :

> Have you considered using Celery for this?  I started offloading
> everything related to subprocesses and message queues to it a while back,
> and have been much happier.
>
> On Monday, January 4, 2021 at 11:20:46 AM UTC-5 tfl...@gmail.com wrote:
>
>> Hi,
>>
>> I need to create custom sub-processes from my main Pyramid application;
>> these processes are used to handle "commands" received from ZeroMQ messages.
>> If I use the components registry to register adapters and utilities which
>> are also required in these processes, is there a way to start these
>> processes so that they can "inherit" from the main application registry ?
>>
>> I'm actually using Pyramid 1.10.5, and I generally do a
>> "config.hook_zca()" on application startup before doing
>> the "config.setup_registry()" and "config.scan()".
>> I tried to do the same operations in my sub-process "run" method, but I
>> always get the "global" registry instead of my "webapp" registry which was
>> configured in my main application...
>>
>> Best regards,
>> Thierry
>> --
>>   https://www.ulthar.net -- http://pyams.readthedocs.io
>>
> --
> You received this message because you are subscribed to the Google Groups
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pylons-discuss+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAPX_VWADs1fgqz2VRyEWupMsrv2hPA-JdD%2BsaB-7ZKekfuMwwg%40mail.gmail.com.

[pylons-discuss] Re: Creating sub-processes from Pyramid application

['Jonathan Vanasco' via pylons-discuss]

Have you considered using Celery for this?  I started offloading everything 
related to subprocesses and message queues to it a while back, and have 
been much happier.

On Monday, January 4, 2021 at 11:20:46 AM UTC-5 tfl...@gmail.com wrote:

> Hi,
>
> I need to create custom sub-processes from my main Pyramid application; 
> these processes are used to handle "commands" received from ZeroMQ messages.
> If I use the components registry to register adapters and utilities which 
> are also required in these processes, is there a way to start these 
> processes so that they can "inherit" from the main application registry ?
>
> I'm actually using Pyramid 1.10.5, and I generally do a 
> "config.hook_zca()" on application startup before doing 
> the "config.setup_registry()" and "config.scan()".
> I tried to do the same operations in my sub-process "run" method, but I 
> always get the "global" registry instead of my "webapp" registry which was 
> configured in my main application...
>
> Best regards,
> Thierry
> -- 
>   https://www.ulthar.net -- http://pyams.readthedocs.io
>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com.