Re: [pylons-discuss] Ensuring the server is running from commandline?

2025-01-10 Thread Jonathan Vanasco 
For deployments, I agree with you.  Our pipeline works similar to what you 
describe.

We open-sourced this tool and I'm cleaning some stuff up for a release.  I 
am trying to safeguard against the most likely user-errors based on past 
experiences.

The hashing is working well, and was inspired by poetry lockfiles.  I just 
compute a md5 of the filepath and filecontents for the config_uri and stick 
it in the settings.  

On Friday, January 10, 2025 at 4:34:18 PM UTC-5 Ian Wilson wrote:

> I didn't understand the issue at first.  The thing you are trying to avoid 
> is that a script is run with the wrong configuration file because multiple 
> configuration files exist on every server instance, ie. a production.ini is 
> on the staging server and a staging.ini is also on the production server?  
> I haven't used the cookie cutter in a while but can't you just only include 
> the correct configuration file with each deployment, and don't send 
> the others, then you don't have to worry about using the wrong file?
>
> I think I'm using exclusively environment variables now but I am using a 
> service which provides a heroku-esque management of those.  But a given 
> server instance just has a preloaded single database url in an environment 
> variable (amongst other things) and does not know about anything else.
>
>
>
> On Fri, Jan 10, 2025 at 12:32 PM Jonathan Vanasco  
> wrote:
>
>> That just overly complicates things by requiring Celery and having to set 
>> it up.  Some commandline scripts require Pyramid to be running, others 
>> don't.  I'd like to keep this simple.
>>
>> On Friday, January 10, 2025 at 3:48:30 AM UTC-5 Luke Crooks wrote:
>>
>>> "I invoke periodic operations that are often long running"
>>>
>>> If you invoke celery to manage these, and have celery execute from 
>>> inside the pyramid, it will be running if the pyramid application is... 
>>>
>>> Then execute any shell scripts with subprocess?
>>>
>>> Regards,
>>> --
>>> Luke Crooks
>>> Solent Wholesale Carpets
>>> www.solentwholesale.com
>>>
>>> On Fri, Jan 10, 2025 at 2:10 AM Jonathan Vanasco  
>>> wrote:
>>>
 In this particular use case:

 * The Pyramid Application may only be occasionally publicly available 
 (firewall rules)
 * The Scripts and Server are on the same machine.

 I've done a test implementation pretty quickly using config-file hashes 
 and that is working well.  A Pyramid route publishes 2 hashes: the config 
 file's filepath, and the config file's contents. Based on that info, a 
 "client" script can reasonably ascertain it is being invoked by the same 
 configuration and is probably communicating with "itself". I've also 
 tested 
 adding this to my exception logging, as it looks promising for 
 troubleshooting and identifying regressions.

 The reason for hashes is that a Pyramid application can function very 
 differently based on the config file - so I can't risk a script invoked 
 with `staging.ini` thinking that is talking to a server running 
 `production.ini`. They may have completely different databases, or 
 functional behaviors.

 The actual use case:

 The Pyramid Application is a TLS Certificate Manager and centralized 
 ACME Client; when running, it responds to public ACME challenges, and 
 offers an API to OpenResty/Nginx servers for dynamic certificate loading.  

 I invoke periodic operations that are often long running, such as 
 renewing enrolled certificates, pulling ARI information, etc.  These all 
 need commandline scripts to run on demand anyways, so I didn't bother with 
 Celery.  I need checks in place to ensure both the server and the script 
 are invoked with the same configuration file.  I guess I could probably 
 add 
 the MAC address into there to identify the machine...
 On Thursday, January 9, 2025 at 5:10:32 PM UTC-5 Luke Crooks wrote:

> Unless I have missed something, why not run celery from inside pyramid?
>
>
> Regards,
> --
> Luke Crooks
> Solent Wholesale Carpets
> www.solentwholesale.com
>
>
> On Thu, 9 Jan 2025 at 15:00, Jonathan Vanasco  
> wrote:
>
>> I have a Pyramid project that has some commandline routines via 
>> registered script entrypoints (e.g. the /scripts/ directory on the 
>> current 
>> cookiecutters).
>>
>> A few of these routines need to be run while the pyramid web 
>> application is running.
>>
>> I'd like to dummyproof this as much as possible and am trying to 
>> think of ways to ensure the server is running.  
>>
>> Has anyone done something like this?
>>
>> First I thought of PIDs/Semaphores, but there could be different 
>> configuration files used.
>>
>> My current thought is calculating a hash of the ini file by both the 
>> commandline and server, and having an endpoint on the server

Re: [pylons-discuss] Ensuring the server is running from commandline?

2025-01-10 Thread Ian Wilson 
I didn't understand the issue at first.  The thing you are trying to avoid
is that a script is run with the wrong configuration file because multiple
configuration files exist on every server instance, ie. a production.ini is
on the staging server and a staging.ini is also on the production server?
I haven't used the cookie cutter in a while but can't you just only include
the correct configuration file with each deployment, and don't send
the others, then you don't have to worry about using the wrong file?

I think I'm using exclusively environment variables now but I am using a
service which provides a heroku-esque management of those.  But a given
server instance just has a preloaded single database url in an environment
variable (amongst other things) and does not know about anything else.



On Fri, Jan 10, 2025 at 12:32 PM Jonathan Vanasco 
wrote:

> That just overly complicates things by requiring Celery and having to set
> it up.  Some commandline scripts require Pyramid to be running, others
> don't.  I'd like to keep this simple.
>
> On Friday, January 10, 2025 at 3:48:30 AM UTC-5 Luke Crooks wrote:
>
>> "I invoke periodic operations that are often long running"
>>
>> If you invoke celery to manage these, and have celery execute from inside
>> the pyramid, it will be running if the pyramid application is...
>>
>> Then execute any shell scripts with subprocess?
>>
>> Regards,
>> --
>> Luke Crooks
>> Solent Wholesale Carpets
>> www.solentwholesale.com
>>
>> On Fri, Jan 10, 2025 at 2:10 AM Jonathan Vanasco 
>> wrote:
>>
>>> In this particular use case:
>>>
>>> * The Pyramid Application may only be occasionally publicly available
>>> (firewall rules)
>>> * The Scripts and Server are on the same machine.
>>>
>>> I've done a test implementation pretty quickly using config-file hashes
>>> and that is working well.  A Pyramid route publishes 2 hashes: the config
>>> file's filepath, and the config file's contents. Based on that info, a
>>> "client" script can reasonably ascertain it is being invoked by the same
>>> configuration and is probably communicating with "itself". I've also tested
>>> adding this to my exception logging, as it looks promising for
>>> troubleshooting and identifying regressions.
>>>
>>> The reason for hashes is that a Pyramid application can function very
>>> differently based on the config file - so I can't risk a script invoked
>>> with `staging.ini` thinking that is talking to a server running
>>> `production.ini`. They may have completely different databases, or
>>> functional behaviors.
>>>
>>> The actual use case:
>>>
>>> The Pyramid Application is a TLS Certificate Manager and centralized
>>> ACME Client; when running, it responds to public ACME challenges, and
>>> offers an API to OpenResty/Nginx servers for dynamic certificate loading.
>>>
>>> I invoke periodic operations that are often long running, such as
>>> renewing enrolled certificates, pulling ARI information, etc.  These all
>>> need commandline scripts to run on demand anyways, so I didn't bother with
>>> Celery.  I need checks in place to ensure both the server and the script
>>> are invoked with the same configuration file.  I guess I could probably add
>>> the MAC address into there to identify the machine...
>>> On Thursday, January 9, 2025 at 5:10:32 PM UTC-5 Luke Crooks wrote:
>>>
 Unless I have missed something, why not run celery from inside pyramid?


 Regards,
 --
 Luke Crooks
 Solent Wholesale Carpets
 www.solentwholesale.com


 On Thu, 9 Jan 2025 at 15:00, Jonathan Vanasco 
 wrote:

> I have a Pyramid project that has some commandline routines via
> registered script entrypoints (e.g. the /scripts/ directory on the current
> cookiecutters).
>
> A few of these routines need to be run while the pyramid web
> application is running.
>
> I'd like to dummyproof this as much as possible and am trying to think
> of ways to ensure the server is running.
>
> Has anyone done something like this?
>
> First I thought of PIDs/Semaphores, but there could be different
> configuration files used.
>
> My current thought is calculating a hash of the ini file by both the
> commandline and server, and having an endpoint on the server publish it 
> for
> comparison.
>
> Does anyone have a better idea?
>
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [email protected].
> To view this discussion visit
> https://groups.google.com/d/msgid/pylons-discuss/8f7b3127-ccbe-4533-a988-1c1034f316dbn%40googlegroups.com
> 
> .
>
 --
>>>

Re: [pylons-discuss] Ensuring the server is running from commandline?

2025-01-10 Thread Jonathan Vanasco 
That just overly complicates things by requiring Celery and having to set 
it up.  Some commandline scripts require Pyramid to be running, others 
don't.  I'd like to keep this simple.

On Friday, January 10, 2025 at 3:48:30 AM UTC-5 Luke Crooks wrote:

> "I invoke periodic operations that are often long running"
>
> If you invoke celery to manage these, and have celery execute from inside 
> the pyramid, it will be running if the pyramid application is... 
>
> Then execute any shell scripts with subprocess?
>
> Regards,
> --
> Luke Crooks
> Solent Wholesale Carpets
> www.solentwholesale.com
>
> On Fri, Jan 10, 2025 at 2:10 AM Jonathan Vanasco  
> wrote:
>
>> In this particular use case:
>>
>> * The Pyramid Application may only be occasionally publicly available 
>> (firewall rules)
>> * The Scripts and Server are on the same machine.
>>
>> I've done a test implementation pretty quickly using config-file hashes 
>> and that is working well.  A Pyramid route publishes 2 hashes: the config 
>> file's filepath, and the config file's contents. Based on that info, a 
>> "client" script can reasonably ascertain it is being invoked by the same 
>> configuration and is probably communicating with "itself". I've also tested 
>> adding this to my exception logging, as it looks promising for 
>> troubleshooting and identifying regressions.
>>
>> The reason for hashes is that a Pyramid application can function very 
>> differently based on the config file - so I can't risk a script invoked 
>> with `staging.ini` thinking that is talking to a server running 
>> `production.ini`. They may have completely different databases, or 
>> functional behaviors.
>>
>> The actual use case:
>>
>> The Pyramid Application is a TLS Certificate Manager and centralized ACME 
>> Client; when running, it responds to public ACME challenges, and offers an 
>> API to OpenResty/Nginx servers for dynamic certificate loading.  
>>
>> I invoke periodic operations that are often long running, such as 
>> renewing enrolled certificates, pulling ARI information, etc.  These all 
>> need commandline scripts to run on demand anyways, so I didn't bother with 
>> Celery.  I need checks in place to ensure both the server and the script 
>> are invoked with the same configuration file.  I guess I could probably add 
>> the MAC address into there to identify the machine...
>> On Thursday, January 9, 2025 at 5:10:32 PM UTC-5 Luke Crooks wrote:
>>
>>> Unless I have missed something, why not run celery from inside pyramid?
>>>
>>>
>>> Regards,
>>> --
>>> Luke Crooks
>>> Solent Wholesale Carpets
>>> www.solentwholesale.com
>>>
>>>
>>> On Thu, 9 Jan 2025 at 15:00, Jonathan Vanasco  wrote:
>>>
 I have a Pyramid project that has some commandline routines via 
 registered script entrypoints (e.g. the /scripts/ directory on the current 
 cookiecutters).

 A few of these routines need to be run while the pyramid web 
 application is running.

 I'd like to dummyproof this as much as possible and am trying to think 
 of ways to ensure the server is running.  

 Has anyone done something like this?

 First I thought of PIDs/Semaphores, but there could be different 
 configuration files used.

 My current thought is calculating a hash of the ini file by both the 
 commandline and server, and having an endpoint on the server publish it 
 for 
 comparison.  

 Does anyone have a better idea?




 -- 
 You received this message because you are subscribed to the Google 
 Groups "pylons-discuss" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to [email protected].
 To view this discussion visit 
 https://groups.google.com/d/msgid/pylons-discuss/8f7b3127-ccbe-4533-a988-1c1034f316dbn%40googlegroups.com
  
 
 .

>>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "pylons-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to [email protected].
>>
> To view this discussion visit 
>> https://groups.google.com/d/msgid/pylons-discuss/b2e4bb56-ee07-4c94-98d4-ff78d8182519n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion visit 
https://groups.google.com/d/msgid/pylons-discuss/7fd0dcf0-13ce-4ce7-a4c6-7be4e57c8607n%40googlegroups.com.

Re: [pylons-discuss] Ensuring the server is running from commandline?

2025-01-10 Thread Luke Crooks 
"I invoke periodic operations that are often long running"

If you invoke celery to manage these, and have celery execute from inside
the pyramid, it will be running if the pyramid application is...

Then execute any shell scripts with subprocess?

Regards,
--
Luke Crooks
Solent Wholesale Carpets
www.solentwholesale.com


On Fri, Jan 10, 2025 at 2:10 AM Jonathan Vanasco  wrote:

> In this particular use case:
>
> * The Pyramid Application may only be occasionally publicly available
> (firewall rules)
> * The Scripts and Server are on the same machine.
>
> I've done a test implementation pretty quickly using config-file hashes
> and that is working well.  A Pyramid route publishes 2 hashes: the config
> file's filepath, and the config file's contents. Based on that info, a
> "client" script can reasonably ascertain it is being invoked by the same
> configuration and is probably communicating with "itself". I've also tested
> adding this to my exception logging, as it looks promising for
> troubleshooting and identifying regressions.
>
> The reason for hashes is that a Pyramid application can function very
> differently based on the config file - so I can't risk a script invoked
> with `staging.ini` thinking that is talking to a server running
> `production.ini`. They may have completely different databases, or
> functional behaviors.
>
> The actual use case:
>
> The Pyramid Application is a TLS Certificate Manager and centralized ACME
> Client; when running, it responds to public ACME challenges, and offers an
> API to OpenResty/Nginx servers for dynamic certificate loading.
>
> I invoke periodic operations that are often long running, such as renewing
> enrolled certificates, pulling ARI information, etc.  These all need
> commandline scripts to run on demand anyways, so I didn't bother with
> Celery.  I need checks in place to ensure both the server and the script
> are invoked with the same configuration file.  I guess I could probably add
> the MAC address into there to identify the machine...
> On Thursday, January 9, 2025 at 5:10:32 PM UTC-5 Luke Crooks wrote:
>
>> Unless I have missed something, why not run celery from inside pyramid?
>>
>>
>> Regards,
>> --
>> Luke Crooks
>> Solent Wholesale Carpets
>> www.solentwholesale.com
>>
>>
>> On Thu, 9 Jan 2025 at 15:00, Jonathan Vanasco  wrote:
>>
>>> I have a Pyramid project that has some commandline routines via
>>> registered script entrypoints (e.g. the /scripts/ directory on the current
>>> cookiecutters).
>>>
>>> A few of these routines need to be run while the pyramid web application
>>> is running.
>>>
>>> I'd like to dummyproof this as much as possible and am trying to think
>>> of ways to ensure the server is running.
>>>
>>> Has anyone done something like this?
>>>
>>> First I thought of PIDs/Semaphores, but there could be different
>>> configuration files used.
>>>
>>> My current thought is calculating a hash of the ini file by both the
>>> commandline and server, and having an endpoint on the server publish it for
>>> comparison.
>>>
>>> Does anyone have a better idea?
>>>
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "pylons-discuss" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> To view this discussion visit
>>> https://groups.google.com/d/msgid/pylons-discuss/8f7b3127-ccbe-4533-a988-1c1034f316dbn%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion visit
> https://groups.google.com/d/msgid/pylons-discuss/b2e4bb56-ee07-4c94-98d4-ff78d8182519n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion visit 
https://groups.google.com/d/msgid/pylons-discuss/CAC0r6X9935Bb%2Bar%3D7dnSmwme_R3c-11jWpgeusanFZOvf0aiYw%40mail.gmail.com.

Re: [pylons-discuss] Ensuring the server is running from commandline?

2025-01-09 Thread Jonathan Vanasco 
In this particular use case:

* The Pyramid Application may only be occasionally publicly available 
(firewall rules)
* The Scripts and Server are on the same machine.

I've done a test implementation pretty quickly using config-file hashes and 
that is working well.  A Pyramid route publishes 2 hashes: the config 
file's filepath, and the config file's contents. Based on that info, a 
"client" script can reasonably ascertain it is being invoked by the same 
configuration and is probably communicating with "itself". I've also tested 
adding this to my exception logging, as it looks promising for 
troubleshooting and identifying regressions.

The reason for hashes is that a Pyramid application can function very 
differently based on the config file - so I can't risk a script invoked 
with `staging.ini` thinking that is talking to a server running 
`production.ini`. They may have completely different databases, or 
functional behaviors.

The actual use case:

The Pyramid Application is a TLS Certificate Manager and centralized ACME 
Client; when running, it responds to public ACME challenges, and offers an 
API to OpenResty/Nginx servers for dynamic certificate loading.  

I invoke periodic operations that are often long running, such as renewing 
enrolled certificates, pulling ARI information, etc.  These all need 
commandline scripts to run on demand anyways, so I didn't bother with 
Celery.  I need checks in place to ensure both the server and the script 
are invoked with the same configuration file.  I guess I could probably add 
the MAC address into there to identify the machine...
On Thursday, January 9, 2025 at 5:10:32 PM UTC-5 Luke Crooks wrote:

> Unless I have missed something, why not run celery from inside pyramid?
>
>
> Regards,
> --
> Luke Crooks
> Solent Wholesale Carpets
> www.solentwholesale.com
>
>
> On Thu, 9 Jan 2025 at 15:00, Jonathan Vanasco  wrote:
>
>> I have a Pyramid project that has some commandline routines via 
>> registered script entrypoints (e.g. the /scripts/ directory on the current 
>> cookiecutters).
>>
>> A few of these routines need to be run while the pyramid web application 
>> is running.
>>
>> I'd like to dummyproof this as much as possible and am trying to think of 
>> ways to ensure the server is running.  
>>
>> Has anyone done something like this?
>>
>> First I thought of PIDs/Semaphores, but there could be different 
>> configuration files used.
>>
>> My current thought is calculating a hash of the ini file by both the 
>> commandline and server, and having an endpoint on the server publish it for 
>> comparison.  
>>
>> Does anyone have a better idea?
>>
>>
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "pylons-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to [email protected].
>> To view this discussion visit 
>> https://groups.google.com/d/msgid/pylons-discuss/8f7b3127-ccbe-4533-a988-1c1034f316dbn%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion visit 
https://groups.google.com/d/msgid/pylons-discuss/b2e4bb56-ee07-4c94-98d4-ff78d8182519n%40googlegroups.com.

Re: [pylons-discuss] Ensuring the server is running from commandline?

2025-01-09 Thread Luke Crooks 
Unless I have missed something, why not run celery from inside pyramid?


Regards,
--
Luke Crooks
Solent Wholesale Carpets
www.solentwholesale.com


On Thu, 9 Jan 2025 at 15:00, Jonathan Vanasco  wrote:

> I have a Pyramid project that has some commandline routines via registered
> script entrypoints (e.g. the /scripts/ directory on the current
> cookiecutters).
>
> A few of these routines need to be run while the pyramid web application
> is running.
>
> I'd like to dummyproof this as much as possible and am trying to think of
> ways to ensure the server is running.
>
> Has anyone done something like this?
>
> First I thought of PIDs/Semaphores, but there could be different
> configuration files used.
>
> My current thought is calculating a hash of the ini file by both the
> commandline and server, and having an endpoint on the server publish it for
> comparison.
>
> Does anyone have a better idea?
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion visit
> https://groups.google.com/d/msgid/pylons-discuss/8f7b3127-ccbe-4533-a988-1c1034f316dbn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion visit 
https://groups.google.com/d/msgid/pylons-discuss/CAC0r6X-JfOJcTEht-ovKAexYB3P6AJR2nz0tJu6ADkf8bTZkEQ%40mail.gmail.com.

Re: [pylons-discuss] Ensuring the server is running from commandline?

2025-01-09 Thread Ian Wilson 
Are the scripts running on the same server as the web app?

How often are they running? Can you just query and arbitrary URL and see if
it returns a 200? Why do you need the hash?  Or maybe that other HTTP
method, HEAD?  Then maybe block it at the proxy and just request it
locally.  I guess you have to make sure something doesn't cache it.

I think on fly.io they call it a health check to make sure the container is
up.

- Ian

On Thu, Jan 9, 2025, 7:00 AM Jonathan Vanasco  wrote:

> I have a Pyramid project that has some commandline routines via registered
> script entrypoints (e.g. the /scripts/ directory on the current
> cookiecutters).
>
> A few of these routines need to be run while the pyramid web application
> is running.
>
> I'd like to dummyproof this as much as possible and am trying to think of
> ways to ensure the server is running.
>
> Has anyone done something like this?
>
> First I thought of PIDs/Semaphores, but there could be different
> configuration files used.
>
> My current thought is calculating a hash of the ini file by both the
> commandline and server, and having an endpoint on the server publish it for
> comparison.
>
> Does anyone have a better idea?
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion visit
> https://groups.google.com/d/msgid/pylons-discuss/8f7b3127-ccbe-4533-a988-1c1034f316dbn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion visit 
https://groups.google.com/d/msgid/pylons-discuss/CADj1%3D671rct86EvRHU%3DcUvJ4-P0yBj71-Gs5fWfgY7nACHrcSw%40mail.gmail.com.