Re: [pylons-discuss] .ini file questions

2025-03-04 Thread Jonathan Vanasco 
I use a lot of env vars, due to CI as well.

In your situation, I would probably try to maintain everything in a JSON 
file -- but not one the application would ever access.  Instead, a 
tool/script/etc would parse that file and generate `.ini` files or populate 
the env vars (or write a script/whatever for your CI system to populate the 
vars).  The underlying idea is that you'd just maintain/edit the 
information in one place, then invoke a command to export the data into 
something your CI or App can read.

I actually have one app that implements this using a Python file as the 
target export.  That package has a  `/settings.py` with stock settings in 
the git repo, but the deployment script will overwrite it based on the 
environment.

i've pulled this off a few different ways, but one way I like is basically 
this structure:

{
   "setting_1": "foo",
   "setting_2": {
"*": "foo",
 "prod1": "bar",
  }
}

When exporting, a string value goes to all environments.  If the value is a 
dict, "*" is the default - but will be overridden if there is a key for the 
target environment.  

Using this strategy, I just maintain a single file for all environments.  
If i have to change a default, or add a new setting, I only edit one file 
in one place -- so the generated config file will always be current.

On Tuesday, March 4, 2025 at 1:57:31 PM UTC-5 Mike Orr wrote:

> I have used parsing '__file__' to get at a custom section 'archive'
> used by a backup routine (read-only database credentials).
>
> I'm now moving more and more things into environment variables, which
> are more Docker Compose and CI (Continuous Integration) friendly when
> different deployments need different configurations. I've got a
> general routine called from 'main()' that looks for variables like
> 'APPNAME__sqlalchemy__url' and overwrite the corresponding setting
> ('sqlalchemy.url').I had to use double underscores because dots aren't
> allowed in environment variable names. The initial double underscore
> is to distinguish "APPNAME__foo" (an envvar overriding an arbitrary
> setting) from "APPNAME_FOO" (an envvar used for a non-setting
> purpose). I didn't want to count on upper/lowercase recognition or
> translation.
>
> Longer-term I'm debating whether to switch to a JSON file or go
> completely to default values overriden by envvars. The reason I
> haven't yet is there are some fifty settings, some for middleware, and
> some that would be split (e.g., four unchanging settings plus one or
> two deployment-specific/secret settings): I don't want to put the
> entire group into envvars, but at the same time I don't want to split
> the group either. So I've kept the INI file until I feel more certain.
> (But the INI file has only the '[app:main]" section, parsed by
> 'pyramid.paster.get_appsettings()'. The logging configuration is a
> separate INI file parsed by 'logging.config.fileConfig()'.
>
> On Tue, Mar 4, 2025 at 8:00 AM Jonathan Vanasco  wrote:
> >
> > This was a fast fix. Thanks all. Code is below in case anyone else needs 
> to do something similar:
> >
> > Background:
> > * a test.ini used an alternate port for CI tests in the background
> > * some tests started failing due to port mismatches.
> > * i caught the port mismatch thanks to a little vanity 'will be serving 
> on' message
> > * this app starts in a deeply nested url, so I print out in the main 
> setup a message like:
> >
> > "APPLICATION will be serving on: 
> http://host:port/path/to/the/nested/start/page";
> >
> > -
> >
> >
> > from pyramid.scripts.common import get_config_loader
> >
> > _config_loader = get_config_loader(config_uri)
> > assert "server:main" in _config_loader.get_sections()
> > _server_settings = _config_loader.get_settings("server:main")
> > _host = _server_settings.get("host")
> > _port = _server_settings.get("port")
> > _admin_server = "http://%s:%s"; %( _host, _port)
> > if _admin_server != config.registry.settings["admin_server"]:
> > print("* == *")
> > print("Updating `admin_server` setting:")
> > print("app:main file states: %s" % 
> config.registry.settings["admin_server"])
> > print("server:main calculates: %s" % _admin_server)
> > print("* == *")
> > config.registry.settings["admin_server"] = _admin_server
> >
> >
> > print(
> > "Application will be serving on: "
> > + config.registry.settings["admin_server"]
> > + config.registry.settings["application_settings"]["admin_prefix"]
> > )
> >
> > On Tuesday, March 4, 2025 at 9:24:42 AM UTC-5 Jonathan Vanasco wrote:
> >>
> >> Thanks. I'll reparse in main!
> >>
> >> On Monday, March 3, 2025 at 4:00:51 PM UTC-5 Michael Merickel wrote:
> >>>
> >>> If you access the plaster loader directly you can query the whole 
> file. However, if you wait until your pastedeploy app factory then 
> everything is hidden from you and you have to parse the file again 
> yourself. The file

Re: [pylons-discuss] .ini file questions

2025-03-04 Thread Mike Orr 
 I have used parsing '__file__' to get at a custom section 'archive'
used by a backup routine (read-only database credentials).

I'm now moving more and more things into environment variables, which
are more Docker Compose and CI (Continuous Integration) friendly when
different deployments need different configurations. I've got a
general routine called from 'main()' that looks for variables like
'APPNAME__sqlalchemy__url' and overwrite the corresponding setting
('sqlalchemy.url').I had to use double underscores because dots aren't
allowed in environment variable names. The initial double underscore
is to distinguish "APPNAME__foo" (an envvar overriding an arbitrary
setting) from "APPNAME_FOO" (an envvar used for a non-setting
purpose). I didn't want to count on upper/lowercase recognition or
translation.

Longer-term I'm debating whether to switch to a JSON file or go
completely to default values overriden by envvars. The reason I
haven't yet is there are some fifty settings, some for middleware, and
some that would be split (e.g., four unchanging settings plus one or
two deployment-specific/secret settings): I don't want to put the
entire group into envvars, but at the same time I don't want to split
the group either. So I've kept the INI file until I feel more certain.
(But the INI file has only the '[app:main]" section, parsed by
'pyramid.paster.get_appsettings()'. The logging configuration is a
separate INI file parsed by 'logging.config.fileConfig()'.

On Tue, Mar 4, 2025 at 8:00 AM Jonathan Vanasco  wrote:
>
> This was a fast fix.  Thanks all.  Code is below in case anyone else needs to 
> do something similar:
>
> Background:
> * a test.ini used an alternate port for CI tests in the background
> * some tests started failing due to port mismatches.
> * i caught the port mismatch thanks to a little vanity 'will be serving on' 
> message
> * this app starts in a deeply nested url, so I print out in the main setup a 
> message like:
>
> "APPLICATION will be serving on:   
> http://host:port/path/to/the/nested/start/page";
>
> -
>
>
> from pyramid.scripts.common import get_config_loader
>
> _config_loader = get_config_loader(config_uri)
> assert "server:main" in _config_loader.get_sections()
> _server_settings = _config_loader.get_settings("server:main")
> _host = _server_settings.get("host")
> _port = _server_settings.get("port")
> _admin_server = "http://%s:%s"; %( _host, _port)
> if _admin_server != config.registry.settings["admin_server"]:
> print("* == *")
> print("Updating `admin_server` setting:")
> print("app:main file states:   %s" % 
> config.registry.settings["admin_server"])
> print("server:main calculates: %s" % _admin_server)
> print("* == *")
> config.registry.settings["admin_server"] = _admin_server
>
>
> print(
> "Application will be serving on:   "
> + config.registry.settings["admin_server"]
> + config.registry.settings["application_settings"]["admin_prefix"]
> )
>
> On Tuesday, March 4, 2025 at 9:24:42 AM UTC-5 Jonathan Vanasco wrote:
>>
>> Thanks. I'll reparse in main!
>>
>> On Monday, March 3, 2025 at 4:00:51 PM UTC-5 Michael Merickel wrote:
>>>
>>> If you access the plaster loader directly you can query the whole file. 
>>> However, if you wait until your pastedeploy app factory then everything is 
>>> hidden from you and you have to parse the file again yourself. The file 
>>> path is contained in the `global_config["__file__"]` key passed to the app 
>>> factory so it is pretty straightforward to do it.
>>>
>>> On Mar 3, 2025, at 08:50, Jonathan Vanasco  wrote:
>>>
>>> Is there any way to access the configured host/port in the 
>>> `project/__init__.py::main(global_config, **settings):` block?  Those vars 
>>> are set in the `[server:main]` block of a `config.ini` file - not 
>>> `[app:main]`. It looks like I can determine them in the context of a 
>>> request due to the implementation details of WSGI, but I need to access 
>>> them in the initial app setup.
>>>
>>> It seems like they may not be passed onto Pyramid and I would need to parse 
>>> the configuration file and pull it out myself.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups 
>>> "pylons-discuss" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to [email protected].
>>> To view this discussion visit 
>>> https://groups.google.com/d/msgid/pylons-discuss/f24eac45-de01-4ed1-812b-1f5d31a85081n%40googlegroups.com.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to [email protected].
> To view this discussion visit 
>

Re: [pylons-discuss] .ini file questions

2025-03-04 Thread Jonathan Vanasco 
This was a fast fix.  Thanks all.  Code is below in case anyone else needs 
to do something similar:

Background:
* a test.ini used an alternate port for CI tests in the background
* some tests started failing due to port mismatches.
* i caught the port mismatch thanks to a little vanity 'will be serving on' 
message
* this app starts in a deeply nested url, so I print out in the main setup 
a message like:

"APPLICATION will be serving on:   
http://host:port/path/to/the/nested/start/page";

-


from pyramid.scripts.common import get_config_loader

_config_loader = get_config_loader(config_uri)
assert "server:main" in _config_loader.get_sections()
_server_settings = _config_loader.get_settings("server:main")
_host = _server_settings.get("host")
_port = _server_settings.get("port")
_admin_server = "http://%s:%s"; %( _host, _port)
if _admin_server != config.registry.settings["admin_server"]:
print("* == *")
print("Updating `admin_server` setting:")
print("app:main file states:   %s" % 
config.registry.settings["admin_server"])
print("server:main calculates: %s" % _admin_server)
print("* == *")
config.registry.settings["admin_server"] = _admin_server


print(
"Application will be serving on:   "
+ config.registry.settings["admin_server"]
+ config.registry.settings["application_settings"]["admin_prefix"]
)

On Tuesday, March 4, 2025 at 9:24:42 AM UTC-5 Jonathan Vanasco wrote:

> Thanks. I'll reparse in main!
>
> On Monday, March 3, 2025 at 4:00:51 PM UTC-5 Michael Merickel wrote:
>
>> If you access the plaster loader directly you can query the whole file. 
>> However, if you wait until your pastedeploy app factory then everything is 
>> hidden from you and you have to parse the file again yourself. The file 
>> path is contained in the `global_config["__file__"]` key passed to the app 
>> factory so it is pretty straightforward to do it.
>>
>> On Mar 3, 2025, at 08:50, Jonathan Vanasco  wrote:
>>
>> Is there any way to access the configured host/port in the 
>> `project/__init__.py::main(global_config, **settings):` block?  Those vars 
>> are set in the `[server:main]` block of a `config.ini` file - not 
>> `[app:main]`. It looks like I can determine them in the context of a 
>> request due to the implementation details of WSGI, but I need to access 
>> them in the initial app setup. 
>>
>> It seems like they may not be passed onto Pyramid and I would need to 
>> parse the configuration file and pull it out myself.  
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "pylons-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to [email protected].
>> To view this discussion visit 
>> https://groups.google.com/d/msgid/pylons-discuss/f24eac45-de01-4ed1-812b-1f5d31a85081n%40googlegroups.com
>>  
>> 
>> .
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion visit 
https://groups.google.com/d/msgid/pylons-discuss/66c7bab8-34fc-4b94-9085-a14fcffbf339n%40googlegroups.com.

Re: [pylons-discuss] .ini file questions

2025-03-04 Thread Jonathan Vanasco 
Thanks. I'll reparse in main!

On Monday, March 3, 2025 at 4:00:51 PM UTC-5 Michael Merickel wrote:

> If you access the plaster loader directly you can query the whole file. 
> However, if you wait until your pastedeploy app factory then everything is 
> hidden from you and you have to parse the file again yourself. The file 
> path is contained in the `global_config["__file__"]` key passed to the app 
> factory so it is pretty straightforward to do it.
>
> On Mar 3, 2025, at 08:50, Jonathan Vanasco  wrote:
>
> Is there any way to access the configured host/port in the 
> `project/__init__.py::main(global_config, **settings):` block?  Those vars 
> are set in the `[server:main]` block of a `config.ini` file - not 
> `[app:main]`. It looks like I can determine them in the context of a 
> request due to the implementation details of WSGI, but I need to access 
> them in the initial app setup. 
>
> It seems like they may not be passed onto Pyramid and I would need to 
> parse the configuration file and pull it out myself.  
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to [email protected].
> To view this discussion visit 
> https://groups.google.com/d/msgid/pylons-discuss/f24eac45-de01-4ed1-812b-1f5d31a85081n%40googlegroups.com
>  
> 
> .
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion visit 
https://groups.google.com/d/msgid/pylons-discuss/e4d11a34-19df-4cce-844a-31b68ecf96f2n%40googlegroups.com.

Re: [pylons-discuss] .ini file questions

2025-03-04 Thread Jonathan Vanasco 
> I ran into a similar issue overriding the 'url_scheme'.

I have a similar use-case.  I define a url in the app settings of the .ini, 
and one of my test systems runs the app on a different port -- so I need to 
catch that and either automatically fix it, or immediately error out during 
startup.  Otherwise random tests fail and it's a pain to figure this out.
On Monday, March 3, 2025 at 3:48:38 PM UTC-5 Mike Orr wrote:

> I ran into a similar issue overriding the 'url_scheme'. I need to
> override it via an environment variable in Docker containers. I ended
> up creating a run script that uses 'pyramid.paster' functions, so now
> I have separate app and logging INI files, and the server
> configuration is handled by the script.
>
> I did find out that 'get_appsettings' requires a 'use=' key, even
> though it seems unused if you call the application's main() yourself.
> So that's another case where pserve/PasteDeploy assumes a pretty
> narrow usage range.
>
> On Mon, Mar 3, 2025 at 7:50 AM Jonathan Vanasco  wrote:
> >
> > Is there any way to access the configured host/port in the 
> `project/__init__.py::main(global_config, **settings):` block? Those vars 
> are set in the `[server:main]` block of a `config.ini` file - not 
> `[app:main]`. It looks like I can determine them in the context of a 
> request due to the implementation details of WSGI, but I need to access 
> them in the initial app setup.
> >
> > It seems like they may not be passed onto Pyramid and I would need to 
> parse the configuration file and pull it out myself.
> >
> > --
> > You received this message because you are subscribed to the Google 
> Groups "pylons-discuss" group.
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to [email protected].
> > To view this discussion visit 
> https://groups.google.com/d/msgid/pylons-discuss/f24eac45-de01-4ed1-812b-1f5d31a85081n%40googlegroups.com
> .
>
>
>
> -- 
> Mike Orr 
>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion visit 
https://groups.google.com/d/msgid/pylons-discuss/da3c5b2b-eb43-4abc-a6ae-d0c216293c67n%40googlegroups.com.

Re: [pylons-discuss] .ini file questions

2025-03-03 Thread Michael Merickel 
If you access the plaster loader directly you can query the whole file. 
However, if you wait until your pastedeploy app factory then everything is 
hidden from you and you have to parse the file again yourself. The file path is 
contained in the `global_config["__file__"]` key passed to the app factory so 
it is pretty straightforward to do it.

> On Mar 3, 2025, at 08:50, Jonathan Vanasco  wrote:
> 
> Is there any way to access the configured host/port in the 
> `project/__init__.py::main(global_config, **settings):` block?  Those vars 
> are set in the `[server:main]` block of a `config.ini` file - not 
> `[app:main]`. It looks like I can determine them in the context of a request 
> due to the implementation details of WSGI, but I need to access them in the 
> initial app setup. 
> 
> It seems like they may not be passed onto Pyramid and I would need to parse 
> the configuration file and pull it out myself.  
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to [email protected] 
> .
> To view this discussion visit 
> https://groups.google.com/d/msgid/pylons-discuss/f24eac45-de01-4ed1-812b-1f5d31a85081n%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion visit 
https://groups.google.com/d/msgid/pylons-discuss/748C4348-D217-473F-B44B-9B3B39BAFE33%40gmail.com.

Re: [pylons-discuss] .ini file questions

2025-03-03 Thread Mike Orr 
I ran into a similar issue overriding the 'url_scheme'. I need to
override it via an environment variable in Docker containers. I ended
up creating a run script that uses 'pyramid.paster' functions, so now
I have separate app and logging INI files, and the server
configuration is handled by the script.

I did find out that 'get_appsettings' requires a 'use=' key, even
though it seems unused if you call the application's main() yourself.
So that's another case where pserve/PasteDeploy assumes a pretty
narrow usage range.

On Mon, Mar 3, 2025 at 7:50 AM Jonathan Vanasco  wrote:
>
> Is there any way to access the configured host/port in the 
> `project/__init__.py::main(global_config, **settings):` block?  Those vars 
> are set in the `[server:main]` block of a `config.ini` file - not 
> `[app:main]`. It looks like I can determine them in the context of a request 
> due to the implementation details of WSGI, but I need to access them in the 
> initial app setup.
>
> It seems like they may not be passed onto Pyramid and I would need to parse 
> the configuration file and pull it out myself.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to [email protected].
> To view this discussion visit 
> https://groups.google.com/d/msgid/pylons-discuss/f24eac45-de01-4ed1-812b-1f5d31a85081n%40googlegroups.com.



-- 
Mike Orr 

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion visit 
https://groups.google.com/d/msgid/pylons-discuss/CAH9f%3Duq%2BvEffQNqRChfLv5dk_WZ8rd3TYTxw5vak4ocLeS2eng%40mail.gmail.com.