Re: declarative settings

[Christian González]

Hi all,

thanks for your fast response - it seems that this is not a "ah, forget
it, boooring." theme.
I'll try to answer all (for me) relevant things at once here - sorry to
write such a huge mail.

*@Adam Johnson:*
> * A class to read django.conf.settings and add defaults/other logic
> through properties - for example
> https://github.com/adamchainz/django-cors-headers/blob/31b9c2ef8a333a40f18081ffc1f1cba9fb34574d/src/corsheaders/conf.py
> . This has the benefit of easily providing defaults, complex logic
> such as deriving one value from multiple settings, and being read from
> settings at runtime rather than import time.

This is a good approach too, but it's not namespaced automatically. And
app settings can't provide and override settings from other apps. This
seems a weird suggest, but when apps depend on another app it should be
able to override settings for that dependency too (because it "uses"
that app anyway).


> I am in favour of most libraries adding a single setting that controls
> them, like DRF has REST_FRAMEWORK. I know django-cors-headers doesn't
> implement the pattern but this is for legacy reasons since I took it over.
Yes, this basically means namespacing. Depending on the size of the
library, this could be one big bunch of a setting.
> I think this pattern solves your first concern Christian.

No it doesn't completely. I used this pattern myself in GDAPS, copied
from graphene (which copied from DRF). But this does _not_ solve the
problem: This pattern  creates a per-app settings object which checks at
the time of using (which is anywhere after main settings.py) if there is
a global override of the local app's variable - and takes the right one.
In this way it works. but this is not very convenient - you always have
to ask the right settings object - it maybe would be more convenient,
simpler and easier to code to just ask for a namespaced setting, e.g. in
a dotted path style.

like settings("django.ROOT_URLCONF") or settings("foo_app.SETTINGA")

This is not a really good approach as it implements strings.

> As for your second concern, I think checks make manual installation of
> apps easier. If a user adds your app to INSTALLED_APPS, the system
> checks will run and can highlight any missing/misconfigured settings
> from following the installation docs. Most libraries take 3 minutes to
> do the initial install so I'm not sure of the value of automating that
> step compared to adding system checks.
Perfect.

*@Matthew Pava*

I have worked with Drupal for a long time before Django, and am
maintaining a few sites with Drupal, one of it rather big, with many
domains & shared content, 40+ modules, API to ther web services, some
modules custom coded. I learned a lot of it, but PHP is really ugly.
Django does much better, but, compared with Drupal, it has a few
drawbacks. I know, Drupal is a CMS, Django is a framework (which you can
build a CMS with) - BUT: Drupal is able (like Wordpress) to add
modules/plugins per web download and on-the-fly - it installs modules
into place, has a very good hooks system and a plugin manager. It also
was a help when I designed GDAPS and was one of my blueprints. Drupal
however keeps ALL of it's settings in the database which creates many
problems on it's own (development/deployment problems how to sync
settings from disk to database and vice versa). I wouldn't do that.
Settings in code/conf files is a good thing IMHO.

The package manager in Drupal is composer, yes, and together with GDAPS,
you could use GDAPS Django packages exactly the sameway. Just install it
using pip/pipenv, and Dajngo/GDAPS finds them automatically.

But this can only be achieved by adding a line to settings.py (call the
plugin manager), like I did:
https://gdaps.readthedocs.io/en/latest/Installation.html#installation

These are just hacks because Django doesn't have declarative settings.
I suggest that if changing to declarative, it MUST be backwards
compatible, yes. So there could be an _internal_ settings.py which loads
environment variables, the bespoken config file, and like I said, **if**
there is a settings.py in the project, Django could use it - to override
settings in a way just code could do it.

*@Jacob Rief and @Tom Forbes*

Sure, JSON was just one idea. It's "xml 2.0", and not suited to be
written by humans easily. yaml or toml would be better, in fact, in a
first approach of GDAPS I used toml as meta config data for my Django
plugins. I decided (because there IS no common framework in Django, and
I didn't want to add another dependency) to let it go and am using an
attribute of AppConfig ATM - code again.

> /Some settings must be taken from the environment./*
*

Sure. DJANGO_SECRET_KEY etc. But this could be handled by the "internal"
settings.py file.

> /So rather than just “allowing people to use JSON files for
settings”// //it would be very interesting to explore what a pluggable
settings backend would look like./

_*Now that's the main point IMHO*__. _

I know 

Re: Google Patch Rewards program

[Dan Davis]

Another good security improvement would be to allow the database password
and other database information to support AWS Secrets Manager, Goolge
Secrets Management, and HashiCorp vault (+ others).
I have done this in a private package used at the National Library of
Medicine, but my package is both private and limited to RDS PostgreSQL and
AWS Secrets Manager.

On Tue, Dec 31, 2019 at 11:25 AM Dan Davis  wrote:

> Taymon Beal writes:
> > First-class integration with one or more secrets management systems,
> both to generally contain secrets better and more specifically
> > so people aren't so tempted to check SECRET_KEYs and database passwords
> into source control. (I think this was mentioned in the list of GSoC
> project ideas.)
>
> What secrets management systems are you thinking about?   I wrote
> confsecrets, whose Django integration never worked right, assuming that I'd
> port it to support the following:
> * HashiCorp vault
> * AWS Secrets Manager
> * Google Secrets Management
>
> I also need to switch it from cryptdome to cryptography or make it
> agnostic on these.
>
> P.S. - I have some very limited applied cryptography knowledge - couple
> chapters of Bruce Schneier and some hands-on learning, such as securing
> single license features with signatures based on my employer's license
> signing private key, and discovering that a customer literally pulled one
> set of feature.xml/feature.sig files from one tarball, and another from
> another tarball to game my home-grown licensing system.  I helped that
> company eventually move to a Secure USB dongle, and that was my last moment
> of systems software development :)
>
> On Sat, Dec 28, 2019 at 11:23 PM Taymon A. Beal 
> wrote:
>
>> (Disclosure: I'm on Google's security team, and my views on this topic
>> are informed by what kinds of things we tend to look for in Web frameworks,
>> but here I don't speak for them, only for myself.)
>>
>> Beyond those already mentioned, here are some potential security
>> improvements I'd like to see in Django:
>>
>>- Support for contextual autoescaping in the template system. The
>>current autoescaping behavior is better than nothing, but it still makes
>>XSS far too easy, since different kinds of strings are XSS sinks in
>>different contexts.
>>
>> https://github.com/google/closure-templates/blob/master/documentation/concepts/auto-escaping.md
>>  shows
>>an example of how to do this sort of thing more securely.
>>- First-class integration with one or more secrets management
>>systems, both to generally contain secrets better and more specifically so
>>people aren't so tempted to check SECRET_KEYs and database passwords into
>>source control. (I think this was mentioned in the list of GSoC project
>>ideas.)
>>- Capability-based authorization. Right now, you have to explicitly
>>check for all relevant permissions everywhere, and if you forget, or even
>>if you accidentally include the wrong variable in a template, you can leak
>>data or worse. It'd be much safer if the allowed permissions could be
>>defined at a single choke point, and from there, all model access within a
>>request could be mediated by the specified authorization rules. Ideally
>>data that the current user isn't supposed to see would not even be fetched
>>from the database.
>>
>> Taymon
>>
>> On Sat, Dec 21, 2019 at 11:29 PM Asif Saif Uddin 
>> wrote:
>>
>>> Really good plans Adam!
>>>
>>> On Saturday, December 21, 2019 at 11:51:11 PM UTC+6, Adam Johnson wrote:

 I just saw Google is expanding their Patch Rewards program for open
 source security improvements:
 https://security.googleblog.com/2019/12/announcing-updates-to-our-patch-rewards.html

 They are offering two tiers of rewards - $5,000 or $30,000 - for  open
 source projects making security improvements. I think Django would find it
 hard to fit in the "small" tier - we generally fix known vulnerabilities
 quickly - but we could use the "large" tier to fund a bigger GSoC style
 project. I suspect it would need active involvement from a DSF member to
 push it through. Not sure how the funding would work in terms of DSF and
 paying for development time on the project.

 Some projects that could fit:

- 2FA built-in to django.contrib.auth (as suggested for GSoC as
well in this thread:

 https://groups.google.com/forum/#!msg/django-developers/ifYT6lX8nmg/1nVO3As1AwAJ
)
- Adding CSP to SecurityMiddleware and shipping some default
(django-csp is a good start but requires users to actively seek it:
https://django-csp.readthedocs.io/en/latest/ )
- Adding CORS to Django itself (I'm maintaining
django-cors-headers, but its design is a bit pants
https://github.com/adamchainz/django-cors-headers )
- Other things in James Bennett's list of suggestions from this
thread in 

Re: Google Patch Rewards program

[Dan Davis]

Taymon Beal writes:
> First-class integration with one or more secrets management systems, both
to generally contain secrets better and more specifically
> so people aren't so tempted to check SECRET_KEYs and database passwords
into source control. (I think this was mentioned in the list of GSoC
project ideas.)

What secrets management systems are you thinking about?   I wrote
confsecrets, whose Django integration never worked right, assuming that I'd
port it to support the following:
* HashiCorp vault
* AWS Secrets Manager
* Google Secrets Management

I also need to switch it from cryptdome to cryptography or make it agnostic
on these.

P.S. - I have some very limited applied cryptography knowledge - couple
chapters of Bruce Schneier and some hands-on learning, such as securing
single license features with signatures based on my employer's license
signing private key, and discovering that a customer literally pulled one
set of feature.xml/feature.sig files from one tarball, and another from
another tarball to game my home-grown licensing system.  I helped that
company eventually move to a Secure USB dongle, and that was my last moment
of systems software development :)

On Sat, Dec 28, 2019 at 11:23 PM Taymon A. Beal 
wrote:

> (Disclosure: I'm on Google's security team, and my views on this topic are
> informed by what kinds of things we tend to look for in Web frameworks, but
> here I don't speak for them, only for myself.)
>
> Beyond those already mentioned, here are some potential security
> improvements I'd like to see in Django:
>
>- Support for contextual autoescaping in the template system. The
>current autoescaping behavior is better than nothing, but it still makes
>XSS far too easy, since different kinds of strings are XSS sinks in
>different contexts.
>
> https://github.com/google/closure-templates/blob/master/documentation/concepts/auto-escaping.md
>  shows
>an example of how to do this sort of thing more securely.
>- First-class integration with one or more secrets management systems,
>both to generally contain secrets better and more specifically so people
>aren't so tempted to check SECRET_KEYs and database passwords into source
>control. (I think this was mentioned in the list of GSoC project ideas.)
>- Capability-based authorization. Right now, you have to explicitly
>check for all relevant permissions everywhere, and if you forget, or even
>if you accidentally include the wrong variable in a template, you can leak
>data or worse. It'd be much safer if the allowed permissions could be
>defined at a single choke point, and from there, all model access within a
>request could be mediated by the specified authorization rules. Ideally
>data that the current user isn't supposed to see would not even be fetched
>from the database.
>
> Taymon
>
> On Sat, Dec 21, 2019 at 11:29 PM Asif Saif Uddin  wrote:
>
>> Really good plans Adam!
>>
>> On Saturday, December 21, 2019 at 11:51:11 PM UTC+6, Adam Johnson wrote:
>>>
>>> I just saw Google is expanding their Patch Rewards program for open
>>> source security improvements:
>>> https://security.googleblog.com/2019/12/announcing-updates-to-our-patch-rewards.html
>>>
>>> They are offering two tiers of rewards - $5,000 or $30,000 - for  open
>>> source projects making security improvements. I think Django would find it
>>> hard to fit in the "small" tier - we generally fix known vulnerabilities
>>> quickly - but we could use the "large" tier to fund a bigger GSoC style
>>> project. I suspect it would need active involvement from a DSF member to
>>> push it through. Not sure how the funding would work in terms of DSF and
>>> paying for development time on the project.
>>>
>>> Some projects that could fit:
>>>
>>>- 2FA built-in to django.contrib.auth (as suggested for GSoC as well
>>>in this thread:
>>>
>>> https://groups.google.com/forum/#!msg/django-developers/ifYT6lX8nmg/1nVO3As1AwAJ
>>>)
>>>- Adding CSP to SecurityMiddleware and shipping some default
>>>(django-csp is a good start but requires users to actively seek it:
>>>https://django-csp.readthedocs.io/en/latest/ )
>>>- Adding CORS to Django itself (I'm maintaining django-cors-headers,
>>>but its design is a bit pants
>>>https://github.com/adamchainz/django-cors-headers )
>>>- Other things in James Bennett's list of suggestions from this
>>>thread in May 2018:
>>>
>>> https://groups.google.com/forum/#!msg/django-developers/DDpkrvFdnvk/J46ZbakxAgAJ
>>>
>>> Thoughts?
>>> --
>>> Adam
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> 

Re: remove_stale_contenttypes doesn't remove entries for renamed apps.

[gowtham kommineni]

Hi Adam

Please link to the ticket here.
>

ticket: https://code.djangoproject.com/ticket/31123
PR: https://github.com/django/django/pull/12257 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e27e1fd5-9ba9-4b9d-b9a0-f477fb353769%40googlegroups.com.

Re: remove_stale_contenttypes doesn't remove entries for renamed apps.

[Adam Johnson]

Hi Gowtham

My Django ticket was closed
>

Please link to the ticket here.

May be we can add an optional kwarg to remove_stale_contenttypes command to
> remove all contenttypes that do not have a corresponding model in existing
> appsv
>

That sounds somewhat reasonable to me. I think there are more implications
for renaming an app pthough. Wouldn't it be better when renaming an app to
add a migration to update the existing ContentType rows to point to the new
names? Then existing rows with Generic Foreign Key should continue to work?

On Tue, 31 Dec 2019 at 07:11, gowtham kommineni 
wrote:

> Currently, it only removes content types in scenarios where you either
> rename a model or delete a model. If I rename an App from "app_label1" to
> "app_label2", it doesn't delete content types for "app_label1" and this is
> causing issues where I use ContentType as a GenericForeignKey.
>
> After I renamed an App and ran remove_stale_contenttypes, I had content
> types like the following:
> app_label1_Post
> app_label2_Post
> app_label1_Comment
> app_label2_Comment
>
>
> My Django ticket was closed saying "I don't think that
> `remove_stale_contenttypes` should remove content types for nonexistent
> apps. We don't automatically remove models for nonexistent apps and I
> don't think we should do this with content types. This can also cause a
> data loss for custom content types (not related to real apps)."
>
>
> May be we can add an optional kwarg to remove_stale_contenttypes command
> to remove all contenttypes that do not have a corresponding model in
> existing apps. That would allow us to use the standard command with that
> optional kwarg while allowing others to keep custom contenttypes. Can you
> please let me how we can add this functionality while respecting others
> using custom contenttypes.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/c306cc5d-4eeb-4324-b210-7017f9269293%40googlegroups.com
> 
> .
>


-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM1NCQaf%3D0EP12DW9xn7h%3Dd%3DB-UBQ%2BzYeHucjH-5zk0vXQ%40mail.gmail.com.

remove_stale_contenttypes doesn't remove entries for renamed apps.

[gowtham kommineni]

Currently, it only removes content types in scenarios where you either 
rename a model or delete a model. If I rename an App from "app_label1" to 
"app_label2", it doesn't delete content types for "app_label1" and this is 
causing issues where I use ContentType as a GenericForeignKey.

After I renamed an App and ran remove_stale_contenttypes, I had content 
types like the following:
app_label1_Post
app_label2_Post
app_label1_Comment
app_label2_Comment


My Django ticket was closed saying "I don't think that 
`remove_stale_contenttypes` should remove content types for nonexistent 
apps. We don't automatically remove models for nonexistent apps and I don't 
think we should do this with content types. This can also cause a data loss 
for custom content types (not related to real apps)."


May be we can add an optional kwarg to remove_stale_contenttypes command to 
remove all contenttypes that do not have a corresponding model in existing 
apps. That would allow us to use the standard command with that optional 
kwarg while allowing others to keep custom contenttypes. Can you please let 
me how we can add this functionality while respecting others using custom 
contenttypes.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c306cc5d-4eeb-4324-b210-7017f9269293%40googlegroups.com.

Re: declarative settings

[Adam Johnson]

When using settings for my third party packages I use:

* A class to read django.conf.settings and add defaults/other logic through
properties - for example
https://github.com/adamchainz/django-cors-headers/blob/31b9c2ef8a333a40f18081ffc1f1cba9fb34574d/src/corsheaders/conf.py
. This has the benefit of easily providing defaults, complex logic such as
deriving one value from multiple settings, and being read from settings at
runtime rather than import time.
* A bunch of system checks to validate them - for example
https://github.com/adamchainz/django-cors-headers/blob/31b9c2ef8a333a40f18081ffc1f1cba9fb34574d/src/corsheaders/checks.py
. For example, if a setting is *required*, you can add a check error; or if
a setting defaults to something unsuitable for production, you can add a
deploy check. System checks make it easy for users to upgrade your package
too, since many users don't read changelogs even for breaking version
changes.

I am in favour of most libraries adding a single setting that controls
them, like DRF has REST_FRAMEWORK. I know django-cors-headers doesn't
implement the pattern but this is for legacy reasons since I took it over.

I think this pattern solves your first concern Christian. As for your
second concern, I think checks make manual installation of apps easier. If
a user adds your app to INSTALLED_APPS, the system checks will run and can
highlight any missing/misconfigured settings from following the
installation docs. Most libraries take 3 minutes to do the initial install
so I'm not sure of the value of automating that step compared to adding
system checks.

Jacob you're right it's a bit of an inconvenience to write a class to wrap
django.conf.settings but I'm not sure there's any general purpose wrapper
that could be provided. There are too many cases for evolution - a setting
changes name, one value combines multiple settings, etc.


On Tue, 31 Dec 2019 at 00:09, Tom Forbes  wrote:

> My two cents: JSON isn’t great as a configuration language - It’s
> annoyingly picky about some things. YAML or TOML are “better” (for some
> definition of better) choices for this domain, in my option. However,
> Django is historically quite hesitant about including third party packages
> and I think including a TOML or YAML parser even as an optional dependency
> might not be the way forward.
>
> So rather than just “allowing people to use JSON files for settings” it
> would be very interesting to explore what a pluggable settings backend
> would look like. It seems that work in this area would be best spent on a
> general abstraction rather than a specific one. There was some discussion
> around this idea in the GSOC thread and I’m sure it’s come up before.
> Personally I think configuration management is an area of Django that is
> quite underdeveloped compared to other frameworks.
>
> On 30 Dec 2019, at 22:46, Christian González <
> christian.gonza...@nerdocs.at> wrote:
>
> Hello,
>
> I recently "finished" my first really working version of GDAPS, my
> Generic Django Application Plugin System. It's noway perfect, but does
> what it should: providing pluggable apps for an app framework, including
> a more or less flexible frontend with each django app.
>
> I had much struggle with it, and one of the lessons I learned was
> Django's setup system, and how it deals with loading apps. Unfortunately
> Django can't load/unload apps on the fly, so it is necessary to restart
> Django whenever a new GDAPS app is installed via pip.
>
> But: I want to resurrect an old theme again which would, in a way,
> improve some of the loading problems I encountered. Django's settings
> are code. Which is, in fact, a very good thing, as it makes it extremely
> flexible and adaptable to different setups. But, as discussed with the
> SECRET_KEY here, some of the settings _have_ to be coded very
> complicated, and it makes some things like per-app-settings extremely
> uncomfortable.
>
> What if - and please don't kill me instantly - yes, I am a newcomer, and
> not a good programmer maybe - but some things are viewed better from
> "outside" - what if Django settings could be "declarative"?
>
> So instead of Python code like
>
> INSTALLED_APPS = [
> 'django.contrib.admin',
> 'django.contrib.auth',
> 'django.contrib.contenttypes'
> ]
>
> This would be in an e.g. JSON file
>
> {
>
> "INSTALLED_APPS": [
> "django.contrib.admin",
> "django.contrib.auth",
> "django.contrib.contenttypes"
> ] ,
> ROOT_URLCONF: "fooproject.urls"
> }
>
> Django's settings.py would look different: It would load that
> settings.json file and set the appropriate values into local code - so
> this wouldn't make much difference.
>
> Except 2 things:
>
> 1. Apps could have (default) settings, and they could be merged MUCH
> easier. Things like namespaced classes that are overwriting values like
> DRF/graphene does, would be completely unnecessary. The main
> settings.json file could be the "last word" in