Re: Making Django more PaaS-friendly

2016-05-10 Thread Anssi Kääriäinen
On Tue, May 10, 2016 at 1:48 AM, Carl Meyer  wrote:
> Hi Anssi,
>
> On 05/09/2016 06:53 AM, Anssi Kääriäinen wrote:
> I'm curious to hear more about this - could you give some example DEPs
> where this was a problem? In the case of the two DEPs I was most closely
> involved with (multiple template engines and rethinking middleware), I
> don't think this is an accurate description of what happened. Both those
> DEPs were written relatively early in the design process and evolved as
> we learned from discussion and implementation.
> They were done quite similarly to how the PEPs I've worked on were done.
>
> I'm also curious about your use of the word "enforced." I don't really
> see the DEP process as something that needs "enforcement" (and I'm
> curious where you see it as having been lacking in "enforcement"). IMO
> the only relevant "enforcement," really, is that the implementation of a
> DEP which is rejected by the technical board (has never happened yet)
> should not be merged.

It's not so much the DEPs that have been written (though the channels
DEP is going to be post-design instead of supporting the design). It's
more about those features that don't have a DEP at all. For example
database schemas and subquery expressions come to mind (these are
ongoing work, PRs available at GitHub). There are various design
choices and the features are somewhat significant, yet the design
hasn't been at all DEP driven.

Also, having DEP requested for channels only at this point, and Andrew
(a technical board member) assuming a DEP wasn't required should point
out there might be a problem. This is not to criticize Andrew but the
way DEPs are sometimes required, sometimes not.

I'm not seeing a problem with the DEP idea in general, with any
particular DEP or with any single feature having or not having a DEP.
I'm criticizing the approach where sometimes patches are called out
randomly for DEP, often at a point where the DEP is just documenting
what has been already decided, not supporting the decision process
itself.

I probably picked the wrong thread to complain about this - if we are
going to want DEPs for features similar to the ideas discussed in this
thread, then the timing for requesting a DEP was accurate.

  - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1E9axuBW5y673LSAE3-2%2BUg45JcPL%2BNLQJVsUrjQ3aLzg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Making Django more PaaS-friendly

2016-05-09 Thread Carl Meyer
Hi Anssi,

On 05/09/2016 06:53 AM, Anssi Kääriäinen wrote:
> I'm very much afraid we are in a situation where we have a documented
> DEP process which isn't actually enforced, nor does it document what
> we actually do.
>
> Python's PEP process is useful as the document evolves while the
> design evolves. What we do with DEPs is different, we first design
> something, then we document the design decision, often mostly because
> DEPs are a required document. We don't get what we could from DEPs as
> they are written too late.

I'm curious to hear more about this - could you give some example DEPs
where this was a problem? In the case of the two DEPs I was most closely
involved with (multiple template engines and rethinking middleware), I
don't think this is an accurate description of what happened. Both those
DEPs were written relatively early in the design process and evolved as
we learned from discussion and implementation.
They were done quite similarly to how the PEPs I've worked on were done.

I'm also curious about your use of the word "enforced." I don't really
see the DEP process as something that needs "enforcement" (and I'm
curious where you see it as having been lacking in "enforcement"). IMO
the only relevant "enforcement," really, is that the implementation of a
DEP which is rejected by the technical board (has never happened yet)
should not be merged.

DEPs, like PEPs, aren't supposed to be a legalistic policy: there is no
hard and fast rule about which features should have DEPs, and I don't
think we need a hard and fast rule. DEPs should happen when someone
observes that the discussion around a feature is getting quite complex
or contentious, and suggests that a DEP would be helpful. DEPs should be
a tool, not a barrier: a tool for helping to focus discussions on
contentious issues, and record the perspectives expressed in those
discussions, both for posterity and to help keep the discussion from
going around in circles.

> Could we try an approach where we *first* require a DEP, then have the
> design discussion?

In the PEP process, it's very typical to start into design discussions
before realizing that the issue is contentious/complex enough to require
a PEP, then someone collects the knowledge generated from that early
discussion into a first draft PEP, which then receives further
discussion. (In the Python world most of this pre-PEP discussion happens
on the python-ideas mailing list, so if you aren't subscribed there you
might not see it.) I think this thread is a great example of that
working precisely the way it should work: first post some general
thoughts, collect ideas and responses to those thoughts, then work that
into a DEP with a specific proposal, followed by more detailed
discussion of that specific proposal.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/573113D6.90503%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Making Django more PaaS-friendly

2016-05-09 Thread Anssi Kääriäinen
I'm very much afraid we are in a situation where we have a documented
DEP process which isn't actually enforced, nor does it document what
we actually do.

Python's PEP process is useful as the document evolves while the
design evolves. What we do with DEPs is different, we first design
something, then we document the design decision, often mostly because
DEPs are a required document. We don't get what we could from DEPs as
they are written too late.

Could we try an approach where we *first* require a DEP, then have the
design discussion?

 - Anssi

On Mon, May 9, 2016 at 2:42 PM, Shai Berger  wrote:
> On Monday 09 May 2016 05:06:47 James Bennett wrote:
>> Whee, this thread got big!
>>
>> The takeaway I'm getting here is that we should be careful about what we
>> adopt into core and when; I'm going to take a couple days and write up some
>> notes on what I think a good conservative approach would look like, and ask
>> for feedback on that.
>>
>
> Looking at what we just had in other threads, this is probably DEP-worthy;
> even if the results end up being a relatively small modification, it would be
> helpful to record why we decided to reject the other suggestions.
>
> Shai.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1HrKje7%3DQLi-51sw4ZRSY3jFE1VTu_BKsL1y4u2EiVRJQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Making Django more PaaS-friendly

2016-05-09 Thread Shai Berger
On Monday 09 May 2016 05:06:47 James Bennett wrote:
> Whee, this thread got big!
> 
> The takeaway I'm getting here is that we should be careful about what we
> adopt into core and when; I'm going to take a couple days and write up some
> notes on what I think a good conservative approach would look like, and ask
> for feedback on that.
> 

Looking at what we just had in other threads, this is probably DEP-worthy; 
even if the results end up being a relatively small modification, it would be 
helpful to record why we decided to reject the other suggestions.

Shai.


Re: Making Django more PaaS-friendly

2016-05-08 Thread James Bennett
Whee, this thread got big!

The takeaway I'm getting here is that we should be careful about what we
adopt into core and when; I'm going to take a couple days and write up some
notes on what I think a good conservative approach would look like, and ask
for feedback on that.

(spoiler: probably the majority of it is going to be adding support for
parsing database URLs in the DATABASES setting)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAL13Cg8yQkkWQFG0zVLi%3DKwOzhEyvp%3DMc63L8jTU4cYxaOURWw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Making Django more PaaS-friendly

2016-04-27 Thread Carl Meyer
Hi Marcin,

On 04/26/2016 08:12 PM, Marcin Nowak wrote:
> But I would like to say my thoughts about "settings" itself.
> They were good (simple) before incuding nested dictionaries.
> After switching to dicts the settings handling went harder way.

I agree that settings grouped into dicts are harder to manage than flat
settings. Django has not made any "switch to dicts" for settings; in
fact, the last time we had to make a decision between a group of flat
settings with a common prefix vs a dict, we chose flat (the SECURE_*
settings).

> As a example I can tell about reconfiguring LOGGING for dev/live settings.
> We're tuning them via direct changes like
> LOGGING['loggers']['some-logger']['level']='...'
> which is very error prone. The flat settings are easier to maintain.

LOGGING is a special case; this dict format is not defined by Django,
but by Python's logging module and its "dictConfig" function.

> Nested settings are good for framework internals and should stay in that
> form, 
> but maybe solution for end users and ENV integration is flattening
> settings again, at the high-level?
> 
> Let's imagine something like INI file which is parsed somehow and
> converted to internal settings object.
> The INI file has a shallow structure (options grouped in sections), 
> can be easily extended / overrided / included (look at Buidout as an
> example),
> and ENV keys will be related to flat INI instead of nested settings.py.
> 
> Let's consider an example:
> 
> |
> [databases]
> default.url =postgres+psycopg2://user:pwd@localhost:5432/exampledb
> default.engine =django.db.backends.postgres_psycopg2
> 
> |
> 
> which can be simply reconfigured by ENV vars like:
> 
> DJANGO_DATABASES_DEFAULT_URL="postgres+psycopg2://user:pwd@localhost:5432/exampledb"
> DJANGO_DATABASES_DEFAULT_ENGINE="django.db.backends.postgres_psycopg2"
> 
> OR add some preprocessing and flexibility:
> 
> |
> 
> 
> [databases]
> default.url =${env:SOME-USER-KEY1}
> default.engine =${env:SOME-USER-KEY2}
> 
> 
> [env]
> SOME_USER_KEY1=a defaultvalue fordefaultdatabase URL
> SOME_USER_KEY2=a defaultvalue fordefaultdatabase engine
> 
> |
> 
> where [env] section will by filled automatically by Django from os.environ.
> 
> This also gives a possibility to extend settings.ini via development.ini, 
> where [databases] can be overridden with developer settings (without env
> access)

I've done something similar before, and I think it's fine, but this is
an example of precisely the sort of more-magical opinionated approach to
settings that I don't think belongs in Django core; it's perfectly
possible to implement it as a third-party package on top of Django's
current settings handling, for those who like it.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5720E786.7020306%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Making Django more PaaS-friendly

2016-04-27 Thread Aymeric Augustin
Hello,

If anything, this discussion shows that people have come up with a wide
variety of customized solutions to meet their use cases. That supports
Django’s historical decision not to be too prescriptive in this area.

I still think James’ proposal is valuable because it will nudge developers
towards best practices when deploying on a PaaS, a path often taken
by those least familiar with deployment best practices in general.

Best regards,

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/B239EA24-C912-4A32-8C04-795B01BD1317%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.


Re: Making Django more PaaS-friendly

2016-04-26 Thread Marcin Nowak
Hi all,

On Wednesday, April 13, 2016 at 9:58:18 PM UTC+2, Carl Meyer wrote:
>
> Hi James et al, 
>
> In general, I like the idea of adding a helper to Django to read 
> settings from the environment. I think this helper should be kept as 
> simple and non-magical as is reasonable. Thus: 
>
> This sounds great. Django should not do anything magical and just in one 
way. 
Just give the tool to the developers.

But I would like to say my thoughts about "settings" itself.
They were good (simple) before incuding nested dictionaries.
After switching to dicts the settings handling went harder way.

As a example I can tell about reconfiguring LOGGING for dev/live settings.
We're tuning them via direct changes like 
LOGGING['loggers']['some-logger']['level']='...'
which is very error prone. The flat settings are easier to maintain.

Nested settings are good for framework internals and should stay in that 
form, 
but maybe solution for end users and ENV integration is flattening settings 
again, at the high-level?

Let's imagine something like INI file which is parsed somehow and converted 
to internal settings object.
The INI file has a shallow structure (options grouped in sections), 
can be easily extended / overrided / included (look at Buidout as an 
example),
and ENV keys will be related to flat INI instead of nested settings.py.

Let's consider an example:

[databases]
default.url = postgres+psycopg2://user:pwd@localhost:5432/exampledb
default.engine = django.db.backends.postgres_psycopg2


which can be simply reconfigured by ENV vars like:

DJANGO_DATABASES_DEFAULT_URL="postgres+psycopg2://user:pwd@localhost:5432/exampledb"
DJANGO_DATABASES_DEFAULT_ENGINE="django.db.backends.postgres_psycopg2"

OR add some preprocessing and flexibility:



[databases]
default.url = ${env:SOME-USER-KEY1}
default.engine = ${env:SOME-USER-KEY2}


[env]
SOME_USER_KEY1=a default value for default database URL
SOME_USER_KEY2=a default value for default database engine


where [env] section will by filled automatically by Django from os.environ.

This also gives a possibility to extend settings.ini via development.ini, 
where [databases] can be overridden with developer settings (without env 
access)


Thanks for reading and sorry for my English. I hope you understood the idea.
Marcin

 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6dbaaaca-8de7-49d3-a7b8-df011b94f147%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Making Django more PaaS-friendly

2016-04-26 Thread Hanne Moa
On 12 April 2016 at 05:35, Martin Owens  wrote:
> There is a difference between settings intended by developers to be used by
> other developers /../ And settings used by systems administrators to define
> the environment they are deployed to as well as any deployment specific
> controls.
>
> Django has always mixed the two into one file, and it causes problems. I've
> seen keys and passwords committed to github, extravagant json or os.env
> based solutions with piles of lines of messy code to what I see most which
> is a local_settings.py with a template version which is copied in at runtime
> with sane defaults.

I used to use extravagant json messes[*] but have (on non-PaaS)
switched to having a directory /etc/django-sites.d, which is on the
python path, containing one python package per site on that host:

/etc/django-sites.d/MyProjectConf/sitetype.py

MyProject.settings.sitetype is referenced by wsgi-scripts. It first
star-imports MyProject.settings.base, then star-imports
MyProjectConf.sitetype inside a try/except.

That way I can have have important overrides/option-merging for a
sitetype in version control, and still have
puppet/chef/ansible/whatever control deploy-and-site-specific options
without touching anything inside the source tree.

> I'd be happy to see a canonical decision made for django, but I'd focus on
> "who should be editing this setting" not just what it's for, if it's a
> secret or if it should be different or the same per instance.

A tag in the settings-docs would be a good start, or a check list of
more site dependent settings. Hey, the docs are restructured text; the
tag could generate an extra index, et voila :)


-- 
HM

[*] I *know* why json doesn't survive that spurious comma at the end
of a list/dict, but I like me that comma so much that I no longer use
json for hand-written config in new stuff.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CACQ%3Drrd3GyFgpxkCJkNegnk62EmR1VzVvS5OToeV6Prns4qWeA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Making Django more PaaS-friendly

2016-04-20 Thread andrew
We deploy django use AWS Elastic Beanstalk and use credstash for secrets:

https://github.com/fugue/credstash

So we can just do PASSWORD = credstash.getSecret('password') which works 
pretty well with the AWS Key Management Service.

What is a pain is that the AWS Health Check does not set the Host header 
and we don't know until runtime what the IP address of the server will be. 
We have to do a call to the local metadata server to find out the IP 
address then add it to ALLOWED_HOSTS which is really quite hideous. There 
is an explanation here:

https://dryan.com/articles/elb-django-allowed-hosts/

Personally I would quite like to see an equivalent of the @csrf_exempt 
decorator perhaps @allowed_hosts_exempt that could be use to mark the 
health check method.

Andrew



-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5868cb37-2d59-4792-95b6-146a56309ad8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Making Django more PaaS-friendly

2016-04-19 Thread Mike Bryant
On 11 April 2016 at 07:33, James Bennett  wrote:
> The main problem with this is that it's really not very elegant. But at the
> moment I can't think of anything better, and so I'd like to throw the floor
> open to ideas on nicer approaches to this. If one can't be found, I do think
> Django 1.10 should at least figure out how to handle database config from
> env since that's such a common use case nowadays, but ideally we'd be able
> to pin down a good API for generically pulling configuration from the
> environment.

Sorry for a rather belated response, but I'd like to share our approach to this.
We wrote django-autoconfig [1] to help with our settings. For some
background, it's main role is to allow reusable apps to define
settings that they must have to work effectively (e.g. additional
middleware),

Of relevance though is, to aid in our deployments into AWS we went
with pulling in environment variables named DJANGO_{setting_name} and
coercing them into valid settings.

I mention this here as it seems to be of relevance to the audience,
and I'd welcome any feedback. Apologies if this is just noise.

Mike

[1] https://github.com/mikebryant/django-autoconfig

-- 


Notice:  This email is confidential and may contain copyright material of 
members of the Ocado Group. Opinions and views expressed in this message 
may not necessarily reflect the opinions and views of the members of the 
Ocado Group. 

 

If you are not the intended recipient, please notify us immediately and 
delete all copies of this message. Please note that it is your 
responsibility to scan this message for viruses. 

 

Fetch and Sizzle are trading names of Speciality Stores Limited, a member 
of the Ocado Group.

 

References to the “Ocado Group” are to Ocado Group plc (registered in 
England and Wales with number 7098618) and its subsidiary undertakings (as 
that expression is defined in the Companies Act 2006) from time to time.  
The registered office of Ocado Group plc is Titan Court, 3 Bishops Square, 
Hatfield Business Park, Hatfield, Herts. AL10 9NE.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAPkekMtB-dq243jKHqXJ7GxUDq2bKjNdi4Ti_3MJw81y_SdzUg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Making Django more PaaS-friendly

2016-04-15 Thread Curtis Maloney

On 14/04/16 05:57, Carl Meyer wrote:

Hi James et al,

In general, I like the idea of adding a helper to Django to read
settings from the environment. I think this helper should be kept as
simple and non-magical as is reasonable. Thus:

- I'm in favor of a flexible helper function that can be manually used
in a settings file to read an explicitly-named env var and coerce it
using any arbitrary callable (that gives enough power to "coerce" using
something as simple as `int` or something as complex as
`parse_database_url` that turns a DSN string into a Django db dict),
with some handling of defaults. Below I give some usage examples of the
function I use for that, and link to its code.


+1

This I've found works well in d-c-s...


- I'm opposed to anything more magical than that, e.g. anything that
tries to set up an all-dancing automatic mapping or translation between
env var names and setting names, or tries to enforce some particular env
var prefix. The minor gains in conciseness with this kind of thing
aren't nearly worth the losses in flexibility in how you construct your
settings file or name your env vars.


+1

Adding more policy will only complicate matters... if you just provide 
the mechanism, people will surprise you with innovative policies :)



- I'm also opposed to conflating the issue of reading env vars with
orthogonal stuff like class-based settings or patterns for using
multiple settings files.


+1

Once again, there's no need to dictate more than necessary.

--
C

--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/57108E2C.9050803%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.


Re: Making Django more PaaS-friendly

2016-04-13 Thread Carl Meyer
Hi James et al,

In general, I like the idea of adding a helper to Django to read
settings from the environment. I think this helper should be kept as
simple and non-magical as is reasonable. Thus:

- I'm in favor of a flexible helper function that can be manually used
in a settings file to read an explicitly-named env var and coerce it
using any arbitrary callable (that gives enough power to "coerce" using
something as simple as `int` or something as complex as
`parse_database_url` that turns a DSN string into a Django db dict),
with some handling of defaults. Below I give some usage examples of the
function I use for that, and link to its code.

- I'm opposed to anything more magical than that, e.g. anything that
tries to set up an all-dancing automatic mapping or translation between
env var names and setting names, or tries to enforce some particular env
var prefix. The minor gains in conciseness with this kind of thing
aren't nearly worth the losses in flexibility in how you construct your
settings file or name your env vars.

- I'm also opposed to conflating the issue of reading env vars with
orthogonal stuff like class-based settings or patterns for using
multiple settings files.

More comments (and sample code) below:

On 04/11/2016 12:26 PM, Aymeric Augustin wrote:
>> On 11 Apr 2016, at 19:39, Curtis Maloney  wrote:
>>
>> 1. All env vars can have a fallback value (the method in my case)
>> 2. You can mark an env var as not having a fallback value, and it will raise 
>> an error at start up if not set.
> 
> There’s an additional complexity here.
> 
> Usually, it’s best for settings read from environment variables:
> 
> a. To use a default value in dev, even if that means a small degradation in
> functionality. This allows developers to start working on the project without
> adding dozens of exports to their $VIRTUAL_ENV/bin/postactivate, and to add
> just the values they need when they work on specific parts like integrations
> with third-party systems.
> 
> b. To crash if no value is provided in prod, in order to catch configuration
> errors upfront.
> 
> One might think of switching the behavior depending on settings.DEBUG, but
> that won't work because the switch is required to load settings properly.

I totally agree with all this. My projects have a separate MODE setting
which is either 'dev' or 'prod', and I use a utility to read settings
from the environment which supports variable defaults (or lack of
default) by mode, exactly as you outline above. Specifically, it works
like this:


env = EnvParser('PRJ_MODE')

SECRET_KEY = env('PRJ_SECRET_KEY', default={'dev': 'dev-secret'})

The effect of this is that the mode ('dev' or 'prod') will be read from
the `PRJ_MODE` env var (usually `PRJ` would be a short code specific to
the project name). Then if the mode is 'dev', no `PRJ_SECRET_KEY` env
var is required, and 'dev-secret' will be used as the default. But if
the mode is 'prod', the server will fail to start up unless
`PRJ_SECRET_KEY` is found in the environment.

I wrote this to allow for any number of arbitrarily-named modes, but in
practice I've never used anything but 'dev' and 'prod' (staging and demo
sites are generally set up as 'prod'). Hardcoding to those two modes
would perhaps allow a nicer syntax for specifying defaults, by using
separate kwargs instead of the dict.

Anyway, FWIW, here's the code I use:
https://gist.github.com/carljm/69b8e351dac87f4c3f5b440632727fdb

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/570EA4C6.6080700%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Making Django more PaaS-friendly

2016-04-13 Thread aRkadeFR
I like the try to move things towards this kind of settings more 
PaaS-friendly.


As a Django user, I'd like to see 'DSN' or 'URL' for the database as a 
built-in in Django. I don't

mind setting it in the dict as so:
```
DATABASES = {
'default': {
'DSN': 'postgres://user:password@ip:port/name_db',
}
}
```

and I use the same configuration layout as aymeric augustin, with most 
of the env name prefix

with ``DJANGO_``.

I'm not exactly a PaaS user, but I package my code with dh-virtualenv 
under .deb which has

some similar constraints as the PaaS out there.


On 04/11/2016 08:33 AM, James Bennett wrote:
Apologies for how late in the process this is; job hunt + moving 
cross-country ate a lot of my time.


At Django Under the Hood I proposed that for Django 1.10 we try to 
find some common best practices for deploying Django on popular 
platform-as-a-service (PaaS) solutions, and building support for them 
into Django. The biggest one of these is probably having the ability 
to read configuration from environment variables instead of 
hard-coding things into settings files.


At the very least I'd like to propose (assuming Kenneth is on board 
with it) integrating dj-database-url[1] or something like it directly 
into Django, so that there's no longer a third-party dependency for 
reading the database configuration from an environment variable. 
Whether this is just porting dj-database-url itself in, or making the 
DATABASES setting understand URLs, I'm unsure of yet and would be 
interested to hear feedback on. Either way I'm willing to put in the 
time to develop the patch.


More generally, I'd like to see Django grow helpers for specifying 
settings that should be read from environment variables, and which 
environment variables to use (the email settings are another common 
case, as is anything that requires an API key to access).


There are a few ways to design this. One option would be just a 
minimal wrapper around os.getenv, perhaps taking an optional type or 
type-coercing function, so that it would be possible in a settings 
file to do:


SECRET_NUMBER_SETTING = env_setting('SECRET_NUMBER', int)

However, this is not much better than the current practice of calling 
os.getenv. A better solution might be the ability to specify a group 
of settings which will be read from the environment, and have Django 
automatically read and set them. For example:


ENV_SETTINGS = [
('SECRET_NUMBER_SETTING', int),
('ACME_API_KEY', str),
('VOLCANO_LAIR_PASSWORD', str),
]

would read the named settings from those environment variables, and 
coerce them to the appropriate types using the function provided.


The main problem with this is that it's really not very elegant. But 
at the moment I can't think of anything better, and so I'd like to 
throw the floor open to ideas on nicer approaches to this. If one 
can't be found, I do think Django 1.10 should at least figure out how 
to handle database config from env since that's such a common use case 
nowadays, but ideally we'd be able to pin down a good API for 
generically pulling configuration from the environment.



[1] https://github.com/kennethreitz/dj-database-url
--
You received this message because you are subscribed to the Google 
Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to django-developers+unsubscr...@googlegroups.com 
.
To post to this group, send email to 
django-developers@googlegroups.com 
.

Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAL13Cg85fYj0i0ysxJGo2SDesqELMeA%2BtnKJ9cdpqNHmQ%3DX3Pg%40mail.gmail.com 
.

For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/570E0733.5040607%40arkade.info.
For more options, visit https://groups.google.com/d/optout.


Re: Making Django more PaaS-friendly

2016-04-11 Thread Martin Owens
I look after many different django websites and I'd like to add something 
conceptual.

There is a difference between settings intended by developers to be used by 
other developers in order to enable and configure their app's behaviour so 
it work with other code. (I call these developer settings)
And settings used by systems administrators to define the environment they 
are deployed to as well as any deployment specific controls. (I call these 
sysadmin settings)

Django has always mixed the two into one file, and it causes problems. I've 
seen keys and passwords committed to github, extravagant json or os.env 
based solutions with piles of lines of messy code to what I see most which 
is a local_settings.py with a template version which is copied in at 
runtime with sane defaults.

I'd be happy to see a canonical decision made for django, but I'd focus on 
"who should be editing this setting" not just what it's for, if it's a 
secret or if it should be different or the same per instance.

Best regards, Martin Owens

On Monday, April 11, 2016 at 7:33:54 AM UTC+1, James Bennett wrote:
>
> Apologies for how late in the process this is; job hunt + moving 
> cross-country ate a lot of my time.
>
> At Django Under the Hood I proposed that for Django 1.10 we try to find 
> some common best practices for deploying Django on popular 
> platform-as-a-service (PaaS) solutions, and building support for them into 
> Django. The biggest one of these is probably having the ability to read 
> configuration from environment variables instead of hard-coding things into 
> settings files.
>
> At the very least I'd like to propose (assuming Kenneth is on board with 
> it) integrating dj-database-url[1] or something like it directly into 
> Django, so that there's no longer a third-party dependency for reading the 
> database configuration from an environment variable. Whether this is just 
> porting dj-database-url itself in, or making the DATABASES setting 
> understand URLs, I'm unsure of yet and would be interested to hear feedback 
> on. Either way I'm willing to put in the time to develop the patch.
>
> More generally, I'd like to see Django grow helpers for specifying 
> settings that should be read from environment variables, and which 
> environment variables to use (the email settings are another common case, 
> as is anything that requires an API key to access).
>
> There are a few ways to design this. One option would be just a minimal 
> wrapper around os.getenv, perhaps taking an optional type or type-coercing 
> function, so that it would be possible in a settings file to do:
>
> SECRET_NUMBER_SETTING = env_setting('SECRET_NUMBER', int)
>
> However, this is not much better than the current practice of calling 
> os.getenv. A better solution might be the ability to specify a group of 
> settings which will be read from the environment, and have Django 
> automatically read and set them. For example:
>
> ENV_SETTINGS = [
> ('SECRET_NUMBER_SETTING', int),
> ('ACME_API_KEY', str),
> ('VOLCANO_LAIR_PASSWORD', str),
> ]
>
> would read the named settings from those environment variables, and coerce 
> them to the appropriate types using the function provided.
>
> The main problem with this is that it's really not very elegant. But at 
> the moment I can't think of anything better, and so I'd like to throw the 
> floor open to ideas on nicer approaches to this. If one can't be found, I 
> do think Django 1.10 should at least figure out how to handle database 
> config from env since that's such a common use case nowadays, but ideally 
> we'd be able to pin down a good API for generically pulling configuration 
> from the environment.
>
>
> [1] https://github.com/kennethreitz/dj-database-url
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f0ff81bc-0eb0-476e-8741-6ba7e7db7926%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Making Django more PaaS-friendly

2016-04-11 Thread Joey Wilhelm
Just to throw another voice in here, I'm currently using django-environ[1],
as mentioned by Sean Brant. In addition, I'm using a settings setup based
on cookiecutter-django[2]. This means having my settings split into
`config.settings.common`, `config.settings.local`,
`config.settings.production`, for example. In common, I have things like
the following:

DATABASES = {
# Raises ImproperlyConfigured exception if DATABASE_URL not in
os.environ
'default': env.db("DATABASE_URL"),
}
EMAIL_BACKEND = env('DJANGO_EMAIL_BACKEND',
default='django.core.mail.backends.smtp.EmailBackend')
# ...etc...

These would obviously propagate up and be used in the production settings.
Then in the local settings, I have things like:

SECRET_KEY = env('DJANGO_SECRET_KEY', default='CHANGEME!!!')
EMAIL_BACKEND = env('DJANGO_EMAIL_BACKEND',
default='django.core.mail.backends.console.EmailBackend')
DATABASES = {
'default': env.db('DATABASE_URL',
default='postgis://django:supersecretpassword@localhost:5432/django'),
}
# ...etc...

I don't know if this is all necessarily the best way to do it, but it has
been working quite well for me. Everything is read out of a .env file, and
I provide a template.env file in my project to show which values need to be
set in there. `DJANGO_SETTINGS_MODULE` defaults to production, so that
missing environment keys will raise alarms by default, and then when
possible, those keys become optional in development settings.

-Joey

[1]: https://github.com/joke2k/django-environ
[2]: https://github.com/pydanny/cookiecutter-django

On Mon, Apr 11, 2016 at 10:39 AM, Curtis Maloney 
wrote:

> Just want to throw my 3c in here...
>
> Firstly, for reference, I wrote django-classy-settings to help with
> various settings related problems, one of which being env-based setting.
>
> https://github.com/funkybob/django-classy-settings
>
> As my env decorators are written to be class properties, they don't quite
> fit this discussion... however, things to consider:
>
> 1. All env vars can have a fallback value (the method in my case)
> 2. You can mark an env var as not having a fallback value, and it will
> raise an error at start up if not set.
> 3. Non-trivial value types are not easy.
>
> The URL series of settings helpers deal with many of these cases fairly
> well, as they allow us to specify a backend (protocol), username, password,
> host, port, and positional and keyword arguments.
>
> Registering more protocols is actually quite easy, all told.
>
> However, it becomes tricky when we allow multiples, such as database and
> cache backends.  Do we settle on, for instance, a pattern of
> "DJANGO_DATABASE_{name}="? Or some other scheme?
>
> Email settings are certainly another big one, IME... as well as AWS keys...
>
> --
> C
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/570BE142.3000209%40tinbrain.net
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADBkHdK5yhxWoDB6yygnFC%2BfmPCn3n6x%3DJoxbym2HuTJ3oyDVQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Making Django more PaaS-friendly

2016-04-11 Thread Curtis Maloney

On 12/04/16 04:26, Aymeric Augustin wrote:

On 11 Apr 2016, at 19:39, Curtis Maloney  wrote:

1. All env vars can have a fallback value (the method in my case)
2. You can mark an env var as not having a fallback value, and it will raise an 
error at start up if not set.


There’s an additional complexity here.

Usually, it’s best for settings read from environment variables:

a. To use a default value in dev, even if that means a small degradation in
functionality. This allows developers to start working on the project without
adding dozens of exports to their $VIRTUAL_ENV/bin/postactivate, and to add
just the values they need when they work on specific parts like integrations
with third-party systems.

b. To crash if no value is provided in prod, in order to catch configuration
errors upfront.

One might think of switching the behavior depending on settings.DEBUG, but
that won't work because the switch is required to load settings properly.


This is all true it's generally not an issue when using 
django-classy-settings as it was engineered to make it easy to switch 
between settings environments such as dev/staging/testing/production.



I’ve seen lots of wrapper that don’t add enough value to be worth it. These
days I just write:

# settings/base.py

...

# settings/dev.py

FOO_TOKEN = os.environ.get('FOO_TOKEN', '')

# settings/prod.py

FOO_TOKEN = os.environ['FOO_TOKEN']

It isn’t as dry as it could be, but at least it’s simple.


This is one thing d-c-s helps you avoid... by letting you declare it all 
in one file, inherit complex/dependent settings through classes, and

switch between those with an env var [or whatever mechanism you like]

However, this is slightly tangential to the actual discussion

--
C

--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/570BED47.7040602%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.


Re: Making Django more PaaS-friendly

2016-04-11 Thread Aymeric Augustin
> On 11 Apr 2016, at 19:39, Curtis Maloney  wrote:
> 
> 1. All env vars can have a fallback value (the method in my case)
> 2. You can mark an env var as not having a fallback value, and it will raise 
> an error at start up if not set.

There’s an additional complexity here.

Usually, it’s best for settings read from environment variables:

a. To use a default value in dev, even if that means a small degradation in
functionality. This allows developers to start working on the project without
adding dozens of exports to their $VIRTUAL_ENV/bin/postactivate, and to add
just the values they need when they work on specific parts like integrations
with third-party systems.

b. To crash if no value is provided in prod, in order to catch configuration
errors upfront.

One might think of switching the behavior depending on settings.DEBUG, but
that won't work because the switch is required to load settings properly.

I’ve seen lots of wrapper that don’t add enough value to be worth it. These
days I just write:

# settings/base.py

...

# settings/dev.py

FOO_TOKEN = os.environ.get('FOO_TOKEN', '')

# settings/prod.py

FOO_TOKEN = os.environ['FOO_TOKEN']

It isn’t as dry as it could be, but at least it’s simple.

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/A0573632-AE0B-4243-AA63-4462C5596D75%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.


Re: Making Django more PaaS-friendly

2016-04-11 Thread Curtis Maloney

Just want to throw my 3c in here...

Firstly, for reference, I wrote django-classy-settings to help with 
various settings related problems, one of which being env-based setting.


https://github.com/funkybob/django-classy-settings

As my env decorators are written to be class properties, they don't 
quite fit this discussion... however, things to consider:


1. All env vars can have a fallback value (the method in my case)
2. You can mark an env var as not having a fallback value, and it will 
raise an error at start up if not set.

3. Non-trivial value types are not easy.

The URL series of settings helpers deal with many of these cases fairly 
well, as they allow us to specify a backend (protocol), username, 
password, host, port, and positional and keyword arguments.


Registering more protocols is actually quite easy, all told.

However, it becomes tricky when we allow multiples, such as database and 
cache backends.  Do we settle on, for instance, a pattern of 
"DJANGO_DATABASE_{name}="? Or some other scheme?


Email settings are certainly another big one, IME... as well as AWS keys...

--
C

--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/570BE142.3000209%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.


Re: Making Django more PaaS-friendly

2016-04-11 Thread Sean Brant
https://github.com/joke2k/django-environ has helpers for loading most of
the backends from urls. We use it with a docker based deployment at the
moment.

On Mon, Apr 11, 2016 at 8:46 AM, Ryan Hiebert  wrote:

> I'd love to see better support for PaaS configuration, especially
> 12-factor. We use Heroku, and I've been directly exposed to the challenges
> and had to come up with some of my own solutions. Here's some thoughts I
> have on the matter, in no particular order.
>
>
> The SECRET_KEY needs to _not_ be required for the `collectstatic` command.
> I haven't tracked down why it currently _is_ required, but it means that I
> have added this line to my project, solely so that my project can be built
> on Heroku.
>
> SECRET_KEY = os.environ.get('SECRET_KEY', 'not-so-secret')
>
> SECRET_KEY is effectively optional (though I'm careful to make sure it's
> set), and I don't see a good reason that should be. Note that environment
> variables aren't normally available during slug build, since changing
> environment variables doesn't re-build the project.
>
>
> I believe that the biggest current impediment to finding a great solution
> to loading Django settings is that the settings are based on a Module.
> Django-Configurations has done an excellent job of making that into a
> class-based API instead, which allows for mixins and inheritance. Having
> built-in support for this would be an excellent way to allow 3rd party
> solutions to build up settings programatically. In the case we are
> discussing, from the environment.
>
> I believe this would make it far easier for projects like Kristian's
> django-12factor and dj-email-url to work well. dj-email-url is a good idea
> (put all the settings in 1 variable, like dj-database-url), except that all
> the settings are split up at the top-level of the module, making what's
> conceptually one multi-facted setting a multi-line pain.
>
> One possibility would be extending `DJANGO_SETTINGS_MODULE` with an
> optional class, separated with a colon, so to get a django-configurations
> style class you'd use
> `DJANGO_SETTINGS_MODULE=myproject.settings:MySettings`. Or we could change
> the variable name `DJANGO_SETTINGS`, which would allow us to avoid the
> colon: `DJANGO_SETTINGS=myproject.settings.MySettings`. Or paint the
> bikeshed some other color.
>
>
> I'd love to see some of the support for environment configurations brought
> into Django, but IMO moving away from a module based settings API is a more
> pressing concern that would enable 3rd party libraries to come up with
> elegant solutions.
>
>
> On Apr 11, 2016, at 4:32 AM, Kristian Glass <
> googleto...@doismellburning.co.uk> wrote:
>
> I wrote django12factor to do something similar. One of the things I like
> least about it is the process of actually using it from your settings.py -
> there's things in there I'd love to see in the generated default.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/2D66F14E-DD71-41B6-A84D-A0C679FD72F5%40ryanhiebert.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAPNuhQxbRKX%3D2FrCd-jg6MJCnO%2BuBC6HbTOpPUdQJAjQjyMDSw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Making Django more PaaS-friendly

2016-04-11 Thread Ryan Hiebert
I'd love to see better support for PaaS configuration, especially 12-factor. We 
use Heroku, and I've been directly exposed to the challenges and had to come up 
with some of my own solutions. Here's some thoughts I have on the matter, in no 
particular order.


The SECRET_KEY needs to _not_ be required for the `collectstatic` command. I 
haven't tracked down why it currently _is_ required, but it means that I have 
added this line to my project, solely so that my project can be built on Heroku.

SECRET_KEY = os.environ.get('SECRET_KEY', 'not-so-secret')

SECRET_KEY is effectively optional (though I'm careful to make sure it's set), 
and I don't see a good reason that should be. Note that environment variables 
aren't normally available during slug build, since changing environment 
variables doesn't re-build the project.


I believe that the biggest current impediment to finding a great solution to 
loading Django settings is that the settings are based on a Module. 
Django-Configurations has done an excellent job of making that into a 
class-based API instead, which allows for mixins and inheritance. Having 
built-in support for this would be an excellent way to allow 3rd party 
solutions to build up settings programatically. In the case we are discussing, 
from the environment.

I believe this would make it far easier for projects like Kristian's 
django-12factor and dj-email-url to work well. dj-email-url is a good idea (put 
all the settings in 1 variable, like dj-database-url), except that all the 
settings are split up at the top-level of the module, making what's 
conceptually one multi-facted setting a multi-line pain.

One possibility would be extending `DJANGO_SETTINGS_MODULE` with an optional 
class, separated with a colon, so to get a django-configurations style class 
you'd use `DJANGO_SETTINGS_MODULE=myproject.settings:MySettings`. Or we could 
change the variable name `DJANGO_SETTINGS`, which would allow us to avoid the 
colon: `DJANGO_SETTINGS=myproject.settings.MySettings`. Or paint the bikeshed 
some other color.


I'd love to see some of the support for environment configurations brought into 
Django, but IMO moving away from a module based settings API is a more pressing 
concern that would enable 3rd party libraries to come up with elegant solutions.


> On Apr 11, 2016, at 4:32 AM, Kristian Glass 
>  wrote:
> 
> I wrote django12factor to do something similar. One of the things I like 
> least about it is the process of actually using it from your settings.py - 
> there's things in there I'd love to see in the generated default.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/2D66F14E-DD71-41B6-A84D-A0C679FD72F5%40ryanhiebert.com.
For more options, visit https://groups.google.com/d/optout.


Re: Making Django more PaaS-friendly

2016-04-11 Thread Kristian Glass
Very very glad to hear this. All too frequently in #django, "please show us 
your settings (and remove any sensitive data)" ends up with a "Now you need 
to reset your SECRET_KEY" etc.

I wrote django12factor to do something similar. One of the things I like 
least about it is the process of actually using it from your settings.py - 
there's things in there I'd love to see in the generated default.

Right now it handles most settings individually and only addresses "the 
most common ones" - a simple boolean handler for DEBUG, string splitting 
for ALLOWED_HOSTS etc., and outsources CACHES / DATABASES / EMAIL* to 
django-cache-url, dj-database-url and dj-email-url respectively. This isn't 
ideal, BUT even without getting into generically useful helpers, I think a 
lot of those could be useful in the standard template...

https://github.com/doismellburning/django12factor/pull/19/files is a branch 
where I prototyped something similar to your type coercion idea; you may 
find it more elegant, ymmv - I'd be interested in your thoughts.

On Monday, April 11, 2016 at 7:33:54 AM UTC+1, James Bennett wrote:
>
> Apologies for how late in the process this is; job hunt + moving 
> cross-country ate a lot of my time.
>
> At Django Under the Hood I proposed that for Django 1.10 we try to find 
> some common best practices for deploying Django on popular 
> platform-as-a-service (PaaS) solutions, and building support for them into 
> Django. The biggest one of these is probably having the ability to read 
> configuration from environment variables instead of hard-coding things into 
> settings files.
>
> At the very least I'd like to propose (assuming Kenneth is on board with 
> it) integrating dj-database-url[1] or something like it directly into 
> Django, so that there's no longer a third-party dependency for reading the 
> database configuration from an environment variable. Whether this is just 
> porting dj-database-url itself in, or making the DATABASES setting 
> understand URLs, I'm unsure of yet and would be interested to hear feedback 
> on. Either way I'm willing to put in the time to develop the patch.
>
> More generally, I'd like to see Django grow helpers for specifying 
> settings that should be read from environment variables, and which 
> environment variables to use (the email settings are another common case, 
> as is anything that requires an API key to access).
>
> There are a few ways to design this. One option would be just a minimal 
> wrapper around os.getenv, perhaps taking an optional type or type-coercing 
> function, so that it would be possible in a settings file to do:
>
> SECRET_NUMBER_SETTING = env_setting('SECRET_NUMBER', int)
>
> However, this is not much better than the current practice of calling 
> os.getenv. A better solution might be the ability to specify a group of 
> settings which will be read from the environment, and have Django 
> automatically read and set them. For example:
>
> ENV_SETTINGS = [
> ('SECRET_NUMBER_SETTING', int),
> ('ACME_API_KEY', str),
> ('VOLCANO_LAIR_PASSWORD', str),
> ]
>
> would read the named settings from those environment variables, and coerce 
> them to the appropriate types using the function provided.
>
> The main problem with this is that it's really not very elegant. But at 
> the moment I can't think of anything better, and so I'd like to throw the 
> floor open to ideas on nicer approaches to this. If one can't be found, I 
> do think Django 1.10 should at least figure out how to handle database 
> config from env since that's such a common use case nowadays, but ideally 
> we'd be able to pin down a good API for generically pulling configuration 
> from the environment.
>
>
> [1] https://github.com/kennethreitz/dj-database-url
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a324953c-95f0-4efa-8e22-993eb48096d1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Making Django more PaaS-friendly

2016-04-11 Thread Raphaël Barrois
Hi James,

>From the experience on our projects, the ``CONFIG.getstr('db.password')`` 
>format works well in settings files:
- It makes it clear that this field is loaded from the environment, and what 
type is expected
- The setting is set and loaded in the same place
- It allows for type checking
- It handles loading other values

This is easier than automagically altering settings based on environment 
variables, as the user might set a value in
some part of its settings file, yet find it overridden by some later 
declaration of ``ENV_SETTINGS``.



In my opinion, a few important questions to address would be:
- What is the format of those environment variables? For instance, using a 
'DJANGO_' prefix will avoid name conflicts
- Can developers use this "environment configuration parser" tool to parse 
extra, custom settings; or are we limited to
  Django's needs?
- Should we only support "loading from the environment", or is it useful to 
allow loading from a ini-style file
  (typically to override some values on a local development environment)?


We wrote a small helper to load configuration from various sources (environment 
and configuration files); this could be
helpful for designing this new feature: 
http://getconf.readthedocs.org/en/latest/

Feel free to ping me if you're interested in more feedback from building and 
using that kind of setup :)


On Sun, 10 Apr 2016 23:33:46 -0700
James Bennett  wrote:

> Apologies for how late in the process this is; job hunt + moving
> cross-country ate a lot of my time.
> 
> At Django Under the Hood I proposed that for Django 1.10 we try to find
> some common best practices for deploying Django on popular
> platform-as-a-service (PaaS) solutions, and building support for them into
> Django. The biggest one of these is probably having the ability to read
> configuration from environment variables instead of hard-coding things into
> settings files.
> 
> At the very least I'd like to propose (assuming Kenneth is on board with
> it) integrating dj-database-url[1] or something like it directly into
> Django, so that there's no longer a third-party dependency for reading the
> database configuration from an environment variable. Whether this is just
> porting dj-database-url itself in, or making the DATABASES setting
> understand URLs, I'm unsure of yet and would be interested to hear feedback
> on. Either way I'm willing to put in the time to develop the patch.
> 
> More generally, I'd like to see Django grow helpers for specifying settings
> that should be read from environment variables, and which environment
> variables to use (the email settings are another common case, as is
> anything that requires an API key to access).
> 
> There are a few ways to design this. One option would be just a minimal
> wrapper around os.getenv, perhaps taking an optional type or type-coercing
> function, so that it would be possible in a settings file to do:
> 
> SECRET_NUMBER_SETTING = env_setting('SECRET_NUMBER', int)
> 
> However, this is not much better than the current practice of calling
> os.getenv. A better solution might be the ability to specify a group of
> settings which will be read from the environment, and have Django
> automatically read and set them. For example:
> 
> ENV_SETTINGS = [
> ('SECRET_NUMBER_SETTING', int),
> ('ACME_API_KEY', str),
> ('VOLCANO_LAIR_PASSWORD', str),
> ]
> 
> would read the named settings from those environment variables, and coerce
> them to the appropriate types using the function provided.
> 
> The main problem with this is that it's really not very elegant. But at the
> moment I can't think of anything better, and so I'd like to throw the floor
> open to ideas on nicer approaches to this. If one can't be found, I do
> think Django 1.10 should at least figure out how to handle database config
> from env since that's such a common use case nowadays, but ideally we'd be
> able to pin down a good API for generically pulling configuration from the
> environment.
> 
> 
> [1] https://github.com/kennethreitz/dj-database-url
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20160411143238.6565a96a%40ithor.polyconseil.fr.
For more options, visit https://groups.google.com/d/optout.


Re: Making Django more PaaS-friendly

2016-04-11 Thread Tim Graham
Here is some past work on "Minimize the risk of SECRET_KEY leaks" [0] that 
was never completed: https://github.com/django/django/pull/2714

[0] https://code.djangoproject.com/ticket/20081

On Monday, April 11, 2016 at 6:51:38 AM UTC-4, Josh Smeaton wrote:
>
> I kind of like the idea of making all settings configurable via the 
> environment by prefixing with DJANGO_SETTINGNAME. Sort of like how click 
> allows environment variables for options: 
> http://click.pocoo.org/5/options/#values-from-environment-variables. 
> Ideally configuring settings from the environment should be as transparent 
> as possible to the settings file, so that users aren't required to 
> os.getenv('NAME', 'default') all over the place to derive value.
>
> Django uses quite a few dicts and lists though, so consideration would 
> need to be made for those. Perhaps double underscores could represent keys 
> within a dict?
>
> DJANGO_CACHES__default__BACKEND='redis'
>
> For lists we could split on whitespace (again like Click): 
>
> DJANGO_INSTALLED_APPS="contrib.admin contrib.auth".
>
> This does not address the type of the value though. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a334c7ec-5b60-43c9-8fe5-6d476e33debb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Making Django more PaaS-friendly

2016-04-11 Thread Josh Smeaton
I kind of like the idea of making all settings configurable via the 
environment by prefixing with DJANGO_SETTINGNAME. Sort of like how click 
allows environment variables for 
options: http://click.pocoo.org/5/options/#values-from-environment-variables. 
Ideally configuring settings from the environment should be as transparent 
as possible to the settings file, so that users aren't required to 
os.getenv('NAME', 'default') all over the place to derive value.

Django uses quite a few dicts and lists though, so consideration would need 
to be made for those. Perhaps double underscores could represent keys 
within a dict?

DJANGO_CACHES__default__BACKEND='redis'

For lists we could split on whitespace (again like Click): 

DJANGO_INSTALLED_APPS="contrib.admin contrib.auth".

This does not address the type of the value though. 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1af1c252-e17a-4e05-816c-c31493e7bcac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Making Django more PaaS-friendly

2016-04-11 Thread Aymeric Augustin
We have a chicken’n’egg problem to support third party database backends with 
dj-database-url’s current API:

- It is each backend’s responsibility to parse the URL and extract the 
configuration for a given database.
- Parsing the URL is a prerequisite to figure which database backend to pick 
for parsing the URL.

I think we need a specific convenience API such as:

DATABASES = {
‘default’: {
‘NAME’: ‘python.path.to.backend’,
‘URL’: ‘backend://...',
}
}

-- 
Aymeric.

> On 11 Apr 2016, at 11:27, Shai Berger  wrote:
> 
> Django-database-url is very nice, but AFAICT has no way to support 3rd-party 
> backends. I think it needs to grow this support before it can be used in 
> Django. We can adapt the backend API to help, but it's a little tricky if we 
> don't want to import unused backend.
> 
> On 11 באפריל 2016 11:13:23 GMT+03:00, Marc Tamlyn  > wrote:
> I like the idea of integrating dj-database-url. There's a similar package for 
> CACHES[0], which is a neat idea but slightly more tricky to justify as I 
> don't know if PaaS tend to send the cache details in that format necessarily.
> 
> I'm not sure about the rest of the mail, although I'd definitely be up for 
> modifying the SECRET_KEY line in particular to make it strongly suggest that 
> this should be different per environment.
> 
> M
> 
> 
> [0] https://github.com/ghickman/django-cache-url 
> 
> 
> On 11 April 2016 at 07:33, James Bennett  > wrote:
> Apologies for how late in the process this is; job hunt + moving 
> cross-country ate a lot of my time.
> 
> At Django Under the Hood I proposed that for Django 1.10 we try to find some 
> common best practices for deploying Django on popular platform-as-a-service 
> (PaaS) solutions, and building support for them into Django. The biggest one 
> of these is probably having the ability to read configuration from 
> environment variables instead of hard-coding things into settings files.
> 
> At the very least I'd like to propose (assuming Kenneth is on board with it) 
> integrating dj-database-url[1] or something like it directly into Django, so 
> that there's no longer a third-party dependency for reading the database 
> configuration from an environment variable. Whether this is just porting 
> dj-database-url itself in, or making the DATABASES setting understand URLs, 
> I'm unsure of yet and would be interested to hear feedback on. Either way I'm 
> willing to put in the time to develop the patch.
> 
> More generally, I'd like to see Django grow helpers for specifying settings 
> that should be read from environment variables, and which environment 
> variables to use (the email settings are another common case, as is anything 
> that requires an API key to access).
> 
> There are a few ways to design this. One option would be just a minimal 
> wrapper around os.getenv, perhaps taking an optional type or type-coercing 
> function, so that it would be possible in a settings file to do:
> 
> SECRET_NUMBER_SETTING = env_setting('SECRET_NUMBER', int)
> 
> However, this is not much better than the current practice of calling 
> os.getenv. A better solution might be the ability to specify a group of 
> settings which will be read from the environment, and have Django 
> automatically read and set them. For example:
> 
> ENV_SETTINGS = [
> ('SECRET_NUMBER_SETTING', int),
> ('ACME_API_KEY', str),
> ('VOLCANO_LAIR_PASSWORD', str),
> ]
> 
> would read the named settings from those environment variables, and coerce 
> them to the appropriate types using the function provided.
> 
> The main problem with this is that it's really not very elegant. But at the 
> moment I can't think of anything better, and so I'd like to throw the floor 
> open to ideas on nicer approaches to this. If one can't be found, I do think 
> Django 1.10 should at least figure out how to handle database config from env 
> since that's such a common use case nowadays, but ideally we'd be able to pin 
> down a good API for generically pulling configuration from the environment.
> 
> 
> [1] https://github.com/kennethreitz/dj-database-url 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to django-developers@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/django-developers 
> .
> To view this discussion on the 

Re: Making Django more PaaS-friendly

2016-04-11 Thread Shai Berger
Django-database-url is very nice, but AFAICT has no way to support 3rd-party 
backends. I think it needs to grow this support before it can be used in 
Django. We can adapt the backend API to help, but it's a little tricky if we 
don't want to import unused backend.

On 11 באפריל 2016 11:13:23 GMT+03:00, Marc Tamlyn  wrote:
>I like the idea of integrating dj-database-url. There's a similar
>package
>for CACHES[0], which is a neat idea but slightly more tricky to justify
>as
>I don't know if PaaS tend to send the cache details in that format
>necessarily.
>
>I'm not sure about the rest of the mail, although I'd definitely be up
>for
>modifying the SECRET_KEY line in particular to make it strongly suggest
>that this should be different per environment.
>
>M
>
>
>[0] https://github.com/ghickman/django-cache-url
>
>On 11 April 2016 at 07:33, James Bennett  wrote:
>
>> Apologies for how late in the process this is; job hunt + moving
>> cross-country ate a lot of my time.
>>
>> At Django Under the Hood I proposed that for Django 1.10 we try to
>find
>> some common best practices for deploying Django on popular
>> platform-as-a-service (PaaS) solutions, and building support for them
>into
>> Django. The biggest one of these is probably having the ability to
>read
>> configuration from environment variables instead of hard-coding
>things into
>> settings files.
>>
>> At the very least I'd like to propose (assuming Kenneth is on board
>with
>> it) integrating dj-database-url[1] or something like it directly into
>> Django, so that there's no longer a third-party dependency for
>reading the
>> database configuration from an environment variable. Whether this is
>just
>> porting dj-database-url itself in, or making the DATABASES setting
>> understand URLs, I'm unsure of yet and would be interested to hear
>feedback
>> on. Either way I'm willing to put in the time to develop the patch.
>>
>> More generally, I'd like to see Django grow helpers for specifying
>> settings that should be read from environment variables, and which
>> environment variables to use (the email settings are another common
>case,
>> as is anything that requires an API key to access).
>>
>> There are a few ways to design this. One option would be just a
>minimal
>> wrapper around os.getenv, perhaps taking an optional type or
>type-coercing
>> function, so that it would be possible in a settings file to do:
>>
>> SECRET_NUMBER_SETTING = env_setting('SECRET_NUMBER', int)
>>
>> However, this is not much better than the current practice of calling
>> os.getenv. A better solution might be the ability to specify a group
>of
>> settings which will be read from the environment, and have Django
>> automatically read and set them. For example:
>>
>> ENV_SETTINGS = [
>> ('SECRET_NUMBER_SETTING', int),
>> ('ACME_API_KEY', str),
>> ('VOLCANO_LAIR_PASSWORD', str),
>> ]
>>
>> would read the named settings from those environment variables, and
>coerce
>> them to the appropriate types using the function provided.
>>
>> The main problem with this is that it's really not very elegant. But
>at
>> the moment I can't think of anything better, and so I'd like to throw
>the
>> floor open to ideas on nicer approaches to this. If one can't be
>found, I
>> do think Django 1.10 should at least figure out how to handle
>database
>> config from env since that's such a common use case nowadays, but
>ideally
>> we'd be able to pin down a good API for generically pulling
>configuration
>> from the environment.
>>
>>
>> [1] https://github.com/kennethreitz/dj-database-url
>>
>> --
>> You received this message because you are subscribed to the Google
>Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it,
>send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to
>django-developers@googlegroups.com.
>> Visit this group at
>https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>>
>https://groups.google.com/d/msgid/django-developers/CAL13Cg85fYj0i0ysxJGo2SDesqELMeA%2BtnKJ9cdpqNHmQ%3DX3Pg%40mail.gmail.com
>>
>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>-- 
>You received this message because you are subscribed to the Google
>Groups "Django developers  (Contributions to Django itself)" group.
>To unsubscribe from this group and stop receiving emails from it, send
>an email to django-developers+unsubscr...@googlegroups.com.
>To post to this group, send email to
>django-developers@googlegroups.com.
>Visit this group at https://groups.google.com/group/django-developers.
>To view this discussion on the web visit

Re: Making Django more PaaS-friendly

2016-04-11 Thread Marc Tamlyn
I like the idea of integrating dj-database-url. There's a similar package
for CACHES[0], which is a neat idea but slightly more tricky to justify as
I don't know if PaaS tend to send the cache details in that format
necessarily.

I'm not sure about the rest of the mail, although I'd definitely be up for
modifying the SECRET_KEY line in particular to make it strongly suggest
that this should be different per environment.

M


[0] https://github.com/ghickman/django-cache-url

On 11 April 2016 at 07:33, James Bennett  wrote:

> Apologies for how late in the process this is; job hunt + moving
> cross-country ate a lot of my time.
>
> At Django Under the Hood I proposed that for Django 1.10 we try to find
> some common best practices for deploying Django on popular
> platform-as-a-service (PaaS) solutions, and building support for them into
> Django. The biggest one of these is probably having the ability to read
> configuration from environment variables instead of hard-coding things into
> settings files.
>
> At the very least I'd like to propose (assuming Kenneth is on board with
> it) integrating dj-database-url[1] or something like it directly into
> Django, so that there's no longer a third-party dependency for reading the
> database configuration from an environment variable. Whether this is just
> porting dj-database-url itself in, or making the DATABASES setting
> understand URLs, I'm unsure of yet and would be interested to hear feedback
> on. Either way I'm willing to put in the time to develop the patch.
>
> More generally, I'd like to see Django grow helpers for specifying
> settings that should be read from environment variables, and which
> environment variables to use (the email settings are another common case,
> as is anything that requires an API key to access).
>
> There are a few ways to design this. One option would be just a minimal
> wrapper around os.getenv, perhaps taking an optional type or type-coercing
> function, so that it would be possible in a settings file to do:
>
> SECRET_NUMBER_SETTING = env_setting('SECRET_NUMBER', int)
>
> However, this is not much better than the current practice of calling
> os.getenv. A better solution might be the ability to specify a group of
> settings which will be read from the environment, and have Django
> automatically read and set them. For example:
>
> ENV_SETTINGS = [
> ('SECRET_NUMBER_SETTING', int),
> ('ACME_API_KEY', str),
> ('VOLCANO_LAIR_PASSWORD', str),
> ]
>
> would read the named settings from those environment variables, and coerce
> them to the appropriate types using the function provided.
>
> The main problem with this is that it's really not very elegant. But at
> the moment I can't think of anything better, and so I'd like to throw the
> floor open to ideas on nicer approaches to this. If one can't be found, I
> do think Django 1.10 should at least figure out how to handle database
> config from env since that's such a common use case nowadays, but ideally
> we'd be able to pin down a good API for generically pulling configuration
> from the environment.
>
>
> [1] https://github.com/kennethreitz/dj-database-url
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAL13Cg85fYj0i0ysxJGo2SDesqELMeA%2BtnKJ9cdpqNHmQ%3DX3Pg%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMwjO1Hd6JFntN49a35E-ktnXMf9pTWjisdh7fwYHC_-bbPPBA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.