from:"dan"

Re: Feature Request: New parameter for render_to_string() to render part of a template using a selector

2023-06-12 Thread Dan Swain

Carlton Gibson has addressed my feature 
request:  https://github.com/carltongibson/django-template-partials

On Friday, July 15, 2022 at 8:21:52 PM UTC-4 Dan Swain wrote:

> With the growing uptake of technologies like HTMX, I would like to see an 
> additional parameter added to render_to_string().  In my rewritten 
> definition of render_to_string() below (taken from the docs), I have named 
> the parameter “part”.  In rendering portions of a template for responses to 
> ajax calls, these template portions end up being stored in separate 
> template snippet files and then {% include(d) %} in the main template.  I 
> think that keeping everything related to a view in a single template file 
> is much more desirable.  Therefore, just include *part=selector* in order 
> to pull out the part of the template that is needed for rendering.  The 
> file structure for managing templates would become much simpler with this 
> approach.
>
>  
>
> *render_to_string**(**template_name**, part=None, **context=None**, *
> *request=None**, **using=None**)¶ 
> <https://docs.djangoproject.com/en/4.0/topics/templates/#django.template.loader.render_to_string>*
>
> *render_to_string()* loads a template like *get_template()* 
> <https://docs.djangoproject.com/en/4.0/topics/templates/#django.template.loader.get_template>
>  and 
> calls its *render()* method immediately. It takes the following arguments.
>
> *template_name*
>
> The name of the template to load and render. If it’s a list of template 
> names, Django uses *select_template()* 
> <https://docs.djangoproject.com/en/4.0/topics/templates/#django.template.loader.select_template>
>  instead 
> of *get_template()* 
> <https://docs.djangoproject.com/en/4.0/topics/templates/#django.template.loader.get_template>
>  to 
> find the template.
>
> *part*
>
> An optional CSS selector that will be used to cause only the portion of 
> the template that matches the selector to be rendered.
>
> *context*
>
> A *dict* <https://docs.python.org/3/library/stdtypes.html#dict> to be 
> used as the template’s context for rendering.
>
> *request*
>
> An optional *HttpRequest* 
> <https://docs.djangoproject.com/en/4.0/ref/request-response/#django.http.HttpRequest>
>  that 
> will be available during the template’s rendering process.
>
> *using*
>
> An optional template engine *NAME* 
> <https://docs.djangoproject.com/en/4.0/ref/settings/#std-setting-TEMPLATES-NAME>.
>  
> The search for the template will be restricted to that engine.
>
>  
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4d54fc0c-51b3-4dfd-9600-65552c338fcen%40googlegroups.com.

Feature Request: New parameter for render_to_string() to render part of a template using a selector

2022-07-15 Thread Dan Swain

With the growing uptake of technologies like HTMX, I would like to see an
additional parameter added to render_to_string(). In my rewritten
definition of render_to_string() below (taken from the docs), I have named
the parameter “part”. In rendering portions of a template for responses to
ajax calls, these template portions end up being stored in separate template
snippet files and then {% include(d) %} in the main template. I think that
keeping everything related to a view in a single template file is much more
desirable. Therefore, just include part=selector in order to pull out the
part of the template that is needed for rendering. The file structure for
managing templates would become much simpler with this approach.

render_to_string(template_name, part=None, context=None, request=None,
using=None)
¶

render_to_string() loads a template like
get_template() and calls its render() method immediately.
It takes the following arguments.

template_name

The name of the template to load and render. If it’s a list of template
names, Django uses
select_template() instead of
get_template() to find the template.

part

An optional CSS selector that will be used to cause only the portion of the
template that matches the selector to be rendered.

context

A dict to be used as
the template’s context for rendering.

request

An optional
HttpRequest that will be available during the template’s rendering
process.

using

An optional template engine
NAME. The search for the template will be restricted to that engine.

Re: Feature Idea: Allow setting session cookie name dynamically

2022-06-06 Thread Dan Strokirk

Hi Carlton, thanks for the response.

An external package might be useful, although the code majority of the code 
would be the copied SessionMiddleware code and the tiny changes to allow a 
dynamic cookie name, so my thoughts is that this might be "too small" for a 
published pypi package?

But since I haven't found a similar request earlier on this mailing list or 
the issue tracker, it seems that it might be really niche, as you say!

Best regards,
Dan

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/01a91019-8d54-4322-a295-dbfdc12dfab9n%40googlegroups.com.

Feature Idea: Allow setting session cookie name dynamically

2022-06-02 Thread Dan Strokirk

Hi,

Currently it's only possible to use a single session cookie, but it can be 
useful in a multi-tenant application to use multiple session cookies. To 
solve this we currently use our own, slightly modified SessionMiddleware 
class that we keep in sync with the official implementation and re-apply 
the patch during each Django upgrade.


Proposal: Adding a new get_cookie_name or get_cookie_params method to 
the SessionMiddleware class might be one way to implement this, which means 
that you can then then use your own subclassed middleware to easily 
override it. It would take the request and response objects as arguments.

If this seems like a reasonable feature to add I'd be glad to supply a 
patch.

(I've previously submitted a WONTFIX ticket for this, perhaps a little 
over-eager :) https://code.djangoproject.com/ticket/33760)

Best regards,
Dan

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/879fbda1-c8f6-4995-a3d0-7b2cbe5bc282n%40googlegroups.com.

Re: Generated Field

2022-04-13 Thread Dan Davis

+1

On Wed, Apr 13, 2022 at 7:01 AM 'Adam Johnson' via Django developers
(Contributions to Django itself)  wrote:

> I'd be interested in seeing this. Generated columns are a useful SQL
> feature that are missing from Django.
>
> Nice initial research on backend coverage - it looks like they're widely
> supported.
>
> Some ideas...
>
> Is it necessary to take a base field? Can we not determine the output
> field type for some kinds of expression? e.g. F("some_integer_field") + 1
> can be assumed to have output field type IntegerField.
>
> Would it be worth blocking assignment to the field? Allowing
> "model.generated_field = ..." would at least be misleading. (This could be
> done with a special descriptor.)
>
> Good luck!
>
> On Wed, Apr 13, 2022 at 11:25 AM Paolo Melchiorre 
> wrote:
>
>> Hi all,
>>
>> I am at PyCon DE 2022 in Berlin with Markus and I shared with him this
>> idea that I have been thinking about for a few months, given his
>> interest I also share it with you.
>>
>> I figured we could add in Django a "GeneratedField" which accepts a
>> "base_field" attribute in a similar way to "ArrayField" and then an
>> expression to generate a value for the field.
>>
>> For example this model:
>>
>> class Album(models.Model):
>>...
>>title = models.CharField(max_length=120)
>>search = GeneratedField(
>>  SearchVectorField(),
>>  F('title')
>>)
>>
>> would generate an SQL code like this in PostgreSQL:
>>
>> CREATE TABLE album (
>>  ...
>>  title char(120),
>>  search tsvector GENERATED ALWAYS AS title STORED
>> );
>>
>> I found some documentation for different database backend about
>> generated column (or similar names)
>>
>> # SQLite
>> https://www.sqlite.org/gencol.html
>> https://www.sqlite.org/releaselog/3_31_0.html
>>
>> # PostgreSQL
>> https://www.postgresql.org/docs/12/ddl-generated-columns.html
>> https://www.postgresql.org/docs/12/release-12.html
>>
>> # Oracle
>> https://oracle-base.com/articles/11g/virtual-columns-11gr1
>>
>> # MySQL
>> https://dev.mysql.com/doc/refman/5.7/en/mysql-nutshell.html
>>
>> https://dev.mysql.com/doc/refman/5.7/en/create-table-generated-columns.html
>>
>> # MariaDB
>> https://mariadb.com/kb/en/changes-improvements-in-mariadb-102/
>> https://mariadb.com/kb/en/generated-columns/
>>
>> I'd love to hear your thoughts on this idea and if you think it might
>> be something worth continuing to investigate.
>>
>> Greetings from PyCon DE 2022 in Berlin,
>> Paolo
>>
>> --
>> https://www.paulox.net
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers  (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/CAKFO%2Bx5GHUEVdzi2awYtH5C17tTPTPh%2ByoDP%3DKC18pF8%2Bi_7PA%40mail.gmail.com
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAMyDDM2CWm0HOkLC%3DBjqQ92928L%3DD%3DXOZzUtA1cHdgc235oYRQ%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFzonYZ8EaQCjD_HayNGvXaZNwGHYE7veLbOyzQi4S0Y%2BZ%3DATg%40mail.gmail.com.

Re: MFA (2FA)

2022-04-09 Thread Dan Davis

My intuition is that Webauthn will be hard to support because of the
varieties of ways to secure the private key (Yubikey, HIDGlobal, etc.) and
the complexities of managing key pairs without devices. PGP/GnuPG followed
a trajectory where we had ways to secure email, but it was too complicated
to achieve wide adoption.

Would it be better to support email only login in core, e.g. to support
passwordless 1FA? This is something that shouldn't require much in terms of
protocol support since Django's email support is so good no one needs to
rewrite or support through libraries.  This is only 1FA, but it is
passwordless.  It could be coupled with the use of something like Google
Authenticator or MS Authenticator - not sure how hard this latter part is
to put in core.

MFA is typically built with some form of federated login, and most
applications will fall into two camps (1) those needing federated business
login for business purposes, or (2) those needing federated social login
for better conversions. In the middle ground, buy vs. build probably
applies since there are so many SaaS services here. Because of this,
implementing login well with MFA is usually a matter of supporting a
federated login protocol such as SAML, OpenID Connect/OAuth2, or CAS. And
it is that server protocol that implements the MFA. Many users may not even
choose their protocol. Once you have these protocols, MFA can be done via a
Google or Github account. Since OIDC federated login can be purchased as
SaaS within AWS, Azure, Okta, etc., that's the best protocol to support
within a "generic" application at this time.

There are already mature libraries that cover SAML, CAS, and OIDC very well
and work with a wide range of Django versions. There is often some
customization necessary for MFA with a particular federated login. Over
many years I've had to write middleware (not Middleware) to support
federated login based on a keypair where the private key never leaves my
PIV card, and the public key (in terms of certificates) is authenticated by
a CA so that it need not be registered with a server/registry. I did a
lightning
talk  at DjangoCon 2016 on this
topic. At that time, we used django-cas-ng  for a
CAS implementation in front of NIH Login.  In the cloud we use
python-social-auth  in
front of AWS Cognito in front of NIH Login. I've also worked with SAML to
enable PIV command-line login to AWS, see
https://nlm-occs.github.io/nlmfedcred/.  All of these have required some
middleware (a Backend and either a middleware or a set of decorators).

P.S. - I don't speak for NLM or NCBI.  NIH Login is no longer as
"antisocial" as it was, it now does have buttons for "Google" and
"Facebook", but NCBI still wraps it to make it easier to click to login -
see
https://account.ncbi.nlm.nih.gov/?back_url=https%3A//pubmed.ncbi.nlm.nih.gov/
.

P.P.S - Most of the versions of OpenSSL out there do not support OpenSC
(smart card), and due to the Heartbleed issue, most security professionals
(if they pay close attention to Django libraries) will want libraries to
link with LibreSSL instead. This may not be as large a deal on the server
side, but it is a big problem with testing this using Python code - until
CPython is built against a version of OpenSSL/LibreSSL that includes
PKCS11, linking up with smart cards and keys will be somewhat difficult.

On Thu, Apr 7, 2022 at 9:18 PM Yonas 
wrote:

> Hi Tobias,
>
> Thanks for the feedback.
>
> There are multiple ways to implement MFA, as you mentioned. But the goal
> here is to provide a simple mechanism. It's "not necessary" to cover every
> use case, and I believe that's where third-party packages come in.
>
> Based on a study conducted by Microsoft, a user account is "99.9% less
> likely to be compromised if" the user uses MFA. So, it would be beneficial
> to have this feature in Django.
> On Thursday, April 7, 2022 at 9:55:32 PM UTC+3 Tobias Bengfort wrote:
>
>> Hi Yonas,
>>
>> the best place to start is probably to look at thrid party packages,
>> e.g.:
>>
>> https://github.com/django-otp/django-otp
>> https://github.com/jazzband/django-two-factor-auth
>> https://github.com/mkalioby/django-mfa2
>> https://gitlab.com/stavros/django-webauthin
>> https://github.com/xi/django-mfa3 (mine)
>>
>> I am not convinced that moving this into django is the best approach
>> though. There is just so many ways you can do MFA/passwordless. Recovery
>> in particular is complex with a lot of different solutions, depending on
>> context.
>>
>> I also noticed that you omitted FIDO2/webauthn from your list of
>> protocols. It is the newest of these protocols and requires a very
>> different architecture, but I guess that IETF and W3C had reasons to
>> choose it.
>>
>> I do agree that a simple, opinionated solution in django itself could
>> push 2FA adaption and therefore general security on the web, which is
>>

Configuration of "level" in LOGGING

2022-04-04 Thread Dan Swain

The current Django LOGGING setup requires a string for the "level".  This is
not intuitive since the Python logging module defines logging.DEBUG,
logging.INFO, etc.  I think one should be able to configure the logging
level using {  'level': logging.DEBUG  } rather than being required to use
{  'level': 'DEBUG'  }.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/010b01d8487b%24683216c0%2438964440%24%40shenberger.org.

Re: Revisiting MSSQL and Azure SQL Support in Django

2022-04-02 Thread Dan Davis

Warren,

We all know support for an RDBMS matters a lot in enterprise corporate
environments. I think having this support in the
https://github.com/microsoft organization is probably better for MSSQL
users.

Coordinating support between a support organization, open source
components, and Django is challenging, and even for projects hosted within
an organization like that. It isn't always clea that support contracts for
MSSQL include support for the Django-mssql extension and all layers in
between. At least clarifying under Features that support fis included in a
support contract will catch the eyes of Django devs who are also
enterprise decision makers and influencers. You could also align things so
that support requests that mention django-mssql always let you know (open a
linked gitub issue, or something).

Clarifying that at DjangoCon will go so very far in making users in
corporate environments happy.

I'm also going to suggest building more coordinated support for publishing
to Power BI from Django applications.  Maybe you can include logic that
understands how to publish to Dataverse based on Django models.  This will
go over a lot of heads, but not mine :).


On Sat, Apr 2, 2022 at 11:44 AM Carlton Gibson 
wrote:

> Hey Tim.
>
> FWIW I have directly asked people at Oracle if they would take up a
> corporate sponsorship of the DSF in order to help support the ongoing
> maintenance of the backend.
> Again, it was almost like I hadn't said anything. Rather than a Yes or a
> No, they did instead offer some cloud hosting, which we looked into but
> which wasn't appropriate.
>
> The Oracle backend would be in big trouble if Mariusz wasn't personally a
> fan. Oracle stepping up to support it would make it much more sustainable.
> If anyone knows how to get to have a sensible conversation with someone
> with the ability to make such a contribution to the DSF please do let us
> know.
>
> MySQL is slightly different, in that there are a number of active
> contributors for it.  Support from Oracle Corp would still be welcome, but
> it's not as pressing IMO.
>
> Personally, I think MSSQL support in core would be awesome, but we can't
> just take that on because it would be a Nice to Have; we have to be
> realistic about how it's going to be maintained.
>
> Kind Regards,
>
> Carlton
>
> On Fri, 1 Apr 2022 at 17:02, Tim Allen  wrote:
>
>> Full disclosure: I've helped maintain SQL Server Django engine packages
>> in the past, and was involved in getting the project started for the
>> official Microsoft Django engine package.
>>
>> First, thanks to Warren and the folks from Microsoft for creating this
>> backend. There are a fair amount of users that need a Microsoft SQL Server
>> backend, and for years we've been having to shuttle between
>> barely-maintained engines, forks, and pip installs to commit hashes while
>> waiting for merges. It is less than ideal. Several big supporters of Django
>> (The Wharton School, RedHat Ansible, to name two) have been among those
>> looking for a solid backend solution for years. Microsoft has provided us a
>> vastly improved option to what we'd been saddled with for the past decade.
>>
>> The DB popularity index at db-engines.com has regularly listed the top
>> four as Oracle, MySQL, Microsoft SQL Server, and PostgreSQL, in that order.
>> I notice some comments in this thread about Microsoft being for-profit...
>> well, what about Oracle? I don't see Oracle on the Support Django page
>> either, yet two of their databases have support in core. MSSQL is the only
>> one of the big-four RDBMS's without support in core Django. That seems to
>> be a pretty big hole in Django's offering.
>>
>> That said, I completely agree that the backend still needs work, and a
>> solid commitment from Microsoft before being brought into core Django. I
>> want to make sure Microsoft knows that many in our community appreciate
>> their efforts in taking the lead on this backend: support for SQL Server in
>> the Django ecosystem is far better than it ever has been in my eight years
>> of being a Djangonaut. They have put a lot of time and effort into this
>> project, and I think they're well on their way to where they need to be for
>> the long-term goal of being in core Django.
>>
>> A lot of the questions being asked of Microsoft in this thread just don't
>> seem fair to me - we're not asking the same of Oracle, Redis Enterprise
>> Software, or any of the other commercial products that Django has built-in
>> support for. Why Microsoft and not the others?
>>
>> Warm regards,
>>
>> Tim
>>
>> On Friday, April 1, 2022 at 8:39:17 AM UTC-4 Salman Mohammadi wrote:
>>
>>> Hi,
>>>
>>> IMO, Django is there to create value for *its* users.
>>>
>>> I'm not aware how MS Team reached this conclusion that merging their
>>> incomplete package into Django core creates more value for Django users
>>> than when it is a third-party package. Would you please tell me how?
>>>
>>> I have access to only two resources to

How to decode data? (trick question)

2021-08-15 Thread Jack Dan

Hello everyone, please help me figure out the problem.

This is a tricky question because I couldn't find anything on the net. I
know how to encode / decode, I do not know where to do it. At what stage.
The Django assembly is standard, it works fine on the local server, not on
the hosting. The detailed situation is below.

Hosting Support Conversation:
*Me:* "On the local server everything works, but you don't. What is the
reason?"
*Support:* "The web server cannot work with Cyrillic in links directly for
work, it is converted accordingly. Therefore, when you open http: //
dictionary.rf/pulpation, the web server converts pulpation to
%25d0%25bf%25d1%2583%25d0%25bb%25d1%258c%25d0%25bf%25d0%25b0%25d1%2586%25d0%25b8%25d1%258f.

Scripts of your site search in the database by the encoded value:
SELECT `dictionary_d_word`.`id`, `dictionary_d_word`.`author_id`,
`dictionary_d_word`.`word`, `dictionary_d_word`.`word_url`,
`dictionary_d_word`.`description`, `dictionary_d_word`.`created_date`,
`dictionary_d_word`.`published_date` FROM `dictionary_d_word` WHERE
`dictionary_d_word`.`word_url` =
'%25d0%25bf%25d1%2583%25d0%25bb%25d1%258c%25d0%25bf%25d0%25b0%25d1%2586%25d0%25b8%25d1%258f'

LIMIT 21

To avoid errors, you should decode the request value before executing the
request, and then execute the request. "

And I have a logical question:
Where to decode this data?

Re: GSOC Proposal: Adding Redis Cache Backend To Django Core.

2021-04-04 Thread Dan Davis

cache coherence can be a problem, but I am thinking more about what happens
when the cache, which is supposed to make things faster, is not available
due to error.  Django ends up waiting for it.  Each request waits for the
cache, and this is a hard problem because there is no common shared place
to write that the cache is down.  However, introducing anything like that
introduces problems also with cache coherence.

On Fri, Apr 2, 2021 at 3:17 AM Girish Sontakke 
wrote:

> Hello,
> If I am not wrong, you are talking about the cache incoherence
>  problem.  Currently, I
> don't have any concrete idea to tackle this problem but I will try to
> figure it out.
> Thanks for bringing this to concern.
>
> Kind Regards,
> Girish
>
> On Thursday, April 1, 2021 at 8:39:59 PM UTC+5:30 dans...@gmail.com wrote:
>
>> Since REDIS is already usable as a cache for Django using 3rd party code,
>> I would rather see a circuit breaker pattern applied to the cache, across
>> all backends.  I'm not that active in maintaining Django right now, but a
>> cache may not be fully redundant, and it also can fail. If the cache is
>> truly used as a cache, then it should be possible to try the cache, and if
>> it fails for several tries, then Django could start to bypass it, and then
>> try to re-enable it later.
>>
>> This is perhaps a bigger effort than a GSOC project - the hard part is
>> doing this across multiple processes and threads.  What does it mean if the
>> cache is only available for some fraction of the instances/processes
>> hosting a project?
>>
>>
>> On Thu, Apr 1, 2021 at 8:25 AM Girish Sontakke 
>> wrote:
>>
>>> Hello Carlton,
>>> As suggested by you I made some revisions to my proposal. Still, it is
>>> not a perfect one, I am trying to improve it. Thanks a lot for the help.
>>>
>>> Kind Regards,
>>> Girish
>>> On Thursday, April 1, 2021 at 12:15:53 AM UTC+5:30 carlton...@gmail.com
>>> wrote:
>>>


 On 31 Mar 2021, at 11:00, Girish Sontakke  wrote:

 Now I have doubt that whether I should remove *Provides Many
 Data-Structures *point from the "Why Redis" section or keep it there?.


 I don’t think it’s vital either way… but it seems a distraction. That
 Redis has HyperLogLog isn’t something that will matter at all to the cache
 backend.

 I’d try to focus on the details of the implementation: Look at the
 backend API, set(), get()  Look at the two existing third-party
 backend, and see what they do there.

 I’d also think about getting the documentation right — there are a few
 options with installs and connections and … — you don’t have to solve all
 of those in the proposal but showing that you’re aware of them would be
 good.

 I hope that helps.

 Kind Regards,

 Carlton

 --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-develop...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-developers/97c757dd-b09f-49f4-9e3f-0ee4a8e4887cn%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/399dbcd7-7127-4715-b2cd-3f5d349d605bn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFzonYbPq07r%2BTr30n30rC87LN9VOmc_0Fu_5bxubEXvgigb0g%40mail.gmail.com.

Re: GSOC Proposal: Adding Redis Cache Backend To Django Core.

2021-04-01 Thread Dan Davis

Since REDIS is already usable as a cache for Django using 3rd party code, I
would rather see a circuit breaker pattern applied to the cache, across all
backends.  I'm not that active in maintaining Django right now, but a cache
may not be fully redundant, and it also can fail. If the cache is truly
used as a cache, then it should be possible to try the cache, and if it
fails for several tries, then Django could start to bypass it, and then try
to re-enable it later.

This is perhaps a bigger effort than a GSOC project - the hard part is
doing this across multiple processes and threads.  What does it mean if the
cache is only available for some fraction of the instances/processes
hosting a project?


On Thu, Apr 1, 2021 at 8:25 AM Girish Sontakke 
wrote:

> Hello Carlton,
> As suggested by you I made some revisions to my proposal. Still, it is not
> a perfect one, I am trying to improve it. Thanks a lot for the help.
>
> Kind Regards,
> Girish
> On Thursday, April 1, 2021 at 12:15:53 AM UTC+5:30 carlton...@gmail.com
> wrote:
>
>>
>>
>> On 31 Mar 2021, at 11:00, Girish Sontakke  wrote:
>>
>> Now I have doubt that whether I should remove *Provides Many
>> Data-Structures *point from the "Why Redis" section or keep it there?.
>>
>>
>> I don’t think it’s vital either way… but it seems a distraction. That
>> Redis has HyperLogLog isn’t something that will matter at all to the cache
>> backend.
>>
>> I’d try to focus on the details of the implementation: Look at the
>> backend API, set(), get()  Look at the two existing third-party
>> backend, and see what they do there.
>>
>> I’d also think about getting the documentation right — there are a few
>> options with installs and connections and … — you don’t have to solve all
>> of those in the proposal but showing that you’re aware of them would be
>> good.
>>
>> I hope that helps.
>>
>> Kind Regards,
>>
>> Carlton
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/97c757dd-b09f-49f4-9e3f-0ee4a8e4887cn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFzonYbS8Tphvvdae6fEs7zb_EH30WKJx7r1_bP5BBBWFWqm9A%40mail.gmail.com.

Re: Improving MSSQL and Azure SQL support on Django

2020-11-27 Thread Dan Davis

I'd suggest someone talk with professional DBAs for MSSQL.  In my work,
which is Federal government, the DBA told me that disconnecting from PSQL
as "appuser" and attempting to connect to database "postgres" in order to
create the test database violated FISMA.  I had to subclass my own
postgresql backend to create the test user while connected to my actual
database.  I would imagine that the original reasons for doing it this way
harkens back to a time when people used the same database server for
production, staging, qa, and integration, and some of these environments
may not have existed. While in some organizations we have only production,
staging/qa, and development/integration, I would guess there are few cases
where the same database server is used for production and the other
environments.

On Fri, Nov 27, 2020 at 1:18 PM r...@whidbey.com  wrote:

> Good news.  I've been using Django on MSSQL for about 8 years.  Couple of
> things:
> - I've been using pyodbc, not django-mssql.  I note your messages from
> 2015 include it as a library to check for compatibility; what was the
> outcome of that?  Is it proposed that django-mssql become the
> "best-practices" interface for SQL Server?
> - Couple of persistent pain points:
>   1. Testing.  The Django code that sets up test databases fails with
> MSSQL, while it succeeds with PostGRE, MySQL and SQLite.  The issues seem
> to revolve around setting constraints as the tables are generated, rather
> than holding off and enabling the constraints at the end of the process.
>   2. Stored Procedures.  These need to be loaded as an additional step in
> the creation of a database, and don't really have any representation in
> Django per se so migrations, etc don't generally have an idea that they
> exist.
>
> I'd be happy to test out what you come up with against our system.  It's
> currently serving a custom REST interface with 2-million-plus rows of
> transactions,with clients world-wide, along with a customer-facing web app,
> a staff site and a suite of Tableau reports.
>
> On Thursday, November 26, 2020 at 5:40:18 PM UTC-8 vwa...@gmail.com wrote:
>
>> Hi All,
>>
>> Microsoft has now committed ongoing resources towards improving MSSQL and
>> Azure SQL support for Django. We're currently focused on internal
>> compliance and forking the ESSolutions django-mssql-backend
>> , adding testing
>> pipelines, refactoring the Django DB engine naming convention, and
>> addressing current test suite errors.
>>
>> We'd love to hear from current mssql-backend maintainers as well as
>> mssql-backend users about the existing issues and feature requests that we
>> should be prioritizing.
>>
>> We looking forward to engaging the community and working towards MSSQL as
>> a first-class supported backend for Django.
>>
>> -Warren
>>
>> On Wednesday, 4 December 2019 at 07:05:25 UTC-8 Tim Allen wrote:
>>
>>> Hi Sean, just an update from what I know.
>>>
>>> We are still waiting for a reply from Microsoft. They're a large
>>> company, so understandably, it takes a little while.
>>>
>>>
>>> For now, if people need to get onto Django 2.2 for long term support
>>> (which will last until April, 2022), you can use this package:
>>>
>>> https://github.com/ESSolutions/django-mssql-backend I've been running
>>> it in production for months without incident. Of course, YMMV.
>>>
>>>
>>> If Microsoft and/or the DSF end up wanting to bring support under the
>>> Django umbrella, the django-mssql-backend repository is a possible
>>> starting point, IMHO.
>>>
>>> The django-mssql-backend is currently being developed and support for
>>> Django 3.0 is being worked on: ESSolutions/django-mssql-backend#18
>>> 
>>>
>>>
>>> Regards,
>>>
>>>
>>> Tim
>>>
>>> On Monday, December 2, 2019 at 11:03:56 AM UTC-5, Sean Martz wrote:

 Hello,

 It seems like this issue has lost momentum. Is this still something
 that's on anyones radar? It looks like django-pyodbc-azure is not actively
 maintained anymore (it looks like Michaya has taken a hiatus from GitHub).
 It also looks like there's a small community potentially popping up that's
 interested in first class MSSQL Server support for Django. (
 https://github.com/FlipperPA/django-mssql-backend). Is Microsoft still
 interested in committing resources to this goal? In my situation, it would
 be a lot easier to sell stakeholders and decision makers on Django if it
 had first class support for MSSQL Server.

 For what it's worth, Django-pyodbc-azure is still working well.

 Cheers,
 Sean

 --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this

Re: f-strings again.

2020-07-21 Thread Dan Davis

+1 iff flake8 can validate  f-srings as well as PyCharm does f-strings!

I think flake8 would mean that SublimeText and Atom can highlight errors.

Background - Arguments based on readability are so subjective. If I am
using Pycharm, f-strings are very readable, and it will check whether a
keyword argument is defined, e.g. what about command-line tools?  My guess
is that validating format and %- strings is an easier problem
because context is localized, but maybe not as solved.

On Tue, Jul 21, 2020 at 9:53 AM Shai Berger  wrote:

> Hi Dave and all,
>
> On Tue, 21 Jul 2020 13:56:53 +0100
> Dave Vernon  wrote:
>
> > More generally:
> >
> > I understand the value of *not* doing a bulk change in a PR, but I was
> > wondering if it's OK to do a PR based purely on improved performance
> > for a specific element? (I don't have one in mind, the question is
> > purely 'in principle')
>
> We are still in discussions here, but if I understand the forming
> concensus correctly, it will be ok to replace older-style formatting
> with f-strings if you're making changes in the code in question, and
> performance-improving changes are, in principal, welcome.
>
> However, whether the performance improvement afforded by such a change
> would, on its own, justify the change -- will need to be judged on a
> case-by-case basis, and my guess is that justifiable cases will be few
> and far between. I believe that in most cases, string formatting is not
> a major performance bottleneck.
>
> My non-shot-calling 2-cents' worth,
>
> Shai.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/20200721165300.70bd9631.shai%40platonix.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFzonYa1hWUYUOw_soGSx%3DykUBDfGw1mspx5qvyNMfk9mbiedQ%40mail.gmail.com.

Re: Making startproject's settings more 12-factor-y

2020-06-24 Thread Dan Davis

 tMost of the world is not as seamless as heroku.  My DevOps won't give me
any more than a handful of environment variables.  I wanted something like
DATABASE_URL, but all I have is DJANGO_LOG_DIR and DJANGO_SETTINGS_MODULE,
and so I need many, many settings files. I think that happens a lot, and
maybe a common pattern.

>From a 12factor perspective, I would like to get it down to local settings
(development) and production settings - yet for a lot of users, DevOps is
not really supporting a full PaaS-like experience any way.

So - all of this has to be optional, which seems to rule out making it part
of the starting project template.  For sure, I've got my personal template,
and work has an on-premise template and a Cloud template as well - but the
department of developers doesn't always use these.  I find databases
containing the tables for other projects, long after the models and
migrations are gone, indicating a start by copy mode.

On Wed, Jun 24, 2020 at 1:35 PM Kit La Touche  wrote:

> Carlton—thanks very much for the feedback. Javier—likewise. In particular,
> the imagined API you describe above is very appealing to me: start with
> `from_env` and then if you learn more about this and want to, add in some
> `EnvFileLoader`.
>
> I want to make clear my motivation and agenda here: I have recently had
> some conversations with newer devs about their experiences with deployment
> of apps they're working on, and with a friend at Heroku about his informal
> research into the problems people have with the same. One recurring
> friction point (and this is not just on Heroku at all, to be clear) is that
> there are a number of things that people *don't know they need to
> configure* for a working deployment.
>
> There are four settings that are recurring particular gotchas that people
> miss: the secret key, debug, static files, and databases. Static files
> seems big and out of scope, databases seems adequately handled by
> dj-database-url for most cases, and if your case is more complex, you'll
> learn it, but the other two (secret key and debug) seemed easy enough to
> flag as "you probably need to configure these!" with this sort of change to
> settings. This would be a first step towards shortening the distance from
> `startproject` to a working deployment.
>
> Newer devs in particular have, based on my conversations and this friend's
> research, been unlikely to (a) know that there are different `startproject`
> templates, and (b) feel equipped to choose one, if they do know.
>
> My hope is to make the smallest possible change to just start us moving
> towards more clearly flagging, especially for newer devs, "these are things
> that will need additional configuration in order to move from 'works on my
> machine' to 'deployed'."
>
> Towards that end, I thought that adding a "you might want to get this from
> the env" helper would be a clear indication to a new dev that this is a
> matter to even consider. Adding other configuration-getting options like
> different secret-store file backends seems like a good next step.
>
> Thanks,
>
> --Kit
>
> On Wed, Jun 24, 2020 at 11:13 AM Javier Buzzi 
> wrote:
>
>> I looked at the libs that do what we want:
>>
>> django-configurations - it looks like they use environment variables /
>> either via loading them from the environ or a key/value pair file. Having
>> classes inside the settings.py might be weird to people.. at the least very
>> different.
>> confucius - very simplistic, only supports environ and is classed based,
>> similar to django-configurations.
>> django-environ - supports env file and environ, non-class based.
>> dynaconf - supports all kinds of loading options (toml, json, ini,
>> environ, .env +) non-class based.
>>
>> In my opinion, django-environ and dynaconf would be the easiest to sell
>> to the community, it would require the least changes/paradigm shifts from
>> how everyone is already using django.
>>
>> Personally, i would really like to see something like this inside my
>> settings.py:
>>
>> from django.conf import from_env  # using standard os.environ
>>
>> DEBUG = from_env.bool("DEBUG", default=False)
>>
>> DATABASES = {
>> "default":  from_env.db("DATABASE_URL")  # crash if it cant find it
>> }
>>
>> ...
>>
>> for more complex examples:
>>
>> from django.conf import EnvFileLoader
>>
>> from_env = EnvFileLoader("path/to/.secret")
>>
>> ...
>>
>> We can have how ever many loaders we want: toml, json, ini ..
>>
>> This is both borrowing heavily from dynaconf and django-environ, making
>> the fewest changes to how people are accustomed to doing things.
>>
>> .. what do you guys think?
>>
>> - Buzzi
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>>

Re: Proposal: django project name

2020-05-14 Thread Dan Davis

>
> Wherever you place the template, users can already override it by creating
> one in their project with the same name, in an app that appears first in
> INSTALLED_APPS. There's a documentation page on this topic:
> https://docs.djangoproject.com/en/3.0/howto/overriding-templates/
> Alternatively, you can tell users to install their own URL mapped to their
> own override template. It would only be one line of configuration, the same
> as using include() to include your URLconf. And it would allow users to
> serve your app at multiple URL's or with more complicated patterns.


Yes! And in apps designed to be re-used, this can be made really good with
clever template design. I generally have *all* my templates extend a base
template, so that my users can override one simple template, and reskin my
pages simply -- assuming they use Bootstrap 3 4. So, mileage may vary.

I have also used drf-yasg, and other third party packages that provide
reusable apps. It truly is not hard to override their templates, but as
Adam is saying, you want to guide your users as to how to extend/embed and
accommodate that.  What I describe above is only one such method.

On Thu, May 14, 2020 at 4:47 PM Adam Johnson  wrote:

> Hi Christian
>
>
>> TL;DR: Django has no (builtin/explicit) settings variable like
>> PROJECT_NAME. Should have.
>>
>
> I can see how it would be useful in several situations. But in most
> long-lived projects I've come across, the "project name" is different to
> the website name (or names, plural, if it has evolved into multiple
> brands). Additionally, there's already the ability for a "Site name"
> through the contrib.sites framework, and translations complicate the idea
> of a single name. I'm not sure *another* axis of naming would make things
> easier. Sticking to Python imports and file paths to glue together things
> in code seems favourable.
>
>> When it comes to frontends, it's worse. In my case, my GDAPS module
>> enables a Vue.js frontend, which needs a index.html which loads Vue. This
>> file is (in SPA apps) the only file which is rendered using Django
>> templates. No problem, put in gdaps' templates directory and go. Works
>> fine, as long as the user of GDAPS is fine with my shipped index.html and
>> the corresponding pre-defined url path that renders that index file using
>> TemplateView.
>>
>> If he wants to override that file using his own template (with maybe
>> another font loaded etc.) I first thought I just change GDAPS to
>> dynamically load the template in GDAPS' urls.py - but therefore it should
>> know the name of the project it is a part of.
>>
> Wherever you place the template, users can already override it by creating
> one in their project with the same name, in an app that appears first in
> INSTALLED_APPS. There's a documentation page on this topic:
> https://docs.djangoproject.com/en/3.0/howto/overriding-templates/
>
> Alternatively, you can tell users to install their own URL mapped to their
> own override template. It would only be one line of configuration, the same
> as using include() to include your URLconf. And it would allow users to
> serve your app at multiple URL's or with more complicated patterns.
>
> Thanks,
>
> Adam
>
> On Wed, 13 May 2020 at 21:13, Christian González <
> christian.gonza...@nerdocs.at> wrote:
>
>> Hi all,
>>
>> I didn't find anything about that in the archive here, so I'd like to
>> come to you with an idea which is either very dumb (please tell me) or I
>> don't know why anyone hasn't thought about it...
>>
>> I'm starting a bigger project with a self-created plugin system (GDAPS,
>> alrady mentioned). Working nicely, but I came across a few Django
>> shortcomings which are sometimes easy to workaround, but could be done
>> better: the django project name.
>>
>> TL;DR: Django has no (builtin/explicit) settings variable like
>> PROJECT_NAME. Should have.
>>
>> Long version:
>>
>> If you have (like in my case) an application which is more than a single
>> django app, and could have various 3rd party additions/plugins, there is no
>> common sense of retrieving the overall "name" of the application. If you
>> create a project using django-admin startproject myproject - the name
>> says it, you name the project. It is written into various places, mostly
>> comments etc.:
>>
>> foo/asgi.py:ASGI config for foo project.
>> foo/asgi.py:os.environ.setdefault('DJANGO_SETTINGS_MODULE',
>> 'foo.settings')
>> foo/settings.py:Django settings for foo project.
>> foo/settings.py:ROOT_URLCONF = 'foo.urls'
>> foo/settings.py:WSGI_APPLICATION = 'foo.wsgi.application'
>> foo/urls.py:"""foo URL Configuration
>> foo/wsgi.py:WSGI config for foo project.
>> foo/wsgi.py:os.environ.setdefault('DJANGO_SETTINGS_MODULE',
>> 'foo.settings')
>> manage.py:os.environ.setdefault('DJANGO_SETTINGS_MODULE',
>> 'foo.settings')
>>
>> But foo is no variable. Needers of this inofficial project name use
>> workarounds like settings.ROOT_URLCONF[0] to get that name. But that's

Re: Generate JWTs with Django

2020-05-11 Thread Dan Davis

The place where JWT begins to get useful and important is when federated
login capabilities end-up in your app. That sort of thing seems more
the domain of python-social-auth packages like social-auth-core and
social-auth-app-django.  Generating an authentication cookie doesn't
require JWT - Django already does that.

On Mon, May 11, 2020 at 9:37 AM Derek Adair  wrote:

> Maybe we can update the docs to show how you you would might use some of
>> the signing primitives instead of JWTs, but this also sounds a bit
>> dangerous 路‍♂️
>>
>
> As someone hoodwinked into believing JWT was the way... I'd absolutely
> LOVE a clear and concise write up on how I might get my single page js apps
> to communicate securely with projects like Django Rest.
>
> Thanks for closing the door on JWT for me James.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/adc7a8eb-6100-4639-af98-4bca9afaad0b%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFzonYarnig9vCJAekv%3DsaKuMJ-jz755XgLd%2B4GKJ0cN5YSBnQ%40mail.gmail.com.

Re: Google Patch Rewards program

2019-12-31 Thread Dan Davis

Another good security improvement would be to allow the database password
and other database information to support AWS Secrets Manager, Goolge
Secrets Management, and HashiCorp vault (+ others).
I have done this in a private package used at the National Library of
Medicine, but my package is both private and limited to RDS PostgreSQL and
AWS Secrets Manager.

On Tue, Dec 31, 2019 at 11:25 AM Dan Davis  wrote:

> Taymon Beal writes:
> > First-class integration with one or more secrets management systems,
> both to generally contain secrets better and more specifically
> > so people aren't so tempted to check SECRET_KEYs and database passwords
> into source control. (I think this was mentioned in the list of GSoC
> project ideas.)
>
> What secrets management systems are you thinking about?   I wrote
> confsecrets, whose Django integration never worked right, assuming that I'd
> port it to support the following:
> * HashiCorp vault
> * AWS Secrets Manager
> * Google Secrets Management
>
> I also need to switch it from cryptdome to cryptography or make it
> agnostic on these.
>
> P.S. - I have some very limited applied cryptography knowledge - couple
> chapters of Bruce Schneier and some hands-on learning, such as securing
> single license features with signatures based on my employer's license
> signing private key, and discovering that a customer literally pulled one
> set of feature.xml/feature.sig files from one tarball, and another from
> another tarball to game my home-grown licensing system.  I helped that
> company eventually move to a Secure USB dongle, and that was my last moment
> of systems software development :)
>
> On Sat, Dec 28, 2019 at 11:23 PM Taymon A. Beal 
> wrote:
>
>> (Disclosure: I'm on Google's security team, and my views on this topic
>> are informed by what kinds of things we tend to look for in Web frameworks,
>> but here I don't speak for them, only for myself.)
>>
>> Beyond those already mentioned, here are some potential security
>> improvements I'd like to see in Django:
>>
>>- Support for contextual autoescaping in the template system. The
>>current autoescaping behavior is better than nothing, but it still makes
>>XSS far too easy, since different kinds of strings are XSS sinks in
>>different contexts.
>>
>> https://github.com/google/closure-templates/blob/master/documentation/concepts/auto-escaping.md
>>  shows
>>an example of how to do this sort of thing more securely.
>>- First-class integration with one or more secrets management
>>systems, both to generally contain secrets better and more specifically so
>>people aren't so tempted to check SECRET_KEYs and database passwords into
>>source control. (I think this was mentioned in the list of GSoC project
>>ideas.)
>>- Capability-based authorization. Right now, you have to explicitly
>>check for all relevant permissions everywhere, and if you forget, or even
>>if you accidentally include the wrong variable in a template, you can leak
>>data or worse. It'd be much safer if the allowed permissions could be
>>defined at a single choke point, and from there, all model access within a
>>request could be mediated by the specified authorization rules. Ideally
>>data that the current user isn't supposed to see would not even be fetched
>>from the database.
>>
>> Taymon
>>
>> On Sat, Dec 21, 2019 at 11:29 PM Asif Saif Uddin 
>> wrote:
>>
>>> Really good plans Adam!
>>>
>>> On Saturday, December 21, 2019 at 11:51:11 PM UTC+6, Adam Johnson wrote:
>>>>
>>>> I just saw Google is expanding their Patch Rewards program for open
>>>> source security improvements:
>>>> https://security.googleblog.com/2019/12/announcing-updates-to-our-patch-rewards.html
>>>>
>>>> They are offering two tiers of rewards - $5,000 or $30,000 - for  open
>>>> source projects making security improvements. I think Django would find it
>>>> hard to fit in the "small" tier - we generally fix known vulnerabilities
>>>> quickly - but we could use the "large" tier to fund a bigger GSoC style
>>>> project. I suspect it would need active involvement from a DSF member to
>>>> push it through. Not sure how the funding would work in terms of DSF and
>>>> paying for development time on the project.
>>>>
>>>> Some projects that could fit:
>>>>
>>>>- 2FA built-in to django.contrib.auth (as suggested for GSoC as
>>>>well in this thread:
>>>>

Re: Google Patch Rewards program

2019-12-31 Thread Dan Davis

Taymon Beal writes:
> First-class integration with one or more secrets management systems, both
to generally contain secrets better and more specifically
> so people aren't so tempted to check SECRET_KEYs and database passwords
into source control. (I think this was mentioned in the list of GSoC
project ideas.)

What secrets management systems are you thinking about?   I wrote
confsecrets, whose Django integration never worked right, assuming that I'd
port it to support the following:
* HashiCorp vault
* AWS Secrets Manager
* Google Secrets Management

I also need to switch it from cryptdome to cryptography or make it agnostic
on these.

P.S. - I have some very limited applied cryptography knowledge - couple
chapters of Bruce Schneier and some hands-on learning, such as securing
single license features with signatures based on my employer's license
signing private key, and discovering that a customer literally pulled one
set of feature.xml/feature.sig files from one tarball, and another from
another tarball to game my home-grown licensing system.  I helped that
company eventually move to a Secure USB dongle, and that was my last moment
of systems software development :)

On Sat, Dec 28, 2019 at 11:23 PM Taymon A. Beal 
wrote:

> (Disclosure: I'm on Google's security team, and my views on this topic are
> informed by what kinds of things we tend to look for in Web frameworks, but
> here I don't speak for them, only for myself.)
>
> Beyond those already mentioned, here are some potential security
> improvements I'd like to see in Django:
>
>- Support for contextual autoescaping in the template system. The
>current autoescaping behavior is better than nothing, but it still makes
>XSS far too easy, since different kinds of strings are XSS sinks in
>different contexts.
>
> https://github.com/google/closure-templates/blob/master/documentation/concepts/auto-escaping.md
>  shows
>an example of how to do this sort of thing more securely.
>- First-class integration with one or more secrets management systems,
>both to generally contain secrets better and more specifically so people
>aren't so tempted to check SECRET_KEYs and database passwords into source
>control. (I think this was mentioned in the list of GSoC project ideas.)
>- Capability-based authorization. Right now, you have to explicitly
>check for all relevant permissions everywhere, and if you forget, or even
>if you accidentally include the wrong variable in a template, you can leak
>data or worse. It'd be much safer if the allowed permissions could be
>defined at a single choke point, and from there, all model access within a
>request could be mediated by the specified authorization rules. Ideally
>data that the current user isn't supposed to see would not even be fetched
>from the database.
>
> Taymon
>
> On Sat, Dec 21, 2019 at 11:29 PM Asif Saif Uddin  wrote:
>
>> Really good plans Adam!
>>
>> On Saturday, December 21, 2019 at 11:51:11 PM UTC+6, Adam Johnson wrote:
>>>
>>> I just saw Google is expanding their Patch Rewards program for open
>>> source security improvements:
>>> https://security.googleblog.com/2019/12/announcing-updates-to-our-patch-rewards.html
>>>
>>> They are offering two tiers of rewards - $5,000 or $30,000 - for  open
>>> source projects making security improvements. I think Django would find it
>>> hard to fit in the "small" tier - we generally fix known vulnerabilities
>>> quickly - but we could use the "large" tier to fund a bigger GSoC style
>>> project. I suspect it would need active involvement from a DSF member to
>>> push it through. Not sure how the funding would work in terms of DSF and
>>> paying for development time on the project.
>>>
>>> Some projects that could fit:
>>>
>>>- 2FA built-in to django.contrib.auth (as suggested for GSoC as well
>>>in this thread:
>>>
>>> https://groups.google.com/forum/#!msg/django-developers/ifYT6lX8nmg/1nVO3As1AwAJ
>>>)
>>>- Adding CSP to SecurityMiddleware and shipping some default
>>>(django-csp is a good start but requires users to actively seek it:
>>>https://django-csp.readthedocs.io/en/latest/ )
>>>- Adding CORS to Django itself (I'm maintaining django-cors-headers,
>>>but its design is a bit pants
>>>https://github.com/adamchainz/django-cors-headers )
>>>- Other things in James Bennett's list of suggestions from this
>>>thread in May 2018:
>>>
>>> https://groups.google.com/forum/#!msg/django-developers/DDpkrvFdnvk/J46ZbakxAgAJ
>>>
>>> Thoughts?
>>> --
>>> Adam
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>>

Re: Migrations in Django 1.7 make unit testing models harder

2019-09-14 Thread Dan Davis

There is no such flag, at least not in 1.11.  I wrote my own "migratetest"
and "cleandb" commands because my DBAs don't let me drop and recreate
databases.   It is a simple matter if you use the testsuite, but it would
probably be better development to call create_test_db manually and such.


On Fri, Sep 13, 2019 at 2:04 PM Dylan Young 
wrote:

> This is an old issue that (as far as I can see) was never resolved.
>
> My proposed solution would be a flag to the test command to automatically
> `makemigrations && migrate`.  When this behaviour is enabled, it would also
> roll back any generated migrations at the end of the test run.   The flag
> might not be necessary (could be default behaviour of `test`)
>
> This way you get the benefits of all worlds:
> 1) uncommitted migration files will be clearly visible after the test run
> (for those inclined to forget to makemigrations).
> 2) The extraneous non-migration creation code can be removed from Django
> (probably already has been at this point).
> 3) Tests are always run against the current state of models
> 4) Tests are always run at current state of migrations (because it
> generates the migrations).
>
> As an extra benefit, it makes testing between branches *so much* easier
> because you don't have to worry about rolling back migrations on the
> test_db (a PITA as far as I can see).
>
> Regarding the last, is there a flag to make migrate target the test_db(s)?
>
> Best,
>
> Casey
>
> On Tuesday, 25 March 2014 14:21:51 UTC-3, Bernie Sumption wrote:
>>
>> Hi Django devs,
>>
>> I've just started a new project in 1.7b, and the development experience
>> working with unit tests on models is more complicated than it was in 1.6.
>> It's all down to how the throwaway test databases are created. In 1.6, the
>> create_test_db function "Creates a new test database and runs syncdb
>> against it." In 1.7, it runs "migrate".
>>
>> While generally speaking, migrate is the new syncdb, this behaviour is
>> not ideal for tests. In 1.6 "syncdb" created a database reflecting the
>> current state of the models in models.py. "migrate" creates a database
>> reflecting the state of the models at the last time makemigrations was run.
>> If you're doing TDD and constantly making small changes to your models then
>> runnning unit tests, you have to run makemigrations before each test run to
>> get your tests to work. You therefore end up with many tiny migration files
>> representing the minute-by-minute history of development.
>>
>> I came up with a pretty effective workaround that is working for me, but
>> I thought I'd post this here as others are sure to encounter this issue,
>> and I think that it makes more sense for the behaviour produced by this
>> workaround to be the default for running tests.
>>
>> If makemigrations has not yet been run, the "migrate" command treats an
>> app as unmigrated, and creates tables directly from the models just like
>> syncdb did in 1.6. I defined a new settings module just for unit tests
>> called "settings_test.py", which imports * from the main settings module
>> and adds this line:
>>
>> MIGRATION_MODULES = {"myapp": "myapp.migrations_not_used_in_tests"}
>>
>> Then I run tests like this:
>>
>> DJANGO_SETTINGS_MODULE="myapp.settings_test" python manage.py test
>>
>> This fools migrate into thinking that the app is unmigrated, and so every
>> time a test database is created it reflects the current structure of
>> models.py.
>>
>> So my feature request is as follows:
>>
>> If the new behaviour is by design and considered desirable, then it is a
>> big change from the previous version and should be prominently documented
>> in the migrations and testing pages, along with the workaround. I'm happy
>> to write this documentation if that's the way you want to go.
>>
>> However, if the new behaviour is not by design but just a by-product of
>> the new migrations feature, I suggest making the workaround the default
>> behaviour. I don't (yet!) know enough about Django internals to volunteer
>> for this however.
>>
>> Thanks for your time,
>>
>> Bernie :o)
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/2aee60d5-e96b-4c96-8f87-150d9de46215%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit

Re: [Feature Request][Model, ORM] Disabling a field before removal to support continuous delivery

2019-06-24 Thread Dan Davis

>
> If does not affect squashing. I think this is roughly reasonable, at least
> for my uses, although I haven't had any real world use for squash myself. I
> recently had to remake the migrations in my repo for $WORK project, and it
> made the most sense to keep them as the default safe after deploy, to match
> what other apps would get. I think in most cases it won't matter when the
> first migration is safe for practical use.


I don't usually use squashmigrations either, being more likely to simply
rewrite or make them initial.   However, if the functionality of your
module were to be added to Django itself, or become more productized, then
these are the issues to think about, right?

I certainly think the use case (support for something like blue/green
deployments with shared database) is very important.
I'm not a super experienced Django contributor, so I'd love to hear what
more experienced hands have to say about it.

On Mon, Jun 24, 2019 at 11:28 PM Ryan Hiebert  wrote:

>
>
> On Mon, Jun 24, 2019, 21:29 Dan Davis  wrote:
>
>>
>> Some questions:
>>
>>- How does the "safe" field of migrations work with other migrations
>>related commands, such as squashmigrations?   It seems to me that only
>>migrations that share the same value of "safe" could be squashed.
>>
>> If does not affect squashing. I think this is roughly reasonable, at
> least for my uses, although I haven't had any real world use for squash
> myself. I recently had to remake the migrations in my repo for $WORK
> project, and it made the most sense to keep them as the default safe after
> deploy, to match what other apps would get. I think in most cases it won't
> matter when the first migration is safe for practical use.
>
>
>>- Can makemigrations figure out which migrations are "safe", such as:
>>   - It is always safe to add a column or table.
>>   - It is never safe to drop or rename a column or table.
>>
>> This is a feature that has sounded neat to me, approximately feasible,
> but hasn't yet been worth the time to implement. We've been happy adding
> the safety annotations manually.
>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CABpHFHSb2AHKb-XhZcAk6gZ3%2B%2Bt5nz7UkBUNyQ_FC%3DsYGgO_Yw%40mail.gmail.com
> <https://groups.google.com/d/msgid/django-developers/CABpHFHSb2AHKb-XhZcAk6gZ3%2B%2Bt5nz7UkBUNyQ_FC%3DsYGgO_Yw%40mail.gmail.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFzonYZpeYZGqGk3SMgYSHPk7%2BZ4vQfDNv-GaQJrjFZLivL0gQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Feature Request][Model, ORM] Disabling a field before removal to support continuous delivery

2019-06-24 Thread Dan Davis

I'd agree that it is a definite use case.  In the dev-ops world, it is
evidently called a "blue torquoise green deployment".   It could be done as
long as the code is not adding a table/field/etc.

My discussion on django-users,
https://groups.google.com/forum/#!topic/django-users/QCmy9reH8cI, also
included this issue, but I was also concerned about:
* Removing a model
* Renaming a field
* etc.

The django-safemigrate 
repository/app seems to be easier to generalize, as such an option could be
added to makemigrations.
Some questions:

   - How does the "safe" field of migrations work with other migrations
   related commands, such as squashmigrations?   It seems to me that only
   migrations that share the same value of "safe" could be squashed.
   - Can makemigrations figure out which migrations are "safe", such as:
  - It is always safe to add a column or table.
  - It is never safe to drop or rename a column or table.


On Mon, Jun 24, 2019 at 9:48 AM Ryan Hiebert  wrote:

> I'm not sure about the solution you mentioned, but the problem you mention
> is one that I definitely do deal with. At my work we have been happy with
> using a "safe" migrate command that only runs migrations that are marked as
> safe to run before the deployment happens, to address exactly this kind of
> scenario. You still need to make sure you don't mix stuff in the same
> migration that delete and add a column, for example, but it makes the
> process much easier, and we've been able to add the safemigrate command to
> our regular automated deployments. The full migrate then only runs when
> manually triggered, to deal with removing columns, etc.
>
> You can check out our repository. We've been pretty happy with how it's
> working for us. https://github.com/aspiredu/django-safemigrate
>
> On Mon, Jun 24, 2019 at 8:15 AM Matthieu Rudelle 
> wrote:
>
>> Hi there,
>>
>> I can't find any previous ticket proposing a solution to this problem so
>> here are my findings:
>>
>> **Use case**:
>> When using continuous delivery several versions of the code can be
>> running in parallel on se same DB. Say for instance that release 2.42 is in
>> production, 2.43 is about to be rolled out and in this release one field
>> (say ''MyModel.my_unused_field'') is not used anymore and was removed.
>> Before rolling out 2.43 the DB is migrated and column ''my_unused_field''
>> of ''MyModel'' is removed. This makes 2.42 crash saying that one column is
>> not found even though 2.42 does not use the field anywhere in the code.
>>
>> **Temporary solution**:
>> Do not makemigrations until de 2.44 release, but it does not scale well
>> with many contributors and CI tools (doing their awesome job of making sure
>> migrations and models are in sync) will complain.
>>
>> **Proposed solution**:
>> Have a ''disabled'' param on Field. When activated this field is not
>> fetched from the DB but replaced by a hardcoded value.
>> In our use case, ''disabled'' is added at the 2.42 release, then when
>> 2.43 rolls out and migrates the DB no error is thrown.
>>
>> **Refs**:
>> - django-users discussion:
>> https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8
>> - the same problem discussed in this article, but with a slightly
>> different solution:
>> https://pankrat.github.io/2015/django-migrations-without-downtimes/#django-wishlist
>> (third item in the wishlist)
>> - Previous django-developpers discussion:
>> https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ
>>
>> What do you guys think?
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/637c3542-c5bd-4493-8b12-44eb61f34a68%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CABpHFHSa4hXVrp7M2MWLcsPXSRnCnvhMQ%3D8D3q1Rg8AsAXU4Cw%40mail.gmail.com
>

Re: Faster Migrations! But at what cost?

2019-05-21 Thread Dan Davis

Migrations are very slow, so I don't run them during test runs, and run
tests with --keepdb even in CI/CD.  This is required for my environment
anyway, because we use Oracle heavily, a "schema" is the same thing as a
"user", and is what I used to think of as a "database" coming from MySQL
and PostgreSQL.

To make migrations testable, I wrote a management command "clearschema"
which can clear the schema of any database alias, or its test analog.  This
"clearschema" command is in a framework app that all the apps at my
workplace include, but I think only I and a couple other projects use it.

Anyway, it seems that if --keepdb is so necessary, that anything we can do
to speed up migrations during tests and during deployment would be
worthwhile, but we have to be very careful to make sure that migrations are
imported once, and that any caching scheme is idempotent - e.g. we can
throw away the cached data.

On Mon, May 20, 2019 at 11:26 PM Raphael Gaschignard 
wrote:

> Hi Markus, Simon,
>
>   Both of you, thank you for the detailed replies and status report on
> this sort of stuff.
>
> > Did you look into squashing these 500 migrations by any chance?
>
> Yeah so I'll go in and squash things, though (partly because effective
> squashing requires moving models around, though we're still at Django
> 1.11 so it might become easier, partly because of cross-app references)
> it's a decent amount of work.
>
> I do like the idea of reusing a "effectively production DB" for things,
> not least because it's actually acurate.
>
> OK so the Operation API is effectively documented. I am tempted to try
> modifying the `database_forwards` and `database_backwards` to not
> require `from_state` (probably through adding a hook that gets called
> _before_ `state_forwards` to allow capturing the small details of the
> old state). It might actually be possible to make this backwards
> compatible by porting Django migrations to use an `apps`-free workflow,
> but falling back to the old "re-render the universe" mechanism for
> operations that don't apply it.
>
> This is very "that xkcd comic about time spent versus time saved"
>
>  > will cause trouble with RunSQL and other operations that use related
> field or model attributes
>
> So one thing I felt like was an invariant in this code was that field
> sharing was expected? From the docstring of ModelState:
>
>  Note that while you are allowed to mutate .fields, you are not allowed
>  to mutate the Field instances inside there themselves - you must
> instead
>  assign new ones, as these are not detached during a clone.
>
> Also maybe you meant to refer to RunPython instead of RunSQL. But I get
> your point in general here. Related models can be a problem
>
> One throwaway idea would be to not allow related model/related field
> access in these models? There's already a lot of documentation related
> to not allowing general model methods (effectively establishing that
> "migration models are _not_ normal models"), so there's a bit of
> precedent. But beyond the general backwards incompatability, it might
> not actually even be obvious how one would implement this. And you kinda
> need this info for foreign keys and the like anyways.
>
>
> Working directly off of `ModelState` is interesting, and I think there
> might be a backwards-compatible way forward there, where we still allow
> for rendering on certain operations but hold off on it on the basic
> ones. Even in our large project, most of our migration operatiosn are
> dirt-simple, so if the core django migrations could work off of
> `ModelState` then we could get a fast path through there.
>
> Thanks for your input, both of you. I have a couple ideas now that I'm
> pretty tempted to try out, mainly around the "fast path and slow path"
> strategies that should offer backwards compatibility.
>
>   Raphael
>
> Markus Holtermann wrote on 2019/05/21 2:26:
> > Thanks Raphael for bringing this topic up and Simon for your input
> already.
> >
> > I just left a note on your PR:
> https://github.com/django/django/pull/11388#issuecomment-494076750 . I'll
> quote it here for ease of readability:
> >
> > As far as I can see right now, a similar caching happened as a first
> approach to the Django 1.8 release but cause significant problems,
> specifically with regards to relational fields. Relational fields
> (ForeignKey, OneToOneField, ManyToManyField) keep an instance reference to
> the related model in `.related_model` or the related fields in
> `.related_fields`. The problem now is, if you reuse a field (and you do
> because you're only calling `list()` on `self.fields` to copy but not
> deepcopy the list), you're screwing up references between the models that
> _will_ cause trouble with `RunSQL` and other operations that use related
> field or model attributes.
> >
> >
> https://github.com/django/django/blob/1d0bab0bfd77edcf1228d45bf654457a8ff1890d/django/db/models/fields/__init__.py#L495-L499
> >
> >  From my work on

Re: New external package - is there any duplication

2019-05-21 Thread Dan Davis

Thank you for the review, Adam.
I like the idea of the post-migrate signal, but it can be complicated, and
sometimes doing things in sequence is important:

With Oracle, if function B runs function A (and so on) then after we run:

CREATE OR REPLACE FUNCTION *A*(...) 


We would need to, in Oracle anyway, run:

ALTER FUNCTION *B *COMPILE REUSE SETTINGS;


Hard then to be an idempotent change then, even with create and replace.

On Tue, May 21, 2019 at 4:48 PM Adam Johnson  wrote:

> Hi Dan,
>
> I think this has not got much of a response because it might be a bit
> niche - it really depends where your teams are struggling to use RunSQL.
>
> Looking for other packages at
> https://djangopackages.org/grids/g/database-migration/ , nothing seems to
> be related. They are all precursors/replacements for Django Migrations.
> Maybe there are other packages somewhere on Django Packages though.
>
> Some responses to your ideas:
>
> 1. Rather than subclass RunSQL to load sql files, just embed SQL in Python
> with triple quoted strings? The checksumming would be unnecessary then,
> presuming team members understand that (applied) migration files are
> immutable.
>
> 2. You could "sync" a folder of (idemptotent) SQL files with a
> post_migrate signal handler. For example, Django syncs the Content Type
> table with such a handler:
> https://github.com/django/django/blob/master/django/contrib/contenttypes/apps.py#L20
>  .
> You could loop over all the SQL files in a given folder to check and run
> them. They could contain custom idempotent SQL such as "CREATE OR REPLACE
> FUNCTION" - presuming you're using RunSQL for DDL that Django Migrations
> doesn't support. I've seen this approach a couple times for things like
> stored procedures, though I'd prefer to use vanilla RunSQL in a migration
> myself. At least one can avoid yet another easily forgotten management
> command.
>
> Thanks,
>
> Adam
>
>
>
> On Fri, 17 May 2019 at 19:08, Dan Davis  wrote:
>
>>
>> I am planning an external package to assist my developers in getting
>> their special-purpose DDL out of the database and into git.   Some teams
>> can handle it, but some teams could use help making sure that they use
>> operations.RunSQL well and wisely.
>>
>> My biggest questions about what I'm doing are:
>> * Are there any packages that already attempt to do what I'm thinking of?
>> * What gotchas have I not considered?
>> * What should it be called?   What should the management command be
>> called?
>> * How do I test it well?
>>
>> Here is a more detailed idea of the packages key contributions (
>> https://github.com/danizen/django-sqlextras/):
>>
>> * Create a migration operation that is a subclass of RunSQL, but takes a
>> forward object that consists of a path and an expected checksum, and a
>> similar optional backwards checksum.
>> * Create a management command that can build migrations using this
>> operation (or you can do it yourself, of course).
>>
>> Feedback is welcome.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/73a56671-3904-4eef-b82a-d04fe175c993%40googlegroups.com
>> <https://groups.google.com/d/msgid/django-developers/73a56671-3904-4eef-b82a-d04fe175c993%40googlegroups.com?utm_medium=email_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> --
> Adam
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAMyDDM0FhWyZrgZRVw1i_OGQ6qMGu8xArL8xwxWZbghC%3Ds6bsg%40mail.gmail.com
> <https://groups.google.com/d/msgid/django-developers/CAMyDDM0FhWyZrgZRVw1i_OGQ6qMGu8xArL8xwxWZbghC%3Ds6bsg%40mail.gmail.com?utm_medium=email_source=footer>
> .

New external package - is there any duplication

2019-05-17 Thread Dan Davis

I am planning an external package to assist my developers in getting their
special-purpose DDL out of the database and into git. Some teams can
handle it, but some teams could use help making sure that they use
operations.RunSQL well and wisely.

My biggest questions about what I'm doing are:
* Are there any packages that already attempt to do what I'm thinking of?
* What gotchas have I not considered?
* What should it be called? What should the management command be called?
* How do I test it well?

Here is a more detailed idea of the packages key contributions (
https://github.com/danizen/django-sqlextras/):

* Create a migration operation that is a subclass of RunSQL, but takes a
forward object that consists of a path and an expected checksum, and a
similar optional backwards checksum.
* Create a management command that can build migrations using this
operation (or you can do it yourself, of course).

Feedback is welcome.

--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/73a56671-3904-4eef-b82a-d04fe175c993%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: injecting settings

2019-05-08 Thread Dan Davis

> Another thing to possibly consider, what should happen if multiple
> packages try to provide different defaults for the same setting? I
> mean, of course, this has kind of been floated in this thread already,
> but it would add one more item to the list of things affected by the
> order of INSTALLED_APPS.

Thus, why Adam called this a Hard Problem(tm).

To resolve this, there would need to be a way for an app in INSTALLED_APPS
to declare that it depends on another app.   Then, despite the order in
INSTALLED_APPS, the ready call could be re-ordered, or rather a new call
could be introduced so that the dependent app could be informed once its
parent has been initialized.The app could also be called before its
"parents" are readied, so that it can tell what the user gave in settings.
 Maybe None is the default for a setting in the parent package, but 'lower'
is to be the default setting in the dependent app...

I am not asserting that such a feature is worth it, just shooting the S**T.

On Wed, May 8, 2019 at 7:38 AM Aymeric Augustin <
aymeric.augus...@polytechnique.org> wrote:

> > On 8 May 2019, at 11:38, Michal Petrucha  wrote:
> >
> > On Tue, May 07, 2019 at 10:53:37PM +0200, Aymeric Augustin wrote:
> >> If we're only talking about providing default values for settings,
> currently there are two straightforward solutions:
> >>
> >> 1. Like Adam suggested, access settings like this: getattr(settings,
> 'MY_SETTING', 'my_default').
> >>
> >> This works well when you access settings just once, probably at import
> time, and cache their value.
> >>
> >> Here's an example:
> https://github.com/aaugustin/django-sesame/blob/070cdb3fcdfa6c7310d7461add328a8095148ff1/sesame/backends.py#L27-L34
> <
> https://github.com/aaugustin/django-sesame/blob/070cdb3fcdfa6c7310d7461add328a8095148ff1/sesame/backends.py#L27-L34
> >
> >
> > This approach, however, makes it impossible to use the decorators and
> > context managers from django.test that override settings. Of course,
> > there are other ways to tune those knobs in tests, but it takes away a
> > standard solution provided by the framework.
>
> I'm not sure I understand what you're referring to. The framework provides
> the setting_changed signal, which seems to work well here:
> https://github.com/aaugustin/django-sesame/blob/070cdb3fcdfa6c7310d7461add328a8095148ff1/sesame/test_signals.py#L8-L18
>
> >> DRF relies on a single Django setting which is a dict containing all
> individual DRF settings. Then it converts that Django setting to an object
> and uses this abstraction to access individual settings as properties.
> >
> > This seems to have become one of the current best practices, but it
> > requires each reusable package that goes down this road to include a
> > bunch of boilerplate just to deal with default values for settings. I
> > suppose that boilerplate could be factored out into its own reusable
> > package, but my gut tells me that there must be a better way to handle
> > something as fundamental as providing defaults for settings.
> > Definitely a very subjective claim, but I just feel like this is
> > something that should be the responsibility of the framework itself.
>
> I'm ambivalent about this. I did it myself (TEMPLATES). I'm not sure that
> was worth doing.
>
> Grouping a bunch of related settings in a dict looks satisfying. However,
> it's less practical than giving them a common prefix, notably to override
> them in tests.
>
> I suppose I could support a well-written proposal, though :-)
>
> >> Here we're talking about something slightly different: formalizing how
> an application can declare default values for its own settings —
> essentially a per-app equivalent of Django's global_settings.py. To do this
> properly, we need two things:
> >>
> >> - a good convention for declaring these settings: I would find it
> elegant to consider every uppercase class attribute of an AppConfig class
> as a setting, but that might be too magic (and perhaps
> backwards-incompatible);
> >
> > I definitely agree that this feels too magic-y. Maybe introducing a
> > class attribute such as “DEFAULT_SETTINGS” that would hold a
> > dictionary could work, though?
>
> Yes, it could work too. I considered it and didn't choose it because it
> increases the distance between what global_settings.py looks like and what
> app settings would look like. Not a very strong argument ;-)
>
> Also there's already magic to include only uppercase variables currently:
> https://github.com/django/django/blob/ef9f2eb69c9396683cefa742bc7d0a0792090e8d/django/conf/__init__.py#L135-L137
>
> >> - a way to insert them properly into the settings object: I tried to
> figure out how LazySettings and friends handle global_settings,
> unfortunately there are more use cases than I was willing to untangle
> tonight.
> >
> > Another thing to possibly consider, what should happen if multiple
> > packages try to provide different defaults for the same setting? I
> > mean, of course, this has

Re: injecting settings

2019-05-07 Thread Dan Davis

Christian,

I do this in my internal and private module that depends on django-cas-ng.
 django-cas-ng provides default settings in an __init__.py file that
predates app.py and ready.   My strategy is that if you wish to depend on
another app in this way, it is best to shadow it entirely.   For example,
django-cas-ng defines an authentication backend, CASBackend.   I had an
empty subclass for a long time until I needed to do something else.  As Tom
Forbes writes, this is a hard problem to solve.

For example, JavaScript closures have allowed npm to solve transitive
dependencies in some novel ways that have introduced new problems.

Another way to solve this is to have settings that use duck typing to
behave like other sorts of settings.   For instance, I support EC2 Secrets
Manager and Django database configuration by having a DBConfig class that
acts like a dict, even down to __copy__ and __deepcopy__, so that whenever
Django asks for the database configuration, I can go get the secret.   This
frees the ops guys from restarting the app when the secret is automatically
rotated.

Because Python is so extensible, these things work, up to a point, but it
is a hard problem.

On Tue, May 7, 2019 at 5:55 PM Christian González <
christian.gonza...@nerdocs.at> wrote:

>
> Sorry about writing, and not testing myself before:
>
> 2. Set a default value at import time:
>
> # apps.py
>
> from django.apps import AppConfig
> from django.conf import settings
>
> class MyAppConfig(AppConfig):
> name = 'my_app'
> verbose_name = "..."
>
> def ready(self):
> if not hasattr(settings, 'MY_SETTING'):
> settings.MY_SETTING = 'my_default'
>
> This works.
>
> I have an app that has the following code in Config.ready():
>
> settings.WEBPACK_LOADER.update(
> {
> "foobar": {
> "STATS_FILE": os.path.join(
> settings.BASE_DIR,
> "{}/frontend/webpack-stats.json".format(
> os.path.abspath(os.path.dirname(__file__))
> ),
> )
> }
> }
> )
>
> And the app webpack_loader is located *after* this app in INSTALLED apps.
> Within webpack-loader, the settings dict WEBPACK_LOADER inclusive "foobar"
> is available. So this is kind of "solved" for me - but - it's a bit of a
> hack.
>
> I'd really appreciate a "standard" approach from Dajngo...
>
> Thanks for your patience, ;-)
>
> Christian
>
> --
> Dr. Christian Gonzálezhttps://nerdocs.at
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/3727cd68-3ed6-1366-2277-0815ae90da5c%40nerdocs.at
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFzonYZ4s65G4Nq4u4Xe2SobBzu9fe00qwVvR5H%3DJe9WH%3DDskg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Proposal to format Django using black

2019-04-16 Thread Dan Davis

+1 isort
-1 black

I think that codestyle checkers are better, because you teach yourself
proper style for python.

On Tue, Apr 16, 2019 at 8:17 PM Josh Smeaton  wrote:

> We aren't talking about code minifiers though, are we? We're talking about
> a very specific tool with very specific rules. No one will ever agree on
> one specific code style, which is why subjectivity is anti-productive.
> Black chooses a specific set of rules and removes ambiguity. Some choices
> will be agreeable, others will not be. And the ones that are agreeable
> aren't agreeable to every person.
>
> > and code passed through autoformatters (especially if author _didn't
> think_ about style) is harder to understand then written by thinking humans
>
> I find this statement to be mostly incorrect with regard to Black. There
> are edge cases, but they are few.
>
> On Wednesday, 17 April 2019 09:45:11 UTC+10, Ivan Anishchuk wrote:
>>
>> Yeah, it's very common to confuse style with formatting. Not having to
>> think is a productivity win when and only when you don't care about your
>> product's quality (in which case I'd suggest using generators and stop
>> thinking about code altogether). Language is a communication tool, a
>> programming language is a tool programmers use to talk to each other, and
>> code passed through autoformatters (especially if author _didn't think_
>> about style) is harder to understand then written by thinking humans. It's
>> pretty much like trying to stop thinking about style in English -- by using
>> some processor to put your commas and whitespace in place you can generate
>> something that is usually possible to understand, at least for the most
>> part, but that and _good style_ would still be worlds apart (yes, natural
>> languages are different, it's much easier to make things worse there, but
>> it's not any easier to make things better in programming languages).
>>
>> There are situations where compromising style for formatting is
>> acceptable and even a good idea, I don't think here is one of them.
>> Removing a significant expressive tool from a language just to ensure your
>> whitespace is arranged according to some arbitrary rules is about as
>> counter-productive and pointless as it can get. Not because sometimes it
>> produces some bad results but exactly because it prevents humans from
>> thinking and expressing themselves properly. What would you think about
>> some processor renaming all your variables so that you don't have to think
>> about naming them? Must be even better for productivity.
>>
>> Ivan.
>>
>> On Wed, Apr 17, 2019 at 12:35 AM Josh Smeaton  wrote:
>>
>>> Ivan, what you’re talking about is subjective code formatting, and lends
>>> itself to extreme bikeshedding and little consensus. Django already has a
>>> formatting standard that mergers try to enforce at review and commit time.
>>> But it’s time consuming, prone to missing things, and requires lots of back
>>> and forth.
>>>
>>> Not having to think of formatting at all, either as a developer or
>>> reviewer, is a major productivity win. Even if some of the formatting
>>> choices are not agreeable.
>>>
>>> On Wed, 17 Apr 2019 at 03:38, 'Ivan Anishchuk' via Django developers
>>> (Contributions to Django itself)  wrote:
>>>
 My two cents: black's mostly ok but I'm against any auto-formatting.
 Code style is much more than whitespace arrangement, it requires looking,
 creative thinking, judging, and sometimes actually compromising rules and
 looks for readability while the usual attitude with such tools is "I have
 this autoformatter tool so I don't have to think about style anymore, see,
 all my whitespace is nice and shining". Using some helpers in your favorite
 editor is great but relying on automatics without looking is almost
 completely counter-productive.

 I find that PR autocommenter is a great way to keep issues detectable
 by flake8/pylint to a minimum.

 Ivan.

 On Sat, Apr 13, 2019 at 6:35 PM Herman S  wrote:

> Hi.
>
> I propose that Django starts using 'black' [0] to auto-format all
> Python code.
> For those unfamiliar with 'black' I recommend reading the the projects
> README.
> The short version: it aims to reduce bike-shedding and non value-adding
> discussions; saving time reviewing code; and making the barrier to
> entry lower
> by taking some uncompromissing choices with regards to formatting.
> This is
> similar to tools such as 'gofmt' for Go and 'prettier' for Javascript.
>
> Personally I first got involved contributing to Django couple of weeks
> back,
> and from anecdotal experience I can testify to how 'formatting of
> code' creates
> a huge barrier for entry. My PR at the time went multiple times back
> and forth
> tweaking formatting. Before this, I had to research the style used by
> exploring
> the docs at length and reading at least 10-20

Re: Should django.contrib.auth.authenticate check is_active?

2019-04-11 Thread Dan Davis

So, federated login systems such as often handle only user identity
authentication, which is what the AuthBackend does. Sometimes, users can
self-register, and systems such as django-cas-ng (for CAS, not Oauth2).
 Oauth2, although actually granting authorization to the identity
provider's platform (e.g. to a Google user's profile, or even their
email/calendar APIs), are often still used just for the purpose of identity
verification.

You are correct that is_active can become a thing that developers must
think about in designing a system, but because it has to do with
authorization to login and use the Django project/site, I think it is still
a concern of authentication.   Also, while the is*_active* field is part of
the standard user model, it is a 1-line change to disable this check:
https://github.com/django/django/blob/bfe279293c0b9a2b4a463be76f33b2025add20b6/django/contrib/auth/backends.py#L108

Many authentication backends subclass the ModelBackend, but they don't have
to do so and can easily override the user_can_authenticate() method.

Does this answer your question?

On Thu, Apr 11, 2019 at 1:43 PM Daniel Tao  wrote:

> This is essentially an RFC. I've become more familiar with Django's
> authentication backend system lately, and something has stood out to me
> that I'd like to draw attention to.
>
> A new site built with Django is likely to use the default ModelBackend,
> which includes a user_can_authenticate
> 
> method that returns False for inactive users (where is_active = False).
> This can create the expectation that inactive users should not be able to
> authenticate as a general rule.
>
> The development team might later choose to introduce some custom
> authentication backends, for example to authenticate with an external
> service or using API tokens, without realizing that these also need to
> check the user's is_active field in order to preserve existing behavior
> (which may have become enshrined in the team's policies, for example if
> they had been setting is_active = False on accounts that had been detected
> performing malicious activity).
>
> In my opinion it would be desirable in a large percentage of cases to move
> the logic in user_can_authenticate out of ModelBackend specifically and
> into the authenticate method itself, so that inactive users are blocked
> from authenticating using *any* backend. This would remove the burden
> from developers of authentication backends to remember to check for
> is_active in each one.
>
> An obvious problem with changing the default behavior is that it would not
> be backwards compatible. How each application interprets is_active is up to
> that application, and some may have authentication backends that allow
> inactive users to authenticate by design. One option for to addressing that
> would be to introduce a setting such as AUTHENTICATION_REQUIRE_ACTIVE, set
> to False by default (but perhaps explicitly set to True for new Django
> projects created with the startproject command), which could be enabled for
> a given application to apply this requirements consistently in a
> future-proof way.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/19cbe676-ab7a-4dc9-98ea-d2cfe767695b%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFzonYYO%3DeS2XtAeravOnEXg4CwSBCrvPi7KANBFuR2JTw07NQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: 2020 Authentication Initiativ

2019-04-11 Thread Dan Davis

Focus is the biggest thing - with so many other packages such as
python-social-auth and django-cas-ng and django-warrant providing some sort
of Federated Login, I don't think it makes sense to try to incorporate
social login.  However, it would be nice if out of the box could do a
register, verify email, login flow, and using email or username as they key
would be great.

Backwards compatibility is also important - with django-cas-ng providing
authentication for all the django projects where I work, I still needed a
back-door to allow AppScan to login as an admin user.   So, I make sure
users gets a random password, even though we are logging in via CAS anyway.

On Thu, Apr 11, 2019 at 10:56 AM Johannes Hoppe 
wrote:

> Hi there,
>
> @Florian, since I had so many PRs pending review, I had to find other ways
> to cause chaos ;)
>
> Besides that I agree, this isn't an easy undertaking and a lot of time
> will need to be spend on the whiteboard to get this going. Maybe it is
> worth actually collecting a little bit of money for this. I am sure there
> are companies that would be willing to invest into a better authentication
> framework in Django.
>
> When it comes to how this could looks like, I see two main pillars:
> 1. It needs to be extendable. Provide a great framework and let the
> community rise to the task implement the various systems.
> 2. There needs to be at least one solid out of the box solution, got get
> users productive from minute one.
>
> With that in mind, I do see something, that would separate the User object
> from an authentication provider. Both being separate and swappable
> entities. In a way this is already the case with the user and
> authentication backend separation. However, the AbstractBaseUser does
> already implement a password, a natural key and suggest the implementation
> of a username.
> All of which are not a given for all authentication mechanisms.
>
> I would suggest as a first step to separate the field names from the
> object methods. Thanks to multi inheritance in Python this should be an
> easy enough task.
>
> I will look into it a bit and propose something as a base for discussion.
>
> Best
> -Joe
>
>
> On Thursday, April 11, 2019 at 1:45:14 PM UTC+2, NOTARO WEB wrote:
>>
>> Hello, this my first message... hi for everybody
>>
>> I think it could be a micro-service... you would not touch what is
>> there,  which is a huge advantage regarding compatibility.  We could start
>> to monolith django and I think with this Auth history it would be a good
>> opportunity. Abstraction for django-monolith with base conversation and
>> then the Auth micro-service... and we open the community can create/improve
>> other potential micro-service
>> Anyway... I would be happy to help.
>>
>> Best regards
>> Jeff Notaro
>>
>> Am 10.04.2019 22:59 schrieb Collin Anderson :
>>
>> Email + password auth is definitely a wanted feature out-of the box, and
>> probably a good first step would be to create a separate AbstractEmailUser
>> or something like that. Seems to me AbstractUser shouldn't be changed for
>> backwards compatibility reasons, but maybe something like a
>> BaseAbstractUser would be helpful?
>>
>> And here's a past discussion and ticket (from 5 years ago):
>> https://groups.google.com/d/topic/django-developers/7feYlp9HqKs/discussion
>> https://code.djangoproject.com/ticket/20824
>>
>> Also related: UserCreationForm by default allows usernames that differ
>> only by case
>> https://code.djangoproject.com/ticket/25617
>>
>> On Wed, Apr 10, 2019 at 7:12 AM Barnaby  wrote:
>>
>> If we go to the most common use case, email + password is the current
>> "default" of the web, rather than username + password. It would make sense
>> for Django to use email + password by default.
>>
>> It also feels like first_name and last_name have no place in AbstractUser
>> and should me moved to NamedAbstractUser or something.
>>
>> So we'd remove username, first_name and last_name by default.
>>
>> Regarding other means of authentication, I don't know if Django should
>> support any out-of-the-box. Magic Links could be a decent default but they
>> do raise security issues and require email setup.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-d...@googlegroups.com.
>> To post to this group, send email to django-d...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/2ec07906-8a2a-4b63-a850-99e8fef95b5a%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this

Re: Support for unittest -k option

2019-03-11 Thread Dan Davis

I personally don't think a short-hand is needed.

On Mon, Mar 11, 2019 at 10:41 AM Tim Graham  wrote:

> -kdb could be a suitable short option.
>
> On Monday, March 11, 2019 at 9:20:37 AM UTC-4, Tobias McNulty wrote:
>>
>> Agreed it's probably better to make the switch now, and I'd be fine
>> without a replacement shorthand alternative for --keepdb.
>>
>> Cheers,
>>
>>
>> *Tobias McNulty*Chief Executive Officer
>>
>> tob...@caktusgroup.com
>> www.caktusgroup.com
>>
>>
>> On Mon, Mar 11, 2019 at 8:19 AM Carlton Gibson 
>> wrote:
>>
>>> Thanks François,
>>>
>>> Just on this, my thought is that if we don't follow `unittest` in
>>> changing `-k` for this, we have a steady trickle of confusion forever-more.
>>> I'd rather avoid that.
>>>
>>> C.
>>>
>>> On Monday, 11 March 2019 13:14:01 UTC+1, François Freitag wrote:

 Hi Django Devs,

 https://code.djangoproject.com/ticket/30245 suggests supporting Python
 unittest `-k` option, to selectively run tests matching a keyword.

 Currently, `-k` is the shorthand for `--keepdb` in Django.
 A `--filter` flag was suggested to preserve backward compatibility.
 Carlton suggested removing the `-k` option from `--keepdb` and reusing
 it for unittest `-k`. That would follow unittest more closely, reduce
 user confusion and ease maintenance.

 What do you think is best? Do you see other options?

 If re-taking the `-k` option for unittest `-k`, should a new shorthand
 be introduced for `--keepdb`?

 Thanks for your time,
 François

>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-developers/c54e52d2-012a-4852-9375-be37add55945%40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/d0f1df05-238c-4ad6-97c2-a852bda939bc%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFzonYYRhKbPbuUvZCb9hM3jM%3D90VPwv9Ug%2B0k%3DdXUuRyDMz_Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Will Django ever CompositePrimaryKeys?

2019-02-26 Thread Dan Davis

Thanks, Josh. I will be cautious if I ever get to taking this on.

On Tuesday, February 19, 2019 at 9:55:49 PM UTC-5, Josh Smeaton wrote:
>
> To add - there's definitely appetite for this feature, but it's a 
> difficult one, and no one has stepped up to do it.
> There are DEP drafts that cover pieces:
>
> https://github.com/django/deps/blob/master/draft/0191-composite-fields.rst
>
> https://github.com/django/deps/blob/master/draft/0192-standalone-composite-fields.rst
>
> Citus Data was looking to sponsor development for this feature, and still, 
> no one with the technical capabilities was willing (or able) to step up: 
>
> https://groups.google.com/forum/#!searchin/django-developers/composite|sort:date/django-developers/wakEPFMPiyQ/ke5OwgOPAQAJ
>
> I would strongly advise not just running off and attempting something on 
> your own. Look over the previous attempts and DEPs, and then document your 
> way forward either by editing the DEP, and by initiating a thread on this 
> list.
>
> Cheers
>
> On Wednesday, 20 February 2019 04:29:41 UTC+11, Dan Davis wrote:
>>
>> James,
>>
>> As a Django user I've had this problem often. My best practice ways to 
>> handle this is as follows:
>>
>>
>>- If the table is read-only, then create a database-level view that 
>>manufactures a primary key by concatenating the primary key columns 
>>together.   Lie to Django and say this combined column is the primary 
>> key, 
>>and keep the Database-level view's model as managed = False at the Django 
>>level.   However, keep the SQL for the view in your git repository, and 
>>when it changes, build a manual migration - manage.py makemigration 
>> fooapp 
>>--empty -n create_replace_bar_view.
>>
>>
>>- If the table is read-write, then either (a) create a view with 
>>instead of update/instead of insert triggers that manage an underlying 
>>table, or (b) just add a unique ID and make the existing primary key a 
>>unique together constraint and index.  With the trigger to add a new ID, 
>>other users of that table shouldn't notice any issues.
>>
>>
>>- Maybe my statement that Django should manage migrations to the 
>>schema seems unworkable. However, one of the biggest gains you can get 
>> from 
>>Django with old schemas like this is to get their DDL into git. Even if 
>> git 
>>doesn't do it, make sure some code does it, and it isn't left unmanaged. 
>>That's been a key challenge and opportunity for me.
>>
>>
>> As a Django developer, I'm a bit green to volunteer to handle this issue, 
>> but this is exactly the kind of issue that leads me to be a developer - I 
>> see the ORM and the all-bells included nature of Django as the killer 
>> combo.  I don't have to go outside the farm to have database migrations, 
>> url routing, etc.  It is almost all there.  So, please ping back in 3 
>> months to see whether I'm up to it.
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/352490e0-ea4c-4628-9cd9-df8d7c583cb6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Test database, --keepdb parameter, and migrations

2019-02-25 Thread Dan Davis

There are a cluster of issues that I find difficult to resolve as a user,
and as a Django developer, I think there is potential to make them easier
through features in Django itself. I'm interested in hearing better
work-arounds than I use now, but I'm more interested in helping clarify how
the product, e.g. Django, can get better at supporting these scenarios
"out-of-the-box". What I am hoping for is a conversation, not a simple bug
report.

"out-of-the-box" Django supports a --keepdb argument. In many of my
environments, I'd like to make this the default in my environment, but I
see no *Good* way to do that unless I wrap the test management command, and
change the default of the keepdb parameter.Many developers use
django-nose or pytest-django to gain additional capabilities, but REUSE_DB=1
for django-nose and --reuse-db for pytest work differently than --keepdb.
I just filed https://github.com/pytest-dev/pytest-django/issues/704 on
pytest-django about that.

It would be nice if "out-of-the-box" Django supported some way to make "
keepdb" the default. and some way to override it if it were the default.
Another way to do that would be to make the "CREATEDB" argument of Oracle
more general and to support it on other databases.

Another work-around for me would be to overwrite the "migrate" management
command to add a "--testdb" option so that it could run migrations on the
test database explicitly, or to write my own "testmigrate" management
command.

Suppose my database settings include this:

'external': {
'ENGINE': 'django.db.backends.oracle',
'NAME': 'EXTERNAL',
'USER': external_username,
'PASSWORD': external_password,
'TEST': {
'CREATE_DB': False,
'CREATE_USER': False,
'NAME': 'EXTERNAL',
'USER': external_username,
'PASSWORD': external_password,
}
}

This is almost like the below:

'external': {
'ENGINE': 'django.db.backends.oracle',
'NAME': 'EXTERNAL',
'USER': external_username,
'PASSWORD': external_password,
'TEST': {
'MIRROR': 'external',
}
}

The user of these sort of capability needs to understand database routers,
or meta options that are similar. But that may not be enough - entering
"./manage.py migrate --database external" would try to migrate the
database, but running tests will always attempt to migrate the database.
Perhaps we need better documentation for "MIRROR", because it is clearly
intended for situations where there are read/write database aliases and
read replica databases, not for read-only external databases that are used
during tests.

I've fixed this by monkey patching the database creation in a test runner,
very similarly to how pytest-django and django-nose do, but that has a code
smell that suggests that Django "out of the box" should have some
capabilities, maybe in the database router, that prevent migrations for
even the DJANGO_MIGRATIONS table.

Another thing that could be done is to provide a cleardb management
command that supports a "--testdb" option, and removes all the
definitions, including DDL, for users unable to drop and recreate their
database. This also supports user who have been running with --syncdb, and
now they wish to make migrations, or they've made a bunch of migrations and
they wish to start fresh and clean things up. How to clean up on a
per-backend basis is a bit complicated, and a place where Django doesn't
hide the RDBMS sufficiently and users end-up working through tools such as
Toad or SQL Developer to clean things up.

Thoughts? Has this come up before?

--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/f39c92c1-512a-4e7b-83d1-ae688adabc81%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django 2.2 and the watchman reloader

2019-02-21 Thread Dan Davis

Claude,

I've tested a Django based application on 2.2b1 without watchman on
Windows, it does tell you about watchman, but it doesn't fail to run.
Apparently, it falls back to the old way of reloading.   Is this not the
behavior on Debian/Ubuntu?

On Thu, Feb 21, 2019 at 12:28 PM Claude Paroz  wrote:

> Hi,
>
> The new watchman-based autoreloader in Django 2.2 is a nice improvement.
> However, the main issue in my opinion is that there are no Debian/Ubuntu
> official packaging of watchman to my knowledge.
> I've been able to compile it without difficulty, but I'm especially
> concerned about newcomers and non-experienced programmers which will be
> faced to a startup warning message and a rather heavy system load when they
> start developing with Django. Telling them they have to compile from
> sources to get rid of that is not very friendly at least.
>
> Tom (Forbes), as the author of this refactoring, could you tell us if it
> would be much work to create a pyinotify reloader to fallback when watchman
> is not installed?
>
> Claude
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/23ecaa54-a148-4439-a16b-16937da9e934%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFzonYbE4fSy7wsO_wDgHy1X%2BRaWppXejQXB-tnbPtC5RjwwoA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Add an defer=True option for model fields

2019-02-19 Thread Dan Davis

What I mean by the below:
> I was not sure whether to tell him to implement a ModelManager with a
get_queryset() method that defers the field,

Of course this works, but I'm not going to maintain this code, and that
sort of sophistication creates a need for more sophisticated maintenance.


On Tue, Feb 19, 2019 at 12:43 PM Dan Davis  wrote:

> I have a developer who stores the binary copy of a file in his table.  In
> ColdFusion, this was acceptable, because he was writing every query by
> hand, and could simply exclude that field.  However, with the Django ORM it
> is a bit of a problem.   The primary table he uses is just for the file,
> and has a file_name, file_type, file_size, and BinaryField.
>
> The problem is that he has a database-level view that incorporates this
> field, and it may be that he needs to keep this because other schemas in
> our big-office Oracle use the view as an exported synonym.
>
> What I advised him to do was to take the BinaryField out of the
> database-level view, to protect the ORM from reading these large files into
> memory, as in:
>
>  [obj for obj in LicensesDBView.objects.all()]
>
> Or, if he cannot do that, to simply defer the field:
>
>  [obj for obj in
> LicensesDBView.objects.defer('scanned_license').all()]
>
> I was not sure whether to tell him to implement a ModelManager with a
> get_queryset() method that defers the field, but it made me wonder whether
> we should have a concept of an "initially deferred" field.
> That is, this is a field that starts deferred, and can be pulled into the
> select using a values iterator or a call to only() or defer(), e.g. the one
> that cancels prior defers.   The concept of "initially deferred" fields
> would certainly require a new queryset method, such as "nodefer" which is
> sort of like only but doesn't cause only those fields to load, or rather
> defer could accept a syntax like defer('-scanned_license') to cancel that
> previous deferred loading field.
>
> I'm afraid I probably don't understand all the implications of this
> feature, so I thought I'd bring it up on the list before filing any sort of
> issue. Its likely this has been discussed before; I cannot do a historical
> search all the time, especially when ancient history may not be today's
> read on this issue.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/ee5c04e5-69d6-42f9-95ff-c01d553b24c1%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/ee5c04e5-69d6-42f9-95ff-c01d553b24c1%40googlegroups.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFzonYaFjKv%3DW7V4RpSrpLhpA5iWUjbYgSdH4-Pb6517BdKWTg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Adding a database-agnostic JSONField into Django

2019-02-19 Thread Dan Davis

I'm for this.   My only advice is that only some versions of Oracle have a 
native JSON type. The oracle backend should probably use some query to 
determine whether the Oracle instance supports JSON field, or there could 
be a flag in OPTIONS about tihs.

On Tuesday, February 19, 2019 at 7:44:40 AM UTC-5, nikesh...@nituk.ac.in 
wrote:
>
> Hello ,
> I would like to work on this. I recently started working on django. I want 
> to participate in GSoC so this might work for me.
> Would you mind helping me. 
>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c308887d-bd65-4883-b7f1-65d6a36491d5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Add an defer=True option for model fields

2019-02-19 Thread Dan Davis

I have a developer who stores the binary copy of a file in his table. In
ColdFusion, this was acceptable, because he was writing every query by
hand, and could simply exclude that field. However, with the Django ORM it
is a bit of a problem. The primary table he uses is just for the file,
and has a file_name, file_type, file_size, and BinaryField.

The problem is that he has a database-level view that incorporates this
field, and it may be that he needs to keep this because other schemas in
our big-office Oracle use the view as an exported synonym.

What I advised him to do was to take the BinaryField out of the
database-level view, to protect the ORM from reading these large files into
memory, as in:

[obj for obj in LicensesDBView.objects.all()]

Or, if he cannot do that, to simply defer the field:

[obj for obj in
LicensesDBView.objects.defer('scanned_license').all()]

I was not sure whether to tell him to implement a ModelManager with a
get_queryset() method that defers the field, but it made me wonder whether
we should have a concept of an "initially deferred" field.
That is, this is a field that starts deferred, and can be pulled into the
select using a values iterator or a call to only() or defer(), e.g. the one
that cancels prior defers. The concept of "initially deferred" fields
would certainly require a new queryset method, such as "nodefer" which is
sort of like only but doesn't cause only those fields to load, or rather
defer could accept a syntax like defer('-scanned_license') to cancel that
previous deferred loading field.

I'm afraid I probably don't understand all the implications of this
feature, so I thought I'd bring it up on the list before filing any sort of
issue. Its likely this has been discussed before; I cannot do a historical
search all the time, especially when ancient history may not be today's
read on this issue.

--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/ee5c04e5-69d6-42f9-95ff-c01d553b24c1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Will Django ever CompositePrimaryKeys?

2019-02-19 Thread Dan Davis

James,

As a Django user I've had this problem often. My best practice ways to
handle this is as follows:

- If the table is read-only, then create a database-level view that
manufactures a primary key by concatenating the primary key columns
together. Lie to Django and say this combined column is the primary key,
and keep the Database-level view's model as managed = False at the Django
level. However, keep the SQL for the view in your git repository, and
when it changes, build a manual migration - manage.py makemigration fooapp
--empty -n create_replace_bar_view.

- If the table is read-write, then either (a) create a view with instead
of update/instead of insert triggers that manage an underlying table, or
(b) just add a unique ID and make the existing primary key a unique
together constraint and index. With the trigger to add a new ID, other
users of that table shouldn't notice any issues.

- Maybe my statement that Django should manage migrations to the schema
seems unworkable. However, one of the biggest gains you can get from Django
with old schemas like this is to get their DDL into git. Even if git
doesn't do it, make sure some code does it, and it isn't left unmanaged.
That's been a key challenge and opportunity for me.

As a Django developer, I'm a bit green to volunteer to handle this issue,
but this is exactly the kind of issue that leads me to be a developer - I
see the ORM and the all-bells included nature of Django as the killer
combo. I don't have to go outside the farm to have database migrations,
url routing, etc. It is almost all there. So, please ping back in 3
months to see whether I'm up to it.

--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/e2b080d8-3b0d-49bf-b0b7-d5f1883e98fa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: CharField with Oracle Backend Null Behaviour

2019-02-13 Thread Dan Davis

This very issue of a not-null constraint causes a problem with 
django-silk.  I'm not sure this matters in practice, because the usage of 
django silk is usually for development; I just added an sqlite3 database 
alias and a router for django-silk.   However, it validates the problem of 
the current approach.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3fd3a890-6669-46d0-b79a-bf0dc86f2538%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Extend FAQ with "How do I get Django and my JS framework to work together?"

2019-02-06 Thread Dan Davis

The problem with ModelForm is translating validation to something
front-end, like jquery Validator.   Here, DRF does better, because swagger
spec includes something like jsonschema (neither are a standard as I used
to know them, in the WSDL/SOAP days), and so you can use jsonschema
validation to validate on the front-end.   I think what is needed is some
way to map validate-only to the back-end, and do a sort of 2-phase commit -
validate before the user presses OK, and then save (PATCH) when the user
presses save/apply/OK.


On Wed, Feb 6, 2019 at 10:03 AM Dmitriy Sintsov 
wrote:

> Speaking of Rails AJAX forms. It's quite trivial to submit Django
> ModelForm in Javascript AJAX request. It's a bit harder to have the
> customizable success action and to display the field validation errors via
> AJAX response.
>
> I implemented AJAX viewmodels for that:
>
>
> https://django-jinja-knockout.readthedocs.io/en/latest/viewmodels.html#ajax-actions
>
> However it's the mixed traditional forms + AJAX approach, not the pure
> SPA. In pure SPA Django becomes the "data supplier" where most of the
> templates, forms, presentation is performed by Javascript framework.
>
> So, the question is, whether the features of Django such as inline
> formsets and custom widgets are required, or should Django become the data
> store only.
>
> On Tuesday, February 5, 2019 at 3:52:42 AM UTC+3, Maciek Olko wrote:
>>
>> I didn't find this topic being discussed before.
>>
>> It seems to me to be good idea to place "How do I get Django and my JS
>> framework to work together?" or similar question and answer to it in FAQ in
>> Django's docs.
>>
>> Having very big popularity of JS frameworks now it is indeed very common
>> question being asked or searched on the Internet. Such question for ReactJS
>> has 65k views on StackOverflow [1].
>>
>> The answer could briefly explain that one can make use of Django backend
>> and produce API to be consumed by JS clients. Probably it would be good to
>> link to Django REST API project as a suggestion for big projects.
>>
>> What do you think about such addition to FAQ?
>>
>> Regards,
>> Maciej
>>
>> [1]
>> https://stackoverflow.com/questions/41867055/how-to-get-django-and-reactjs-to-work-together
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/0cafc96d-bc4f-4316-9022-28e5134a1bcd%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFzonYZvqTVigfjohY6GUqJicuJeOH4pzwd32KAsR4XmnHt9hA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Extend FAQ with "How do I get Django and my JS framework to work together?"

2019-02-04 Thread Dan Davis

I kind of disagree that saying it works with DRF is enough.  One issue that
needs to be addressed is matching any possible path with re_path, so that
an SPA that takes over the browser history API will work with bookmarks.

Django is opinionated.  The winning strategy for npm frameworks is to let
the CSS/JS/assets come from the framework.  While many projects have used
django-pipeline or django-compressor to good effect, if you use
webpack+react or angular cli, then the output can be a couple of pre-cache
busted bundles, and the solution is:
   (a) tell the javascript framework to put its output files in a
particular directory,
   (b) tell Django to look there with STATICFILES_DIRS,
   (c) Write something to generate tags for the static assets, either a
templatetag or a staticfiles finder/storage.

I think it would be good to include with Django some template tags that can
find a cache-busting bundle by the bundle name, maybe using a manifest
generated during create static.   My code isn't great, but it works like
this:

from django.templatetags.static import static
from ng_loader.utils import BundleMap


logger = logging.getLogger(__name__)

if not settings.DEBUG:
_map = BundleMap(settings.STATIC_ROOT)

register = template.Library()


@register.simple_tag
def ng_load_cssbundle(name):
if settings.DEBUG:
fname = '{}.bundle.css'.format(name)
else:
fname = _map.get_bundle('css', name)
logger.debug('Serving css bundle {} from {}'.format(name, fname))
return mark_safe(''.format(static(fname)))


@register.simple_tag
def ng_load_jsbundle(name):
if settings.DEBUG:
fname = '{}.bundle.js'.format(name)
else:
fname = _map.get_bundle('js', name)
logger.debug('Serving js bundle {} from {}'.format(name, fname))
return mark_safe(''.format(static(fname)))

With the bundle map being created like this:

import os
import re


class BundleMap(object):

def __init__(self, rootpath):
self.rootpath = rootpath
self.rebuild()

def rebuild(self):
self.map = { 'css': {}, 'js': {} }
for name in os.listdir(self.rootpath):
m = re.match(r'([a-z]+)\.[a-z0-9]+\.bundle.(js|css)', name)
if m is not None:
bname = m.group(1)
btype = m.group(2)
if btype in self.map:
self.map[btype][bname] = name

def get_bundle(self, btype, name):
return self.map[btype][name] if name in self.map[btype] else None



On Mon, Feb 4, 2019 at 9:31 PM Jason Johns  wrote:

> Tom Christie wrote this about what DRF brings to the table over plain
> Django:
>
> Django REST framework isn't required, but it helps you get a lot of things
> right that will be time consuming and error prone if you're working from
> core Django.
>  • Serializers
> The Django serializers are not really suitable for anything other than
> dumping and loading fixture data - they don't allow you to customize the
> representation in any substantial way.
> Using Django Forms for validation isn't suitable either as they're
> intended for HTML only validation and can't eg handle nested validation.
> REST frameworks serializers are designed for API usage and cover both JSON
> or form validation, as well as being able to represent as either HTML Forms
> or formats such as JSON. They also give you lots of scope for handling the
> representation of relationships, such as using hyperlinked relations.
>  • Authentication and permissions
> REST framework's authentication will gracefully handle both session based
> and token based schemes at the same time, and get the CSRF behavior right.
> You'll find that really awkward to do if using plain Django. It also helps
> ensure you're issuing failure responses that are suitable for API clients
> (eg get 401 vs 403 responses right)
> The auth, permissions and throttling are also more flexible because
> they're defined at a view level, rather than as middleware or a view
> decorator. This makes it easier to eg combine multiple schemes, or to apply
> different schemes to different parts of your application.
>  • Views
> Django's generic class based views are suitable to HTML applications. REST
> framework's generic class based views are suitable for API services.
> Typicallly API views have slightly different behavior by convention. Eg
> create in an HTML application might typically redirect the user to the
> created item, whereas an API will respond with a 201 CREATED response.
> There's stacks of other functionality and behavior that makes using Django
> REST framework simpler, quicker and likely more correct than if you start
> with plain Django. But those are some of the obvious differences to get
> started with.
>
> https://reddit.com/r/django/comments/3h9oj8/_/cu5pzu9/?context=1
>
> Seems pretty concise and self explanatory to me. This could easily be
> adapted to be in the docs.
>
> --
> You received this message because you are subscribed to the

Re: Proposal: Track used headers and use that information to automatically populate Vary header.

2019-01-25 Thread Dan Davis

On Fri, Jan 25, 2019 at 8:27 PM James Bennett  wrote:

> My immediate thought here is: if people already aren't taking the time to
> patch using the existing mechanism, they also aren't going to take the time
> to opt out of patching. So what you're proposing is effectively still "any
> accessed header patches Vary". And that seems like it's as bad as the
> problem it's trying to solve.
>

The people who do take the time to patch the Vary header probably can
easily adapt to the new mechanism, its the people who don't take the time
to do this that we can help with such a feature.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFzonYafego8WvaQNss46%3DeeSsdvxFKa%2B-F-4-Dzj7guz5Kqmw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Proposal: Track used headers and use that information to automatically populate Vary header.

2019-01-25 Thread Dan Davis

I would like this - Django is a framework with batteries, and my
development group tells me "Django is too hard".  This is because they
don't understand HTTP; mostly they understand HTML/CSS and SQL, with maybe
some easy jquery level of SQL. So, this kind of solution would fit well for
my developers. The young engineers all love that we switched to Django,
however, so any such solution should be opt-out.

On Fri, Jan 25, 2019 at 10:56 AM Adam Johnson  wrote:

> Accessing the value of a header doesn't necessarily mean that the response
> varies based up on, for example it might simply be accessed for storage in
> informational logs. Additionally, Request.headers is not the only way to
> access the values now, Request.META has not been removed. I don't believe
> any of Django's internal header lookups have been changed to use
> Request.headers and it's unlike third party packages or applications will
> ever all be moved.
>
> Anyway I'm pretty sure you can write such a middleware yourself, replacing
> Request.headers with a proxy object (and maybe Request.META too), then
> adding 'Vary' on the way out based upon accessed keys, at least as a proof
> of concept. If it gets some usage that would prove it could be valuable to
> add to Django itself.
>
>
>
> On Fri, 25 Jan 2019 at 14:46, Linus Lewandowski <
> linus.lewandow...@netguru.pl> wrote:
>
>> Right now, it's hard to report Vary header correctly. Headers might get
>> accessed in many different places, like middlewares, subroutines (which
>> can't use patch_vary_headers as they don't have access to the response
>> object), etc - and all those cases should be reflected in the Vary header,
>> or something might get cached incorrectly.
>>
>> However, thanks to the newly added request.headers property (see
>> https://code.djangoproject.com/ticket/20147 and
>> https://github.com/django/django/commit/4fc35a9c3efdc9154efce28cb23cb84f8834517e),
>> we now have a single place which is used to access request headers. We can
>> track which ones were accessed, and then set Vary header automatically, for
>> example in a middleware.
>>
>> What do you think about:
>> 1) adding some code to track accessed headers in request.headers,
>> 2) adding a new middleware (or expanding an existing one), that sets the
>> Vary header based on 1),
>> 3) deprecating patch_vary_headers function
>> and vary_on_headers/vary_on_cookie decorators and recommending to use
>> request.headers instead?
>>
>> Thanks,
>> Linus
>>
>> PS. This is a follow-up to the
>> https://code.djangoproject.com/ticket/28533 ticket.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/81484ff1-552e-4103-9fa8-8a3348512b84%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> --
> Adam
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAMyDDM3uDHaDywYFa6fM_-jJw8j81yoTP9%3DyLO2Rx2JDOk45jw%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFzonYZV_oh4aotsZ9Suz6DQOx4-TMAZGrfdZshsnr%2BiUM6t9A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: revisiting the Python version support policy

2019-01-25 Thread Dan Davis

My employer is still using CPython 3.4.6 on the servers, and CPython 3.5.1
on the desktop.   I've been instrumental in developing a plan to move
forward.  I know of one established company and one start-up, by name,
where they are still using CPython 2.7 (and a horrendously old version of
Django), because it is hard to upgrade.   As long as Django has a LTS
version, such as 1.11, and soon 2.2, I think forcing an upgrade is OK.
 But it is good to be aware of what's out there.

On Fri, Jan 25, 2019 at 9:43 AM Tom Forbes  wrote:

> This message really resonated with me, especially after helping a few
> beginners get started with Python and watching them struggle with exactly
> this kind of thing.
>
> I'd be +1 on following Python. Looking through the diff there is not a
> huge amount of things to remove and IMO none of them are really holding us
> back or all that serious. We've fixed some issues with mangling cached
> property names, some workarounds for ModuleNotFoundError/ImportError and an
> issue with sqlite3 on 3.5.
>
> On 24 January 2019 at 20:33:42, Carlton Gibson (carlton.gib...@gmail.com)
> wrote:
>
> To be honest, I'm surprised there's even one person who comes within a
> 1000 miles of this list who's using Python 3.5. :)
>
> My reason for thinking we should follow Python's supported versions is
> users, and particularly beginning users, who have got they-don't-know
> version and find a tutorial just what... no sorry need... `pip3 install
> Django` to work, and give them the version of Django that corresponds to
> what they see when they visit docs.djangoproject.com.
>
> I don't agree this is theoretical at all.
>
> It's not just Debian. (Which doesn't fit my mental model here really...)
>
> It's all those few-years-old computers out there.
>
> It's for example Raspbian, which as of this month is still shipping Python
> 3.5.
>
> So my boy, who's 10, says,
>
> - What would you use?
> - Well I'd use Django (obviously)
> - Can I use that?
> - Yeah...
>
> If we do drop Python 3.5 I have to say, "Well, no. But you can use this
> old one." That's not as cool.
> But there will be people who are more seriously affected.
>
> > Who is saying, "I want to use the latest version of Django, but I want
> to use a really old version of Python."
>
> No one is saying this. The notion of versions doesn't come into it. We're
> well beyond the barrier-to-entry before we get there.
> I (just my opinion on this) think we mistake our audience if we forget
> this.
> (For this reason I don't think the deployment issue is the relevant one.
> It's about people learning to programme, not professionals.)
>
> We can't support everything forever, and I'm as keen as anyone to push
> forward, but following Python is (for me) the thing we should do.
> I think Django's position in the Python eco-system requires it.
>
> Of course if we don't, things are easier for us, yes.
>
> Again, just my opinion.
> C.
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/19e099cf-6087-4efd-9138-d338f12bbf2c%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAFNZOJPdaWYJ9LCPZsG9dE66ka%2B8M_8Y4_%2BzMwFNz4er1pXi8w%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web

Re: Begninner

2019-01-24 Thread Dan Davis

George,

If you are an experienced programmer in some other language, I recommend
Fluent Python.   It is not a good book for someone new to programming.

On Thu, Jan 24, 2019 at 7:19 AM george ngugi  wrote:

> hey,
>
> Am new in python programming, can kindly can someone help me on which is
> the best site to learn the python
>
> BR George
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/ee66af07-4ffb-4e0b-b8fc-648e1aca35ee%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFzonYbocP8zmuQroc992tOaL4Q6740x92ZhvwBBfwZNf%2B_%2BTg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: A faster paginator for django

2018-12-23 Thread Dan Davis

To be honest, I just entered it as a word, and the client made it a URL
because it ends with a top-level domain, and looks like a domain name.

On Sun, Dec 23, 2018 at 7:15 PM Dan Davis  wrote:

> Yes, https://datatables.net/, often miscalled jquery datatables, it is
> more like php datatables in its CGI parameters ;)
>
> On Sun, Dec 23, 2018 at 2:51 PM Adam Johnson  wrote:
>
>> (I think you meant https://datatables.net/ ? ) :)
>>
>> On Sun, 23 Dec 2018 at 19:25, Dan Davis  wrote:
>>
>>> Also, it can be worse than one count query. When interacting with
>>> datables.net serverSide, you will need multiple count queries.
>>>
>>> On Sat, Dec 15, 2018, 10:32 AM Kye Russell >>
>>>> It might also be worth looking at the alternative pagination methods
>>>> offered by Django REST Framework as a source of inspiration.
>>>>
>>>> On Wednesday, December 5, 2018 at 8:15:22 PM UTC+8, Saleem Jaffer wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> The default paginator that comes with Django is inefficient when
>>>>> dealing with large tables. This is because the final query for fetching
>>>>> pages uses "OFFSET" which is basically a linear scan till the last index 
>>>>> of
>>>>> the current page. Does it make sense to have a better paginator which does
>>>>> not use "OFFSET".
>>>>>
>>>>> If this sounds like a good idea, I have some ideas on how to do it and
>>>>> with some help from you guys I can implement it.
>>>>>
>>>>> Saleem
>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Django developers (Contributions to Django itself)" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to django-developers+unsubscr...@googlegroups.com.
>>>> To post to this group, send email to django-developers@googlegroups.com
>>>> .
>>>> Visit this group at https://groups.google.com/group/django-developers.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/django-developers/0f431ffc-5ebf-4703-9e01-91240007c154%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/django-developers/0f431ffc-5ebf-4703-9e01-91240007c154%40googlegroups.com?utm_medium=email_source=footer>
>>>> .
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-developers+unsubscr...@googlegroups.com.
>>> To post to this group, send email to django-developers@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-developers/CAFzonYYvXG7u6AFUKKHULun7H5%2Bv8dzKNqqOAmyrUxwiGDr-4g%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/django-developers/CAFzonYYvXG7u6AFUKKHULun7H5%2Bv8dzKNqqOAmyrUxwiGDr-4g%40mail.gmail.com?utm_medium=email_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>> --
>> Adam
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/CAMyDDM0s1N9%3DiZAdXrHE5UNDzp24wkOcxoW4HwKS5H6Gn2O-cg%40mail.gmail.com
>> <https://groups.google.com/d/msgid/django-developers/CAMyDDM0s1N9%3DiZAdXrHE5UNDzp24wkOcxoW4HwKS5H6Gn2O-cg%40mail.gmail.com?utm_medium=email_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFzonYYapkGGGv9%3DiPBQam7ScrYJCnT1iWPRSgj7ijN-H5-K5g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: A faster paginator for django

2018-12-23 Thread Dan Davis

Yes, https://datatables.net/, often miscalled jquery datatables, it is more
like php datatables in its CGI parameters ;)

On Sun, Dec 23, 2018 at 2:51 PM Adam Johnson  wrote:

> (I think you meant https://datatables.net/ ? ) :)
>
> On Sun, 23 Dec 2018 at 19:25, Dan Davis  wrote:
>
>> Also, it can be worse than one count query. When interacting with
>> datables.net serverSide, you will need multiple count queries.
>>
>> On Sat, Dec 15, 2018, 10:32 AM Kye Russell >
>>> It might also be worth looking at the alternative pagination methods
>>> offered by Django REST Framework as a source of inspiration.
>>>
>>> On Wednesday, December 5, 2018 at 8:15:22 PM UTC+8, Saleem Jaffer wrote:
>>>>
>>>> Hi all,
>>>>
>>>> The default paginator that comes with Django is inefficient when
>>>> dealing with large tables. This is because the final query for fetching
>>>> pages uses "OFFSET" which is basically a linear scan till the last index of
>>>> the current page. Does it make sense to have a better paginator which does
>>>> not use "OFFSET".
>>>>
>>>> If this sounds like a good idea, I have some ideas on how to do it and
>>>> with some help from you guys I can implement it.
>>>>
>>>> Saleem
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-developers+unsubscr...@googlegroups.com.
>>> To post to this group, send email to django-developers@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-developers/0f431ffc-5ebf-4703-9e01-91240007c154%40googlegroups.com
>>> <https://groups.google.com/d/msgid/django-developers/0f431ffc-5ebf-4703-9e01-91240007c154%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/CAFzonYYvXG7u6AFUKKHULun7H5%2Bv8dzKNqqOAmyrUxwiGDr-4g%40mail.gmail.com
>> <https://groups.google.com/d/msgid/django-developers/CAFzonYYvXG7u6AFUKKHULun7H5%2Bv8dzKNqqOAmyrUxwiGDr-4g%40mail.gmail.com?utm_medium=email_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> --
> Adam
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAMyDDM0s1N9%3DiZAdXrHE5UNDzp24wkOcxoW4HwKS5H6Gn2O-cg%40mail.gmail.com
> <https://groups.google.com/d/msgid/django-developers/CAMyDDM0s1N9%3DiZAdXrHE5UNDzp24wkOcxoW4HwKS5H6Gn2O-cg%40mail.gmail.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFzonYZS6dYUqe%2Bgie00SfcybFKJXaGP35ESdM1gMi%3Dged%3DR-Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: A faster paginator for django

2018-12-23 Thread Dan Davis

Also, it can be worse than one count query. When interacting with
datables.net serverSide, you will need multiple count queries.

On Sat, Dec 15, 2018, 10:32 AM Kye Russell  It might also be worth looking at the alternative pagination methods
> offered by Django REST Framework as a source of inspiration.
>
> On Wednesday, December 5, 2018 at 8:15:22 PM UTC+8, Saleem Jaffer wrote:
>>
>> Hi all,
>>
>> The default paginator that comes with Django is inefficient when dealing
>> with large tables. This is because the final query for fetching pages uses
>> "OFFSET" which is basically a linear scan till the last index of the
>> current page. Does it make sense to have a better paginator which does not
>> use "OFFSET".
>>
>> If this sounds like a good idea, I have some ideas on how to do it and
>> with some help from you guys I can implement it.
>>
>> Saleem
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/0f431ffc-5ebf-4703-9e01-91240007c154%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFzonYYvXG7u6AFUKKHULun7H5%2Bv8dzKNqqOAmyrUxwiGDr-4g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Widening participation (Thoughts from DjangoCon)

2018-12-10 Thread Dan Davis

And there might be no need to develop code for this, only configuration:
https://github.com/dnephin/TracAdvancedSearchPlugin


On Mon, Dec 10, 2018 at 11:22 AM Dan Davis  wrote:

> I think that TracSearch could use improvement.   As an expert in
> Information Retrieval, I would be happy to help with this.   The particular
> suggestions I have:
>
>
>- Provide quick filters (e.g. facets) after a search is done based on
>ticket keywords, stage, and whether a result is a ticket, wiki, milestone,
>etc.
>- Use a Solr/Elasticsearch inverted index for better word
>segmentation, automated phrase and field boosting, etc.
>
> A search system such as TracSearch is biased towards "Recall", e.g. find
> all relevant tickets, rather than "Precision", e.g. show me only relevant
> results, and only the most relevant results.   The limits on word
> segmentation provided by RDBMS-backed full-text search often makes both
> recall and precision worse.
>
>
> On Mon, Dec 10, 2018 at 11:14 AM Dan Davis  wrote:
>
>> > I strongly dislike Trac in nearly every way. It's hard to search and
>> the filters are next to useless, and the categorisation features we use are
>> not very useful.
>> > I believe the better way to search Trac is to use google and site:
>> code.djangoproject.com which is a red flag itself.
>>
>> Is there a separate list for the infrastructure team where we could
>> continue this discussion?   I feel sure that using a better indexing plugin
>> for Trac could improve the problem.
>>
>>
>> On Sun, Oct 28, 2018 at 6:04 PM Josh Smeaton 
>> wrote:
>>
>>> I strongly dislike Trac in nearly every way. It's hard to search and the
>>> filters are next to useless, and the categorisation features we use are not
>>> very useful. I believe the better way to search Trac is to use google and
>>> site:code.djangoproject.com which is a red flag itself.
>>>
>>> On Saturday, 27 October 2018 11:09:38 UTC+11, Tom Forbes wrote:
>>>>
>>>> How much of this would you attribute to the current ticketing system
>>>> itself, rather than tickets being tagged appropriately?
>>>>
>>>> I know when I started contributing I found trac to be pretty
>>>> intimidating in terms of complexity, especially the search. I still prefer
>>>> to use the 'Search Trac' field in the root code.djangoproject.com page
>>>> than fiddle with the myriad of options and drop downs in the browse tickets
>>>> section.
>>>>
>>>> If we think of getting new people onboard as a conversion funnel we
>>>> need to stop dropoff as much as possible, and that extends to the UI of the
>>>> ticket tracker as well I believe.
>>>>
>>>> Tom
>>>>
>>>> On Fri, 26 Oct 2018, 22:43 Ian Foote,  wrote:
>>>>
>>>>> Hi Carlton,
>>>>>
>>>>> I've had similar thoughts sitting in the back of my mind for at least
>>>>> a couple of months, so thank you for sharing this. I agree that finding
>>>>> tickets is one of the big problems here, both for new contributors and for
>>>>> sprint leaders. At Pycon UK I took on the role of sprint leader along with
>>>>> Adam Johnson and directing people to appropriate tickets was a definite
>>>>> difficulty. I was also unaware of the django core mentorship list and will
>>>>> be joining that soon. I'm willing to spend some time mentoring a small
>>>>> number of people, life permitting.
>>>>>
>>>>> Ian
>>>>>
>>>>> On Fri, 26 Oct 2018 at 14:44, Carlton Gibson 
>>>>> wrote:
>>>>>
>>>>>> Hi All.
>>>>>>
>>>>>> OK, so last week I was at DjangoCon US in San Diego. (Thank you if
>>>>>> you organised that! Hi! if we met and chatted.)
>>>>>> I gave a talk ("Your web framework needs you!") inspired by the
>>>>>> discussion on the
>>>>>> <https://groups.google.com/d/topic/dsf-members/GWOzvsOAGUs/discussion>
>>>>>> DSF list and the proposal to dissolve Django Core
>>>>>> <https://github.com/django/deps/pull/47>. (Can’t see the DSF list? Join
>>>>>> the DSF
>>>>>> <https://docs.google.com/forms/d/e/1FAIpQLSd5lbWxAO-sylEEjHVKBNIpmHlhdJRf0_LCo8glnLUWd-Q2Sw/viewform>
>>>>>> .)
>>>>>> I was asking for more participation in general, and partic

Re: Widening participation (Thoughts from DjangoCon)

2018-12-10 Thread Dan Davis

I think that TracSearch could use improvement.   As an expert in
Information Retrieval, I would be happy to help with this.   The particular
suggestions I have:


   - Provide quick filters (e.g. facets) after a search is done based on
   ticket keywords, stage, and whether a result is a ticket, wiki, milestone,
   etc.
   - Use a Solr/Elasticsearch inverted index for better word segmentation,
   automated phrase and field boosting, etc.

A search system such as TracSearch is biased towards "Recall", e.g. find
all relevant tickets, rather than "Precision", e.g. show me only relevant
results, and only the most relevant results.   The limits on word
segmentation provided by RDBMS-backed full-text search often makes both
recall and precision worse.


On Mon, Dec 10, 2018 at 11:14 AM Dan Davis  wrote:

> > I strongly dislike Trac in nearly every way. It's hard to search and the
> filters are next to useless, and the categorisation features we use are not
> very useful.
> > I believe the better way to search Trac is to use google and site:
> code.djangoproject.com which is a red flag itself.
>
> Is there a separate list for the infrastructure team where we could
> continue this discussion?   I feel sure that using a better indexing plugin
> for Trac could improve the problem.
>
>
> On Sun, Oct 28, 2018 at 6:04 PM Josh Smeaton 
> wrote:
>
>> I strongly dislike Trac in nearly every way. It's hard to search and the
>> filters are next to useless, and the categorisation features we use are not
>> very useful. I believe the better way to search Trac is to use google and
>> site:code.djangoproject.com which is a red flag itself.
>>
>> On Saturday, 27 October 2018 11:09:38 UTC+11, Tom Forbes wrote:
>>>
>>> How much of this would you attribute to the current ticketing system
>>> itself, rather than tickets being tagged appropriately?
>>>
>>> I know when I started contributing I found trac to be pretty
>>> intimidating in terms of complexity, especially the search. I still prefer
>>> to use the 'Search Trac' field in the root code.djangoproject.com page
>>> than fiddle with the myriad of options and drop downs in the browse tickets
>>> section.
>>>
>>> If we think of getting new people onboard as a conversion funnel we need
>>> to stop dropoff as much as possible, and that extends to the UI of the
>>> ticket tracker as well I believe.
>>>
>>> Tom
>>>
>>> On Fri, 26 Oct 2018, 22:43 Ian Foote,  wrote:
>>>
>>>> Hi Carlton,
>>>>
>>>> I've had similar thoughts sitting in the back of my mind for at least a
>>>> couple of months, so thank you for sharing this. I agree that finding
>>>> tickets is one of the big problems here, both for new contributors and for
>>>> sprint leaders. At Pycon UK I took on the role of sprint leader along with
>>>> Adam Johnson and directing people to appropriate tickets was a definite
>>>> difficulty. I was also unaware of the django core mentorship list and will
>>>> be joining that soon. I'm willing to spend some time mentoring a small
>>>> number of people, life permitting.
>>>>
>>>> Ian
>>>>
>>>> On Fri, 26 Oct 2018 at 14:44, Carlton Gibson 
>>>> wrote:
>>>>
>>>>> Hi All.
>>>>>
>>>>> OK, so last week I was at DjangoCon US in San Diego. (Thank you if
>>>>> you organised that! Hi! if we met and chatted.)
>>>>> I gave a talk ("Your web framework needs you!") inspired by the
>>>>> discussion on the
>>>>> <https://groups.google.com/d/topic/dsf-members/GWOzvsOAGUs/discussion>
>>>>> DSF list and the proposal to dissolve Django Core
>>>>> <https://github.com/django/deps/pull/47>. (Can’t see the DSF list? Join
>>>>> the DSF
>>>>> <https://docs.google.com/forms/d/e/1FAIpQLSd5lbWxAO-sylEEjHVKBNIpmHlhdJRf0_LCo8glnLUWd-Q2Sw/viewform>
>>>>> .)
>>>>> I was asking for more participation in general, and participation that
>>>>> is more representative of the wider Django community in particular.
>>>>>
>>>>> There was lots of good input from many people, including (but not, at
>>>>> all, limited to) representatives of groups such Pyladies, DjangoGirls, and
>>>>> so on.
>>>>>
>>>>>
>>>>> The recurring themes seem to me to fit into three categories:
>>>>>
>>>>>1. The importance of *mentoring*.

Re: Widening participation (Thoughts from DjangoCon)

2018-12-10 Thread Dan Davis

> I strongly dislike Trac in nearly every way. It's hard to search and the
filters are next to useless, and the categorisation features we use are not
very useful.
> I believe the better way to search Trac is to use google and site:
code.djangoproject.com which is a red flag itself.

Is there a separate list for the infrastructure team where we could
continue this discussion?   I feel sure that using a better indexing plugin
for Trac could improve the problem.

On Sun, Oct 28, 2018 at 6:04 PM Josh Smeaton  wrote:

> I strongly dislike Trac in nearly every way. It's hard to search and the
> filters are next to useless, and the categorisation features we use are not
> very useful. I believe the better way to search Trac is to use google and
> site:code.djangoproject.com which is a red flag itself.
>
> On Saturday, 27 October 2018 11:09:38 UTC+11, Tom Forbes wrote:
>>
>> How much of this would you attribute to the current ticketing system
>> itself, rather than tickets being tagged appropriately?
>>
>> I know when I started contributing I found trac to be pretty intimidating
>> in terms of complexity, especially the search. I still prefer to use the
>> 'Search Trac' field in the root code.djangoproject.com page than fiddle
>> with the myriad of options and drop downs in the browse tickets section.
>>
>> If we think of getting new people onboard as a conversion funnel we need
>> to stop dropoff as much as possible, and that extends to the UI of the
>> ticket tracker as well I believe.
>>
>> Tom
>>
>> On Fri, 26 Oct 2018, 22:43 Ian Foote,  wrote:
>>
>>> Hi Carlton,
>>>
>>> I've had similar thoughts sitting in the back of my mind for at least a
>>> couple of months, so thank you for sharing this. I agree that finding
>>> tickets is one of the big problems here, both for new contributors and for
>>> sprint leaders. At Pycon UK I took on the role of sprint leader along with
>>> Adam Johnson and directing people to appropriate tickets was a definite
>>> difficulty. I was also unaware of the django core mentorship list and will
>>> be joining that soon. I'm willing to spend some time mentoring a small
>>> number of people, life permitting.
>>>
>>> Ian
>>>
>>> On Fri, 26 Oct 2018 at 14:44, Carlton Gibson 
>>> wrote:
>>>
 Hi All.

 OK, so last week I was at DjangoCon US in San Diego. (Thank you if you
 organised that! Hi! if we met and chatted.)
 I gave a talk ("Your web framework needs you!") inspired by the
 discussion on the

 DSF list and the proposal to dissolve Django Core
 . (Can’t see the DSF list? Join
 the DSF

 .)
 I was asking for more participation in general, and participation that
 is more representative of the wider Django community in particular.

 There was lots of good input from many people, including (but not, at
 all, limited to) representatives of groups such Pyladies, DjangoGirls, and
 so on.

 The recurring themes seem to me to fit into three categories:

1. The importance of *mentoring*.
2. The difficulty of *finding tickets*.
3. The importance of *sprints*.

 The rest here is a summary of that. Hopefully it’s useful.

 Mentoring

 For whatever reasons, the exiting *Contributing How-To*

 doesn’t lead to contributions from a demographic that matches the wider
 Django Community.

 The point that came up again and again about this was that *mentoring*
 is one of the best (perhaps the best?) tool in helping to change this.

 Django Core Mentorship

 We don’t have an official mentoring programme but we do have the 
 django-core-mentorship
 list .

 This must be about the best-kept secret in the Django world: it’s gets
 ≈0 traffic, but I told everybody at DjangoCon about it, and that they
 should use it.

 If you are not on django-core-mentorship, and you’re willing to help
 prospective contributors, please sign-up. I’m hoping we can drive some
 traffic to it.

 Maybe there’s call for something more formal, but at least until DCM is
 actually being used, that seems (to me) like something we can postpone.

 Finding Tickets

 The next thing was that there’s not enough guidance on what to work on.

 The guidance is to look for *Easy Pickings*. There are ≈1300 accepted
 open tickets in TRAC. 13 of these are marked *Easy Pickings*.

 That’s not enough. I think we’re too tight with it (or need another
 grade).

 There are *many* tickets which aren’t super hard: I put it that, most
 of our

Re: Allow usage of widgets in generic class-based views?

2018-12-05 Thread Dan Frankowski

Ah, I see.

It looks like I can use modelform_factory. So:

class PatientCreate(LoginRequiredMixin, UserOrgRequiredMixin, CreateView):
model = models.Patient
fields = ['name', 'caregiver_name', 'sex', 'birth_date',
  'residence', 'country']

becomes

from django.forms.models import modelform_factory

_patient_create_form = modelform_factory(
models.Patient,
fields=['name', 'caregiver_name', 'sex', 'birth_date',
'residence', 'country'])

class PatientCreate(LoginRequiredMixin, UserOrgRequiredMixin, CreateView):
form_class = _patient_create_form
template_name = 'healthdbapp/patient_form.html'

and then I can add the widgets:

from django.forms import HiddenInput

from django.forms.models import modelform_factory

_patient_create_form = modelform_factory(
models.Patient,
fields=['name', 'caregiver_name', 'sex', 'birth_date',
'residence', 'country'],
widgets={'country': HiddenInput()})


class PatientCreate(LoginRequiredMixin, UserOrgRequiredMixin, CreateView):
form_class = _patient_create_form
template_name = 'healthdbapp/patient_form.html'


Works for me.

On Tue, Dec 4, 2018 at 5:04 PM charettes  wrote:

> I agree with Tim that this is a slippery slope.
>
> Maybe that adding a ModelFormMixin.get_form_class_options() that returns a
> {'fields': self.fields} dict by default and is passed to
> modelform_factory(**kwargs)
> in get_form_class() would be a good compromise?
>
> Best,
> Simon
>
> On Tuesday, December 4, 2018 17:28:39 UTC-5, Tim Graham wrote:
>>
>> What I meant is that modelform_factory() also has these parameters:
>>
>> localized_fields is a list of names of fields which should be localized.
>>
>> labels is a dictionary of model field names mapped to a label.
>>
>> help_texts is a dictionary of model field names mapped to a help text.
>>
>> error_messages is a dictionary of model field names mapped to a
>> dictionary of error messages.
>> field_classes is a dictionary of model field names mapped to a form
>> field class.
>>
>> If we accept the patch for widgets, then are we headed down the road of
>> adding support for the rest of those arguments? Customizing a form directly
>> rather than indirectly via the view seems like better design to me. It
>> doesn't require adding features to ModelFormMixin as they are added to
>> ModelForm.
>>
>> On Tuesday, December 4, 2018 at 4:57:40 PM UTC-5, Dan F wrote:
>>>
>>> https://code.djangoproject.com/ticket/24589
>>>
>>> This ticket has a patch to pass through the "widgets" argument for
>>> generic class-based views.
>>>
>>> It was closed with "won't fix" with this comment: "*I don't think the
>>> possibility of saving a few lines of user code justifies the complexity of
>>> reimplementing the parameters to modelform_factory() as CBV parameters and
>>> methods.*"
>>>
>>> I don't understand. The patch was quite small (doesn't seem complex),
>>> and it could give everyone the ability to overload widgets.
>>>
>>> Aside from just saving lines of code, it also would act more as
>>> expected. I tried using "widgets" before seeing that it didn't work. Also,
>>> it would support not making a new form class where every field just has to
>>> be copied (useless boilerplate).
>>>
>>> Can you accept the patch?
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/0281164b-9bf7-485f-83ad-
> c0301630c91a%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/0281164b-9bf7-485f-83ad-c0301630c91a%40googlegroups.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CA%2BriwVRmoxLBNdrGDzyDmpw8ap6jY5J93ZmV6yco4YoB7R0N4A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Allow usage of widgets in generic class-based views?

2018-12-04 Thread Dan F

https://code.djangoproject.com/ticket/24589

This ticket has a patch to pass through the "widgets" argument for generic 
class-based views.

It was closed with "won't fix" with this comment: "*I don't think the 
possibility of saving a few lines of user code justifies the complexity of 
reimplementing the parameters to modelform_factory() as CBV parameters and 
methods.*"

I don't understand. The patch was quite small (doesn't seem complex), and 
it could give everyone the ability to overload widgets.

Aside from just saving lines of code, it also would act more as expected. I 
tried using "widgets" before seeing that it didn't work. Also, it would 
support not making a new form class where every field just has to be copied 
(useless boilerplate).

Can you accept the patch?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/781e9f85-c278-46f7-b712-bf49842a9259%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Removing Oracle from Django core in 3.0

2018-11-26 Thread Dan Davis


Another related question - 
https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/unit-tests/#testing-other-python-versions-and-database-backends
 
provides some terse advice for running the unit tests with different 
backends.   Is that essentially what is happening with a test 
like https://djangoci.com/job/django-oracle-1.11/, or is it more specific?

On Monday, November 26, 2018 at 2:41:32 PM UTC-5, Tim Graham wrote:
>
> That's the query I would use. The 'oracle' keyword might not be assigned 
> completely but you can scan through all the "Database layers" tickets 
> fairly easily and add it to any that are missing.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5bfa1fe2-38b2-4086-bc57-f9badec93489%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Moving database backends out of the core

2018-11-26 Thread Dan Davis


I think this would only work if most database specific backends were 
maintained by the djangoproject itself, allowing for integration tests that 
test compatibility.
To my mind, a strong ORM to backend API is a great thing, but we also need 
stronger backend integration tests.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/23759c70-4148-4ac7-8dfa-ca97f1dae97d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Removing Oracle from Django core in 3.0

2018-11-26 Thread Dan Davis

Related question - how would I search for Oracle specific issues.   I found 
this query:

https://code.djangoproject.com/query?status=assigned=new=~oracle=Database+layer+(models%2C+ORM)=id=summary=status=component=owner=type=version=1=id

However, I'm not sure how much I can rely on the keywords.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/affc60fa-b182-4767-aae8-086f5ba8fe4a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Removing Oracle from Django core in 3.0

2018-11-25 Thread Dan Davis

My employer is an Oracle shop.  I would dedicate myself to Oracle specific
bugs to prevent removing Oracle from core.   That said, we'll probably be
off Oracle and onto the cloud and Postgresql by 3.0.

On Sun, Nov 25, 2018 at 1:36 PM Adam Johnson  wrote:

> Interestingly, I didn't receive your first email Johannes, only Tim's
> reply. I can't even find it in spam. Maybe Gmail's filters highly associate
> mentions of "Oracle" with spam? :/
>
> I agree that with Tim that it's going to be easier to keep it in core if
> development is going to continue. Any suggestion that unbundling it would
> improve its support lifecycle should be well backend by Django+Oracle users
> themselves, which I take it you aren't Johannes.
>
> On Sun, 25 Nov 2018 at 17:54, Tim Graham  wrote:
>
>> I can't find a past discussion specific to Oracle, but it's not a new
>> proposal. See
>> https://groups.google.com/d/topic/django-developers/O-g06EM6XMM/discussion
>> for "Moving database backends out of the core."
>>
>> I think removing Oracle from core would only increase the maintenance
>> burden. Since Oracle has edge cases, it's useful to test those along with
>> new Django features. If the Oracle backend is in a separate repo, then
>> adding new features will often require commits to two repositories and I
>> don't know how we would run the tests with pull request X for Django and
>> pull request Y for the Oracle backend. Then we also have to release the
>> Oracle backend separately.
>>
>> djangoci.com isn't reporting any Oracle failures on master. If you've
>> found an issue, please open a ticket with details.
>>
>> We don't run the Oracle tests with pull requests because they take about
>> an hour, while other databases take about 10 minutes. It hasn't been
>> difficult to identify which pull requests require running the tests on
>> Oracle and to trigger that build with the trigger phrase.
>>
>> On Sunday, November 25, 2018 at 5:21:02 AM UTC-5, Johannes Hoppe wrote:
>>>
>>> Hi there,
>>>
>>> I have recently refactored some bits in the database backend and came to
>>> realize that a lot of the complexity in there comes from the poor
>>> implementation of the Oracle backend.
>>> Fun fact, did you know that Oracle tests don't run by default and that
>>> the current master, fails on oracle ;)
>>>
>>> Anyhow, I want to come to a conclusion about the following matter:
>>>
>>> Should we remove the Oracle database backend from Django core in the 3.0
>>> release?
>>>
>>> Here are a couple of reasons, why I believe this to be a good idea:
>>>
>>>- License
>>>- Oracle is  Proprietary software
>>>- Money
>>>   - Oracle is not a sponsor of the Django Foundation, but makes
>>>   40bn in revenue
>>>- Technical:
>>>   - Oracle does not support may features
>>>   - due to its lack of features, a lot of edge case handling to the
>>>   base database backend which drives overall complexity
>>>- Development:
>>>   - Oracle does not run in the regular CI suite, in fact master is
>>>   broken right now
>>>   - entrance barrier for first time contributors is high
>>>  - one needs to accept a non open source license
>>>  - register with oracle
>>>  - go through a very complex setup process
>>>
>>> Of course there are some users who use Oracle and I don't want to keep
>>> them hanging. I simply believe the database backend should be developed
>>> separately from Django.
>>> This could even be helpful for the Oracle community. Since oracle is
>>> enterprise only, they usually looks for longer support cycles than what
>>> Django want's to offer.
>>>
>>> Ok, I made my case, I am curious, what do you guys think?
>>>
>>> Best
>>> -Joe
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/d354b7a9-d116-41e9-9b4c-f8335931957f%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> --
> Adam
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this

Re: skipping elidable migrations

2018-11-13 Thread Dan Watson

Hi Adam,

All of your suggestions are perfectly valid things to do (and are in fact 
things I've done in the past). This particular project uses data migrations 
fairly heavily, and the developers haven't been particularly good about 
making sure they run cleanly on fresh databases. We are using Django's test 
framework, but a) with migrations disabled for performance reasons and b) 
using --keepdb, which masks these types of problems.

It just got me thinking that if you've gone out of your way to mark an 
operation as basically "safe to delete", it seems unimportant to run them 
during an initial migration. Bad programming aside, it seems like an easy 
performance bump for projects that use these types of operations. I think 
of elidable as a "soft delete" option that gets garbage collected next 
squash. This would just be an optimization step to ignore what is already 
marked for death. Maybe it's not an optimization we want to reason about, 
or maybe it isn't a big enough use case to be worth coding for, but I 
figured I'd put it out there.

Regards,
Dan


On Saturday, November 10, 2018 at 8:13:27 AM UTC-5, Adam Johnson wrote:
>
> Do you have an example? If you're using Django's default testing 
> framework, it normally creates a fresh database, so from the moment a new 
> data migration is written it would be tested with the empty database 
> scenario.
>
> Afaiu it is possible to write RunPython operations in a way that no-ops on 
> empty databases, e.g. starting with the equivalent of "if not 
> Model.object.exists(): return." It is also possible with RunSQL on database 
> backends that support conditional logic around SQL statements,  or they can 
> be easily converted to RunPython using a function that uses 
> cursor.execute(). And if a project develops problems with past migrations, 
> it's always possible to (carefully) edit them to include such conditions.
>
> On Fri, 9 Nov 2018 at 16:11, Dan Watson > 
> wrote:
>
>> Hi All,
>>
>> I was wondering if anyone had any thoughts on an option to the "migrate" 
>> command (--skip-elidable?) that would skip running elidable migrations. The 
>> use case here is that data migrations that build up over time may act on 
>> certain assumptions (existing tables/data) that may not be true when 
>> migrating a new database. It seems that since they were explicitly marked 
>> as able to be deleted when squashing, they would be safe to not run when 
>> creating a new database. Maybe we don't go so far as to make this the 
>> default behavior when migrating a fresh database, but an option would be 
>> nice. I realize you could simply squash your migrations, but that's not 
>> without penalty of code churn, testing, etc. especially when your existing 
>> migration graph is otherwise fine (and performant).
>>
>> If there's some consensus about this being worthwhile, or at least no 
>> strong objections to it, I can take a stab at the implementation.
>>
>> Regards,
>> Dan
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/fba7a58a-444b-4c90-b139-151580423366%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/django-developers/fba7a58a-444b-4c90-b139-151580423366%40googlegroups.com?utm_medium=email_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> -- 
> Adam
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9cbccecc-0831-4c62-869d-ba7ca619ded7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Pluggable secret key backend

2018-11-10 Thread Dan Davis

Maybe a LoFi way to accomplish this is just to make sure that the
SECRET_KEY is cast to bytes() before use.   That way, a non-bytes object
placed there during settings will be asked to convert it to bytes before
use.   I use the same trick with an internal module that retrieves database
passwords from a web service.   For cx_Oracle, I only had to implement
__str__, but for PostgreSQL, MySQL, and pyodbc to SQL Server I eventually I
collected many other string methods to be duck typed as a string.

The same trick might work today with SECRET_KEY, depending on how it is
used.   If anyone ever does a check that isinstance(settings.SECRET_KEY,
bytes), then we'd have problems, but if Django has the discipline to
iterate it, get its length, and cast it to bytes before use, then it would
be OK.

On Sat, Nov 10, 2018 at 9:36 AM Andreas Pelme  wrote:

> On 10 Nov 2018, at 13:29, Adam Johnson  wrote:
> >
> > Hi Andreas
> >
> > I like your proposal, moving to a backend is an elegant way of solving
> both the immediate problem and opening up the other possibilities you
> mentioned.
>
> Thanks Adam, I am glad you like the proposal. :)
>
> > I think it would also be nice to have an "out of the box" way of
> rotating the key, without needing to implement a custom backend. Perhaps a
> second setting OLD_SECRET_KEYS that may contain a list of old keys that are
> returned for verification too? Or we could allow SECRET_KEY to be a
> list/tuple, and if so, sign with the first and verify with all of them.
>
> Agreed, I will add something like that then! :)
>
> Cheers,
> Andreas
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/A16A11DF-1439-46EF-BF0D-85C483F53608%40pelme.se
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFzonYbcFn2h0-g8nk9Bj1fgprUU0AFLxph9L_1H%2B0KEMLZ1%3DQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

skipping elidable migrations

2018-11-09 Thread Dan Watson

Hi All,

I was wondering if anyone had any thoughts on an option to the "migrate"
command (--skip-elidable?) that would skip running elidable migrations. The
use case here is that data migrations that build up over time may act on
certain assumptions (existing tables/data) that may not be true when
migrating a new database. It seems that since they were explicitly marked
as able to be deleted when squashing, they would be safe to not run when
creating a new database. Maybe we don't go so far as to make this the
default behavior when migrating a fresh database, but an option would be
nice. I realize you could simply squash your migrations, but that's not
without penalty of code churn, testing, etc. especially when your existing
migration graph is otherwise fine (and performant).

If there's some consensus about this being worthwhile, or at least no
strong objections to it, I can take a stab at the implementation.

Regards,
Dan

--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/fba7a58a-444b-4c90-b139-151580423366%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: CharField with Oracle Backend Null Behaviour

2018-11-08 Thread Dan Davis

Part of the reason to use Django is to assure some level of database
portability.  I use Oracle exclusively at work, although there are some
plans to move towards PostgreSQL.  I think blank=True should govern whether
the field may be null or not.

So, from my perspective, we should limit ourselves with tinkering with how
the backend represents such constraints at the database level.

On Thu, Nov 8, 2018 at 2:14 PM Mariusz Felisiak 
wrote:

> Hi,
>
> Oracle treats empty strings and NULLs in the same way, that's why we
> had to decide on some workaround that will cover both python's (other Dbs')
> cases i.e. NULLs and "" (see
> https://docs.djangoproject.com/en/2.1/ref/databases/#null-and-empty-strings).
> I don't see much value in changing current behavior from *"empty strings
> always"* to *"NULLs always"*, because it will be backward incompatible
> and IMO it doesn't change anything, still non-Oracle people will be
> surprised by this behavior.
> I agree that creating a *"NOT NULL" *constraint in case when
> "null=False*" *(or even "blank=False") may be a helpful addition to the
> current behavior but I'm not sure how doable it is if we take into account
> migration of existing projects.
>
> Best,
> Mariusz
>
> W dniu czwartek, 8 listopada 2018 18:33:52 UTC+1 użytkownik
> vaf...@exscientia.co.uk napisał:
>>
>> Hi Florian,
>>
>> Yes you are correct, backwards compatibility would break with this
>> change.
>> As Django already mimics the required checks in middleware for nulls,
>> only difference I can see going forward would be that newly inserted blank
>> strings would be read back back as None, instead of '' if nulls are
>> allowed. And where nulls are not allowed, neither would make it into the db.
>>
>> Look forward to hearing your thoughts Felix.
>>
>> Many Thanks,
>> Vackar
>>
>> On Thursday, 8 November 2018 17:06:05 UTC, Florian Apolloner wrote:
>>>
>>> Hi Vackar,
>>>
>>> Thank you, now we are getting somewhere!
>>>
>>> On Thursday, November 8, 2018 at 5:36:53 PM UTC+1,
>>> vaf...@exscientia.co.uk wrote:

 My main concern currently is that required fields are not enforced at
 the db level, which makes using it with other clients difficult. I would
 much prefer that constraints be added, and accept that empty strings cannot
 be inserted into required columns.

>>>
>>> Okay, now I get you. I have no strong feelings how our oracle backend
>>> should behave here; other than your suggested change would be (highly?)
>>> backwards incompatible and that alone might prevent it from getting merged.
>>> Maybe Felix can chime in here with his oracle knowledge.
>>>
>>> FWIW, Django has quite a few places where it does stuff in the framework
>>> itself instead of at the database level. Examples include the handling of
>>> database default values (we don't, defaults are in the application, even
>>> for simple integers and strings) as well as handling of cascades on
>>> deletes. If one cannot live with those issues, it is imo currently best to
>>> manage the database independent from Django. We are certainly open to
>>> improving in those areas though.
>>>
>>> Cheers,
>>> Florian
>>>
>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/403d5d50-660b-426c-a911-579d82273259%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFzonYaJUKr8NZBX_sPGVxMr%3DgJyk-AMhRgmVkEc%3DNjNzFvsKg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: backend specific tests

2018-11-07 Thread Dan Davis

Tom,

I'm interested in this - I have a software architect's goal of getting our
Django systems to go to the cloud with Docker, but for Django development,
I am eager to clutter up my local box with these tools.   I know systemctl
quite well, and I have no problem changing postgresql and mysql to be
services not started by default.   I'm a better postgresql dba than mysql,
but I do OK.

On Wed, Nov 7, 2018 at 5:45 AM Tom Forbes  wrote:

> Hey Dan,
>
> I’ve been working on a project called django_docker_box (
> https://github.com/orf/django-docker-box) that might help with this.
> Docker is pretty good at spinning up various databases without needing to
> clutter your local machine, spend time configuring authentication or
> dealing with issues like this.
>
> You might find this helps with you. You can run the entire test suite with
> it, or you can just spin up a Postgres database and connect to it from your
> local machine (docker-compose run postgres-db) with django:django.
>
> Alternatively you could run:
>
> docker run -p 5432:5432 -e POSTGRES_PASSWORD=django postgres:9.6
>
> And connect to it with postgres:django on localhost:5432
>
> Mysql can be done in the same way:
>
> docker run -p 3306:3306 -e MYSQL_PASSWORD=django mysql:5.7
>
> Hope this helps,
>
> Tom
>
>
>
>
> On 7 November 2018 at 03:51:47, Dan Davis (dansm...@gmail.com) wrote:
>
> What about mysql?   I have 5.7 installed, and tried to run with root
> privileges, which is what worked for postgresql.  It still failed quite a
> number of times.
>
> On Tue, Nov 6, 2018 at 7:41 PM charettes  wrote:
>
>> Exactly.
>>
>> Given you should be running tests against a throwaway or at least
>> non-critical
>> PostgreSQL cluster anyway I don't think requiring superuser privileges is
>> an
>> issue.
>>
>> Simon
>>
>> Le mardi 6 novembre 2018 18:43:47 UTC-5, Dan Davis a écrit :
>>>
>>> So, a developer using PostgreSQL doesn't need superuser privileges, but
>>> you do to run Django's unit tests, because it will test these contributed
>>> postgres operations.
>>>
>>> On Monday, November 5, 2018 at 5:48:20 PM UTC-5, Josh Smeaton wrote:
>>>>
>>>> I don't think there's a full list of extensions the test suite uses,
>>>> but
>>>> https://docs.djangoproject.com/en/2.1/ref/contrib/postgres/operations/
>>>> would be close to a full set I'd imagine.
>>>>
>>>> On Sunday, 4 November 2018 14:43:23 UTC+11, Dan Davis wrote:
>>>>>
>>>>> So, the contributor guidelines page about unit tests mentions running
>>>>> database specific tests:
>>>>>
>>>>>
>>>>> https://docs.djangoproject.com/en/2.1/internals/contributing/writing-code/unit-tests/#testing-other-python-versions-and-database-backends
>>>>>
>>>>> I am working on ticket 29984, and it seems to me that since the
>>>>> TruncDay is attempting to cast to the timezone on the database level, it 
>>>>> is
>>>>> working, and its job is done.  So, the fix should be at the backend level,
>>>>> and the ticket provides one for Postgres.
>>>>>
>>>>> Following the unit test advice, I created a settings file.   But not
>>>>> using my Postgresql admin user.   I get the following errors:
>>>>>
>>>>> psycopg2.ProgrammingError: permission denied to create extension
>>>>> "btree_gin"
>>>>> HINT:  Must be superuser to create this extension.
>>>>>
>>>>> I'm not going to do that - do you have any list of required extensions?
>>>>>
>>>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/d252a3e0-f8f4-4026-b1eb-9dfd79993618%40googlegroups.com
>> <https://groups.google.com/d/msgid/django-developers/d252a3e0-f8f4-4026-b1eb-9dfd79993618%40googlegroups.com?utm_medium=email_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google

Re: backend specific tests

2018-11-06 Thread Dan Davis

What about mysql?   I have 5.7 installed, and tried to run with root
privileges, which is what worked for postgresql.  It still failed quite a
number of times.

On Tue, Nov 6, 2018 at 7:41 PM charettes  wrote:

> Exactly.
>
> Given you should be running tests against a throwaway or at least
> non-critical
> PostgreSQL cluster anyway I don't think requiring superuser privileges is
> an
> issue.
>
> Simon
>
> Le mardi 6 novembre 2018 18:43:47 UTC-5, Dan Davis a écrit :
>>
>> So, a developer using PostgreSQL doesn't need superuser privileges, but
>> you do to run Django's unit tests, because it will test these contributed
>> postgres operations.
>>
>> On Monday, November 5, 2018 at 5:48:20 PM UTC-5, Josh Smeaton wrote:
>>>
>>> I don't think there's a full list of extensions the test suite uses, but
>>> https://docs.djangoproject.com/en/2.1/ref/contrib/postgres/operations/
>>> would be close to a full set I'd imagine.
>>>
>>> On Sunday, 4 November 2018 14:43:23 UTC+11, Dan Davis wrote:
>>>>
>>>> So, the contributor guidelines page about unit tests mentions running
>>>> database specific tests:
>>>>
>>>>
>>>> https://docs.djangoproject.com/en/2.1/internals/contributing/writing-code/unit-tests/#testing-other-python-versions-and-database-backends
>>>>
>>>> I am working on ticket 29984, and it seems to me that since the
>>>> TruncDay is attempting to cast to the timezone on the database level, it is
>>>> working, and its job is done.  So, the fix should be at the backend level,
>>>> and the ticket provides one for Postgres.
>>>>
>>>> Following the unit test advice, I created a settings file.   But not
>>>> using my Postgresql admin user.   I get the following errors:
>>>>
>>>> psycopg2.ProgrammingError: permission denied to create extension
>>>> "btree_gin"
>>>> HINT:  Must be superuser to create this extension.
>>>>
>>>> I'm not going to do that - do you have any list of required extensions?
>>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/d252a3e0-f8f4-4026-b1eb-9dfd79993618%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/d252a3e0-f8f4-4026-b1eb-9dfd79993618%40googlegroups.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFzonYYhELA0RRpM07KGsYX08CmM09UjeN8vZRovwfmCUTz_2g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: backend specific tests

2018-11-06 Thread Dan Davis

So, a developer using PostgreSQL doesn't need superuser privileges, but you 
do to run Django's unit tests, because it will test these contributed 
postgres operations.

On Monday, November 5, 2018 at 5:48:20 PM UTC-5, Josh Smeaton wrote:
>
> I don't think there's a full list of extensions the test suite uses, but 
> https://docs.djangoproject.com/en/2.1/ref/contrib/postgres/operations/ 
> would be close to a full set I'd imagine.
>
> On Sunday, 4 November 2018 14:43:23 UTC+11, Dan Davis wrote:
>>
>> So, the contributor guidelines page about unit tests mentions running 
>> database specific tests:
>>
>>
>> https://docs.djangoproject.com/en/2.1/internals/contributing/writing-code/unit-tests/#testing-other-python-versions-and-database-backends
>>
>> I am working on ticket 29984, and it seems to me that since the TruncDay 
>> is attempting to cast to the timezone on the database level, it is working, 
>> and its job is done.  So, the fix should be at the backend level, and the 
>> ticket provides one for Postgres.
>>
>> Following the unit test advice, I created a settings file.   But not 
>> using my Postgresql admin user.   I get the following errors:
>>
>> psycopg2.ProgrammingError: permission denied to create extension 
>> "btree_gin"
>> HINT:  Must be superuser to create this extension.
>>
>> I'm not going to do that - do you have any list of required extensions?
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/0955d1f6-c7eb-4303-b334-629dfc80a65a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

django.utils.timezone.make_aware confusion

2018-11-05 Thread Dan Davis

Working on ticket #29984, I noticed that the following raises:

from datetime import datetime
import pytz
from django.utils.timezone import make_aware

euberlin = pytz.timezone('Europe/Berlin')s a
dt1 = datetime(2018, 10, 24, tzinfo=euberlin)
dt2 = make_aware(datetime(2018, 10, 24))
assert dt1 == dt2


It is certainly necessary to have make_aware, because datetime may come in 
without a timezone.  I'm familiar with the various solutions - user picks a 
timezone in their profile, and/or JavaScript detects the timezone in the 
browser and provides that to the backend at some point.  I'm even urging a 
coworker to fix his public Django webapp to implement the latter solution.

However, it is disturbing that the datetimes are not equal.   Can someone 
explain why not?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/46a382ef-2b19-4b81-87b8-babb1f93b9cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

backend specific tests

2018-11-03 Thread Dan Davis

So, the contributor guidelines page about unit tests mentions running 
database specific tests:

https://docs.djangoproject.com/en/2.1/internals/contributing/writing-code/unit-tests/#testing-other-python-versions-and-database-backends

I am working on ticket 29984, and it seems to me that since the TruncDay is 
attempting to cast to the timezone on the database level, it is working, 
and its job is done.  So, the fix should be at the backend level, and the 
ticket provides one for Postgres.

Following the unit test advice, I created a settings file.   But not using 
my Postgresql admin user.   I get the following errors:

psycopg2.ProgrammingError: permission denied to create extension "btree_gin"
HINT:  Must be superuser to create this extension.

I'm not going to do that - do you have any list of required extensions?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/dfae7d19-a356-4b39-aab5-3148e0468082%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Requiring sqlparse for sqlite introspection

2018-11-03 Thread Dan Davis

I just joined as a contributor, but I've shipped an appliance install 
running using rpms, anaconda (the other one), and pungi.   Depending on 
sqlparse doesn't seem to me a big deal.  It already gets invoked for me 
during migrations.  I cannot recall what caused it to be installed. One 
thing we should definitely do however, is more clearly document that RunSQL 
migrations will split your SQL on statement boundaries to the extent it can 
unless you run manual SQL migrations like this:

RunSQL([forward_compound_statement], [backward_compound_statement])


Creating functions with Oracle doesn't work without this.

On Saturday, November 3, 2018 at 4:47:30 PM UTC-4, charettes wrote:
>
> > So you want to add it to Django's install_requires even though it won't 
> be necessary if SQLite isn't used?
>
> That's my understanding and what I was advocating for.
>
> Simonon 
>
> Le samedi 3 novembre 2018 10:09:55 UTC-4, Tim Graham a écrit :
>>
>> So you want to add it to Django's install_requires even though it won't 
>> be necessary if SQLite isn't used? It seems okay to me. It's an extra 40k 
>> or so of disk space but that's not much compared to all the extra stuff 
>> Django comes with that every Django project doesn't necessarily use. We 
>> also have the requires_sqlparse_for_splitting feature flag which we could 
>> remove if we can assume sqlparse is installed.
>>
>> On Sunday, October 28, 2018 at 4:59:54 AM UTC-4, Adam Johnson wrote:
>>>
>>> I'm fine with adding it as a dependency, my experience has been that 
>>> it's a stable, well-maintained package over the past few years I've used it.
>>>
>>> On Sun, 28 Oct 2018 at 05:00, charettes  wrote:
>>>
 After a bit of work to minimize the cases where sqlparse would be a 
 required at runtime for SQLite to AddConstraint/RemoveConstraint 
 operations 
 [0] I came to the conclusion that it would make more sense to make 
 sqlparse 
 an hard dependency of Django 2.2.

 The two reasons backing this conclusions are

 1. Given we run the suite with sqlparse installed on CI it will be 
 really hard to prevent inadvertently breaking the promise of a soft 
 dependency on sqlparse for Meta.constraints only. I guess we could 
 have a daily CI job but that would quickly get out of hand once we have to 
 perform backport and such.

 2. There's a few instances of fragile regex parsing that could be made 
 more reliable if sqlparse was an hard dependency for SQLite. Two 
 examples are
  - 
 https://github.com/django/django/blob/f892781b957f674806a227a10c58768f66a48c07/django/db/backends/sqlite3/introspection.py#L90-L146
  - 
 https://github.com/django/django/blob/f892781b957f674806a227a10c58768f66a48c07/django/db/backends/sqlite3/schema.py#L91-L127

 Given the transition to require pytz in Django 1.11 went smoothly and 
 our needs for sqlparse are similar to provide a solid experience on SQLite 
 I'd be +1 on requiring it from Django 2.2 to a point where the lowest 
 version of SQLite we support has better introspection capabilities.

 Simon

 [0] https://github.com/django/django/pull/10572

 Le mercredi 10 octobre 2018 14:53:53 UTC-4, Tim Graham a écrit :
>
> sqlparse is already installed as part of Django's tests. The question 
> is whether sqlparse should be mandatory for SQLite users (i.e. when 
> getting 
> started with a new project an error message will say, "You must install 
> sqlparse to use SQLite" (I don't think there's a way to install it 
> automatically unless we install it for all users, regardless of database) 
> or if we should try to make it optional and only raise an error if a 
> project's migrations require it.
>
> The question is "what percentage of SQLite projects are going to end 
> up needing sqlparse based on their migrations? If it's high enough, it 
> might make for a better user experience to have users install it when 
> they're getting started instead of a lazy error when get_constraints() 
> is called."
>
> On Monday, October 8, 2018 at 3:22:32 AM UTC-4, Andrew Godwin wrote:
>>
>> Adding sqlparse into the introspection code for SQLite specifically 
>> means it's going to be a runtime dependency for anything to do with 
>> migrations.
>>
>> I'm alright having that be the case, but I do think we should make 
>> sure the tests run with SQLite as otherwise a large section of the most 
>> complicated code in migrations won't be tested properly.
>>
>> Andrew
>>
>>
>> On Mon, 8 Oct 2018, 00:59 Ian Foote,  wrote:
>>
>>> Hi all,
>>>
>>> On my pull request (https://github.com/django/django/pull/10406) 
>>> refactoring how Django creates database constraints I introduced a 
>>> dependency on sqlparse in the sqlite introspection code. This allows 
>>> Django 
>>> to correctly

Re: Introspection of querysets

2018-11-03 Thread Dan Davis

OK - why borrow trouble. There is plenty to do just maintaining current 
interfaces, and adding major features - like the new db functions in 2+.

On Saturday, November 3, 2018 at 7:36:00 PM UTC-4, Adam Johnson wrote:
>
> I have code in my package Django-MySQL that uses the internals of 
> querysets. It’s not so bad to maintain with enough test coverage and 
> ensuring it keeps up to date with the latest Django. The fact the interface 
> is private means that it could change at any time, but practically that 
> doesn’t mean often, and rarely or never in a patch release.
>
> See code in/around 
>
> https://github.com/adamchainz/django-mysql/blob/master/django_mysql/models/query.py
>
> On Sat, 3 Nov 2018 at 22:22, Dan Davis > 
> wrote:
>
>> I was thinking of providing a package containing a couple of DRF 
>> renderers that work directly from querysets.   DRF Serializers are great 
>> when you need to render a hierarchical structure, and/or when you expect to 
>> parse a POST body containing the structure.  For returning non-hierarchical 
>> results, querysets ought to be fine.   when I attempted to code this up, I 
>> realized that I needed for the renderer to know which fields are in the 
>> body to write a CSV or XLSX.  I eventually decided that the renderers would 
>> assume that their input queryset was a values or values list queryset.
>>
>> There is also the issue of simply being more DRY within Django itself - 
>> multiple Iterables combine list(values_select) + list(annotation_select) + 
>> list(extra_select), so a general public function could make that could 
>> slightly better.
>>
>> It isn't too bad to look at _iterable_class - it just is clearly not a 
>> public interface.
>>
>> On Friday, November 2, 2018 at 1:34:07 PM UTC-4, Tim Graham wrote:.
>>
>>> Could you explain the use case for the code that needs to handle the 
>>> different types of iterators? list
>>>
>>  
>>
>>> On Friday, November 2, 2018 at 11:18:34 AM UTC-4, Dan Davis wrote:
>>>>
>>>> I'm wondering two things:
>>>>
>>>>- Is there any non-internals way to know what sort of iterable a 
>>>>queryset is set to do?   Should there be?   Background - users should 
>>>> not 
>>>>write code to look at isinstance(queryset._iterable_class, 
>>>>FubarIterator) because that is a private interface (and it may be 
>>>>code no longer in 2.x, since I'm currently looking at 1.11.x).
>>>>- Is looking at query.values_select, query.annotation_select, and 
>>>>extra_select as is done by the ValuesIterable, ValuesListIterable, 
>>>>and FlatValuesListItereable a supported interface?   Is there 
>>>>documentation for this?   Should there be?
>>>>
>>>> Thanks,
>>>>
>>>> -Dan
>>>>
>>>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/7f162ffb-b970-4064-87ef-af67fc0da917%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/django-developers/7f162ffb-b970-4064-87ef-af67fc0da917%40googlegroups.com?utm_medium=email_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
> -- 
> Adam
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3759d99d-1279-4b2c-b0cb-b219b06b6ff9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Introspection of querysets

2018-11-03 Thread Dan Davis

I was thinking of providing a package containing a couple of DRF renderers 
that work directly from querysets.   DRF Serializers are great when you 
need to render a hierarchical structure, and/or when you expect to parse a 
POST body containing the structure.  For returning non-hierarchical 
results, querysets ought to be fine.   when I attempted to code this up, I 
realized that I needed for the renderer to know which fields are in the 
body to write a CSV or XLSX.  I eventually decided that the renderers would 
assume that their input queryset was a values or values list queryset.

There is also the issue of simply being more DRY within Django itself - 
multiple Iterables combine list(values_select) + list(annotation_select) + 
list(extra_select), so a general public function could make that could 
slightly better.

It isn't too bad to look at _iterable_class - it just is clearly not a 
public interface.

On Friday, November 2, 2018 at 1:34:07 PM UTC-4, Tim Graham wrote:.

> Could you explain the use case for the code that needs to handle the 
> different types of iterators? list
>

> On Friday, November 2, 2018 at 11:18:34 AM UTC-4, Dan Davis wrote:
>>
>> I'm wondering two things:
>>
>>- Is there any non-internals way to know what sort of iterable a 
>>queryset is set to do?   Should there be?   Background - users should not 
>>write code to look at isinstance(queryset._iterable_class, 
>>FubarIterator) because that is a private interface (and it may be 
>>code no longer in 2.x, since I'm currently looking at 1.11.x).
>>- Is looking at query.values_select, query.annotation_select, and 
>>extra_select as is done by the ValuesIterable, ValuesListIterable, 
>>and FlatValuesListItereable a supported interface?   Is there 
>>documentation for this?   Should there be?
>>
>> Thanks,
>>
>> -Dan
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7f162ffb-b970-4064-87ef-af67fc0da917%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Introspection of querysets

2018-11-02 Thread Dan Davis

I'm wondering two things:

   - Is there any non-internals way to know what sort of iterable a 
   queryset is set to do?   Should there be?   Background - users should not 
   write code to look at isinstance(queryset._iterable_class, FubarIterator) 
   because that is a private interface (and it may be code no longer in 2.x, 
   since I'm currently looking at 1.11.x).
   - Is looking at query.values_select, query.annotation_select, and 
   extra_select as is done by the ValuesIterable, ValuesListIterable, and 
FlatValuesListItereable 
   a supported interface?   Is there documentation for this?   Should there 
   be?

Thanks,

-Dan

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ec429a48-28b6-4289-bb9b-37b975e7a949%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Widening participation (Thoughts from DjangoCon)

2018-10-28 Thread Dan Davis

Trac can be made easier to search with Apache Solr 
- 
https://www.pycon.it/conference/talks/full-text-search-for-trac-with-apache-solr

On Sunday, October 28, 2018 at 6:04:12 PM UTC-4, Josh Smeaton wrote:
>
> I strongly dislike Trac in nearly every way. It's hard to search and the 
> filters are next to useless, and the categorisation features we use are not 
> very useful. I believe the better way to search Trac is to use google and 
> site:code.djangoproject.com which is a red flag itself.
>
> On Saturday, 27 October 2018 11:09:38 UTC+11, Tom Forbes wrote:
>>
>> How much of this would you attribute to the current ticketing system 
>> itself, rather than tickets being tagged appropriately?
>>
>> I know when I started contributing I found trac to be pretty intimidating 
>> in terms of complexity, especially the search. I still prefer to use the 
>> 'Search Trac' field in the root code.djangoproject.com page than fiddle 
>> with the myriad of options and drop downs in the browse tickets section.
>>
>> If we think of getting new people onboard as a conversion funnel we need 
>> to stop dropoff as much as possible, and that extends to the UI of the 
>> ticket tracker as well I believe.
>>
>> Tom
>>
>> On Fri, 26 Oct 2018, 22:43 Ian Foote,  wrote:
>>
>>> Hi Carlton,
>>>
>>> I've had similar thoughts sitting in the back of my mind for at least a 
>>> couple of months, so thank you for sharing this. I agree that finding 
>>> tickets is one of the big problems here, both for new contributors and for 
>>> sprint leaders. At Pycon UK I took on the role of sprint leader along with 
>>> Adam Johnson and directing people to appropriate tickets was a definite 
>>> difficulty. I was also unaware of the django core mentorship list and will 
>>> be joining that soon. I'm willing to spend some time mentoring a small 
>>> number of people, life permitting.
>>>
>>> Ian
>>>
>>> On Fri, 26 Oct 2018 at 14:44, Carlton Gibson  
>>> wrote:
>>>
 Hi All. 

 OK, so last week I was at DjangoCon US in San Diego. (Thank you if you 
 organised that! Hi! if we met and chatted.) 
 I gave a talk ("Your web framework needs you!") inspired by the 
 discussion on the 

 DSF list and the proposal to dissolve Django Core 
 . (Can’t see the DSF list? Join 
 the DSF 

 .)
 I was asking for more participation in general, and participation that 
 is more representative of the wider Django community in particular.

 There was lots of good input from many people, including (but not, at 
 all, limited to) representatives of groups such Pyladies, DjangoGirls, and 
 so on. 

 The recurring themes seem to me to fit into three categories:

1. The importance of *mentoring*. 
2. The difficulty of *finding tickets*. 
3. The importance of *sprints*. 

 The rest here is a summary of that. Hopefully it’s useful. 

 Mentoring

 For whatever reasons, the exiting *Contributing How-To* 

 doesn’t lead to contributions from a demographic that matches the wider 
 Django Community. 

 The point that came up again and again about this was that *mentoring* 
 is one of the best (perhaps the best?) tool in helping to change this. 

 Django Core Mentorship

 We don’t have an official mentoring programme but we do have the 
 django-core-mentorship 
 list . 

 This must be about the best-kept secret in the Django world: it’s gets 
 ≈0 traffic, but I told everybody at DjangoCon about it, and that they 
 should use it. 

 If you are not on django-core-mentorship, and you’re willing to help 
 prospective contributors, please sign-up. I’m hoping we can drive some 
 traffic to it. 

 Maybe there’s call for something more formal, but at least until DCM is 
 actually being used, that seems (to me) like something we can postpone. 

 Finding Tickets

 The next thing was that there’s not enough guidance on what to work on. 

 The guidance is to look for *Easy Pickings*. There are ≈1300 accepted 
 open tickets in TRAC. 13 of these are marked *Easy Pickings*. 

 That’s not enough. I think we’re too tight with it (or need another 
 grade). 

 There are *many* tickets which aren’t super hard: I put it that, most 
 of our community solve harder problems every day *using Django* than 
 most tickets require. 

 Yes, they still require time, love, energy, etc — and maybe some 
 mentoring — but it’s not primary research, in the main.

 I talked to people

Re: Ticket/Issue Tracker

2018-10-27 Thread Dan Davis

Collin, 

Are you using an inverted index (search engine) behind that for relevance 
ranking, better word segmentation/stemming/synonyms, and peirformance?
I think that something like that is really needed when the problem is 
finding stuff.

On Friday, October 26, 2018 at 9:50:43 PM UTC-4, Collin Anderson wrote:
>
> A few years ago I realized I really liked the UI of Trello.com, so I tried 
> creating a trello-style view of django tickets:
>
> https://djello.collinand.org/ (beware, I'm not a designer :)
>
> It's basically just 100 lines of JS and a little CSS. It's still my go to 
> for finding tickets.
>
> I thought I'd share that as an idea for how to improve find-ability.
>
>
> -- Forwarded message -
> From: Tom Forbes
> Date: Fri, Oct 26, 2018 at 8:09 PM
> Subject: Re: Widening participation (Thoughts from DjangoCon)
>
> How much of this would you attribute to the current ticketing system 
> itself, rather than tickets being tagged appropriately?
>
> I know when I started contributing I found trac to be pretty intimidating 
> in terms of complexity, especially the search. I still prefer to use the 
> 'Search Trac' field in the root code.djangoproject.com page than fiddle 
> with the myriad of options and drop downs in the browse tickets section.
>
> If we think of getting new people onboard as a conversion funnel we need 
> to stop dropoff as much as possible, and that extends to the UI of the 
> ticket tracker as well I believe.
>
> Tom
>
> On Fri, 26 Oct 2018, 22:43 Ian Foote wrote:
>
>> Hi Carlton,
>>
>> I've had similar thoughts sitting in the back of my mind for at least a 
>> couple of months, so thank you for sharing this. I agree that finding 
>> tickets is one of the big problems here, both for new contributors and for 
>> sprint leaders. At Pycon UK I took on the role of sprint leader along with 
>> Adam Johnson and directing people to appropriate tickets was a definite 
>> difficulty. I was also unaware of the django core mentorship list and will 
>> be joining that soon. I'm willing to spend some time mentoring a small 
>> number of people, life permitting.
>>
>> Ian
>>
>> On Fri, 26 Oct 2018 at 14:44, Carlton Gibson wrote:
>>
>>> Hi All. 
>>>
>>> OK, so last week I was at DjangoCon US in San Diego. (Thank you if you 
>>> organised that! Hi! if we met and chatted.) 
>>> I gave a talk ("Your web framework needs you!") inspired by the 
>>> discussion on the 
>>>  DSF 
>>> list and the proposal to dissolve Django Core 
>>> . (Can’t see the DSF list? Join 
>>> the DSF 
>>> 
>>> .)
>>> I was asking for more participation in general, and participation that 
>>> is more representative of the wider Django community in particular.
>>>
>>> There was lots of good input from many people, including (but not, at 
>>> all, limited to) representatives of groups such Pyladies, DjangoGirls, and 
>>> so on. 
>>>
>>>
>>> The recurring themes seem to me to fit into three categories:
>>>
>>>1. The importance of *mentoring*.
>>>2. The difficulty of *finding tickets*.
>>>3. The importance of *sprints*.
>>>
>>> The rest here is a summary of that. Hopefully it’s useful. 
>>>
>>> Finding Tickets
>>>
>>> The next thing was that there’s not enough guidance on what to work on. 
>>>
>>> The guidance is to look for *Easy Pickings*. There are ≈1300 accepted 
>>> open tickets in TRAC. 13 of these are marked *Easy Pickings*. 
>>>
>>> That’s not enough. I think we’re too tight with it (or need another 
>>> grade). 
>>>
>>> There are *many* tickets which aren’t super hard: I put it that, most 
>>> of our community solve harder problems every day *using Django* than 
>>> most tickets require. 
>>>
>>> Yes, they still require time, love, energy, etc — and maybe some 
>>> mentoring — but it’s not primary research, in the main.
>>>
>>> I talked to people who had (at the conference) got the test suite 
>>> running and such, but been overawed by the (for want of a better phrase) 
>>> *sheer 
>>> face* of issue tracker. 
>>>
>>> We would do well to invite people better here. (I don’t have instant 
>>> solutions.) 
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d52ed4e1-ad0d-42d8-9d3f-7f3e4be2a0b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Widening participation (Thoughts from DjangoCon)

2018-10-26 Thread Dan Davis

Thanks.   Although I don't solve the demographic problem, I should respond
to the call for broader participation.  I should make myself put aside time
to contribute - and I should force my boss to let me ;)

On Fri, Oct 26, 2018 at 9:44 AM Carlton Gibson 
wrote:

> Hi All.
>
> OK, so last week I was at DjangoCon US in San Diego. (Thank you if you
> organised that! Hi! if we met and chatted.)
> I gave a talk ("Your web framework needs you!") inspired by the
> discussion on the
> 
> DSF list and the proposal to dissolve Django Core
> . (Can’t see the DSF list? Join
> the DSF
> 
> .)
> I was asking for more participation in general, and participation that is
> more representative of the wider Django community in particular.
>
> There was lots of good input from many people, including (but not, at all,
> limited to) representatives of groups such Pyladies, DjangoGirls, and so
> on.
>
>
> The recurring themes seem to me to fit into three categories:
>
>1. The importance of *mentoring*.
>2. The difficulty of *finding tickets*.
>3. The importance of *sprints*.
>
> The rest here is a summary of that. Hopefully it’s useful.
>
> Mentoring
>
> For whatever reasons, the exiting *Contributing How-To*
>  doesn’t
> lead to contributions from a demographic that matches the wider Django
> Community.
>
> The point that came up again and again about this was that *mentoring* is
> one of the best (perhaps the best?) tool in helping to change this.
>
> Django Core Mentorship
>
> We don’t have an official mentoring programme but we do have the 
> django-core-mentorship
> list .
>
> This must be about the best-kept secret in the Django world: it’s gets ≈0
> traffic, but I told everybody at DjangoCon about it, and that they should
> use it.
>
> If you are not on django-core-mentorship, and you’re willing to help
> prospective contributors, please sign-up. I’m hoping we can drive some
> traffic to it.
>
> Maybe there’s call for something more formal, but at least until DCM is
> actually being used, that seems (to me) like something we can postpone.
>
> Finding Tickets
>
> The next thing was that there’s not enough guidance on what to work on.
>
> The guidance is to look for *Easy Pickings*. There are ≈1300 accepted
> open tickets in TRAC. 13 of these are marked *Easy Pickings*.
>
> That’s not enough. I think we’re too tight with it (or need another
> grade).
>
> There are *many* tickets which aren’t super hard: I put it that, most of
> our community solve harder problems every day *using Django* than most
> tickets require.
>
> Yes, they still require time, love, energy, etc — and maybe some mentoring
> — but it’s not primary research, in the main.
>
> I talked to people who had (at the conference) got the test suite running
> and such, but been overawed by the (for want of a better phrase) *sheer
> face* of issue tracker.
>
> We would do well to invite people better here. (I don’t have instant
> solutions.)
>
> Sprints
>
> I’m not historically a Django-sprinter. (I have too many children for that
> TBH, but they’re getting older…)
>
> I always thought it was for a hard-core to work on hard issues.
>
> Shows what I know… 
>
> It was strongly impressed upon me that the real benefit of sprints is
> being able to give new contributors the support they need to make their
> first (or second or…) contribution.
>
> In particular, groups such as Pyladies can organise a sprint event with
> the specific goal of helping members of the community get across the
> barriers to contributing. This can reach numbers that otherwise simply
> aren’t possible. (So wow. Basically.)
>
> Sprints & Mentoring
>
> Obviously having mentors at sprints is a key component.
>
> But even if you (or I) can’t attend a sprint, maybe we can (sometimes) be
> available at the right time to contribute remotely. Maybe, in fact, remote
> mentors are a key resource in more remote parts of the Django-sphere.
>
> It turns out just being on e.g. Twitter can be enough here.
>
> If we’re all on django-core-mentorship, maybe sprint organisers could post
> notice of an upcoming sprint.
>
> Sprints & Finding Tickets
>
> It turns out it’s equally hard for a sprint organiser to work out what
> tasks to give sprinters.
>
> At DjangoCon (some) people have a topic and asks others to join them. But,
> maybe if you’re short of experts and so on, that’s not necessarily a model
> that allows that scales out in other contexts.
>
> It was put to me that, if we had something like curated project boards
> (think Trello or GitHub projects) with groups of tickets… perhaps some
> easier, some harder… Perhaps with a *curating mentor*, even if remote… —

Re: New Password Validators

2018-08-31 Thread Dan Davis

Mehmet,

If you need such complexity validators, then they are easy to add as
package.  I think the reason why Django doesn't include more is that many
use Oauth2 or other corporate authentication to validate.
You can see how this is done in one case by looking at
https://github.com/mingchen/django-cas-ng, which I use in my corporate
environment.

You can see other examples at
https://python-social-auth.readthedocs.io/en/latest/configuration/django.html

In the modern world, do we really need the ModelBackend much?   I hope not.

On Fri, Aug 31, 2018 at 2:07 PM Mehmet Dogan  wrote:

> 2 points I forgot to mention:
>
>
>
>1. I think it would be interesting to look at what other web
>frameworks are doing, e.g., Ruby on Rails
>2. If what I offered is not added, I think it makes more sense to
>remove similar ones (e.g., NumericPasswordValidator) from Django to make it
>self consistent.
>
>
>
> Mehmet
>
>
>
> *From: *Mehmet Dogan 
> *Sent: *Friday, August 31, 2018 12:59 PM
> *To: *django-developers@googlegroups.com
> *Subject: *RE: New Password Validators
>
>
>
> Hey James,
>
>
>
> Thank you for the resources you provided. I really learned a lot. Here are
> a few points (references/details at the very bottom):
>
>
>
>1. *Blacklisting: *Seems to be most effective, I agree. However,
>Django does not seem to be up to date on this either. The list of 1000 most
>common password it uses seem to be taken from an URL which is not available
>anymore, and the page it redirects to is dated 2011! So the list is at
>least around 10 years old, and it does not catch all of top 25 words listed
>on Wikipedia. Samples of passwords it allows: qwertyuiop, 987654321,
>1234567890, abcdefgh
>
>
>
>1. *Are Complexity Requirements Deprecated?: *I checked changing my 
> password
>for 3 major tech companies: Gmail, Microsoft account, and for my Apple Id.
>Only Gmail did not require password complexity. So, most do require
>password complexity although they do have 2-factor authentication in place.
>Most Django deployments, more than 95% I assume, would not have 2-factor
>authentication, so password complexity is more needed.
>
>
>
>1. *Do Complexity Requirements Work?: *
>   1. *Research: *From the resources you linked, I read one article
>   (ref’ed below). Here are key findings:
>
> *  i.  *A
> complex 8-character password had entropy of 34.3 where as a basic
> 8-character one had 29.43 (calculated with the specific method mentioned in
> the article). Not a huge difference, however not too bad either, that is
> about *17%* gain.
>
>  ii.  188/972
> (=*20%*) of the basic-8 passwords was cracked (with the tool mentioned in
> the article), whereas this number is *0 (zero)* for the complex-8 group.
> A big difference there.
>
>iii.  About
> 15% of the basic group wrote down their passwords, either electronically or
> paper and pencil, where as this number is about 27% for the complex.
> Simpler passwords is the winner here, however one should not forget,
> written passwords are expected have their own protection (of some sort).
> Another thing to consider is that, if one is targeted individually, a
> written password might be a big vulnerability; however, for general
> account/password screenings this may not be as bad.
>
> All in all, I say, password complexity has benefits; although not as much
> as one would expect.
>
>1. *Complexity and Blacklisting Can Interoperate: *For the simple case
>   of “Password1!” mentioned below, the best method to tackle, I think, 
> would
>   be 1) remove the numbers and characters at the end 2) lowercase the 
> first
>   character 3) lookup in the black list. In other words, tackle the 
> thinking
>   pattern algorithmically, not by listing cases in a text file.
>   2. *Complexity support blacklisting:* Check this list of common
>   passwords, for example:
>   
> https://github.com/danielmiessler/SecLists/blob/master/Passwords/Common-Credentials/500-worst-passwords.txt
>   (provided by the Author of the Django’s list of common passwords). Most
>   items there are either all lowercase or all numbers etc. By just 
> requiring
>   1 upper case character, one could know off 60% off them. Adding a 
> number,
>   and a symbol requirement would easily bring that up to 95%.. Sure, those
>   who are committed to not password protecting their accounts would find
>   something in the remaining 5% (as in the example of “Password1!”),
>   howerver, policies would be made for the mainstream, not for the 
> extremes.
>
>
>
> Bottom line; I think password complexity do have some benefits, and
> inclusion in Django would provide options, and save time to those who would
> like to use them.

Re: Help wanted testing proposal for the Migration Graph algorithm

2018-08-28 Thread dan

Hi Carlton,

Adam asked me to take a look at this. I've run it on our codebase with 
~1100 migrations and ~380 apps.

Running on Django 2.1, the results of this script (the updated version) are:

> 1min 23s ± 1.1 s per loop (mean ± std. dev. of 7 runs, 1 loop each)


Running on the branch, installed with [1], the results are:

> 1min 20s ± 524 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

There were no exceptions thrown - the script completed cleanly, although I 
haven't actually migrated with it or checked that the migrations work. I 
assume the script is enough to ensure consistent migrations? Around a 3.5% 
speedup.

I hope this is what you were looking for, apologies if I've got something 
wrong - I'm lacking full context on this. Hope this helps!


Dan


[1]: pip install 
https://github.com/monoidal/django/archive/optimize_migrations.zip




On Monday, 20 August 2018 10:54:29 UTC+1, Carlton Gibson wrote:
>
> Making the script less noisy here I commented out the actual work.
> (So it didn't function as a test.) 
> (No comment. )
>
> Corrected version:
>
> ```
> import sys
>
> from django.db import connection
> from django.db.migrations.loader import MigrationLoader
>
> loader = MigrationLoader(connection)
>
> backwards = loader.graph.root_nodes()
> forwards = loader.graph.leaf_nodes()
>
> print('Calculating backward plans:')
> for root in backwards:
> loader = MigrationLoader(connection)
> sys.stdout.write('.')
> plan = loader.graph.backwards_plan(root)
> #print(plan)
> sys.stdout.write('\n')
> print('Calculating forward plans:')
> for leaf in forwards:
> loader = MigrationLoader(connection)
> sys.stdout.write('.')
> plan = loader.graph.forwards_plan(leaf)
> # print(plan)
>
> sys.stdout.write('\nRun Done\n\n')
>
> ```
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/cab6baa9-bb5f-42c7-be8b-0223b36cf826%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: rlwrap and manage.py dbshell

2018-06-28 Thread Dan Davis

Sure, but not everyone has rlwrap installed at all, or installed in the
same place.
I guess the thing to do is to use shutil.which first.   I will prepare such
a patch.


On Thu, Jun 28, 2018 at 2:55 PM Adam Johnson  wrote:

> Won't this do the trick?
>
> diff --git a/django/db/backends/oracle/client.py
> b/django/db/backends/oracle/client.py
> index 102e77fd15..1a0e1ca10b 100644
> --- a/django/db/backends/oracle/client.py
> +++ b/django/db/backends/oracle/client.py
> @@ -8,5 +8,5 @@ class DatabaseClient(BaseDatabaseClient):
>
>  def runshell(self):
>  conn_string = self.connection._connect_string()
> -args = [self.executable_name, "-L", conn_string]
> +args = ['rlwrap', '--histsize', '2000', self.executable_name,
> "-L", conn_string]
>  subprocess.check_call(args)
>
> I'd suggest trying it in the third party backend first. Also the history
> size probably doesn't need an easy way to specify, you could just give it
> sensible default on a class level attribute on the DatabaseClient class so
> if it *really* needs specifying, projects can subclass their own backend.
>
> On Thu, 28 Jun 2018 at 17:56, Dan Davis  wrote:
>
>>
>> After some work, I've modified django.db.backends.oracle.client in my
>> virtual environment so that it will invoke Oracle's sqlplus within the
>> excellent utility rlwrap.
>> In my shell, I have
>>
>> * alias sqlplus='rlwrap --histsize 2000 sqlplus'
>>
>> However, it is not secure for a project like Django to use
>> subprocess.check_call with shell=True.
>>
>> To turn this into a permanent feature, I'd need to:
>>
>>- Figure out how users should specify the path to rlwrap, or how to
>>look for the rlwrap utilityp
>>- Figure out how users should specify the size of history.
>>
>>
>> This could also be integrated with a 3rd party backend, such as
>> django-oraclepool and django-oracle-drcp (the latter I've contributed to).
>>
>> In any case, how would developers suggest I proceed to make this a basic
>> option of the Oracle backend?
>>
>> Thanks,
>>
>> -Dan
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/5af25634-2093-451f-8b50-2c1dffb93cf8%40googlegroups.com
>> <https://groups.google.com/d/msgid/django-developers/5af25634-2093-451f-8b50-2c1dffb93cf8%40googlegroups.com?utm_medium=email_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> --
> Adam
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAMyDDM0tg6iw85tx8ZuE%3DqyKyZouzxjBWc66sDBMkKiTf-F6XQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/django-developers/CAMyDDM0tg6iw85tx8ZuE%3DqyKyZouzxjBWc66sDBMkKiTf-F6XQ%40mail.gmail.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFzonYbViyoNf5XPAvar3t_g_QnMfoU%3Dv%3D0neP-16Yr4yV9D1w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

rlwrap and manage.py dbshell

2018-06-28 Thread Dan Davis


After some work, I've modified django.db.backends.oracle.client in my 
virtual environment so that it will invoke Oracle's sqlplus within the 
excellent utility rlwrap.
In my shell, I have

* alias sqlplus='rlwrap --histsize 2000 sqlplus'

However, it is not secure for a project like Django to use 
subprocess.check_call with shell=True.

To turn this into a permanent feature, I'd need to:

   - Figure out how users should specify the path to rlwrap, or how to look 
   for the rlwrap utilityp
   - Figure out how users should specify the size of history.


This could also be integrated with a 3rd party backend, such as 
django-oraclepool and django-oracle-drcp (the latter I've contributed to).

In any case, how would developers suggest I proceed to make this a basic 
option of the Oracle backend?

Thanks,

-Dan

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5af25634-2093-451f-8b50-2c1dffb93cf8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: about ticket 28588- has_perm hide non-existent permissions

2017-09-25 Thread Dan Watson

Seems like maybe it would be more helpful if has_perm logged a note about 
the permission not existing (probably only in debug), rather than just 
returning False. In fact, I'd argue it should still return True -- if the 
permission did exist, the superuser would have it. And there's a 
backwards-compatibility argument. Think of superusers more as "permissions 
don't apply to me" than "I have all permissions".

Dan

On Sunday, September 24, 2017 at 10:56:40 AM UTC-4, moshe nahmias wrote:
>
> Hi,
> I am a python developer and like to use Django for web development.
> Since I like the framework I want to contribute back, so I looked at the 
> open tickets to find something I can start with contributing and found 
> ticket 28588.
>
> This ticket is about when checking if the user has permission for some 
> action if the user is super user he/she gets it all the time, even when the 
> permission doesn't exist, and this is not developer friendly because the 
> developer can mistakenly think that everything is fine even when the 
> permission doesn't exist.
>
> As I understand (and correct me if I'm wrong) there should be a discussion 
> about if we want to do this.
>
> If accepted I would like to do this, I think it's an easy enough change 
> for a new contributor like me.
>
> As I understand the ticket the problem is that a developer gets confused 
> on this behaviour (and it's illogical) that the super user is having a 
> permission that doesn't exist.
>
> What do you think? (I think I will discuss my solution or optional 
> solutions after we decide if we want to change this behaviour)
>
> [1] https://code.djangoproject.com/ticket/28588
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d0430a0f-04d9-484d-b888-bddf6f53ff95%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Suggested documentation: PG server-side cursors used in .iterator() only work inside a transaction

2017-08-07 Thread Dan Watson

Declaring a cursor WITH HOLD means it can be used outside transactions, but 
it seems like the server-side cursors are already set up WITH HOLD when 
autocommit is on:

https://github.com/django/django/blob/master/django/db/backends/postgresql/base.py#L212

So I'm guessing maybe you've disabled transaction management entirely 
(AUTOCOMMIT=False)? In that case, you could try enabling autocommit 
manually when you need to use .iterator(), but maybe the backend could 
check "connection.autocommit or not connection.in_atomic_block" when 
setting WITH HOLD, so even if autocommit is off, WITH HOLD would be used if 
not using an atomic block?

As for the behavior when trying to iterate a cursor after the transaction 
has been exited, I'd expect an error. The cursor should be destroyed at the 
end of the transaction (unless WITH HOLD was used, which it shouldn't be 
inside a transaction), so trying to FETCH from it should raise an error.

Dan

On Sunday, August 6, 2017 at 10:34:35 PM UTC-4, Josh Smeaton wrote:
>
> Yes we should be documenting edge cases and unexpected results. We have a 
> page that discusses some issues with server side cursors: 
> https://docs.djangoproject.com/en/dev/ref/databases/#transaction-pooling-server-side-cursors
>
> Is there anyway we could make SSC work without a transaction? We'd prefer 
> to fix than document if possible.
>
> On Thursday, 3 August 2017 07:22:23 UTC+10, Evan Heidtmann wrote:
>>
>> Hey all,
>>
>> The docs for qs.iterator() say that, in Django 1.11 on Postgres, a 
>> server-side cursor is used.
>>
>> Oracle and PostgreSQL 
>>> <https://docs.djangoproject.com/en/1.11/ref/databases/#postgresql-server-side-cursors>
>>>  use 
>>> server-side cursors to stream results from the database without loading the 
>>> entire result set into memory.
>>
>>
>> https://docs.djangoproject.com/en/1.11/ref/models/querysets/#iterator
>>
>> I discovered that this is only true if the query is run inside a 
>> transaction. Outside a transaction, it appears that Django falls back to a 
>> regular SELECT query, which could be extremely expensive and is certainly 
>> unexpected.
>>
>> I don't know what happens if you call .iterator() inside a transaction 
>> block and then exit that block. Undefined behavior?
>>
>> Therefore I suggest the following docs edit (changes in italic):
>>
>> Oracle and PostgreSQL 
>>> <https://docs.djangoproject.com/en/1.11/ref/databases/#postgresql-server-side-cursors>
>>>  use 
>>> server-side cursors to stream results from the database without loading the 
>>> entire result set into memory. *On PostgreSQL, server-side cursors are 
>>> used only if .iterator() is called within a transaction.*
>>
>>
>> And thanks for this new feature -- it's transformative for my workload.
>>
>> -Evan
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/89ff1578-93d6-4146-b0a5-792344e64000%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Contribution towards the project

2015-12-19 Thread Dan Stephenson

Welcome Yash, the "Advice for new contributors" area is probably a good 
first stop for dipping your toe in the water..

https://docs.djangoproject.com/en/1.9/internals/contributing/new-contributors/



On Wednesday, 16 December 2015 10:55:14 UTC, Yash Sharan wrote:
>
> Hello my name is yash sharan. I am a first year student from BITS Pilani 
> Kk Birla Goa Campus, India. I have basic knowledge about python. In what 
> ways can I enhance my current skills so that I may be Able to contribute 
> towards your project in Gsoc 2016. 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/77611387-93e7-4963-895a-89127fb61160%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: annoyance with Python 3.2 support in Django 1.8

2015-12-03 Thread Dan Stephenson

Do we currently raise any warnings/exceptions in cases where Python support 
has / or is about to be dropped (particularly mid LTS)..   As a suggestion, 
I was thinking it could be helpful to people affected we raised exception 
msg indicating the last Django version to support their current Python 
version?I'd be happy to build if thought useful.




On Thursday, 3 December 2015 23:30:42 UTC, Chris Streeter wrote:
>
> Donald could probably provide more information, but this post from April 
> shows the Python 3.2 numbers downloading from PyPI are constant, and pretty 
> small [https://caremad.io/2015/04/a-year-of-pypi-downloads/] His take was 
> that CI systems (like Django's!) were doing most of the Python 3.2 package 
> downloading.
>
> On Thu, Dec 3, 2015 at 2:18 PM, Josh Smeaton  > wrote:
>
>> I agree with Tim. Unless someone puts their hand up to say they 
>> definitely require python 3.2 support for 1.8, I think it makes sense to 
>> drop support in the next dot release of 1.8. 3.2 isn't an easy python to 
>> find in the wild as far as I know, so I'd be surprised if there was any 
>> real support for it on 1.8 by users.
>>
>> On Friday, 4 December 2015 02:50:24 UTC+11, Tim Graham wrote:
>>>
>>> No, using pypy3 doesn't make things easier. There are a handful of test 
>>> failures with pypy3 and it doesn't solve the issue that 
>>> unittest-xml-reporting doesn't work with Python 3.2.
>>>
>>> Issues aside, the main thing I'm trying to find out is, are we providing 
>>> any substantial value supporting Django on an unsupported version of 
>>> Python? So far no one has indicated "yes". If you care about Django 
>>> security updates, shouldn't you care about Python security updates too?
>>>
>>> On Wednesday, December 2, 2015 at 5:22:46 PM UTC-5, Shai Berger wrote:

 On Wednesday 02 December 2015 21:05:00 Tim Graham wrote: 
 > 
 > Given that no one reading this indicated that they plan a long-term 
 > deployment of Python 3.2, how about if in the next 1.8.x release we 
 > advertise that Python 3.2 support for Django 1.8 will end January 1, 
 2017? 
 > (we won't break anything intentionally after that, but we won't have 
 to 
 > worry about testing and can spin down our 12.04 machine before it's 
 EOL a 
 > few months later) 
 > 

 Since you brought the issue up yourself -- shouldn't we "swap" PyPy3 
 for 
 Python 3.2? Would that make running tests on ubuntu 14.04 easier? 

 Just a half-baked thought, 

 Shai. 

>>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at http://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/0a0631c3-4733-46fc-8884-c4050c8b8701%40googlegroups.com
>>  
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/933a22e5-3f88-4663-b25a-9fdedc83d9c1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: FAQ on Model-View-Controller

2015-08-08 Thread Dan S

Hello Aymeric,

I apologize for using an aggressive tone, it is not my intent to insult 
anyone, just to request that content be updated.   xkcd/386 on the other 
hand makes it personal, referring to some*one*, not some*thing* :-)

I suppose I intended to say that MVC is not defined within the context of a 
Django project, large or small.   In the context of the English language, 
and engineering practice, and even  the unimpeachable source 
Wikipedia , a controller controls (makes changes to) a thing, while 
a view provides a particular vantage point (one of many) for observing that 
thing without changing it.

I am an old Django newbie working on a small project, and because the only 
hooks for Python code I've discovered so far are in views.py, that file 
necessarily touches both V and C code.  I've tried to keep them separate, 
retaining all the template-driving code in views.py and pushing all 
model-updating code (the interesting part of my project) into different 
modules.   So I suppose I have discovered your large project approach 
without even knowing there was a distinction or needing to use more than a 
single datastore.  The data-model-centric design of Django makes it hugely 
attractive to me regardless of any quibbles over where V and C get 
instantiated.

So I humbly suggest that the wording of the FAQ be adjusted to make clear 
that the "meaning" of V and C are not debatable, that templates and 
template-feeding code always serve the purpose of Viewing the model, and 
that Controller code can be either included in views.py or refactored 
elsewhere.

Dan

On Saturday, August 8, 2015 at 9:55:44 AM UTC-4, Aymeric Augustin wrote:
>
> Hello Dan,
>
> Depending on the size and complexity of a Django project, there are two 
> ways to implement MVC. (I’m talking of a traditional Django project, not 
> Django-as-an-API-for-a-JS-app.)
>
> 1) Small project: the straightforward way
>
> M = models.py
> V = template.html
> C = thick views.py: most of the application is implemented there
>
> 2) Large project: the scalable way
>
> M = Django models, other data stores, various APIs, with ad-hoc mechanisms 
> to ensure consistency
> V = thin views.py: business logic gets pushed out to other layers for 
> testability + template.html
> C = a custom abstraction layer: most of the application is implemented 
> there
>
> The FAQ was written 10 years ago when (1) was the dominant pattern and (2) 
> hadn’t been formalized as a best practice for large projects.
>
> If I understand correctly you’re advocating for (2). Perhaps we can 
> improve this FAQ entry but (1) remains a perfectly valid way to write 
> simple applications.
>
> -- 
> Aymeric.
>
> PS: in case you didn’t realize it, your email is worded in a rather 
> aggressive way. Saying hello and being polite wouldn’t hurt your 
> credibility, on the contrary. Thanks!
>
>
> On 8 août 2015, at 13:17, Dan S <dms...@outlook.com > wrote:
>
> The Django FAQ says:
>  
>
> *Django appears to be a MVC framework, but you call the Controller the 
> “view”, and the View the “template”. How come you don’t use the standard 
> names?*¶ 
> <https://docs.djangoproject.com/en/1.8/faq/general/#django-appears-to-be-a-mvc-framework-but-you-call-the-controller-the-view-and-the-view-the-template-how-come-you-don-t-use-the-standard-names>
>
> Well, the standard names are debatable.
>
> In our interpretation of MVC, the “view” describes the data that gets 
> presented to the user. It’s not necessarily *how* the data *looks*, but 
> *which* data is presented. The view describes *which data you see*, not *how 
> you see it.* It’s a subtle distinction.
>
> So, in our case, a “view” is the Python callback function for a particular 
> URL, because that callback function describes which data is presented.
>
> Furthermore, it’s sensible to separate content from presentation – which 
> is where templates come in. In Django, a “view” describes which data is 
> presented, but a view normally delegates to a template, which describes 
> *how* the data is presented.
>
> Where does the “controller” fit in, then? In Django’s case, it’s probably 
> the framework itself: the machinery that sends a request to the appropriate 
> view, according to the Django URL configuration.
>
> If you’re hungry for acronyms, you might say that Django is a “MTV” 
> framework – that is, “model”, “template”, and “view.” That breakdown makes 
> much more sense.
>
> At the end of the day, of course, it comes down to getting stuff done. 
> And, regardless of how things are named, Django gets stuff done in a way 
> that’s most logical to us.
>
>
> Sorry, but the standard names are not debatable, the developers have 
> simply chosen to misuse terms that have well-accept

FAQ on Model-View-Controller

2015-08-08 Thread Dan S

The Django FAQ says:
 

*Django appears to be a MVC framework, but you call the Controller the 
“view”, and the View the “template”. How come you don’t use the standard 
names?*¶ 


Well, the standard names are debatable.

In our interpretation of MVC, the “view” describes the data that gets 
presented to the user. It’s not necessarily *how* the data *looks*, but 
*which* data is presented. The view describes *which data you see*, not *how 
you see it.* It’s a subtle distinction.

So, in our case, a “view” is the Python callback function for a particular 
URL, because that callback function describes which data is presented.

Furthermore, it’s sensible to separate content from presentation – which is 
where templates come in. In Django, a “view” describes which data is 
presented, but a view normally delegates to a template, which describes 
*how* the data is presented.

Where does the “controller” fit in, then? In Django’s case, it’s probably 
the framework itself: the machinery that sends a request to the appropriate 
view, according to the Django URL configuration.

If you’re hungry for acronyms, you might say that Django is a “MTV” 
framework – that is, “model”, “template”, and “view.” That breakdown makes 
much more sense.

At the end of the day, of course, it comes down to getting stuff done. And, 
regardless of how things are named, Django gets stuff done in a way that’s 
most logical to us.


Sorry, but the standard names are not debatable, the developers have simply 
chosen to misuse terms that have well-accepted definitions.   A Controller 
is something that manipulates the model, not something that performs one 
step or a different step of viewing the model.  For example, in a payroll 
application the model would contain records for employees' time and 
attendance and records for W-2 tax statements.  A View for a W-2 statement 
is certainly a Python callback to pull data from the model, as well as a 
template to present the W-2 to the user on the screen.   But a View cannot 
create a W-2 record in the database - that is the job of a Controller, 
which takes employee records containing SSN and salary, timesheet records 
containing hours worked, and information derived from tax laws and creates 
the W-2 records.

Instead of claiming that MVC terms can be debated or redefined, the FAQ 
should state that Django supports the MVC design pattern, that it provides 
two elements (views.py and templates.html) that support the View component, 
and that developers create their own business logic to implement Controller 
elements that produce model data.  In journalism staff people creating 
articles and the public commenting on articles are sort of acting as 
controllers, but in most domains the computer plays a more central 
controller role - translating the position of the foot on an accelerator 
pedal into the amount of gas and air injected into an engine, or 
translating employee data into tax data.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/274923c2-99f5-4013-8914-0f7142d15f78%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Using AbstractBaseUser without django.contrib.auth

2015-04-02 Thread Dan Watson

On Thursday, April 2, 2015 at 10:38:48 AM UTC-4, Marc Tamlyn wrote:
>
> Apologies, I was confusing abstract base user and abstract user there. 
> Seems your proposal should work. Have you opened a ticket?
>

I have:

https://code.djangoproject.com/ticket/24564
https://github.com/django/django/pull/4438

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a46911df-7bfa-477c-a2cd-3e5b0462880f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Using AbstractBaseUser without django.contrib.auth

2015-04-02 Thread Dan Watson

On Thursday, April 2, 2015 at 2:27:51 AM UTC-4, Marc Tamlyn wrote:
>
> Moving them into another module won't make much difference as their 
> definition requires Permission and Group and therefore they'd still need to 
> import Permission and Group. We'd need an "AbstractAbstractBaseUser" which 
> would be silly.
>
>
 AbstractBaseUser doesn't require any models from contrib.auth (as opposed 
to AbstractUser, which inherits PermissionsMixin), and I'm fine with 
leaving PermissionsMixin in models.py since it does require Group and 
Permission.

Of course, there is no requirement to use AbstractBaseUser for your custom 
> model at all, though this does result in some otherwise unnecessary code 
> duplication. I would say your choice is either two empty tables in your 
> database, or code duplication between Django's source and your custom user. 
> Personally I'd prefer the former.
>

Very true, but it seems like AbstractBaseUser was designed to be used by 
systems not wanting Django's permission structure. It's even documented[1] 
how to make AbstractBaseUser subclasses work with the admin. I can live 
with either wart (copying code or empty tables) if need be, just wanted to 
explore some alternatives. I'll open a ticket.

[1] 
https://docs.djangoproject.com/en/1.8/topics/auth/customizing/#custom-users-and-django-contrib-admin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/297c3eb4-7832-424a-a269-4af471eae23a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Using AbstractBaseUser without django.contrib.auth

2015-04-01 Thread Dan Watson

On Wednesday, April 1, 2015 at 10:50:17 PM UTC-4, Curtis Maloney wrote:
>
> Does your model inherit from PermissionsMixin?
>
> If you're using Admin, or any of Django's permissions machinery, you will 
> need django.contrib.auth in there to use the Group and Permission models.
>
>
I'm using my own permissions system, but implementing 
everything PermissionsMixin does (without subclassing it) so the admin 
still works.
 

> --
> Curtis
>
> On 2 April 2015 at 13:47, Dan Watson <dcwa...@gmail.com > 
> wrote:
>
>> While trying out Django 1.8 with one of my sites that uses a custom User 
>> model (and doesn't have django.contrib.auth in INSTALLED_APPS), I noticed a 
>> few unexpected deprecation warnings:
>>
>> /Users/dcwatson/Documents/Environments/reader/lib/python3.4/site-packages/django/contrib/auth/models.py:41:
>>  
>> RemovedInDjango19Warning: Model class django.contrib.auth.models.Permission 
>> doesn't declare an explicit app_label and either isn't in an application in 
>> INSTALLED_APPS or else was imported before its application was loaded. This 
>> will no longer be supported in Django 1.9.
>>   class Permission(models.Model):
>>
>> Same thing for User and Group. Is it required to have django.contrib.auth 
>> installed to use custom User models now? Seems like it would be easy enough 
>> to move AbstractBaseUser (and probably UserManager/BaseUserManager) to a 
>> separate modules and import them from django.contrib.auth.models, so sites 
>> could use them without triggering these errors. If this sounds reasonable, 
>> I can open a ticket/PR.
>>
>> Dan
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at http://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/02f98a6f-9fbf-4c57-8285-80f3c2c3e2ef%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/django-developers/02f98a6f-9fbf-4c57-8285-80f3c2c3e2ef%40googlegroups.com?utm_medium=email_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/44df83a3-5006-4be0-bbb1-cb5d0803381d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Using AbstractBaseUser without django.contrib.auth

2015-04-01 Thread Dan Watson

While trying out Django 1.8 with one of my sites that uses a custom User 
model (and doesn't have django.contrib.auth in INSTALLED_APPS), I noticed a 
few unexpected deprecation warnings:

/Users/dcwatson/Documents/Environments/reader/lib/python3.4/site-packages/django/contrib/auth/models.py:41:
 
RemovedInDjango19Warning: Model class django.contrib.auth.models.Permission 
doesn't declare an explicit app_label and either isn't in an application in 
INSTALLED_APPS or else was imported before its application was loaded. This 
will no longer be supported in Django 1.9.
  class Permission(models.Model):

Same thing for User and Group. Is it required to have django.contrib.auth 
installed to use custom User models now? Seems like it would be easy enough 
to move AbstractBaseUser (and probably UserManager/BaseUserManager) to a 
separate modules and import them from django.contrib.auth.models, so sites 
could use them without triggering these errors. If this sounds reasonable, 
I can open a ticket/PR.

Dan

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/02f98a6f-9fbf-4c57-8285-80f3c2c3e2ef%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: sys.exit(1) from makemigrations if no changes found

2014-10-30 Thread Dan Poirier

Wouldn't enabling this by default cause a problem for any automated
deploys?  Ours typically run the migrate command on every deploy, just
in case there are new migrations, and any command that returns a non-0
exit status is going to look like a deploy failure.

Having the behavior change with a new command line option would be just
fine.

Dan

On Wed. 2014-10-29 at 03:44 AM EDT, Marc Tamlyn <marc.tam...@gmail.com> wrote:

> I agree number 1 is fine. For most general interaction with the
> command users won't notice.
>
> Marc
>
> On 29 Oct 2014 02:04, "Andrew Godwin" <
> and...@aeracode.org> wrote:
>
> I'm actually fine with option 1 - always exiting with a status
> code if no migrations are found. Since the status code does
> nothing useful at the moment, I don't see any backwards
> compatibility issues, and as long as it's a suitably small patch,
> it should be fine.
>
> (As for being surprising compared to grep, there are many other
> commands with different exit codes one could draw parallels to;
> I'm not sure being consistent with a very different utility is a
> worthwhile cause).
>
> Andrew
>
> On Tue, Oct 28, 2014 at 6:22 PM, Tim Heap  o5...@public.gmane.org> wrote:
>
> Hi all,
>
> I have created a ticket for this (https://
> code.djangoproject.com/ticket/23728) but I would like some
> input before I work on it. I will copy the content of the
> ticket below for ease of reading:
>
> It would be very useful for continuous deployment, testing,
> commit hooks, and other applications if django-admin
> makemigrations signaled via an exit code if any migrations
> were found. Commits in projects could be rejected if
> migrations were outstanding, continuous deployment systems
> could fail the build on outstanding migrations, and
> potentially other uses. No more would hasty commits break
> things when developers forgot to make migrations!
>
> Changes to the code to make this happen are easy enough, but
> I am unsure how the command should behave. The grep unix
> utility is a example to copy. Under normal operation, grep
> always exits 0 unless an error happens, regardless of whether
> it found any matches. Invoking grep with the -q/--quiet flag
> causes grep to be silent, not printing anything, as well as
> exiting 0 if matches are found and 1 if nothing is found.
>
> I am proposing django-admin makemigrations should exit with 1
> (or anything non-zero) if no migrations to make were found,
> or exit 0 if migrations to make were found. As the command is
> instructed to make migrations, not making any is the error
> case.
>
> I am unsure how this new functionality should be selected by
> the user when invoking makemigrations. The options I see are:
>  1. Enable this always. This is very simple to implement and
> easy to understand. Good unixy tools use error codes
> extensively to signal errors. This may be surprising
> behaviour when compared to grep though, and breaks
> backwards compatibility in a minor way.
>  2. Enable this when the --dry-run flag is enabled. Now this
> flag can be used to check for migrations that need to be
> created both visually via the printed text, and composed
> in shell commands.
>  3. Add a new flag -e/--exit (or similar). The sole purpose
> of this flag would be to exit with 1 when no migrations
> were found. This could be combined with --dry-run to just
> check for migrations that need to be made.
>  4. Add a new flag -q/--quiet that copies the behaviour of
> greps -q/--quiet flag: silences output and exits with 1
> when no migrations were found. This duplicates
> functionality though, as logging can be silenced using
> -v0 already.
> My personal preference is for option 2. I was surprised when
> enabling --dry-run did not signal its result via the exit
> code. 3 would be the cleanest and most composable option, as
> 4 could be emulated using -ev0.
>
> I will implement this change using 2, unless other people
> have opinions on the matter.
>
> Regards,
> Tim
>
> --
> You received this message because you are subscribed to the
> Google Groups "Django developers (Contribu

Re: Django Trac ticket keywords

2013-09-02 Thread Dan Poirier

On Thu. 2013-08-29 at 06:15 PM EDT, "Daniele Procida"  wrote:

> Would there be any objection if I used a keyword ("afraid_to_commit"
> or something) to mark tickets that I think would be suitable for
> first-time committers doing the "Don't be afraid to commit" tutorial
> to tackle?

My first reaction to that keyword name is "I should be especially afraid
to commit this ticket". Not sure that's the intention.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Feature request: collectstatic shouldn't recopy files that already exist in destination

2012-10-08 Thread Dan Loewenherz

> > On Mon, Oct 1, 2012 at 12:47 AM, Jannis Leidel <lei...@gmail.com> wrote:
> Then, frankly, this is a problem of the storage backends, not Django's.
> The S3BotoStorage backend *does* have a modified_time method:
>
>
> https://bitbucket.org/david/django-storages/src/1574890d87be/storages/backends/s3boto.py#cl-298
>
> What storage backend do you use that doesn't have a modified_time method?
>

I don't think you're seeing the problem I'm having. I'm working with a
distributed team using git. This means when we check out files, the local
modified time is the time at which I checked the files out, not the time
which the files were actually last modified.

As a result, it's a questionable metric for figuring out if a file is the
same or not, since every team member's local machine thinks they were all
just created! We end up re-uploading the file every time.

> This is a bit confusing...why call it last_modified when that's doesn't
> necessarily reflect what it's doing? It would be more flexible to create
> two methods:
>
> It's called modified_time, not last_modified.
>

Sorry, typo.


>
> > def modification_identifier(self):
> >
> > def has_changed(self):
> >
> > Then, any backend could implement these however they might like, and
> collectstatic would have no excuse in uploading the same file more than
> once. Overloading last_modified to also do things like calculate md5's
> seems a bit hacky to me, and confusing for any developer maintaining a
> custom storage backend that doesn't support last modified.
>
> I disagree, modified_time is perfectly capable of handling your use case.
>

No it does not address my needs, as I described above.

Dan

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Feature request: collectstatic shouldn't recopy files that already exist in destination

2012-10-07 Thread Dan Loewenherz

This issue just got me again tonight, so I'll try to push once more on this
issue. It seems right now most people don't care that this is broken, which
is a bummer, but in which case I'll just continue using my working solution.

Dan

On Sat, Oct 6, 2012 at 10:48 AM, Dan Loewenherz <d...@dlo.me> wrote:

> Hey Jannis,
>
> On Mon, Oct 1, 2012 at 12:47 AM, Jannis Leidel <lei...@gmail.com> wrote:
>
>>
>> On 30.09.2012, at 23:41, Dan Loewenherz <d...@dlo.me> wrote:
>>
>> > Many backends don't support last modified times, and even if they all
>> did, it's incorrect to assume that last modified time is an accurate
>> heuristic for whether a file has already been uploaded or not.
>>
>> Well but it's an accurate way to decide whether a file has been changed
>> on the filesystem, and that's what collectstatic cares about. The storage
>> backend *is* the API to extend that when needed, so feel free to use it.
>>
>
> It's accurate *only* in certain situations. And on a distributed
> development team, I've run into a lot of issues with developers re-upload
> files that have already been uploaded because they just recently updated
> their repo.
>
> A checksum is the only true accurate method to determine if a file has
> changed.
>
> Additionally, you didn't address my point that I quoted from. Storage
> backends don't just reflect filesystems--they could reflect files stored in
> a database, S3, etc. And some of these filesystems don't support last
> modified times.
>
> > It might be a better idea to let the backends decide when a file has
>> been changed (instead of just calling the backend's last modified method).
>>
>> I don't understand, you can easily implement exactly that in the
>> last_modified method if you'd like.
>>
>
> This is a bit confusing...why call it last_modified when that's doesn't
> necessarily reflect what it's doing? It would be more flexible to create
> two methods:
>
> def modification_identifier(self):
>
> def has_changed(self):
>
> Then, any backend could implement these however they might like, and
> collectstatic would have no excuse in uploading the same file more than
> once. Overloading last_modified to also do things like calculate md5's
> seems a bit hacky to me, and confusing for any developer maintaining a
> custom storage backend that doesn't support last modified.
>
> Dan
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Feature request: collectstatic shouldn't recopy files that already exist in destination

2012-10-06 Thread Dan Loewenherz

Hey Jannis,

On Mon, Oct 1, 2012 at 12:47 AM, Jannis Leidel <lei...@gmail.com> wrote:

>
> On 30.09.2012, at 23:41, Dan Loewenherz <d...@dlo.me> wrote:
>
> > Many backends don't support last modified times, and even if they all
> did, it's incorrect to assume that last modified time is an accurate
> heuristic for whether a file has already been uploaded or not.
>
> Well but it's an accurate way to decide whether a file has been changed on
> the filesystem, and that's what collectstatic cares about. The storage
> backend *is* the API to extend that when needed, so feel free to use it.
>

It's accurate *only* in certain situations. And on a distributed
development team, I've run into a lot of issues with developers re-upload
files that have already been uploaded because they just recently updated
their repo.

A checksum is the only true accurate method to determine if a file has
changed.

Additionally, you didn't address my point that I quoted from. Storage
backends don't just reflect filesystems--they could reflect files stored in
a database, S3, etc. And some of these filesystems don't support last
modified times.

> It might be a better idea to let the backends decide when a file has been
> changed (instead of just calling the backend's last modified method).
>
> I don't understand, you can easily implement exactly that in the
> last_modified method if you'd like.
>

This is a bit confusing...why call it last_modified when that's doesn't
necessarily reflect what it's doing? It would be more flexible to create
two methods:

def modification_identifier(self):

def has_changed(self):

Then, any backend could implement these however they might like, and
collectstatic would have no excuse in uploading the same file more than
once. Overloading last_modified to also do things like calculate md5's
seems a bit hacky to me, and confusing for any developer maintaining a
custom storage backend that doesn't support last modified.

Dan

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Feature request: collectstatic shouldn't recopy files that already exist in destination

2012-09-27 Thread Dan Loewenherz

On Thu, Sep 27, 2012 at 4:13 PM, Carl Meyer <c...@oddbird.net> wrote:

> Hi Dan,
>
> On 09/27/2012 04:47 PM, Dan Loewenherz wrote:
> > Just updated the ticket.
> >
> > As I commented, the heuristic for checking if a file has been modified
> > lies in line 282 of collectstatic.py:
> >
> > *if not prefixed_path in self.copied_files:*
> > *
> > return self.log("Skipping '%s' (already copied earlier)" % path)
> >
> > *
> >
> https://github.com/django/django/blob/master/django/contrib/staticfiles/management/commands/collectstatic.py#L282
> >
> > This seems off, since a path may stay the same but a file's contents may
> > change.
>
> That's not checking whether the file has been modified, that's checking
> whether the same source file was previously copied in the same
> collectstatic run (due to overlapping/duplicate file sources of some kind).
>
> The check for modification date is up on line 234 in the delete_file
> method, which is called by both link_file and copy_file.
>

Thanks, I missed that.

I still see an issue here. In any sort of source control, when a user
updates their repo, local files that were updated remotely show up as
modified at the time the repo is cloned or updated, not when the file was
actually last saved by the last author. You then have the same scenario I
pointed to earlier: when multiple people work on a project, they will
re-upload the same files multiple times.

Don't get me wrong here--I'm happy to see there is some sort of check here
to avoid collectstatic'ing the same file, but I think it's warranted to
push back on the use of last modified as the heuristic. I think using a
checksum would work much better, and solves the problem this is trying to
solve in a much more foolproof way. With all this said, I don't think this
logic belongs in a "delete_file" method. IMO it'd be better to separate out
the logic of "does this file already exist, if so, skip it" from "delete
this file if it exists on the target" (delete_file).

@Karen--thanks for digging up that SO post. It actually is super relevant
here. As mentioned, when uploading files to S3, it's quite time consuming
to perform a network round trip to pick up file metadata (such as last
modified time) for each file you're uploading. This was initial reason I
chose to store this data in a single location that I'd only need to grab
once.

Dan

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Feature request: collectstatic shouldn't recopy files that already exist in destination

2012-09-27 Thread Dan Loewenherz

Just updated the ticket.

As I commented, the heuristic for checking if a file has been modified lies
in line 282 of collectstatic.py:

*if not prefixed_path in self.copied_files:*
*
return self.log("Skipping '%s' (already copied earlier)" % path)

*
https://github.com/django/django/blob/master/django/contrib/staticfiles/management/commands/collectstatic.py#L282

This seems off, since a path may stay the same but a file's contents may
change.

Also, the existing functionality doesn't work when multiple people are
running collectstatic (or the same person is running it on multiple
computers).

Dan

On Thu, Sep 27, 2012 at 3:12 PM, Karen Tracey <kmtra...@gmail.com> wrote:

> On Thu, Sep 27, 2012 at 12:51 PM, Dan Loewenherz <dloewenh...@gmail.com>wrote:
>
>> The problem I've run into is that collectstatic copies all files,
>> regardless of whether they already exist on the destination.
>
>
> No, as noted in the ticket, which has been closed needsinfo, staticfiles
> already only copies modified files. And I don't think that's a
> recently-added feature, so I suspect the behavior you are seeing has more
> to do with the S3 side than the staticfiles code. This post:
>
>
> http://stackoverflow.com/questions/6618013/django-staticfiles-and-amazon-s3-how-to-detect-modified-files
>
> looks enlightening though even that is not all that new. It sounds like
> the problem here is efficiently getting the last-modified time for the
> files out in S3, not with staticfiles.
>
> Karen
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Feature request: collectstatic shouldn't recopy files that already exist in destination

2012-09-27 Thread Dan Loewenherz

Hey all!

This is a feature request / proposal (one which I'm willing to build out,
given that I've already developed a solution for my own uploader).

I run a consulting business that helps small startups build initial MVPs.
When the time ultimately comes to deciding how to store static assets, my
preference (as is that of many others) is to use Amazon S3, with Amazon
CloudFront acting as a CDN to the appropriate buckets. For the purposes of
this ticket, s/S3/your object storage of choice/g.

Now, Django has an awesome mechanism for making sure static assets are up
on S3. With the appropriate static storage backend, running ./manage.py
collectstatic just searches through your static media folders and copies
the files.

The problem I've run into is that collectstatic copies all files,
regardless of whether they already exist on the destination. Uploading
5-10MB of files is pretty wasteful (and time consuming) when none of the
files have changed and no new files have been added.

As I wrote in the trac ticket (https://code.djangoproject.com/ticket/19021),
my current solution was to write a management command that does essentially
the same thing that collectstatic does. But, there are a few differences.
Here's a rundown (copied straight from the trac ticket).

I currently solve this problem by creating a file containing metadata of
> all the static media at the root of the destination. This file is a JSON
> object that contains file paths as keys and checksum as values. When an
> upload is started, the uploader checks to see if the file path exists as a
> key in the dictionary. If it does, it checks to see if the checksums have
> changed. If they haven't changed, the uploader skips the file. At the end
> of the upload, the checksum file is updated on the destination.


I'll contribute the patch. I know there is not a lot of time before the
feature freeze, but I'm willing to make this happen if there's interest.

If we don't want to change behavior, perhaps adding a flag such as
--skip-unchanged-files to the collectstatic command is the way to go?

All the best,
Dan

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Model inheritance extended.

2012-09-25 Thread Dan Fairs

> On 24 syys, 16:55, Jonathan Slenders <jonathan.slend...@gmail.com>
> wrote:
>> This may be interesting to some of you. I created a small library for
>> inheritance of *a set of* models.
>> It's best to go quickly through the Readme on the site below.
>> 
>> We felt a need for this, but I'm wondering whether some kind of inheritance
>> like this has been discussed before. And whether, if useful, this would
>> make a candidate for django.db.
>> 
>> https://github.com/citylive/django-model-blueprint
> 
> Looks interesting! It is nice to see how little code you needed to
> implement this. Inheriting a system of models should be useful in some
> special circumstances. I know I have missed that feature at least
> once...
> 

fwiw, I've implemented something like this in another project, as well. It did 
more stuff (model partitioning for large data volumes, handling the case of 
deriving from generated base classes, shared models between 'blueprints'), but 
the core of yours looks pretty neat.

Cheers,
Dan

--
Dan Fairs | dan.fa...@gmail.com | @danfairs | www.fezconsulting.com

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Ticket #18855: persist socket in runserver

2012-08-25 Thread Dan LaMotte

First time contributing to Django, but I've used it for a few years now. 
 I'd like to enhance to the dev runserver to persist a socket through 
autoreloads.  I've submitted a ticket and a pull request on Github is 
linked in the ticket.

If someone wouldn't mind reviewing the code and giving me some feedback, 
I'd like to help in any way I can to get this into Django.

With this change, the default behavior of runserver is changed, but the 
user is allowed to opt-out passing "--nopersistsocket" to runserver.

Thanks for your time.
- Dan LaMotte

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/veb_dBZ_YNoJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

django.core.urlresolvers swallowing AttributeErrors

2012-08-14 Thread Dan Passaro

Hello,

I was working on a project and noticed that urlresolvers.py would swallow 
attribute errors in user code.

The way the get_callable function is written, the line

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/0KLOMKSwDV8J.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Proposal: unique by default on SlugFields

2012-06-27 Thread Dan Watson

On Wednesday, June 27, 2012 7:15:03 AM UTC-4, Meshy wrote:
>
> Perhaps it's just me, but I've very rarely wanted a SlugField that wasn't 
> unique. Would this not be a sensible default? I realise that a lot of apps 
> will rely upon this default, but objectively speaking would this not be 
> better? Perhaps this change would be appropriate for django 2.0.
>
> At the moment, slug = models.SlugField()creates a non-unique field, and 
> if you want it to be unique, then you must add unique=True. I feel this 
> is wrong.
>
> It seems to me that unique should be default, and if you don't want a 
> unique slug, you should explicitly state that:
> slug = models.SlugField(unique=False)
>

I'd agree that slugs are often unique, but I think it's just as common that 
they are unique together with another field (think blogs -- 
/blog/2012/06/27/some-slug/ -- where the slug is unique_together with the 
date). For this case, you'd have to remember to specify unique=False *and* 
unique_together.
 

> I've added an issue on the tracker for this: 
> https://code.djangoproject.com/ticket/18525
>
> I realise this may be a contentious issue...or that I may even get shot 
> down in flames on this one ;P Go easy on me! :)
>
> What do you all think?
>

It doesn't seem like a burden to have to say unique=True if you truly want 
a singularly unique slug. Making it explicit, aside from being good python 
zen, also helps to see the natural keys of your model at a glance.

Dan

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/8V3D_HcVhf8J.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: FYI: Trac downtime today

2012-04-25 Thread Dan McGee

I'm unable to attach files, although ticket creation worked fine. Here is 
the error page:

Oops… 
*Trac detected an internal error:* 

OSError: [Errno 13] Permission denied: 
'/home/trac/code.djangoproject.com/trac-env/attachments/ticket/18210'

 There was an internal error in Trac. It is recommended that you notify 
your local Trac administrator with the information needed to reproduce the 
issue.
 
To that end, you could  a ticket.
 
The action that triggered the error was:

POST: /attachment/ticket/18210/



On Wednesday, April 25, 2012 10:32:58 AM UTC-5, Jacob Kaplan-Moss wrote:
>
>  OK, done! Thanks for your patience. 
>
> Jacob
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/y3VccW9ge5AJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Migrating to 1.4

2012-04-16 Thread Dan Fairs

Hi,

>> Both of these were fixed by using the new implementations from Django
>> itself, and re-applying our small changes.
>> 
>> Should I raise backwards-incompatibility tickets for these? They're pretty
>> obscure, and I can't imagine many people are doing them. I raised #17891
>> because the code involved a clear extension point.
> 
> Thanks for sharing this story, Dan! It's great to hear. Let us know
> about the performance differences (if any), too.


Well, we're finally getting into the performance side of things now!

The first big regression for us was that our test suite took 20% longer to run. 
I traced this down to the new default password hasher. This is clearly by 
design - and we'll just use the PASSWORD_HASHERS setting to use a faster (and 
much less secure) hasher for test runs.

Fortunately our app uses an external authentication service for most of its 
users, so this won't affect us in production - but for those running sites with 
high signup rates, this could be a surprise. Is that worth a mention in the 
release notes?

Cheers,
Dan
--
Dan Fairs | dan.fa...@gmail.com | www.fezconsulting.com

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

1 2 >

1 - 100 of 163 matches

Mail list logo