Re: Model Generation for CSV, XLS Files

[Jon Dufresne]

Is django-import-export at all along the lines of what you're looking for?

https://github.com/django-import-export/django-import-export

> django-import-export is a Django application and library for importing
and exporting data with included admin integration.
> ...
> support multiple formats (Excel, CSV, JSON, ...)

On Wed, Nov 25, 2020 at 11:53 AM Muskan Vaswan 
wrote:

> Hi everyone,
> I am Muskan and am very new to this community however not that new to
> django itself. Contributing to django is something I would really like to
> do, and I might also be participating in GSoc for the same.
>
> I have an vague idea of what I want to fix, because I myself have used
> django and just want to add things as a developer that I would've wanted as
> a user. So my question is understanding based around this idea of mine.
>
> *I want to know what functionality already exists that makes it easier to
> directly load a large data set into a django model from a CSV file (going
> with the simplest format or now).* When I had to do it as a user it took
> me quite a while and was a lot more complicated than I had expected, after
> being used to smooth transitions with load data. I could not find any
> better methods to do it, if there indeed exists no other methods, this is
> something I would like to work on So this is just to confirm if my
> research was thorough enough (very possibly wasn't).
>
> Thank you! I'm excited to begin helping out!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/a3946181-cd91-4b83-b9d6-d8d8f786b6acn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b4Mx902FKtRuB1KHke_7GKSX4e%2BmV%2Bij579WPVN6ec2rg%40mail.gmail.com.

Re: HttpResponse headers interface

[Jon Dufresne]

>
> I would prefer "one right way to do it", but I also don't see a compelling
> reason to deprecate the old interface.
>

For me, the reason to eventually deprecate the old interface is to help new
developers learning a new codebase. When these developers have been taught
to use response.headers but inevitably happen upon the old interface, there
will be a period of head scratching, diving into docs, code, whatever to
learn what this means only to come to the conclusion that it is an alias.
To me, that is an experience that could be improved.

The other benefit I see in deprecating the old interface is that new
developers don't need to learn or care how this fits in a particular
organization's coding style. Perhaps the organization requires using only
response.headers. Someone that uses the old interface out of habit will
need to unlearn this habit and perhaps deal with the frustration that comes
up during a review to fix it.

Consolidating to only a single interface would resolve both of these, IMO.
I'm quite happy to see the deprecation occur over a longer time than
normal, so I think the "may be deprecated in the future" is a good choice
to start and then we can act only after we feel enough time has passed.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b6WwZpW_P9cNePMpxnrc1xVQwqWFhM4J0ROqUNiy%3Dh2AQ%40mail.gmail.com.

Re: HttpResponse headers interface

[Jon Dufresne]

> I don't see a reason to deprecate it at all just now (though perhaps in
_my_ ideal world that would happen at some point), but I'm not sure if it's
worth keeping the current interface in the documentation at all?

IMHO, we should eventually take the advice from the zen of Python "There
should be one-- and preferably only one --obvious way to do it.". While we
should not take this as dogma, I do think it is generally wise.

If there are increased concerns about existing projects, perhaps we could
delay the initial deprecation or apply some kind of extended deprecation
period that would allow projects more time to migrate. Removing the old
interface from the docs is a great first step.

But ultimately, I think having two interfaces to solve the same tasks
confuses new library users and makes project coding styles more difficult
than necessary.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b6qijpNCCv_M17%2B11RTUGG-MA3AG%3Dhe3RccMrciV%3DNHTA%40mail.gmail.com.

Re: Removing url() ?

[Jon Dufresne]

+1 for deprecating for eventual removal.

On Tue, May 5, 2020 at 9:33 AM Mariusz Felisiak 
wrote:

> Hi,
>
>I think it's worth to deprecate url() and remove it in Django 4.0
> (that's why I accepted this ticket). Aymeric proposed Django 3.1 as a
> beginning of deprecation period in a discussion about DEP 201 [1], it was
> three years ago. Moreover we removed it from the docs with the new
> implementation [2] (also three years ago) and only versions with re_path()
> are now supported. Also, Django 3.2 is a LTS supported until April 2024 and
> it will contain url().
>
> Best,
> Mariusz
>
> [1]
> https://groups.google.com/d/msg/django-developers/D44LSp0bPg8/hKybIqNiBAAJ
> [2]
> https://github.com/django/django/commit/df41b5a05d4e00e80e73afe629072e37873e767a
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/85de5898-ee51-44d8-b20b-dbe3a8865996%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b71kPY%3DTCnmKNv%2BYK1mKvs7HPT%2BB6D30%3Dj3RC0s%2BE8Nog%40mail.gmail.com.

Re: Ticket #25236: Remove ifequal from the template language

[Jon Dufresne]

Hi,

I'd like to raise this topic for renewed discussion. I think it is time to
begin deprecating the obsolete template tags. So +1 for removal.

I had all but forgotten about the {% ifequal %} template tag and then
eventually stumbled across it doing unrelated work. Upon rediscovering it
as a practical duplicate of {% if %}, I thought I'd help out Django by
deprecating it for eventual removal so that we're back to "one -- and
preferably only one -- obvious way to do it". I created the PR to begin the
deprecation here: https://github.com/django/django/pull/12851

We're now coming up on 10 years since Django 1.2 was released with the
smart if template tag. Projects and third party apps have had a generous
extended time to update their code.

For projects that don't follow Django releases in detail, they may not be
aware that the {% ifequal %} tag is obsolete and so have not had any
motivation to change. Adding a deprecation warning may help such projects
to understand they should update their templates.

For projects that find it is a burden to update, they've had 10 years to do
so. I think adding more time isn't what they need at this point. The actual
upgrade can be mostly automated with sed (I could even put a recommended
command in the release notes). So the update doesn't need to be very time
consuming.

While perhaps we agree leaving old code around can mostly go unnoticed,
technical debt has subtle ways of showing up. For example, time was spent
by me rediscovering this feature exists and wondering why there are two way
to write {% if ... %} (this could happen to other contributors studying the
template tag system). Time has been spent on this mailing list discussing
what to do with old unused code. Its tests -- which mostly don't matter at
this point -- run on every PR and in local environments. If there is ever
an attempt at an internal refactoring of the template code, the ifequal tag
will need to be considered and maintained. And so on.

On Sun, Aug 9, 2015 at 12:37 PM Collin Anderson 
wrote:

> Hi All,
>
> I really like the "don't use this on new code, but there's no rush in
> replacing it" category. I think that's really important to have that in a
> project that's this old. I think it would be great to minimize the amount
> required changes that need to be done.
>
> Thanks,
> Collin
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/2ddf20f1-af1a-47c1-b07c-669818846b53%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b5%3D3A03OjPTGa7mUgm-7braWgJwLUU_xW5pFJrgPP6Uow%40mail.gmail.com.

Re: Remove automatic date-naming of migrations (00XX_auto_YYYMMDD)

[Jon Dufresne]

> I'd like to propose using this new full coverage of operation naming to
remove the "auto_MMDD" behaviour

+1 on this proposal. As you've noted in your blog post, I find the default
auto name extremely opaque. Projects I work on normally have an internal
policy to change these names to something more informative. Inevitably,
this means there is always a bit of a learning process for new team
members. If Django handled this more automatically, it would remove some of
this learning process.

> and instead always combine operations' "suggested migration names"

This sounds reasonable to me. If there is concern about words blending into
one another, we could separate them by a double underscore:
0026_remove_book_title__add_book_description. I would personally be fine
with either.

> up until a limit of say 60 characters

Did you have a suggestion for this situation? Revert back to auto-naming or
request the user to name the migration?

On Wed, Apr 22, 2020 at 6:06 AM Adam Johnson  wrote:

> Hi djangonauts,
>
> In a blog post earlier this year I outlined my technique to prevent Django
> creating migration files with automatic date names. I had a lot of response
> with other techniques and ended up adding two more techniques to the post.
> It's at
> https://adamj.eu/tech/2020/02/24/how-to-disallow-auto-named-django-migrations/
> .
>
> The problem with such migration names:
>
> When you run Django’s manage.py makemigrations, it will try to generate a
>> name for the migration based upon its contents. For example, if you are
>> adding a single field, it will call the migration 0002_mymodel_myfield.py.
>> However when your migration contains more than one step, it instead uses a
>> simple ‘auto’ name with the current date + time, e.g.
>> 0002_auto_20200113_1837.py. You can provide the -n/--name argument to
>> makemigrations, but developers often forget this.
>>
>> Naming things is a known hard problem in programming. Having migrations
>> with these automatic names makes managing them harder: You can’t tell which
>> is which without opening them, and you could still mix them up if they have
>> similar names due to being generated on the same day.
>>
> Django *currently* sets the migration name to something other than the
> automatic date name in two cases:
>
>- If the migration contains a single operation, it uses a name based
>on that operation, e.g. 00023_remove_model_field, or 0024_add_model_field
>(but not for all operation types)
>- If the migration consists *only* of model creation operations, it
>combines their operation names names, which come as just the lower-cased
>model names e.g. 0025_book_author_sale
>
> I opened a PR to expand the operation naming for the single case to cover
> all built-in operations: https://github.com/django/django/pull/12131
>
> I'd like to propose using this new full coverage of operation naming to
> remove the "auto_MMDD" behaviour, and instead always combine
> operations' "suggested migration names" up until a limit of say 60
> characters. I made a commit for that here:
> https://github.com/adamchainz/django/commit/c2bc6893a05c2c8099e1fb68e688618446641ed6
>
> This would lead to migration names such as:
>
>- 0026_remove_book_title_add_book_description
>
> Whilst perhaps long, they're explict and imo easier to work with than the
> auto_MMDD behaviour.
>
> Mariusz wrote on the PR:
>
> Personally, I'm against removing auto named migrations. IMO chaining
>> operation names is even more confusing.
>>
> --
> Adam
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAMyDDM10ACO6wEMZXukN6_xy766Bxa8PQOc7teFEQhRmzWxqwA%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b4-LHwmKpMTwO2%2BFwLSUCKYwPzwQs28fbH5edPdqjNGRw%40mail.gmail.com.

Re: Use "raise from" where appropriate, all over the codebase

[Jon Dufresne]

> > I think it's rare enough that personally, I would have liked to have a
> > Flake8 / PyLint rule like that enforces it, and allow ignoring it
> > with a comment. (As much as I hate Lint ignore comments.)
>
> That makes sense. And you know, flake8 supports plugins... a couple of
> web searches and "pip search flake8 | grep {raise,from,chain}" didn't
> pull anything which seems relevant, but if so, it can be written.
>

As a weekend project, I wrote a flake8 plugin to handle this at:
https://pypi.org/project/flake8-raise/

I'm not advocating this necessarily included by the Django test suite, but
it can be used to review the PR. Either way, it was fun to write.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b4m40efQkorgji9-erXri9qSAb1VJBv%3DZ3HrOL6ksOJAQ%40mail.gmail.com.

Re: Use "raise from" where appropriate, all over the codebase

[Jon Dufresne]

+1 on chaining exceptions. I think the information is useful.

> Is there anything we can do to control the way python displays them?

I don't think we should do anything non-standard to display exceptions.
Over time, Python programmers have become accustomed to how these
exceptions are displayed. To alter that would be to introduce something
unfamiliar from all their other Python debugging experiences.

If exception displaying should be improved, I think the effort should be
upstream to benefit the larger Python community.

> And how would we ensure the format is kept going forwards? Is there a
flake8 rule/plugin we could activate to enforce it?

At the very least, we could add a note in the code style guidelines that
chained exceptions should normally be used.

Cheers

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b41caYYAXS%3DypOs6wCQ%3D%2BcSE%3D53Uxk6ia4009ZOo8XiNA%40mail.gmail.com.

Re: remove_stale_contenttypes doesn't remove entries for renamed apps.

[Jon Dufresne]

An optional command line argument makes sense to me. +1

On Thu, Jan 2, 2020 at 2:11 PM Adam Johnson  wrote:

> I guess an optional kwarg would be okay then.
>
> On Thu, 2 Jan 2020 at 03:08, Javier Buzzi  wrote:
>
>> @adam I agree with your points, about data loss, but this can still see
>> this as being beneficial, perhaps the approach was just too harsh. Perhaps
>> adding a flag in the management command would get everyone on board? The
>> flag being off by default and only turns on if you know what you’re doing
>> and enable it. At any rate i believe from what i can i see in the code it
>> will still prompt you to delete the items it finds unless you added the
>> ——no—input which prevents helps with data loss.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers  (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/5c55fdaa-8d89-4baa-a08d-68b0e84c610c%40googlegroups.com
>> .
>>
>
>
> --
> Adam
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAMyDDM3Hu_ZKWUj46%3D0BYN8XECG0OT_kMLNUgaopKXhJOEfJ5Q%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b47-4UMxEQjG147dMzSbHijL6a8v8Ue8mxrL8A2B1ShWA%40mail.gmail.com.

Re: Redis cache support in core

[Jon Dufresne]

The django-redis maintainer is currently in the process of stepping down
and is moving away from the project. Should we propose moving it to
Django's GitHub? For details, see
https://github.com/niwinz/django-redis/issues/400

On the other hand, if there is interest in supporting a Redis backend in
core, given django-redis's status, now might be a good time to consider it.
As long as memcache is in core, I'm +1 on including Redis as well.

On Thu, Sep 5, 2019 at 7:40 AM Aymeric Augustin <
aymeric.augus...@polytechnique.org> wrote:

> Hello,
>
> This is an interesting point — and one we tend to forget too often.
>
> The documentation for memcached says that it doesn't support Windows:
> https://github.com/memcached/memcached/wiki/Install#installation
>
> The documentation for redis says that it supports Windows, with some
> limitations, and that there's an effort by Microsoft to lift these
> limitations:
>
> https://redislabs.com/ebook/appendix-a/a-3-installing-on-windows/a-3-1-drawbacks-of-redis-on-windows/
>
> So the Windows situation seems slightly better with Redis than with
> memcached, even if it isn't ideal.
>
> Best regards,
>
> --
> Aymeric.
>
> Le jeu. 5 sept. 2019 à 15:29, Matthew Pava  a
> écrit :
>
>> I’d just like to point out that Redis support on Windows is limited at
>> best. All other technologies that Django uses, as far as I can recall, do
>> support Windows.
>>
>>
>>
>> *From:* django-developers@googlegroups.com [mailto:
>> django-developers@googlegroups.com] *On Behalf Of *Jacob Rief
>> *Sent:* Thursday, September 5, 2019 1:33 AM
>> *To:* django-developers@googlegroups.com
>> *Subject:* Re: Redis cache support in core
>>
>>
>>
>> I'm also in favor on having it as part of the core, since memcache is
>> also supported.
>>
>>
>>
>> One of the nice features Redis provides, is the possibility to invalidate
>> one or more cached object by using a wildcard key.
>>
>> It namely is the method delete_pattern() added by django-redis-cache to
>> the given Django caching backend. That
>>
>> (or a similar method) then should be part of the other Django caching
>> backends as well, such as the dummy cache or in-memory cache.
>>
>>
>>
>> Jacob
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/CAJ_HXxo9jpr-%3DFmLXcYwwR6tdn-NVDfrEoo-qYnB-vQwKv9O%3DA%40mail.gmail.com
>> 
>> .
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/b9357775f9684aa888091bce83f2852b%40iss2.ISS.LOCAL
>> 
>> .
>>
>
>
> --
> Aymeric.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CANE-7mU9m23CWkGrvEETxOhyCFTsU-uP1dS0qnn%3DvPXfemTdqg%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b4rFPmVSUHFJ2%3DMpPimzCdsRnSB%2BufgH7FSiPer6mbmAw%40mail.gmail.com.

Re: Proposal to format Django using black

[Jon Dufresne]

> One thing we have not considered here is that after running black on
Django a huge portion of our outstanding merge requests will have
conflicts, some of which might be tricky to rebase. I’m not sure there is
much we can do about that though.

With a little bit of git-foo, this is actually not that bad. In past
projects, this is the approach I took:

$ git checkout my-branch
# Rebase on the parent of the commit introducing Black.
$ git rebase ^
# Fix normal merge conflicts.
# ...
# Rebase on the Black commit, instructing git to prefer the rebased
changes
# over the black changes. For rebase, the terminology is backwards,
hence
# "theirs".
$ git rebase  -X theirs
# Reformat your changes with black.
$ black .
# Amend the commit with the Black formatting changes.
$ git commit --ammend -a

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b7JD6C-wqyKgPVDK0JOvX_W_vgzAe%2Be6TcSkC6ArPm5jA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Proposal to format Django using black

[Jon Dufresne]

> 2. there are issues with git history in doing "the great commit".

Tools make a real difference here. I know everyone has their preferred
tools and workflows, but there are tools that let one move through
iterations of git blame. This is something I do quite frequently. Just one
example, in Emacs the vc-annotate command (bound to "C-x v g") shows a view
of git blame. The user can hit "a" to go to the previous diff the line
appeared on which might be more interesting. So one could use this feature
to skip over the Black commit on to a more interesting one. This is just a
single example. Florian's suggested tools may handle this even more
elegantly, but I haven't used them. This only goes to show that the history
and git blame concerns don't need to be a show stopper and can be solved.
The feature people are asking for, "skipping over" git commits in blame,
does exist for some tools. It even exists in GitHub's blame view. The
button is labeled "View blame prior to this change".


On Sun, Apr 14, 2019 at 1:22 AM Curtis Maloney  wrote:

> So to summarise the discussion so far:
>
> 1. automated code formatting will be a great boon - reduce work, lower
> barrier for new committers, reduce work for the fellows, etc.
>
> 2. there are issues with git history in doing "the great commit".
>
> 3. there are issues with black's formatting choices.
>
>
> So to address 1:
>
> I am entirely +1 for automated code formatting to take the work out of
> our hands.
>
> Can such a tool be automated into, say, github in a way that doesn't
> create extra commit noise?
>
> To address 2:
>
> I side with those who favour a progressive solution, whereby new code
> only has the new tool applied.
>
> That said, there might be cause to suggest a deadline [perhaps a N.0
> release?] where all remaining code is "cleaned".
>
> And finally 3:
>
> My perspective on the goal of any code formatting tool is this:
>
> When we as developers approach a piece of code, our goal is the
> understand its intent and its implementation of that intent.
>
> In the process of reaching that, we pass through the stages of (a)
> identifying the relevant code, (b) understanding the action of the code,
> and (c) understanding the intent of the code.
>
> Good code formatting choices will remove or reduce the cognitive load in
> (b) and (c).
>
> In my experience with using black [we use it at work], there are
> numerous choices (including those demonstrated in this list already)
> where it can significantly _increase_ the cognitive load in simply
> parsing the code.
>
> As simple as black can make the job of code formatting, I feel I'd
> rather see a different tool that retained the benefits of "trivial code
> reformatting", but still allowed us to retain some of Django's existing
> code formatting rules.
>
> (An interesting [and defensible] choice, we had a module with a lot of
> strings wrapped across lines. black opted to push them onto the same
> line, but NOT to merge them.  This is because in Python prior to 3.7, it
> would have altered the generated AST - one of the guides black uses)
>
> --
> Curtis
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/56392596-0246-b62b-4725-628a2b0801ae%40tinbrain.net
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b4Udk1ooc0zdagMPecx-ZGRk%2B01dGCGVr%2BjYOA8vFp8Ag%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Proposal to format Django using black

[Jon Dufresne]

Big +1

I've used black in quite a few projects now and found it has greatly
reduced the mental energy involved in code formatting while aligning all
contributors expectations.

I'm in favor of adding no configuration. That is, use black's default line
length and string delimiters. IMO, the less configuration the better. The
more the Django code style aligns with the larger Python community, the
better and easier it is few new contributors. But these secondary choices
are less important to me than the improvement of adopting a tool to
auto-format code.

On Sat, Apr 13, 2019 at 8:35 AM Herman S  wrote:

> Hi.
>
> I propose that Django starts using 'black' [0] to auto-format all Python
> code.
> For those unfamiliar with 'black' I recommend reading the the projects
> README.
> The short version: it aims to reduce bike-shedding and non value-adding
> discussions; saving time reviewing code; and making the barrier to entry
> lower
> by taking some uncompromissing choices with regards to formatting.  This is
> similar to tools such as 'gofmt' for Go and 'prettier' for Javascript.
>
> Personally I first got involved contributing to Django couple of weeks
> back,
> and from anecdotal experience I can testify to how 'formatting of code'
> creates
> a huge barrier for entry. My PR at the time went multiple times back and
> forth
> tweaking formatting. Before this, I had to research the style used by
> exploring
> the docs at length and reading at least 10-20 different source – and even
> those
> were not always consistent. At the end of the day I felt like almost 50%
> of the
> time I used on the patch was not used on actually solving the issue at
> hand.
> Thinking about code formatting in 2019 is a mental energy better used for
> other
> things, and it feels unnecessary that core developers on Django spend
> their time
> "nit-picking" on these things.
>
> I recently led the efforts to make this change where I work. We have a
> 200K+
> LOC Django code-base with more than 30K commits. Some key take-aways: it
> has
> drastically changed the way we work with code across teams, new engineers
> are
> easier on-boarded, PR are more focused on architectural choices and "naming
> things", existing PRs before migration had surprisingly few conflicts and
> were
> easy to fix, hot code paths are already "blameable" and it's easy to blame
> a
> line of code and go past the "black-commit", and lastly the migration went
> without any issues or down-time.
>
> I had some really fruitful discussions at DjangoCon Europe this week on
> this
> very topic, and it seems we are not alone in these experiences. I would
> love to
> hear from all of you and hope that we can land on something that will
> enable
> *more* people to easier contribute back to this project.
>
> I've set up how this _could_ look depending on some configurables in Black:
>
> * Default config: https://github.com/hermansc/django/pull/1
> * Line length kept at 119: https://github.com/hermansc/django/pull/3
> * Line length kept at 119, no string normalization:
> https://github.com/hermansc/django/pull/2
>
> Please have a look at the Black documentation. It explains the benefits
> better
> than I possibly could do here.
>
> With kind regards,
> Herman Schistad
>
> [0]: https://github.com/ambv/black
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAN%3DnMTx0EE5WfXuccv_e3MBuCxp9u_pAV_ow5MxNST6MptTDBw%40mail.gmail.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b4K6ox%3DZ2YPqF%2BWpxGAFkdF1gjViudJ9QhkVn7UKjy%3DRA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Can we remove FILE_CHARSET?

[Jon Dufresne]

> Is that always available these days? (I'd guess yes.)

I too would guess yes. I believe any reasonably modern text editor will
support
UTF-8 and even likely default to saving in that encoding. I know mine does.

> Is is something we want to impose? Not sure. Are there people doing
> otherwise? (No idea.)

For templates, it wouldn't be imposed. Users can still override the template
engine's encoding with the 'file_charset' option.

For static files, without imposing it, we're back to the third-party app
concern. Just like DEFAULT_CHARSET, it would difficult to change
FILE_CHARSET
_and_ integrate third party apps. The third party apps have likely encoded
their static files using UTF-8, so setting FILE_CHARSET to some other value
will break.

Cheers,
Jon


On Wed, Oct 3, 2018 at 12:14 PM Carlton Gibson 
wrote:

> Thanks for the follow-up Jon.
>
> I'll let Vasili follow-up on his use-case if possible/relevant.
>
> TBH I'm not at all sure about the SQL data files bit, which is in part why
> I asked here.
> (Encoding issues!)
>
> > Maybe that sentence should be rephrased to "template
> files, static files, and translation catalogs".
>
> OK, so IF it's just this, then I'm on Windows doing development in UTF-8
> no problem (and can't really envisage doing much different as it stands)
> but:
>
> * Is that always available these days? (I'd guess yes.)
> * Is is something we want to impose? Not sure. Are there people doing
> otherwise? (No idea.)
>
> (If we can drop a setting, that'd be )
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/13cc5f04-967b-4f53-92f6-cbe155014edc%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b4-7vc-hZgKP_%2BYOmUkbRC%2B8r7i2t74d0-9b0vGrVcmkA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Adding a database-agnostic JSONField into Django

[Jon Dufresne]

> Anyone know of an SQLite implementation?

A quick search shows the JSON1 extensions exists:
https://www.sqlite.org/json1.html

According to the release history (https://sqlite.org/changes.html) this was
added in version 3.9.0 (2015-10-14).

But I have no direct experience working with it.

On Thu, Oct 4, 2018 at 12:19 AM Carlton Gibson 
wrote:

> This has come up again, with an example implementation for Oracle:
>
> https://code.djangoproject.com/ticket/29821
>
> Anyone know of an SQLite implementation?
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/215a0e1c-6947-4375-ba1d-38d310c32411%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b4dAkdiOeZFxEqGYGj6DYrUrYFVaGT9rewrhVr_Ha%2B3MA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Can we remove FILE_CHARSET?

[Jon Dufresne]

> So Jon, are you basically saying that Vasili's concern shouldn't come up?

Yeah, I think it shouldn't come up. But I'm not sure I fully understand
Vasili's concern . Maybe if it was more specific with more details, I could
better understand it.

Django's documentation states:

https://docs.djangoproject.com/en/dev/ref/unicode/#creating-the-database

> Make sure your database is configured to be able to store arbitrary string
> data. Normally, this means giving it an encoding of UTF-8 or UTF-16. If
you
> use a more restrictive encoding – for example, latin1 (iso8859-1) – you
won’t
> be able to store certain characters in the database, and information will
be
> lost.
>
> ...
>
> All of Django’s database backends automatically convert strings into the
> appropriate encoding for talking to the database. They also automatically
> convert strings retrieved from the database into strings. You don’t even
need
> to tell Django what encoding your database uses: that is handled
> transparently.

So, if these non-UTF-8 articles are stored in the database, this doesn't
involve FILE_CHARSET. Are the articles stored as text or binary data? If
text,
this violates existing Django documentation & assumptions. The database is
expected to be configured for UTF-8. If binary data, then the project's code
will be responsible for decoding it to a text string.

If, on the other hand, these articles are stored as files, how are they
being
loaded? If they are being loaded through a Django code path, which one such
that FILE_CHARSET is involved? Or, are these articles loaded by project code
such that the encoding can be specified.

So, IIUC, it doesn't seem like FILE_CHARSET should be involved for this use
case.

> That the whole "SQL data files" bit is misleading...?

I was unable to find any code with an interaction between FILE_CHARSET &
"SQL
data files". If it exists, do you have a link? I think this text may be
outdated or obsolete. Maybe that sentence should be rephrased to "template
files, static files, and translation catalogs".


On Wed, Oct 3, 2018 at 7:55 AM Carlton Gibson 
wrote:

> Thanks for the input everyone.
>
> So Jon, are you basically saying that Vasili's concern shouldn't come up?
> (That the whole "SQL data files" bit is misleading...?)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/6d555b4d-0b38-452c-8f40-8e690f9b33a6%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b6JU3oVYovQHG%3DO1oz0Ntw1bHTWF_iOkJcnrVPSBQh36w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Can we remove FILE_CHARSET?

[Jon Dufresne]

I'm the one that proposed this setting be removed.

The settings is used in the following areas:

> ./django/template/backends/django.py:23:
options.setdefault('file_charset', settings.FILE_CHARSET)

I suppose this is its main use case. The Django template engine defaults to
loading files from disk using the encoding specified by FILE_CHARSET. If a
project needs to load templates using a different encoding, it can continue
to do so by specifying an OPTION in the TEMPLATES setting:

TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'OPTIONS': {
'file_charset': 'latin1',
},
},
]


> ./django/core/management/commands/makemessages.py:106:encoding =
settings.FILE_CHARSET if self.command.settings_available else 'utf-8'

The makemessages management command loads files to preprocess using the
encoding specified by FILE_CHARSET.


> ./django/contrib/staticfiles/storage.py:287:content =
original_file.read().decode(settings.FILE_CHARSET)

The HashedFilesMixin loads files to preprocess using the encoding specified
by FILE_CHARSET.


> ./django/template/backends/dummy.py:31:with
open(template_file, encoding=settings.FILE_CHARSET) as fp:

The dummy template backend loads files using the encoding specified by
FILE_CHARSET. This dummy backend is used for internal testing purposes only
and is not a documented or public API. So I think it could safely be
modified without affecting projects or users.


That's it!

I think this setting has the same issue that was identified by
DEFAULT_CONTENT_TYPE. That is, if a projects sets FILE_CHARSET to a
different value, interactions with third-party apps may be problematic. The
third-party app likely encode templates and static files using UTF-8 so the
use cases above may not work properly.

Projects using a different encoding will still have a deprecation period to
see the Django warnings, adjust the setting, and re-encode files. The
removal won't be immediate. If such projects re-encode files to UTF-8
early, the projects will be both backwards and forwards compatible with
current and future Django versions.

FWIW, I was unable to find examples of a changed FILE_CHARSET by searching
GitHub.

Using a different value for FILE_CHARSET is currently untested internally
(although I believe it works as designed).

On Wed, Oct 3, 2018 at 5:03 AM Claude Paroz  wrote:

> We are not talking about general data encodings here, FILE_CHARSET is used
> to read Django text files from disk (template files, static files (css, js)
> or translation catalogs). So the question is mainly about encoding usage in
> text editors.
>
> Claude
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/ba98d239-479f-4b21-b899-8c9b39b921a3%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b5Zz9zLOKVZQOJjGNogh%3DYVpAVP0-tWbeTkj3WRr6UrZg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: HTML5 and XHTML5 documents

[Jon Dufresne]

Django started adopting HTML5 features since before 2.1. I found changes as
early as 1.11. From the release notes:

https://docs.djangoproject.com/en/dev/releases/1.11/#miscellaneous

> The checked and selected attribute rendered by form widgets now uses
HTML5 boolean syntax rather than XHTML’s checked='checked' and
selected='selected'.

FWIW, I prefer the concise syntax of HTML5 over XHTML and seems to be the
de facto preferred style across much of the web. Style guides such as
Google's recommend it. From
https://google.github.io/styleguide/htmlcssguide.html#HTML:

> Document Type
>
> Use HTML5.
>
> HTML5 (HTML syntax) is preferred for all HTML documents: .
>
> (It’s recommended to use HTML, as text/html. Do not use XHTML. XHTML, as
application/xhtml+xml, lacks both browser and infrastructure support and
offers less room for optimization than HTML.)
>
> Although fine with HTML, do not close void elements, i.e. write , not
.

Cheers,
Jon


On Sun, Aug 12, 2018 at 10:34 AM Nils Fredrik Gjerull 
wrote:

> Den 12. aug. 2018 12:20, skrev Adam Johnson:
>
> You didn't pose an exact question in your message, but I guess it's "Can
> Django officially support XHTML5?"
>
>
> Django has supported XHTML for years, so it came as a surprise to me that
> Django 2.1 broke it. So it is more like "Can Django continue to support
> XHTML, even for HTML5?". But the principle is the same.
>
>
> As it's a stricter subset of HTML5, it seems a valid idea. But at the same
> time, we need to be able to support it going forwards, with test coverage
> and knowledge that this is what we aim for on every PR that adds HTML. Also
> I doubt you'll find many third party packages support XHTML5 too.
>
>
> I use a number of third party libraries, for my site that uses XHTML5. I
> had to patch a few of them, but only small changes.
>
>
> You say there's more to be fixed that just reversing #29038
> . What else is there? Would
> you be willing to do all the work?
>
> I am willing to do the work. It is not much that need to be done. Just
> some query-replace to change the generated HTML into valid XML.
>
> Regards
>
> --
> Nils Fredrik Gjerull
> -
> "Ministry of Eternal Affairs"
> Computer Department
> ( Not an official title :) )
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/d0645f61-836c-c908-98a9-b99dec62d5f6%40gjerull.net
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b70Qe22rVHbBb5w3HFCmVBB8%2B9jk37gdNcPKXhY8DU15Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Are there use cases for storing null bytes in CharField/TextField?

[Jon Dufresne]

On Mon, May 15, 2017 at 10:30 AM, Tim Chase 
wrote:

> On 2017-05-15 08:54, Tim Graham wrote:
> > Does anyone know of a use case for using null bytes in
> > CharField/TextField?
>
> Is this not what BinaryField is for?  It would seem to me that
> attempting to store binary NULL bytes in a CharField/TextField should
> result in an error condition.
>

The null byte is also a valid Unicode code point [0].

I guess I'm a bit surprised that a valid code point can't be stored in a
PostgreSQL text column. This does appear to be documented for the char(int)
string function [1], although without justification.

> The NULL (0) character is not allowed because text data types cannot
store such bytes.

I'm curious behind PostgreSQL's decision to prohibit this code point. If
anyone has additional information to share on their reason, please pass it
along.


[0] http://www.fileformat.info/info/unicode/char//index.htm
[1] https://www.postgresql.org/docs/current/static/functions-string.html

Cheers

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b6egRaJ8t23HRXwjfqTHjTJx%3DjRPSL8P%3D6MUTyotFsP6w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: should nonexistent template filter arguments resolve to string_if_invalid rather than raise VariableDoesNotExist?

[Jon Dufresne]

Just my opinion, but I think raising an exception is more helpful to
developers and will result in fewer unnoticed bugs.

More generally, I know the template engine has a history of silently
converting unknown variables to string_if_invalid but this is more harmful
than helpful in my experience. It continues to be a source of hard to catch
bugs on the projects I work on. So I'd prefer we avoid introducing more of
this pattern if possible.

On Wed, May 31, 2017 at 8:03 AM, Tim Graham  wrote:

> Should nonexistent template filter arguments raise an exception? Current
> behavior:
>
> {{ value|filter:nonexistent_template_var}}
>
> raises VariableDoesNotExist for nonexistent_template_var.
>
> I guess the proposal would be to make nonexistent_tempatle_var resolve to
> string_if_invalid. As for me, I think the current behavior is less error
> prone.
>
> Related tickets:
>
> https://code.djangoproject.com/ticket/13167
> This was first closed incorrectly as "fixed" by Jacob (actually a
> different issue was fixed), then amended as wontfix by Karen with the note,
> "I'm fine with wontfixing that, though it does rather seem to go against
> "template errors don't raise exceptions" philosophy."
>
> https://code.djangoproject.com/ticket/28172
> A follow up ticket requesting the same thing as #13167.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/02727bd0-5a67-4cf9-9c3f-
> b0a0a7ea0a3a%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b6adbyeG_mQwZBhsu2TJq9D_EGZgKgakr1fnqqMN7tWFA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: CITextField base class

[Jon Dufresne]

> Does it make sense to add a CICharField for these cases or you can just
override the form/admin.

In a past Django developers discussion, Aymeric suggested building this
functionality with a mixin so the citext type could apply to multiple DB
fields which would then affect how the form field is rendered. Personally,
I like this idea and believe it would provide the most flexibility such
that a TextField, CharField, EmailField, etc. could all be backed by the
citext type.

https://groups.google.com/d/msg/django-developers/SW7_qI81G58/i6aJVRnJBAAJ

>From Aymeric in the previous thread:

---
I'm forseeing a small difficulty in terms of API. This is a behavior I'd
like
to "mix in" to some fields but I can't say if that will be easy to
implement.

The general ideas would be:

# A mixin

class CITextField(TextField):
# ...

# Case-insensitive versions of some built-in Django default fields
# (if we consider that makes sense)

class CIEmailField(CITextField, EmailField):
pass

# The possibility for users to make custom fields case insensitive

class CITagField(CITextField, TagField):
pass
---

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b5GvZMSq9rUiGOHawfbBBoP0rf1%3D74Zi-yzmxcnvxndxA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: CITextField base class

[Jon Dufresne]

I believe this will also change the default form widget from  to . Is that also desired? IME, I most often use
citext with short text inputs, such as email.

On Wed, Feb 8, 2017 at 7:11 AM, Tim Graham  wrote:

> Since that's a release blocker for a feature that hasn't been released
> yet, it's fine to reference/reopen the original ticket rather than create a
> new one. I've created a PR, https://github.com/django/django/pull/8034.
>
> On Wednesday, February 8, 2017 at 7:40:02 AM UTC-5, Adam Johnson wrote:
>>
>> Pretty sure this is a new ticket since it's effectively a bug report
>>
>> On 8 February 2017 at 11:00, Mads Jensen  wrote:
>>
>>>
>>>
>>> On Wednesday, February 8, 2017 at 11:32:22 AM UTC+1, Adam Johnson wrote:

 Sounds legit, make a ticket

>>>
>>> Shouldn't https://code.djangoproject.com/ticket/26610 just be reopened?
>>>
>>>
 On 8 February 2017 at 00:37, Sean Brant  wrote:

> I noticed the new postgres citext[1] field is a subclass of CharField.
> Wouldn't it be more correct to subclass TextField? Subclassing CharField
> means we need to add a max_length which is ignored in the db.
>
> [1] https://github.com/django/django/blob/master/django/cont
> rib/postgres/fields/citext.py
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-develop...@googlegroups.com.
> To post to this group, send email to django-d...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/0e50eae1
> -8806-47fc-b223-9cac7152bf1e%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



 --
 Adam

>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit https://groups.google.com/d/ms
>>> gid/django-developers/e2be5fd6-3fc0-44d5-8d07-da713eb58db4%
>>> 40googlegroups.com
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>> Adam
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/cac9ac5f-cdcd-4e33-bba3-
> b1a9de700365%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b6%3Dc7EAzbjP%3DFxDb%3D7O2-8x6V4jOoQhx2OTvcAH1P_xEg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: AbstractBaseUser.set_unusable_password() why a random string instead of an empty string?

[Jon Dufresne]

Thanks Ramiro and Alex. That does clear it up and helps.

I withdraw the suggestion.

On Wed, Oct 19, 2016 at 7:02 PM, Alexander Hill <a...@hill.net.au> wrote:

> Hi Jon,
>
> Here's the ticket where this was proposed: https://code.djangoproject.
> com/ticket/20079
>
> And the commit that implements it: https://github.com/django/
> django/commit/aeb1389442d0f9669edf6660b747fd10693b63a7
>
> Cheers,
> Alex
>
>
> On Thu, 20 Oct 2016 at 08:20 Jon Dufresne <jon.dufre...@gmail.com> wrote:
>
> When set_unusable_password() is called on a user object, the user's
> password is set to a random string starting with "!" [0]. The "!" is then
> used by is_password_usable() [1] to determine that this password isn't
> usable.
>
> My question is, why is a random string used instead of an empty string? An
> empty string would appear to make the code both simpler and slightly more
> efficient. Is the random string more secure or solving some other issue I'm
> not aware of?
>
> I tested this idea and all tests pass: https://github.com/jdufresne/
> django/commit/2332194b449fe4a336c8ea515221955ba0ea3aeb
>
> The change is relatively easy as all interactions with unusable passwords
> are nicely abstracted by functions
>
> Thoughts on me following through with this code simplification?
>
> [0] https://github.com/django/django/blob/90c3b11e873a326219a68c14a3b5dc
> a0181e7b3c/django/contrib/auth/hashers.py#L78
> [1] https://github.com/django/django/blob/90c3b11e873a326219a68c14a3b5dc
> a0181e7b3c/django/contrib/auth/hashers.py#L27
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/CADhq2b7v6BxetkaS6GOW5%3DO1EoXFX9Ke1FWzmCLUs%
> 3DLDjQ5DsA%40mail.gmail.com
> <https://groups.google.com/d/msgid/django-developers/CADhq2b7v6BxetkaS6GOW5%3DO1EoXFX9Ke1FWzmCLUs%3DLDjQ5DsA%40mail.gmail.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/CA%2BKBOKySg7uPpgYDXJ9aq0gL92-
> bztoZ5FwKbXyNXPDi4yeqng%40mail.gmail.com
> <https://groups.google.com/d/msgid/django-developers/CA%2BKBOKySg7uPpgYDXJ9aq0gL92-bztoZ5FwKbXyNXPDi4yeqng%40mail.gmail.com?utm_medium=email_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b5CZNTaoHktp6tEq3LQ2dEGO00rcQmRXn4o8SpV8WcDPg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

AbstractBaseUser.set_unusable_password() why a random string instead of an empty string?

[Jon Dufresne]

When set_unusable_password() is called on a user object, the user's
password is set to a random string starting with "!" [0]. The "!" is then
used by is_password_usable() [1] to determine that this password isn't
usable.

My question is, why is a random string used instead of an empty string? An
empty string would appear to make the code both simpler and slightly more
efficient. Is the random string more secure or solving some other issue I'm
not aware of?

I tested this idea and all tests pass:
https://github.com/jdufresne/django/commit/2332194b449fe4a336c8ea515221955ba0ea3aeb

The change is relatively easy as all interactions with unusable passwords
are nicely abstracted by functions

Thoughts on me following through with this code simplification?

[0]
https://github.com/django/django/blob/90c3b11e873a326219a68c14a3b5dca0181e7b3c/django/contrib/auth/hashers.py#L78
[1]
https://github.com/django/django/blob/90c3b11e873a326219a68c14a3b5dca0181e7b3c/django/contrib/auth/hashers.py#L27

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b7v6BxetkaS6GOW5%3DO1EoXFX9Ke1FWzmCLUs%3DLDjQ5DsA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Exceptions caught by the "include" template tag make it hard to rely on tests

[Jon Dufresne]

> Has anyone relied on the exception silencing behavior as a "feature" when
using {% include %}?

Not here.

+1 on deprecating the silencing behavior.

I have created a ticket [0] and PR [1] so others may get a sense of the
bigger picture change and how it could affect Django. Feedback welcome.

More generally, I find the template engine's behavior of replacing
exceptions with an empty string to cause more bugs than it prevents. In my
experience, loud errors will more likely be caught during development, unit
tests, and manual testing.

[0] https://code.djangoproject.com/ticket/27175
[1] https://github.com/django/django/pull/7208

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b7%3Daxpcduaas68ihWugLhiffTPmAx0bQoyTy%3Dsa8Sq7GA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Proposal: Use HTML5 boolean attribute for checked on checkbox/radio inputs

[Jon Dufresne]

> I was wondering if it causes any HTML validation problems for other
doctypes?

True. A strict validator for XHTML will flag HTML5 syntax as an error. It
isn't a part of the XHTML spec. From my testing it seems like modern
browsers can handle this, but you're right, validators will catch it.

> If so, we might document that Django's default HTML rendering targets the
HTML5 doctype.

Makes sense to me. I've updated PR with additional documentation that
mentions this.

Cheers,
Jon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b5UsjaAKQhe%2Bn0eYabCuBPTG-72RvYEHz1dC7oZgefvGA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Proposal: Use HTML5 boolean attribute for checked on checkbox/radio inputs

[Jon Dufresne]

Hi,

I would like to propose that Django renders the "checked" attribute of
checkbox and radio inputs using the HTML5 boolean style attributes.

Django has supported HTML5 boolean attributes since 1.8 [0]. It has used
them internally for the "disabled" attribute since 1.9 [1] and the
"required" attribute starting with 1.10 [2]. So there is some precedent to
using the HTML5 style. I find the newer style cleaner and more in line with
modern conventions.

I have created a ticket [3] with this proposal as well as a PR [4].

One concern raised in the ticket is backwards compatibility with non-HTML5
doctypes. I'm not aware of any such issues with modern browsers. I have
tested older doctypes on Firefox and Chrome, both accept the HTML5 boolean
style with HTML4 and XHTML doctypes. Currently, I do not have access to IE,
so I am unable to test those cases. If anyone is interested to test, there
is a very simple test case in the ticket.

Additionally, if there is an issue with older doctypes, presumably this
issue already exists with the disabled and required attributes.

Just reaching out for feedback, concerns, and comments.

Thanks!

Cheers,
Jon


[0] https://docs.djangoproject.com/en/dev/releases/1.8/#forms
[1]
https://github.com/django/django/blob/stable/1.9.x/django/forms/boundfield.py#L88-L89
[2]
https://github.com/django/django/blob/stable/1.10.x/django/forms/boundfield.py#L88-L89
[3] Ticket: https://code.djangoproject.com/ticket/26928
[4] PR: https://github.com/django/django/pull/6961

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b6WBJfd0_XWPXPbApdG7KkYAYGNK69joOF3uGgcEvqCFg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: change commit message format to present tense?

[Jon Dufresne]

On Fri, Jun 24, 2016 at 10:48 AM, Carl Meyer  wrote:

> To be clear, the recommended git style is not present tense, it is
> imperative mood. So it should _not_ be "Fixes #12345 -- Regulates the
> frobnicator", it should be "Fix #12345 -- Regulate the frobnicator."
>

Do you have a link to an authoritative source stating this as the
recommended style or is it just common knowledge?

I'm not arguing against the proposal (in fact, I agree with it), I'm just
curious if there is documentation to support one style over the other.

Cheers,
Jon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b4zz8Dd8%3Dh7PSuMvFOFok3Tn-K0JsQkVkNNiBKphkBH%2Bg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Would a patch to add allow_empty_strings=True to CharField be accepted?

[Jon Dufresne]

Would recent commit solve the problem for you:
https://github.com/django/django/commit/267dc4a2882182f71a7f285a06b1d4b15af0

Relevant ticket: https://code.djangoproject.com/ticket/4136

On Fri, Jun 17, 2016 at 3:27 AM, Luke Benstead  wrote:

> Hi!
>
> We've recently hit upon a problem where we need empty strings in a
> CharField to be coerced to NULL. Obviously nullable CharField's are frowned
> upon but we have the requirement that non-empty values must be unique, so
> the empty state must be NULL.
>
> We've written a custom subclass of CharField to deal with this, but when I
> found the bug report[1] for this issue, the wontfix justification was that
> it would make things backwards incompatible. What I'm proposing instead is
> we add another option to CharField's __init__ which defaults to true called
> "allow_empty_strings". If set to False, and the field is nullable, this
> would coerce empty values to None. If the field was not nullable, this
> would raise an exception (ValueError?, IntegrityError?).
>
> Is this an acceptable solution to support this use-case?
>
> Thanks,
>
> Luke.
>
> [1] https://code.djangoproject.com/ticket/9590
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/ad3d155c-6386-4d4f-a4f3-07544c2f127e%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b5cU_Ce6g_9k%2BHeCfgNSFbOV3YFtpAffoAsHsEhwNwi5A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

New CharField attribute to handle CharField(null=True, blank=True, unique=True) in model forms

[Jon Dufresne]

Hi,

Occasionally I'll need to define a CharField on a model that is unique but
also allow blank values. At the database level, this is easily handled by
storing NULL for the blank values. (Storing the empty string multiple times
will result in a DB unique constraint violation.) This use case has
surfaced several times as evident by the 9 year old ticket #4136 [1].

I have created a POC solution to the ticket at PR 6624 [2]. The change adds
the attribute "empty_value" to forms.CharField. The value specified by
empty_value is used as the Python empty value for the field. This value
defaults to the empty string (current behavior) but could also be changed
to None. The change also modifies model forms to set empty_value to None
for CharField when null=True is set. The model forms change allows the use
of the admin to save blank, unique char values without unique constraint
violations.

This choice to create the empty_value attribute API was based on the prior
art of the empty_value attribute of TypedChoiceField [3].

In the PR Tim Graham requested I raise the new API on the developers list.
I'm looking for concerns or feedback regarding the new attribute or any
other issue with the PR.

Thanks!

Cheers,
Jon


[1] https://code.djangoproject.com/ticket/4136
[2] https://github.com/django/django/pull/6624
[3]
https://github.com/django/django/blob/218175b/django/forms/fields.py#L832-L856

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b6dp46Z2MROPTanNt%2Bo3wAVfuq3A32jTGrRo87-W8Wo-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Add HTML5 required attribute on form widgets

[Jon Dufresne]

On Thu, May 5, 2016 at 11:29 AM, Collin Anderson 
wrote:

> If anyone is running into hidden required fields preventing forms from
> submitting (like me), I've been using this jQuery code for a somewhat-hacky
> quickfix:
>
> $(':hidden[required]').removeAttr('required')
>
>
The changes made on master should not be adding required to hidden inputs.
I added a check to prevent this. Are you experiencing different?

Are these inputs hidden by Django or from another source?

Cheers,
Jon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b44N4VxvtoB-QYykapRoQtCXHbrNpOzxbay0KzmpSjHqQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Add HTML5 required attribute on form widgets

[Jon Dufresne]

On Wed, Mar 30, 2016 at 7:01 AM, Alex Riina  wrote:

> What's the plan for formsets with extra?
>
> I could see the required only getting applied to the first min forms but
> I'm not sure there is an actual workable case there. It seems like it will
> get too messy with adding and deleting at the same time.
>
> If can_delete is false and extra is 0, it seems like the required
> attribute could at least be used. Because of this, I think it should
> probably be an initialization argument, default to false, or be overridden
> when constructing forms in formsets.
>
> https://github.com/gregmuellegger/django-floppyforms/issues/75
>
>
Thanks for highlighting this.

I'll investigate implementing the suggestion "If can_delete is false and
extra is 0, it seems like the required attribute could at least be used.
Because of this, I think it should probably be an initialization argument,
default to false, or be overridden when constructing forms in formsets."
Thanks.

I think with this concern, this feature can't be solved entirely by
template-base widgets as something other then Field.required is necessary
for the formset case.

Cheers,
Jon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b5G_Bif-EB5C_XcQmLR_XZZboXOxmrc_Ja92rfEXfwydQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Deprecating User.is_anonymous ?

[Jon Dufresne]

> Do we know of any custom user models with this unconventional behaviour?

FWIW, I have a project with the following code snippet:

---

class AuthenticatedAnonymousUser(AnonymousUser):
def is_authenticated(self):
return True


class AnonymousAuthenticationMiddleware(AuthenticationMiddleware):
def process_request(self, request):
response = super().process_request(request)
if not response and not request.user.is_authenticated():
if check_api_auth():
request.user = AuthenticatedAnonymousUser()
else:
response = HttpResponseForbidden()
return response

---

This is a small, internal, private project. Users can access the app though
HTML pages by logging in (traditional web app) or through a user-less API.
Many scripts use the API.

This allows views to check user.is_authenticated(). This passes for the
AuthenticatedAnonymousUser, but the user is still treated as anonymous.

While I find this convenient, I could easily solve this some other way. But
I wanted to point out that this use exists.

Cheers,
Jon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b4wYyGzND916rJdqYkMTAC%3DaJyzZXpOCP%3DbiWcEqMMp1A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Add HTML5 required attribute on form widgets

[Jon Dufresne]

Hi,

I'm working on ticket #22383 to add the HTML5 required attribute to form
widgets.

The hope is this will add an additional level of validation at the client
side. The feedback provided to the user is much faster than a server round
trip.

This is something I've done manually in my own projects. I'll frequently
add "required" to widget attrs to take advantage of this validation. As I
already use this frequently, I'm motivated to see it adopted by the Django
core.

This feature have been discussed in the ticket [1] as well as a previous
email thread [2].

In the past, there were two discussed concerns:

* Allow a project to opt-out of using the required attribute
* Add a deprecation warning that new version of Django will include the
required attribute

Past discussion settled on using the class attribute
Form.use_required_attribute to decide if a form's widget should render the
HTML5 required attribute. As this was the previous consensus I have
implemented a PR [3]. To be on the safe side, this PR also adds the
deprecation warning.

Upon review, Tim Graham replied with the following:

I fear the deprecation will be quite annoying if every form in a project
> needs to be modified to silence all warnings.
> ...
> I guess I'm not sure if a deprecation path provides more value than making
> a backwards-incompatible change. For example, if we expect a majority of
> projects to adopt this change, then a deprecation will require every Django
> project to silence the warning instead of a subset of users to opt-out.
>

No problem for me. I'm OK skipping the deprecation path. I don't see it as
necessary, I was simply trying to accommodate other opinions. I have
created a second PR which is identical to the first, but without the
deprecation warning [4].

Also in the review Tim suggested the following:

I wonder if template-based widget rendering (#15667
> ) might ease this change. A
> project could provide custom widget templates if they don't want the
> required attribute (or if they want required='required'.
>

This is a different approach than previously discussed. I see the merit in
this suggestion and think it could be a better implementation. With this
idea, we could avoid adding the Form.use_required_attribute, entirely.
However, template-based widget is still very much a WIP. I worry waiting on
that feature may mean this simpler feature may miss a release cycle. Is it
reasonable to expect the template-based widget rendering to land before the
next alpha/beta cut?

Tim asked I post these ideas to the mailing to get other opinions and
feedback on these two points.

Thanks,
Jon


[1] https://code.djangoproject.com/ticket/22383
[2]
https://groups.google.com/forum/#!topic/django-developers/obw18wSc4xU/discussion
[3] https://github.com/django/django/pull/6341
[4] https://github.com/django/django/pull/6352

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b7z08aYuY_wpzCCkCaEfOAT5uQvh74wsPCJoZT0hF5aGg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

% vs {} string formatting for public APIs

[Jon Dufresne]

Hi,

I noticed that some Django public APIs use the % old-style string
formatting while others use the {} new-style formatting.

For example:

{} style

* success_url [0] (Converted to {} in 1.8)
* format_html [1]

% style

* ValidationError [2]
* related_name [3]


Is this difference intentional?

As success_url recently moved to {}, I'm guessing {} is Django's
preferred style?

If this different is not intentional, I would like to propose a change
to ValidationError to use the {} style formatting. This change would
go through the same deprecation path as used by success_url. Thoughts?

I think this would create a greater API consistency throughout the
framework. I think using a single, consistent string formatting API
follows the design philosophy "principle of least surprise".

Ultimately, I don't care what format is preferred, but I think one
should be used consistently.

I have created a proof of concept as to what this would mean as a
change. The change still requires new tests and docs; so it still WIP.

https://github.com/django/django/compare/master...jdufresne:remove-percent-placeholder

If there is support for this change, I would like to follow through a
ticket and finished PR.

Cheers,
Jon


[0] 
https://docs.djangoproject.com/en/1.9/ref/class-based-views/mixins-editing/#django.views.generic.edit.ModelFormMixin.success_url
[1] 
https://docs.djangoproject.com/en/1.9/ref/utils/#django.utils.html.format_html
[2] 
https://docs.djangoproject.com/en/1.9/ref/forms/validation/#raising-validationerror
[3] 
https://docs.djangoproject.com/en/1.9/topics/db/models/#be-careful-with-related-name

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b6RCWUhHFAAKsKeYbgpA35TQ%3DoQrHyR5vFL8UmTHn3QAQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: remove support for unsalted password hashers?

[Jon Dufresne]

On Tue, Feb 2, 2016 at 10:35 AM, charettes  wrote:
>> That hasher, being the fastest non-plaintext hasher around, is quite
>> useful
>> when running tests: it allows login checks to be performed much faster.
>
> This argument came up a couple of time in the past and from what I remember
> the Django test suite itself spends a significant amount of time hashing
> passwords.
>
> I understand that we want to avoid shipping a plaintext password hasher in
> `django.contrib.auth.hashers` for the sake of not exposing a footgun-API but
> what if we shipped one in the `django.contrib.auth.test` package instead?
>
> We could document its use for testing purposes only and would make the
> complete deprecation of unsafe hashers easier.

For testing, Django provides force_login(). This method skips password
hashing and logs a user in for a test. For tests that want a logged in
user and do not care about how the password was verified, this seems
like a good choice. Assuming this was used throughout, would this be
sufficient for handling this performance concern?

Tests that verify the password hasher will need to run the hash
algorithm regardless.

https://docs.djangoproject.com/en/1.9/topics/testing/tools/#django.test.Client.force_login

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b5%3DnqrD2GOTTg4hjT4KxRUFGGT0tZZHCYypdggH-0ePqQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Referrer Policy Delivery, Django shouldn't do strict referrer check anymore?

[Jon Dufresne]

On Wed, Dec 2, 2015 at 10:29 AM, Flávio Junior  wrote:
> Also, I can't imagine now why, but some
> developer might want to disable referer header altogether, and can easily do
> so by setting policy to No Referrer.

Why is it unimaginable that I may want to maximize privacy for my
users? The domain my users are coming from is between me and my users.
If CSRF would work securely without any referer header (and not leak
the same data some other way), I would choose to disable it entirely.
Often the domain is enough to leak what kind of data a user was
reading or interacting with. For example, what might one assume
(rightly or wrongly) if the referer header was
"https://wikileaks.org;?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b62tY-izLJ2WrgsskJmbBT5ArezGL17HuinbHv4hYAg3g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: makemigrations --exit; exit status feels backwards from conventions

[Jon Dufresne]

Thanks everyone for the useful feedback. I've created a ticket where
the development of the idea can continue:
https://code.djangoproject.com/ticket/25604

On Fri, Oct 23, 2015 at 7:33 AM, Chris Foresman <fores...@gmail.com> wrote:
> Jon,
>
> You proposal seems like a sane and welcome change that aligns the exit
> status of --exit with long-standing convention.
>
>
> Thanks,
> Chris
>
>
>
> On Wednesday, October 21, 2015 at 10:20:09 AM UTC-5, Jon Dufresne wrote:
>>
>> On Tue, Oct 20, 2015 at 9:29 PM, Gavin Wahl <gavi...@gmail.com> wrote:
>> > In your case, successfully creating a migration indicates a failure.
>>
>> Only if the --check flag is on. The --check flag indicates that one is
>> explicitly checking that all model changes have migrations. A non-zero
>> exist status indicates that migrations were missing. If you feel the
>> help text could be improved to communicate this, I can work towards
>> that.
>>
>> > Why do you object to using a ! to communicate this?
>>
>> With the --check flag or the --exit flag?
>>
>> I think I covered this in the OP. But just to clarify:
>>
>> My use case:
>>
>> Continuous integration server check's developers' commits for
>> correctness. One aspect of correctness is that all model changes have
>> migrations.
>>
>> In shell scripting and CI servers an exit status of 0 indicates
>> true/pass and an exit status of non-zero indicates false/failure.
>>
>> Therefore, the command should return 0 when everything is OK and
>> correct and non-zero otherwise. Commits are correct when all model
>> changes are accounted for.
>>
>> The current --exit behavior, returns non-zero when everything is
>> correct. To account for this in CI, one must negate the exit status
>> with !, this goes against conventional behavior.
>>
>> Further, if something goes terribly wrong and there is an unhandled
>> exception, negating the exit status will make the CI stage appear to
>> pass. This is backwards! CI can't tell the difference between "all
>> changes accounted for" and "Python had an unhandled exception".
>>
>>
>> Cheers,
>> Jon
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/4a641f83-2334-4f64-b9d5-438c094dbe27%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b7ajz50vU_NEpwmQ8F8%3DmyVo-pKGoOkTDYbL90LrN7tYg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: makemigrations --exit; exit status feels backwards from conventions

[Jon Dufresne]

On Tue, Oct 20, 2015 at 9:29 PM, Gavin Wahl  wrote:
> In your case, successfully creating a migration indicates a failure.

Only if the --check flag is on. The --check flag indicates that one is
explicitly checking that all model changes have migrations. A non-zero
exist status indicates that migrations were missing. If you feel the
help text could be improved to communicate this, I can work towards
that.

> Why do you object to using a ! to communicate this?

With the --check flag or the --exit flag?

I think I covered this in the OP. But just to clarify:

My use case:

Continuous integration server check's developers' commits for
correctness. One aspect of correctness is that all model changes have
migrations.

In shell scripting and CI servers an exit status of 0 indicates
true/pass and an exit status of non-zero indicates false/failure.

Therefore, the command should return 0 when everything is OK and
correct and non-zero otherwise. Commits are correct when all model
changes are accounted for.

The current --exit behavior, returns non-zero when everything is
correct. To account for this in CI, one must negate the exit status
with !, this goes against conventional behavior.

Further, if something goes terribly wrong and there is an unhandled
exception, negating the exit status will make the CI stage appear to
pass. This is backwards! CI can't tell the difference between "all
changes accounted for" and "Python had an unhandled exception".


Cheers,
Jon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b4N24e1TipB-DPCRZxBk71JW0NEs6F_8gKSLDk_ZtQ-tg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: makemigrations --exit; exit status feels backwards from conventions

[Jon Dufresne]

On Tue, Oct 20, 2015 at 1:48 PM, Jon Dufresne <jon.dufre...@gmail.com> wrote:
> On Tue, Oct 20, 2015 at 1:18 PM, Shai Berger <s...@platonix.com> wrote:
>> On second thought, I take that back.
>>
>> As is evident from the discussion Jon cited[1], the --exit flag was intended 
>> to
>> be used as Marc's --check. Jon is basically correct in his analysis, and I
>> think the root issue is that "--exit" is just a bad name for "--check".
>>
>> [1] 
>> https://groups.google.com/d/topic/django-developers/I3M6B-wYYd8/discussion
>
> How would you feel about putting "--exit" on a deprecation path and
> replacing it with a "--check" flag? The "--check" flag would return 0
> if no migrations are missing and non-zero if they are missing.

To get a sense of what this would mean, I've created a POC with this
idea at: https://github.com/django/django/pull/5453

If the idea and design is well received, I'll create a ticket and
continue with it.

Thanks,
Jon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b47ioAcHa9wB0baV1KNGaO3uOdGhw68Pt5S4s1Y-GZhZQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

makemigrations --exit; exit status feels backwards from conventions

[Jon Dufresne]

Before posting this, I've read through the full thread "sys.exit(1)
from makemigrations if no changes found" at
.

I fully agree with the spirit of the change. I already find the
feature useful in CI. However, after using this feature on a CI
server, I find the exit status backwards compared to typical commands.
The makemigrations command returns status 0 to indicate CI failure
(migrations missing) and 1 to indicate CI pass (continue to the next
CI stage).

Typically status 0 indicates pass and non-zero indicates failure. By
following the typical exit status conventions, commands can explicitly
return a non-zero status when detecting a failure or the Python
runtime may return a non-zero status if something goes terribly wrong;
such as an unhandled exception.

Someone that is accustomed to typical exit status conventions might
naively use the makemigrations command:

./manage.py makemigrations --dry-run --exit

The expectation: the next stage should continue if there are no new
migrations (the developer did everything they were supposed to do and
included migrations). However, the above command will return status 1,
not 0, if there are no new migrations.

OK, we can test for that. Maybe change the command to:

! ./manage.py makemigrations --dry-run --exit

That is, interpret the exit status opposite of what one would
typically expect. Immediately, this looks backwards compared to
typical shell scripting. But what happened to the "terribly wrong"
scenario? For example, what if a developer mistakenly added an
incorrect setting that caused an ImproperlyConfigured error? If this
were to happen, I would want the above command to fail and stop the CI
pipeline.

So maybe the next step would be to check explicitly for exit code 1.

./manage.py makemigrations --dry-run --exit; test $? -eq 1

Now it looks like we're hacking around something.

Additionally, Python exits with status 1 for unhandled exceptions. So
the above command would still pass the CI stage for an unhandled
exception. Further Django's BaseCommand.run_from_argv() also exits
with status code 1 on CommandError, so again, it would pass the CI
stage for anything that triggers this sort of error.

It seems exit status 1 is overloaded to mean "all migrations are
accounted for, continue to the next stage of CI", and "something went
terribly wrong".

This is why I feel the exit status is backwards from what is typically expected.

I would like to suggest we find a better way to interface with CI
servers. That is return 0 when there are no migrations (continue to
the next stage) and non-zero for both "migrations missing" and
"something went terribly wrong".

I suggest maybe adding a system check for missing migrations. The
check could report an error, when they are missing.  The check
framework seems like a natural command to be used by CI servers
anyway, so this seems like a good place. The missing migration
detection already exists, so the same code could be leveraged for this
check.

I'm also open to other suggestions on creating a more convention exit status.

If there is agreement and the proposal sounds good, I can follow
through with a ticket and code changes.

Cheers,
Jon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b5YDr-HB5sUdwKK-K2awQZk7qUhJJdaU%2B4SH_6nx9x%3D5w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Fate of sql* management commands

[Jon Dufresne]

On Sun, Mar 29, 2015 at 4:57 PM, Russell Keith-Magee
 wrote:
> It would probably be *possible* to refactor manage.py sqlall to use the new
> table creation logic; but the question would then be "why do you want it"?

Speaking from experience, I have found these commands to be a
extremely helpful when migrating a legacy application to Django. In my
situation, the legacy application would do whatever it wanted with the
schema and was not very opinionated nor well maintained. When moving
to Django having a quick way to dump Django's *ideal* SQL
representation was a very useful way to debug what needed to change in
the legacy application. There may even be a interim where SQL
migrations are handled outside Django but with the goal of moving
towards Django's representation. Having a way to diff actual vs
expected is very useful for this situation.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b5x66TN1mky3PoZoJNd78RtVA0ee951kd7_k4Qy%3DVPHHg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Refactor year, month, day lookups?

[Jon Dufresne]

On Tue, Mar 24, 2015 at 9:24 PM, Josh Smeaton  wrote:
> Hi,
>
> Firstly (and least importantly) opening a PR makes it a lot easier to review
> code, especially when there are lots of commits. A [WIP] pull request is
> common and useful. If you get a chance, you should open one with this
> change.
>
> I think it's a good idea. So much so that I opened a ticket about a year
> ago: https://code.djangoproject.com/ticket/22394. You'll note some comments
> there about retaining the Year based behaviour as a `BETWEEN X and Y` rather
> than `Extract(YEAR)`. Otherwise, I think the support is rather positive. At
> a high level, your code looks fairly solid and I think would be a useful
> addition.

Thank you for pointing me to this. I have added a PR to that ticket.
Future review and discussion of the changes can continue in the ticket
and PR.

> Another thing I would really like to see is transform based annotations. I'm
> not 100% sure on whether .annotate(F('X__transform')) is supported or not,
> but if it is, we'll get some really nice behaviour from the use of
> transforms.

AFAICT this does not work. Both before and after my change doing:

Article.objects.annotate(month=F('pub_date__month'))

Yields:
---
  File "/home/jon/devel/django/tests/lookup/tests.py", line 257, in
test_values_with_month_lookup
values = Article.objects.values('pub_date__month')
  File "/home/jon/devel/django/django/db/models/manager.py", line 127,
in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/jon/devel/django/django/db/models/query.py", line 703, in values
clone = self._values(*fields)
  File "/home/jon/devel/django/django/db/models/query.py", line 698, in _values
query.add_fields(field_names, True)
  File "/home/jon/devel/django/django/db/models/sql/query.py", line
1603, in add_fields
name.split(LOOKUP_SEP), opts, alias, allow_many=allow_m2m)
  File "/home/jon/devel/django/django/db/models/sql/query.py", line
1377, in setup_joins
names, opts, allow_many, fail_on_missing=True)
  File "/home/jon/devel/django/django/db/models/sql/query.py", line
1345, in names_to_path
" not permitted." % (names[pos + 1], name))
FieldError: Cannot resolve keyword u'month' into field. Join on
'pub_date' not permitted.
---

Just to be clear, are you suggesting support for this be a part of my
change? Personally, I see this feature as orthogonal to the transform
refactoring. Seeing as it doesn't work with the built-in lookups nor
transforms there should be little BC concerns.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b6Z9P-XH487MKgJ_Sfm_TnMGry-5-WAg6pq33580we5cg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Refactor year, month, day lookups?

[Jon Dufresne]

Hi,

I have been spending some time learning and investigating the custom
lookups feature that was newly introduced in 1.7 [0]. While
investigating, I wanted to learn by example so I started looking
through the Django code. In the end, I didn't find many examples.
However, I did notice that there exists lookups for the year, month,
day, (and more) value from a database date or datetime value [1].
These appear to be implemented as special case "builtin" lookups and
not through the new lookup mechanism. Is there a specific reason for
this or is it historical momentum?

I started investigating if these could be refactored to use the common
code path and implemented using the new lookup mechanism. To my
delight it was not very difficult and I now have all tests passing
after refactoring these lookups. Right now, this lives in a branch of
mine and not in a ticket or pull request. The WIP branch is located
at: 

Would this be something welcome as a ticket and pull request? While
there is no outward change in functionality, I see it as a beneficial
refactoring because with this change:

1. The year, month, day, etc lookups are no longer treated as special
cases, but instead use the common code path.
2. There now exists complete and useful examples of registering new
lookups in the Django code itself. This might help others build more
lookups.
3. The lookups are now limited to the correct Field types where as
previously this was not true. I demonstrate this with a unit test.

If this looks like it could be a welcome change I will can go forward
with a typical ticket and pull request.

Cheers,
Jon

[0] https://docs.djangoproject.com/en/dev/howto/custom-lookups/
[1] https://docs.djangoproject.com/en/dev/ref/models/querysets/#year

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b4C4y5ZUHvNEugizHrqjL4EWoQJQcc8nne%2BjxsmU-L3uA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: CSRF REASON_NO_REFERER with meta referrer tags

[Jon Dufresne]

On Wed, Feb 4, 2015 at 5:59 AM, Erik Romijn <erom...@solidlinks.nl> wrote:
>
> On 03 Feb 2015, at 16:44, Jon Dufresne <jon.dufre...@gmail.com> wrote:
>> However some URLs are accessed by a unique URL
>> containing a nonce without a login. Login is not an option for these
>> URLs. Sharing this URL is considered very bad and I would like to
>> avoid it happening unintentionally.
>
> I'm not following this: to prevent this case, you are actively
> instructing all your users to disable referer headers in their browsers?
> If not, how are you controlling what referrers your users send?

I am not instructing my users to do anything with their headers. This
would never be feasible for me. I mentioned some users may do so
independently.

I *want* to help control referrers using the new meta referrer tag.
This is a new feature not yet supported in all major browsers. See:

https://blog.mozilla.org/security/2015/01/21/meta-referrer/
https://w3c.github.io/webappsec/specs/referrer-policy/#referrer-policy-delivery-meta
https://developer.mozilla.org/en-US/docs/Web/HTML/Element/meta (search
referrer on page)

>
> URLs without login, which contain a secret nonce, are indeed sensitive
> to the nonce leaking through the referer. Dropbox ran into this a
> while ago:
> https://blog.dropbox.com/2014/05/web-vulnerability-affecting-shared-links/
>
> This also affected Evernote for some time. The common resolution seems
> to be not to disable referer headers, which is a client-side issue, but
> to mask it by sending all external links through a specific URL first
> without the nonce, which works as a simple redirector.
>
> Far from ideal, especially when dealing with more complicated links like
> when sharing office documents. But it seems to work for Dropbox and
> Evernote. You'll notice for example that when viewing a PDF on Dropbox,
> you're not using your in-browser PDF viewer but Dropbox' custom viewer,
> which I imagine also modifies all external links.

Thanks. Aymeric suggested a similar scheme to me earlier. I will take
it under consideration.

However, In my opinion, the user's privacy needs go beyond this one
scenario. When going cross origin, it is never the business of the
final destination where I started. Interestingly enough, the links
above have an "origin-when-crossorigin" mode (but not
"no-referrer-when-crossorigin"). However, this is not supported on
Chrome and so it defaults to "no-referrer" which, once again, breaks
Django POST over HTTPS.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b7MYsFmnXdbOREzmv-5HtnzEdEr0ci8%2B7EfbGx%3DABSjPQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: CSRF REASON_NO_REFERER with meta referrer tags

[Jon Dufresne]

On Tue, Feb 3, 2015 at 11:45 PM, Aymeric Augustin
<aymeric.augustin.2...@polytechnique.org> wrote:
> Le 4 févr. 2015 à 03:31, Jon Dufresne <jon.dufre...@gmail.com> a écrit :
>>
>> Prevent the application from being served in an attacker's iframe: use
>> X-Frame-Options. (Supported by all major browsers [1])
>
> That's irrelevant in the scenario we're discussing here. The iframe Paul 
> talks about would be injected by a MITM. It's under the attacker's control, 
> not your control.

Thank you for the calcification.

Is there detailed documentation on the type of attack this REFERER
check guards against? Whether Django documentation or external
documentation is fine. I would like to have a firm understanding of
the real problem and how this solves it.

Cheers,
Jon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b7EuwAuyJAvge00qNjgttFTjtJ0q9TPy4S1hBZ1XfsdmQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: CSRF REASON_NO_REFERER with meta referrer tags

[Jon Dufresne]

On Tue, Feb 3, 2015 at 7:09 PM, Josh Smeaton  wrote:
> Just quickly, HSTS[0] and X-Frame-Options[1] are supported and recommended
> in the security documentation already. As you point out though, HSTS isn't
> yet a full solution, and, frankly, it scares me a little. Personally, I
> redirect the / path to HTTPS from HTTP and drop all other HTTP connections.

I think that enhances and is consistent with my existing question.

If there are better mechanisms to secure against these attacks *and*
they are already recommended by Django, what is the CSRF REFERER check
doing that isn't already solved by these mechanisms?

By using these other security mechanisms to secure against the attack,
developers can then use the meta referrer tag to help with users'
privacy.

Cheers,
Jon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b7D2m%2BD4TJ%2BVF8XaeDFQ%3Dtfn2wdE4BFXzWWkhpFALyiMw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: CSRF REASON_NO_REFERER with meta referrer tags

[Jon Dufresne]

On Tue, Feb 3, 2015 at 2:12 PM, Paul McMillan  wrote:
> The referer check is primarily there to help make users who choose not
> to use HSTS safer.
>
> Without HSTS, a mitm can set CSRF cookies (e.g. by serving an HTTP
> page emulating your domain in an iframe on a different unencrypted
> page, even if you only ever serve your own page from HTTPS), and then
> post (from wherever) to your secured page. By forcing the post to come
> from the same domain, we've made this attack significantly less
> convenient.
>
> I agree that it's really unfortunate that we don't have a better
> mechanism for this in the browsers, but this still seems to be the
> best tradeoff we have right now.

Aren't there existing security mechanisms to handle these situations?
It seems like this is sending the wrong message. Django should be
pushing developers and admins for the best solution to security and
privacy concerns.

Prevent HTTP when using HTTPS: use HSTS. (Apparently not supported by IE11 [0])

Prevent the application from being served in an attacker's iframe: use
X-Frame-Options. (Supported by all major browsers [1])

These seem like much better solutions to the scenario you present.
What does the REFERER check do that is not handled by the above?

At that point, the user's privacy concerns could be handled with the
meta referrer tag.

[0] 
https://developer.mozilla.org/en-US/docs/Web/Security/HTTP_strict_transport_security#Browser_compatibility
[1] 
https://developer.mozilla.org/en-US/docs/Web/HTTP/X-Frame-Options#Browser_compatibility

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b4-E9tS5yqho%2BVKERE4-66XenoN_vBDWwjwNUTqPtGdAA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: CSRF REASON_NO_REFERER with meta referrer tags

[Jon Dufresne]

On Tue, Feb 3, 2015 at 11:52 AM, Aymeric Augustin
 wrote:
> You can fix that problem by saving some authentication info in the user's 
> session, most likely with a custom auth backend — see django-sesame for an 
> example of how to do this. Then redirect immediately to an URL that doesn't 
> contain the nonce. Of course all this must happen over HTTPS to reduce the 
> likelihood of leaving the nonce in the logs of various caches or reverse 
> proxies.

My application is 100% over HTTPS, HTTP traffic is not allowed.

Thanks for these pointers. This may not work exactly for me, but it
certainly is something interesting to think about. I'll look into it
more.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b6SXqY78qiNdB7BkAQUcBHzAFUON%3DY69mEddu6Q55SWdg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: CSRF REASON_NO_REFERER with meta referrer tags

[Jon Dufresne]

On Tue, Feb 3, 2015 at 11:39 AM, Aymeric Augustin
<aymeric.augustin.2...@polytechnique.org> wrote:
> Le 3 févr. 2015 à 16:54, Jon Dufresne <jon.dufre...@gmail.com> a écrit :
>> Assuming this MITM already has the correct CSRF value, what is
>> stopping this MITM from adding a REFERER to the HTTPS request?
>
> While MITM of HTTP is trivial, MITM of HTTPS isn't possible (under Django's 
> security model, which doesn't account for government-level attacks, etc.)

Agreed. My application is 100% over HTTPS, so why do I need the CSRF
referrer check. As you state, MITM is theoretically impossible?

However I'm referring to this comment:

> We're talking about a MITM of an HTTP connection that is then used for
> posting a form over an HTTPS connection. Check the comment in the first
> message of this thread for details.

So the MITM is over HTTP, which we both agree is trivial. This MITM
then makes an HTTPS POST request. Making a HTTPS POST request on its
own is also trivial. To circumvent the CSRF protection the MITM will
need the CSRF token as well as to set the referrer header. If the MITM
is capable of obtaining the CSRF, adding a header to a request seems
like the trivial part.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b6PjnZx_XfusSX5odDjo295dgwN80wq26K_mwHqsQzMAw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: CSRF REASON_NO_REFERER with meta referrer tags

[Jon Dufresne]

On Tue, Feb 3, 2015 at 6:18 AM, Aymeric Augustin
 wrote:
> We're talking about a MITM of an HTTP connection that is then used for
> posting a form over an HTTPS connection. Check the comment in the first
> message of this thread for details.

Assuming this MITM already has the correct CSRF value, what is
stopping this MITM from adding a REFERER to the HTTPS request?

Jon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b42%3D-8Lp4XtJUC4Mbn9neVoKVxGY4DNGyWb81rcxYbQ2A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: CSRF REASON_NO_REFERER with meta referrer tags

[Jon Dufresne]

On Tue, Feb 3, 2015 at 1:31 AM, Aymeric Augustin
 wrote:
> Your request boils down to "make Django's CSRF protection of HTTPS
> pages vulnerable to MITM attacks" which isn't acceptable.

Please. That is a very straw-man like way to have a discussion.

The first thing I asked for was for additional information.
Information on why this helps with CSRF security and information on
what other web technologies use this technique. IMO, the code comment
is very limited in details for such a big jump. I'm not suggesting the
comment be expanded, just wanted to read some external links or
details. Upon receiving that information it would help me understand
if the code is doing something useful and if the code should be
modified.

> http://www.w3.org/TR/referrer-policy/#referrer-policy-states doesn't have
> a policy to send the referrer only for requests to the same origin.
>
> "Origin Only" or "Origin When Cross-Origin" are quite close and alleviate
> privacy concerns. The domain name can still leak privacy-sensitive data
> for instance if you're building a help site for people suffering from some
> illness they don't want to reveal.
>
> However, you're framing this as a security matter, which I don't get. Can
> you clarify how removing Referer headers improves security?

You're right. Privacy is a probably better term for what I'm trying to
achieve. I started with the assumption to share as little information
as possible. In my private application, 99% of the URLs are secured
behind a login. However some URLs are accessed by a unique URL
containing a nonce without a login. Login is not an option for these
URLs. Sharing this URL is considered very bad and I would like to
avoid it happening unintentionally. Additionally, I don't want to leak
any information about my users. Including the fact that they were
using my application. Some of this is real privacy some of this is
common courtesy to the users.

Thanks for the link but I read through that document before choosing "none".

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b43-h5f%2BxsofcQHHiiK8%2BfMzp%3D2fSzvtNMfW7etPnSAxQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

CSRF REASON_NO_REFERER with meta referrer tags

[Jon Dufresne]

Hi,

In the interest of security, I recently started using meta referrer
tags in my HTML [0]. To share the least amount of data as possible, I
opted for the "none" policy [1].

This new HTML5 feature breaks Django POST views. The reason: the CSRF
mechanism checks that, when serving over HTTPS, that a HTTP_REFERER
header is set in order to validate the CSRF check. Otherwise, the CSRF
check fails with REASON_NO_REFERER [2]. There is comment in the code
that reads as follows:

# Suppose user visits http://example.com/
# An active network attacker (man-in-the-middle, MITM) sends a
# POST form that targets https://example.com/detonate-bomb/ and
# submits it via JavaScript.
#
# The attacker will need to provide a CSRF cookie and token, but
# that's no problem for a MITM and the session-independent
# nonce we're using. So the MITM can circumvent the CSRF
# protection. This is true for any HTTP connection, but anyone
# using HTTPS expects better! For this reason, for
# https://example.com/ we need additional protection that treats
# http://example.com/ as completely untrusted. Under HTTPS,
# Barth et al. found that the Referer header is missing for
# same-domain requests in only about 0.2% of cases or less, so
# we can use strict Referer checking.

As of right now Chrome is the only browser (that I've tested) that
actually implements the meta referrer tag, but Firefox's next release
will as well. This combination prevented successful login for users.
Unfortunately I didn't detect this problem early in development as I
develop using HTTP, but serve production using HTTPS.

I'm not convinced that this check is really protecting the user from
an attack. Is there additional documentation, beyond this comment, on
why this is more secure? Is there evidence of other web frameworks
using this same technique? I would like to understand more.

Additionally, as a web user, one may use a Firefox/Chrome extension
that purposely strips the HTTP_REFERER from all requests. Django
applications would be effectively broken for such users. In my
opinion, the HTTP_REFERER can't be considered reliable or trustworthy
for such critical operation.

The question: I'm wondering how Django developers feel about removing
this check (or altering it?) in order to help support meta referrer
tags for developers and projects that wish to use it. If there is
general agreement I'll file a ticket and create a pull request.

Cheers,
Jon


[0] https://blog.mozilla.org/security/2015/01/21/meta-referrer/
[1] http://www.w3.org/TR/referrer-policy/#referrer-policy-state-none
[2] 
https://github.com/django/django/blob/27dd7e727153cbf12632a2161217340123687c44/django/middleware/csrf.py#L135

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b4_ph2pCRw%3DukmCmHBxmP56xFxykCiyqo8Li4P0tzM1ig%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Setting database default values in migrations (postgres)

[Jon Dufresne]

On Fri, Oct 31, 2014 at 9:46 AM, Andrew Godwin  wrote:
> So, bear in mind that you can easily set these defaults yourself in a
> migration with RunSQL if you need them for legacy purposes; that way they'll
> get applied

Absolutely. I effectively have such a system in place at the moment.

But, my point is I am also making an effort to match Django's expected
schema while moving away from the legacy schema. I would prefer not to
drift too far from Django's expectations as the goal is move entirely
to Django. This is just one more thing to keep track of and handle
semi-manually.

All I'm saying is that if the described feature existed, it would
benefit me and others that share my use case.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b7zb4SfwTpE8AvuaCkuNLVO%3DYQoFNQZK3ks6fpR0%3DjVYA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Setting database default values in migrations (postgres)

[Jon Dufresne]

On Fri, Oct 31, 2014 at 6:48 AM, Shai Berger  wrote:
> Peter, you said you're "interested in getting database migrations to support
> setting database-level default values", but you haven't said why; unless
> there's a convincing use-case, I'm going to be -1 on this (per the new rules,
> this is no longer a veto, but still).

I am not the original poster, but, FWIW, I too would benefit from this feature.

My Django application interacts with an existing Legacy application
and database. These two applications share the existing database.
Parts of the Legacy application have been ported to Django, other
parts haven't. The existing Legacy application and legacy scripts
relied on these database defaults to handle some data. I've made an
effort to shape the database and Legacy application to the "Django
way", but sometimes it is inconvenient given the existing Legacy
compatibility.

For my particular use case, just handling "primitave" values would be
sufficient: True, False, numbers, text. I understand arbitrary Python
functions can't be easily represeted by databases in a cross platform
way, but that is already beyond the common case. Those functions would
need to be implemented by the Legacy application already.

As far as I'm concerned, Django's ORM doesn't even need to rely on
these defaults when creating objects, but if the migration system
_created_ them (and kept them up to date) it would be very helpful to
people in my situation.

Cheers,
Jon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b7wNsZfoY9Gnq2QejigOCCO08uzQtmo7ZK8AX1whHnSRg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Should reverse() return a Unicode string?

[Jon Dufresne]

On Fri, Sep 19, 2014 at 5:13 AM, Tom Christie  wrote:
> One point of clarity is that we ought to return the same type for each of
> `reverse`, `request.path`, `request.get_full_path`, `request.path_info`, and
> the values in the `request.GET` dictionary. Given that, the answer is
> clearly "it should be a string".

>From my testing, these are all returning text strings except `reverse()`.

The pull request for the reverse() fix can be found here:


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b61tSxmGkpTnCgQ_51OZCkLzqHhKB1BwLy8jPoaXHRNAw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Should reverse() return a Unicode string?

[Jon Dufresne]

Hi,

In my Django application, I'm making a strong attempt to always deal
with Unicode strings at the application layer. Byte strings are only
handled at the protocol layer -- sending data out on the "wire". If my
application tests if an object is a string, I'll use isinstance(obj,
unicode) (Python2).

One gotcha that I noticed is that reverse() will always return a byte
string. Tracing this through the code, I see this happens during the
call to iri_to_uri(), as this function creates a string consisting
only of ASCII characters, other characters are escaped.

Now, reverse() is often used to grab a URL and handle it at the
application layer. It is not reserved only for the protocol layer. An
example would be presenting a URL inside a HTML template, (as an href
or as text), mail, or JSON.

In my opinion, reverse() should return a Unicode string, even if that
string consists only of ASCII characters. It is not until the string
hits the wire that it ought to be forced to bytes.

To verify this, I have created a unit test that I placed in
"urlpatterns_reverse.tests.URLPatternReverse" to demonstrate this is
at the Django layer.

def test_reverse_unicode(self):
name, expected, args, kwargs = test_data[0]
self.assertIsInstance(
reverse(name, args=args, kwargs=kwargs),
six.text_type)

What do you think? If others agree, I can file a bug and create a pull
request to fix this.

Thanks,
Jon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b7WtCq%2BZ52Y-wV5TpmBQu_iRBZU5%3DSP%2B_aJODVxex04Tg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Querying across FK produces too many OUTER JOINs (1.6) or FieldError (1.7)

[Jon Dufresne]

On Fri, Jul 4, 2014 at 7:44 PM, Anssi Kääriäinen
 wrote:
> I wasn't able to reproduce this - are you sure the query is exactly the same
> both on 1.6 and 1.7? Are models are set up correctly before generating the
> query? If after double-checking you still get the error, could you post a
> sample project somewhere?

I have created a test repository at


Upon recreating the test scenario, I have discovered that this is a
false alarm. In my original testing I didn't add my app with the
models to the list of INSALLED_APPS. Without this step (which is
clearly a bug in my test application) the FieldError occurs.

Testing, I see that the OUTER JOIN in 1.6 did in fact change to an
INNER JOIN in 1.7! Thanks for that! Sorry about the confusion and my
mistakes.

Cheers,
Jon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b4CEwX%3DZhW-bzoU59F8T4%2BtWWtrsCvfxjsiVegL%2BVy2YQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Querying across FK produces too many OUTER JOINs (1.6) or FieldError (1.7)

[Jon Dufresne]

> ... and this is more appropriate on django-users

Sorry, but I felt like I was reporting information interesting to developers.

1.) Wrong JOIN type (OUTER vs INNER) producing inefficient queries
2.) FieldError in 1.7 where there wasn't one in 1.6

If these are of no interest, I will not continue the discussion.

Cheers,
Jon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b5-QZffBC1eVbfKZuKfjJdSUBXP4e9Ad5HGK8-U%2B4o8AA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Querying across FK produces too many OUTER JOINs (1.6) or FieldError (1.7)

[Jon Dufresne]

On Fri, Jul 4, 2014 at 2:54 PM, Javier Guerra Giraldez
 wrote:
> what purpose does the "container__item__previous__isnull=True"
> argument serve here?

To filter on the initial item in the list. I'm looking for flag=True
*only* on the first item in the list. flag=True could be set on other
items in the same chain, but I don't care about those. The goal of the
query is:

I'm looking for all *current* items such that the *first item* in the
chain has *flag = true*.

In raw SQL I want the query to be:

SELECT myapp_item.*
FROM "myapp_item"
INNER JOIN "myapp_container"
ON ( "myapp_item"."container_id" = "myapp_container"."id" )
INNER JOIN "myapp_item" T3
ON ( "myapp_container"."id" = T3."container_id" )
WHERE ("myapp_item"."current" = True  AND T3."previous_id" IS NULL AND
T3."flag" = True )

So the WHERE condition flag=True, only applies to the initial item.

>i think it means "an item that belongs to a container that has an item with no 
>'previous'"

Correct, but not just any container, but the same container as the
item being filtered.

> which if it's a linked list, then any non-empty container would comply

Right, except then the "container__item__flag=True" should influence it further.

> and since you start the query from the item, then the container is non-empty 
> by definition.

Yes, correct.

> btw, a linked list in a database?  can you elaborate on that idea?

It is just one way to think about it. It is not really a linked list.
I simply meant the items are linked together by the "previous" point.

In my actual application item represent a step with history (previous
being that history). Container represents the process of the step (has
actual fields). So each step has a previous step in the process.
Sometimes I want to query for the current step in the process based on
the initial step.

Cheers,
Jon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b7Uy6nY0Vo0__%2B3Ka1%2B_Hv7_smnJrDb4rtL%2B6sF%2B3QE5A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Querying across FK produces too many OUTER JOINs (1.6) or FieldError (1.7)

[Jon Dufresne]

Suppose I have the following models:

---
class Container(models.Model):
pass

class Item(models.Model):
container = models.ForeignKey(Container)
previous = models.ForeignKey('self', null=True)
current = models.BooleanField()
flag = models.BooleanField()
---

Item represents a chain of items, all grouped by a container (like a
linked list). The field "current" represents the most recent item
(front of the list). The field "flag" is simply something to query on.

Suppose I perform the following query:

---
Item.objects.filter(current=True,
container__item__previous__isnull=True, container__item__flag=True)
---

That is, I'm looking for all current items such that the first item in
the chain has flag = true. Django 1.6 produces the following SQL for
this query:

---
SELECT ...
FROM "myapp_item"
INNER JOIN "myapp_container"
ON ( "myapp_item"."container_id" = "myapp_container"."id" )
LEFT OUTER JOIN "myapp_item" T3
ON ( "myapp_container"."id" = T3."container_id" )
WHERE ("myapp_item"."current" = True  AND T3."previous_id" IS NULL AND
T3."flag" = True )
---

The OUTER JOIN is the problem. Why is this not a more efficient INNER
JOIN? This query is very inefficient as the the database gets larger.
This causes slow downs on pages.

Interestingly, if I try this on 1.7, this query is apparently not even
supported. I get the following error:

---
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/jon/djtest/djtest/test.py", line 3, in 
qs = Item.objects.filter(current=True,
container__item__previous__isnull=True, container__item__flag=True)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/models/manager.py",
line 92, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/models/query.py",
line 689, in filter
return self._filter_or_exclude(False, *args, **kwargs)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/models/query.py",
line 707, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/models/sql/query.py",
line 1287, in add_q
clause, require_inner = self._add_q(where_part, self.used_aliases)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/models/sql/query.py",
line 1314, in _add_q
current_negated=current_negated, connector=connector)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/models/sql/query.py",
line 1181, in build_filter
lookups, value)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/models/fields/related.py",
line 1506, in get_lookup_constraint
raise exceptions.FieldError('Relation fields do not support nested lookups')
FieldError: Relation fields do not support nested lookups
---

A slow query is better than no query at all. Any particular reason
this was removed? How would one model and query what I'm trying to
achieve moving forward to avoid inefficient queries and exceptions?

Should I file this as a bug?

Cheers,
Jon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b7%3DcsYSOU2gSCL%3DwTD8Mztc9On%2ByRLWmUef%2B-y-fWryuA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: modelformset_factory and unique_together don't always validate unique fields

[Jon Dufresne]

On Thu, Jul 3, 2014 at 10:48 AM, Florian Apolloner
<f.apollo...@gmail.com> wrote:
>
>
> On Thursday, July 3, 2014 3:03:25 PM UTC+2, Jon Dufresne wrote:
>>
>> > Also, even if we find a place to show
>> > the errors, the user is (usually) in no position to correct them (after
>> > all,
>> > there is no field he could change to fix it).
>>
>> I don't follow. In my specific example the user is able to change the
>> "name" field. In my opinion, the form should fail to validate because
>> the _user_ entered "newname" twice, for two different names when they
>> should be unique. The user is in the position to 1) make these
>> conflict and 2) correct them.
>
>
> I don't follow; if the user is able to change the "name" field; it's
> included in the form and uniqueness is checked, I thought we are talking
> about fields __not__ included in a form.
>

The "name" field is included in the form, but the "container" field is
not. The name and container fields are unique_together. The formset
only validates unique fields when all fields are in the form. Here is
the example again.



models.py


class Container(models.Model):
pass

class Item(models.Model):
container = models.ForeignKey(Container)
name = models.CharField(max_length=
100)

class Meta:
unique_together = ('container', 'name')

ItemFormSet = modelformset_factory(model=Item, fields=['name'])


tests.py

class ItemFormSetTestCase(TestCase):
def test_unique_item_name(self):
container = Container()
container.save()
item1 = Item(container=container, name='item1')
item1.save()
item2 = Item(container=container, name='item2')
item2.save()
data = {
'form-TOTAL_FORMS': 2,
'form-INITIAL_FORMS': 2,
'form-MAX_NUM_FORMS': 2,
'form-0-id': str(item1.pk),
'form-0-name': 'newname',
'form-1-id': str(item2.pk),
'form-1-name': 'newname',
}
formset = ItemFormSet(
data,
queryset=Item.objects.filter(container=container))
self.assertFalse(formset.is_valid())
---

This test fails because the uniqueness of name is not actually
validated. If I were to go ahead an save this "valid" form, I receive
the following error:

---
Traceback (most recent call last):
  File "/home/jon/djtest/djtest/myapp/tests.py", line 27, in
test_unique_item_name
formset.save()
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/forms/models.py",
line 636, in save
return self.save_existing_objects(commit) + self.save_new_objects(commit)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/forms/models.py",
line 753, in save_existing_objects
saved_instances.append(self.save_existing(form, obj, commit=commit))
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/forms/models.py",
line 623, in save_existing
return form.save(commit=commit)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/forms/models.py",
line 457, in save
construct=False)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/forms/models.py",
line 103, in save_instance
instance.save()
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/models/base.py",
line 590, in save
force_update=force_update, update_fields=update_fields)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/models/base.py",
line 618, in save_base
updated = self._save_table(raw, cls, force_insert, force_update,
using, update_fields)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/models/base.py",
line 680, in _save_table
forced_update)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/models/base.py",
line 724, in _do_update
return filtered._update(values) > 0
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/models/query.py",
line 598, in _update
return query.get_compiler(self.db).execute_sql(CURSOR)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/models/sql/compiler.py",
line 1003, in execute_sql
cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/models/sql/compiler.py",
line 785, in execute_sql
cursor.execute(sql, params)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/backends/utils.py",
line 65, in execute
return self.cursor.execute(sql, params)
  File "/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/utils.py",
line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)

Re: modelformset_factory and unique_together don't always validate unique fields

[Jon Dufresne]

On Thu, Jul 3, 2014 at 7:02 AM, Tim Graham  wrote:
> Another suggestion: could you include the field on the form but use a
> HiddenInput widget?

I could, but originally I wanted to avoid this. The container_id
should not be editable by the user. Including this in the form seems
to make a statement against this. I could experiment making this
readonly so the form validates that the value is unchanged. But in the
end, this seems to be a workaround to implement my original
suggestion. That is, when instantiating the formset, pass a dictionary
of "static" fields and their data, that are not rendered as part of
the form HTML, but could be used in:

* Default queryset (could always be overridden)
* Creating new instances
* Validating form data

I suppose readonly hidden inputs would achieve the same effect, but it
seems there is no need for the data to be a part of the rendered HTML
form.

I'll play around with this, thanks for the suggestion.

> I think in your case you
> should call model.full_clean() after form validation and handle any unique
> errors there…

I experimented with this, but this will not work. The method
model.full_clean() validates against the database. But the form needs
to validate against unsaved data. I imagine this is why
BaseModelFormSet.validate_unique() was written in the first place.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b5f_oN%3D2-V-WDqv3omPyu2zpKddXTiDNXhWQNd3-u%3DU-Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: modelformset_factory and unique_together don't always validate unique fields

[Jon Dufresne]

On Thu, Jul 3, 2014 at 1:12 AM, Florian Apolloner  wrote:
> To me this is no shortfall but expected and good behaviour, including other
> fields in the validation (i.e. fields not on the form) would be very
> confusing. Where would errors show up?

Right now unique violations show up as non-field errors on the forms
with duplicate values. So I my proposal would do the same.

> Also, even if we find a place to show
> the errors, the user is (usually) in no position to correct them (after all,
> there is no field he could change to fix it).

I don't follow. In my specific example the user is able to change the
"name" field. In my opinion, the form should fail to validate because
the _user_ entered "newname" twice, for two different names when they
should be unique. The user is in the position to 1) make these
conflict and 2) correct them.

> I think in your case you
> should call model.full_clean() after form validation and handle any unique
> errors there…

I will certainly experiment with this thanks.

Cheers,
Jon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b7_LiNG6LnikpMWwJew_kQZ3J9Hd5%3DsT4sp9Y_m-cd4GA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

modelformset_factory and unique_together don't always validate unique fields

[Jon Dufresne]

I'm reporting this to the developers list as I feel this shows a
shortfall in (to me) expected behavior. I'm not sure this is exactly a
bug or simply a use case the unique validation wasn't designed for.

Basically, sometimes I want to create model formsets that use a
limited number of model fields. These model fields may have a unique
constraint using unique_together. The other field (or fields) of
unique together may not be included in the formset. Upon validating
the form, unique_together fields are only checked if all fields are
contained in the form. This means, my unique fields will not be
validated automatically. To achieve the validation I usually have to
copy parts of Django form validation, missing out on DRY.

I think one solution would be for model formsets to have a "static" or
"constant" fields argument. This could be a single value per field
that all forms in the formset would share for a particular set of
fields. These fields would not be a part of the rendered form, but
could be used for validating the forms and creating new models
instances. Thoughts?

An example where I might do this: suppose I have a "container" model.
This model has many "item" models created through a FK relationship.
Perhaps a field is unique together with the container. This example
could apply to any container/item relationship. I might create a
formset for all items of just one container for a bulk (unique)
rename. I have created a simple small example that illustrates my
point:

models.py


class Container(models.Model):
pass

class Item(models.Model):
container = models.ForeignKey(Container)
name = models.CharField(max_length=100)

class Meta:
unique_together = ('container', 'name')

ItemFormSet = modelformset_factory(model=Item, fields=['name'])


tests.py

class ItemFormSetTestCase(TestCase):
def test_unique_item_name(self):
container = Container()
container.save()
item1 = Item(container=container, name='item1')
item1.save()
item2 = Item(container=container, name='item2')
item2.save()
data = {
'form-TOTAL_FORMS': 2,
'form-INITIAL_FORMS': 2,
'form-MAX_NUM_FORMS': 2,
'form-0-id': str(item1.pk),
'form-0-name': 'newname',
'form-1-id': str(item2.pk),
'form-1-name': 'newname',
}
formset = ItemFormSet(
data,
queryset=Item.objects.filter(container=container))
self.assertFalse(formset.is_valid())
---

This test fails because the uniqueness of name is not actually
validated. If I were to go ahead an save this "valid" form, I receive
the following error:

---
Traceback (most recent call last):
  File "/home/jon/djtest/djtest/myapp/tests.py", line 27, in
test_unique_item_name
formset.save()
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/forms/models.py",
line 636, in save
return self.save_existing_objects(commit) + self.save_new_objects(commit)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/forms/models.py",
line 753, in save_existing_objects
saved_instances.append(self.save_existing(form, obj, commit=commit))
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/forms/models.py",
line 623, in save_existing
return form.save(commit=commit)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/forms/models.py",
line 457, in save
construct=False)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/forms/models.py",
line 103, in save_instance
instance.save()
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/models/base.py",
line 590, in save
force_update=force_update, update_fields=update_fields)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/models/base.py",
line 618, in save_base
updated = self._save_table(raw, cls, force_insert, force_update,
using, update_fields)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/models/base.py",
line 680, in _save_table
forced_update)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/models/base.py",
line 724, in _do_update
return filtered._update(values) > 0
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/models/query.py",
line 598, in _update
return query.get_compiler(self.db).execute_sql(CURSOR)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/models/sql/compiler.py",
line 1003, in execute_sql
cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/models/sql/compiler.py",
line 785, in execute_sql
cursor.execute(sql, params)
  File 
"/home/jon/djtest/venv/lib/python2.7/site-packages/django/db/backends/utils.py",
line 65, in execute
return self.cursor.execute(sql, params)
  File 

Re: DRY date_format and datetime objects

[Jon Dufresne]

On Sun, Mar 9, 2014 at 10:36 AM, Jon Dufresne <jon.dufre...@gmail.com> wrote:
> On Sun, Mar 9, 2014 at 10:05 AM, Aymeric Augustin
> <aymeric.augus...@polytechnique.org> wrote:
>> Furthermore, there are two reasons why I'm against making this change
>> in Django:
>> - it's a large backwards incompatibility;
>> - it deviates from Python's behavior in a surprising and unexpected way.
>
> Both these concerns could be solved with a setting to turn this
> feature on and be off by default.
>
> What I find surprising is setting a DATE_FORMAT in settings.py has
> little effect on how many dates are formatted.

I went back and re-read the documentation on this setting (emphasis mine):

---
<https://docs.djangoproject.com/en/dev/ref/settings/#date-format>

**The default formatting to use for displaying date fields in any part
of the system.** Note that if USE_L10N is set to True, then the
locale-dictated format has higher precedence and will be applied
instead. See allowed date format strings.
---

>From my experience this does not seem to be the case. As best I can
tell, it is only the default formatting to use for displaying dates in
templates. When dates are displayed through other means (such as
string formatting), this format sepcification is ignored. This is what
I find confusing and what I hope I could fix.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b6V%3DRLm-i6O9U8XUcUh%2BRgdQbZ4ccUNy7JT5LAD3wdymw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: DRY date_format and datetime objects

[Jon Dufresne]

On Sun, Mar 9, 2014 at 10:05 AM, Aymeric Augustin
 wrote:
> ... assuming your application is only ever used in one locale. We cannot
> make this assumption for all Django applications.
>
> -MM-DD is unambiguous and consistent. You could argue that
> DD/MM/ is just as good, but Americans tend to interpret it
> backwards (they read 10/12/2014 as October 12th instead of December
> 10th).

I understand what you're saying, but how is this any different than
rendering a date as a string at template time? The Locale needs to be
considered in both cases.

>
> Furthermore, there are two reasons why I'm against making this change
> in Django:
> - it's a large backwards incompatibility;
> - it deviates from Python's behavior in a surprising and unexpected way.

Both these concerns could be solved with a setting to turn this
feature on and be off by default.

What I find surprising is setting a DATE_FORMAT in settings.py has
little effect on how many dates are formatted.

> If you want that feature, I would suggest to subclass DateTimeField in
> your projects.

I'll certainly consider this, thanks. But this will not work for any
third party apps. It will also mean my apps are not re-usable outside
my projects.

Cheers,
Jon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b6ozK%3DXi0CceRAujCsQs8eHV5hbH4EWZarmeqs%3DTXkS2A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: DRY date_format and datetime objects

[Jon Dufresne]

On Sat, Mar 8, 2014 at 6:39 PM, Shai Berger  wrote:
> Can you present your use-case in more detail? Why do you have to format the
> dates explicitly in Python code, rather than use the template filter "date"?

Sometimes this happens in __unicode__ methods for models. A model may
contain two dates: start and finish. I want the __unicode__ method to
display this range. The method ends up looking like:

def __unicode__(self):
return u'%s (%s–%s)' % (
self.name,
date_format(self.start_date),
date_format(self.end_date))

I use "date_format" so the dates are always formatted correctly and
consistently. I'll sometimes create other utility functions in the
same spirit. With what I'm suggesting, I'd expect to write the
function as:

def __unicode__(self):
return u'%s (%s–%s)' % (
self.name,
self.start_date,
self.end_date)

If the dates always format themselves correctly without me thinking
about it, it seems like and opportunity to stop repeating myself.

Cheers,
Jon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b4VGaViL0bAGdhdQkCTxoMtbdr7VCYuH1E2UKQp%3D_pCbA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

DRY date_format and datetime objects

[Jon Dufresne]

Hi,

I am using Django 1.6. I find myself using date_format semi-regularly
to convert datetime objects to strings in the DATE_FORMAT format. I
feel like I am repeating myself quite a bit and need to remember to do
this, else I receive the format -MM-DD, which, in my opinion, is
less pleasant to read.

I am considering a pull request that extends Python's datetime class
that overrides the __str__, __unicode__ methods to always return the
date formatted with the configured DATE_FORMAT. Then all uses
throughout the Django code could use this extended datetime instead of
Python's datetime.

Any thoughts on this? If I do the work, is this something that has a
chance of being accepted?

Cheers,
Jon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b6SE4a1JxfLpdNi415pkuBMYu7rfSP%2BfvWhBNkBDwpE1w%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Question about password salt and encryption

[Jon Dufresne]

On Sat, Jun 15, 2013 at 2:24 PM, Luke Plant  wrote:

> 2) Should Django's security be improved by an additional salt that isn't
> stored in the database?
>
> Regarding number 2, this is not likely to happen quickly, due to
> backwards compatibility issues, and the need to introduce a new setting
> etc. (That may help you to decide question 1).
>
> It's definitely worth considering, of course. We would have to consider
> whether it is worth the work. For many installations, if an attacker has
> the database they are very likely to have the source code too. Of
> course, we should try to layer security so that it isn't all or nothing.
> But given the difficulties of changing things, we'd have to consider
> whether the increase in security, in a typical setup, would justify the
> change.
>

Are you suggesting this should be a change to Django itself? A new password
hasher that uses a per user salt as well as a static salt stored outside
the database? Should I file this as a ticket or is this just hypothetical?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Question about password salt and encryption

[Jon Dufresne]

Thank you Luke.


On Sat, Jun 15, 2013 at 12:55 AM, Luke Plant  wrote:

> The reason that SECRET_KEY is not used is that SECRET_KEY is used for
> other applications which might require key cycling - typically for short
> lived data where key cycling isn't going to cause too much of a problem.
>
> So if it was used as a salt for passwords, you would lose the ability to
> check passwords when you cycled the key.
>
> That consideration doesn't stop you from using another value as a salt,
> of course.
>
> If you are trying to get authentication to match an existing system, it
> seems like a better approach would be to write a custom Django
> authentication backend, rather than the other way around, especially if
> you don't want to lose the security features of what you have already:
>
>
> https://docs.djangoproject.com/en/1.5/topics/auth/customizing/#writing-an-authentication-backend
>
>
>From reading the Django documentation, it appears I would need to create a
custom password hasher that is identical to the default hasher, but passes
(salt + settings.SECRET_KEY) to the pbkdf2() function instead of simply
salt. This seems easy enough in principle.

However, part of the appeal and motivation for porting the existing
application to Django is moving away from the NIH of the existing
application. Django gets so much of the foundation right. The existing
application is currently implementing all this foundation itself; sometimes
it does a good job, other times a poor job. I don't claim to be an expert
on password encryption, so I would prefer to leave that sort of thing to
the people that are smarter than me. This seems like I'd be losing a small
portion of that value. I'd have to once again be implementing a piece of
the password hasher.

I guess I need to decide which way to go. Either a custom password hasher
that uses a static salt, or use Django's existing password hasher and not
think about it.

Thanks for the help.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Question about password salt and encryption

[jon . dufresne]

I am in the process of porting an existing application to use Django. I am 
modifying the authentication portion of the existing application to be 
compatible with Django's authentication system. One thing that struck me, 
Django appears to use a single per user salt, stored in the database. 
However, the existing application uses two salts, one static salt stored 
outside the database, and a per user salt stored in the database. From all 
the advice I've received about secure authentication it seems the two salt 
method is standard behavior and considered a best practice. Is there a 
reason Django does not use the SECRET_KEY (or some other static salt) when 
encrypting passwords? Is this still considered a secure encryption 
mechanism? This feels like a step backwards for the authentication of this 
application.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.