Re: Deprecations vs. backwards-incompatible changes for tightening behavior that hides probable developer error

2015-12-21 Thread Cristiano Coelho
Hello, the select_related change was a really good one, after updating I 
found around 3 or 4 queries that had a typo in select_related which was 
obviously never noticed before. In this project finding those errors was 
not complicated at all, but I believe that on a big project that also has 
poor testing, that change might have given the team a few days/weeks of bug 
hunting.
Now the question is, even if it goes through a deprecation cycle, what if 
the team isn't even looking at debug warnings? The issues might still go 
undetected until it is too late and start crashing everywhere, so I'm not 
really sure how big of a difference it would have made.

El lunes, 21 de diciembre de 2015, 12:09:31 (UTC-3), Tim Graham escribió:
>
> I'd like to ask for opinions about whether or not deprecations are more 
> useful than making small backwards incompatible changes when they move 
> Django in a direction toward unhiding probable developer error.
>
> An example from a past release is the validation of fields in 
> select_related() [1]. This was done as a backwards-incompatible change in 
> 1.8: invalid fields immediately raised an error. Would you rather a change 
> like that go through the normal deprecation cycle (pending deprecation, 
> deprecation, error)? My reasoning against it is that it continues to hide 
> probable errors in new and old code (especially in the first release since 
> PendingDeprecationWarning isn't loud by default) and the cost for users of 
> fixing existing code is likely low compared to the overhead of a 
> deprecation cycle in Django. Did anyone find this was not the case in their 
> own projects?
>
> The question came up again with the behavior of the default error views 
> silencing TemplateDoesNotExist when a custom template name is specified [2].
>
> [1] 
> https://docs.djangoproject.com/en/dev/releases/1.8/#select-related-now-checks-given-fields
> [2] https://code.djangoproject.com/ticket/25697
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/cf412600-9f63-44ed-b867-393610e0d7f7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [Question] MySQL Microseconds stripping

2015-12-21 Thread Cristiano Coelho
I think a simple setting allowing to use the old behaviour should be 
enough, shouldn't it? How does it handle other db backends? I'm not sure if 
oracle has an option for datetime precision, but if it does, it makes sense 
to have a global setting for datetime precision, as right now you are 
pretty much forced to always go with a precision of 6 (at least on mysql?) 
and that might be just too much if you want a simpler datetime.

El lunes, 21 de diciembre de 2015, 19:54:29 (UTC-3), Josh Smeaton escribió:
>
> I think this is a fairly big oversight that should be fixed in the most 
> backwards compatible way, so users don't need to change their code, or only 
> have to change it minimally. I'm with Aymeric here. Does Django have 
> visibility of the field constraints at insert/select queryset time? Ideally 
> Django would handle the differences transparently. If that's not possible 
> then we should have a migration or script that'll do the conversion on 
> behalf of users once off.
>
> ./manage.py mysql-upgrade-microseconds && ./manage.py migrate ?
>
>
> On Monday, 21 December 2015 19:39:44 UTC+11, Aymeric Augustin wrote:
>>
>> 2015-12-20 22:57 GMT+01:00 Cristiano Coelho :
>>
>>> Thanks for the suggestion, I think that work around might just add too 
>>> much code, so I'm probably going the way of converting every datetime 
>>> column of every table to datetime(6) and afford the extra storage (and 
>>> probably a little performance impact ?).
>>> I think the documented change might need a little more of attention, and 
>>> mention something about that any equality query will stop working if you 
>>> either don't strip microseconds or update datetime columns to datetime(6) 
>>> (and not even datetime(3) will work...)
>>>
>>
>> If that's the solution we end up recommending -- because the horse has 
>> left the barn months ago... -- then we must document it in detail.
>>
>> This is a large backwards incompatibility that may result in subtle bugs 
>> and requires non-trivial steps to fix. It doesn't live up to Django's 
>> standards.
>>
>> -- 
>> Aymeric.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/78eb038d-a11f-47ec-bf85-ef454341af01%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Deprecations vs. backwards-incompatible changes for tightening behavior that hides probable developer error

2015-12-21 Thread Shai Berger
On Monday 21 December 2015 20:45:26 Carl Meyer wrote:
> 
> Obviously the first question is whether the behavior in question is
> documented. If it is, then a deprecation path is definitely required. On
> the other hand, if the current behavior contradicts the documentation,
> then it's a bug and can just be fixed.
> 

+1

> But the two examples you cite have an edge case with behavior that is
> unspecified in the documentation. The next question I would ask in those
> cases is "Can I imagine a scenario in which someone is intentionally
> taking advantage of the current behavior - it's not just hiding an error
> in their code? How likely is that scenario? How reasonable is their use
> case? And if we conclude that there may be people out there with a
> reasonable use case for the current behavior, how hard would it be for
> them to change their code to continue working as-is, if Django's
> behavior is changed?"
> 

Again, +1.

Another example where we changed immediately was when we disallowed 
select_related in automatic transaction mode, in 1.6.3. In that case, we had a 
stronger argument: 1.6 changed automatic mode, so that code which had been 
correct suddenly gained a potential for data-corruption.

Shai.


Re: FK constraints are not checked at the end of nested atomic blocks

2015-12-21 Thread Shai Berger
On Monday 21 December 2015 19:42:22 Anssi Kääriäinen wrote:
> The only generic way to check the constraints is to run through all
> tables and their constraints. The constraint check itself is a check
> that a matching row in a foreign table exists for each row of the
> checked table. This means that the check could take hours on large
> databases. In addition the check isn't safe against concurrency
> issues. I'm afraid that if we have a flag for this for atomic(), then
> users are going to think it is a good idea to run the check just in
> case.
> 

We can handle that by naming the argument "lengthy_constraint_check" or some 
such.

> We could add a documented API to check wanted models for foreign key
> constraint violations (.check_constraints(Model1, Model2) would be a
> likely API). Then lets just document that depending on the database
> this check can be slow. On some databases we can get away with SET
> CONSTRAINTS IMMEDIATE
> [http://www.postgresql.org/docs/9.1/static/sql-set-constraints.html],
> which should be fast.

Then why don't we implement check_constraints() as 
"SET CONSTRAINTS IMMEDIATE; SET CONSTRAINTS DEFERRED" on PG? That is 
documented to only check outstanding changes in the transaction.

I am not sure about the performance implications on Oracle, but the Oracle 
docs suggest "Making constraints immediate at the end of a transaction is a 
way of checking whether COMMIT can succeed."[1]

As far as I know, neither MySql nor Sqlite support deferrable constraints.

FWIW, MSSQL does not support deferring as such -- it only supports explicitly 
turning the constraints off and on; when you turn them on, you can choose 
between "with check", which checks every single row in your db, and "with no 
check", which allows corrupt data into your DB; this makes loading fixtures 
excruciatingly slow (I believe Anssi was involved in a recent to improve this 
recently).

> I do think DEFERRABLE INITIALLY IMMEDIATE constraints would make a lot
> of sense for what Django is doing with the constraints, that is we
> want to keep the constraints in check except in special cases.
> Unfortunately I can't see an easy way to switch constraints to
> INITIALLY IMMEDIATE mode due to backwards compat and migrations
> issues.
> 

We could handle the migrations if deferrability with an attribute of a FK, I 
think; we could then handle backwards compatibility with a deprecation cycle 
forcing deferrability to be explicit and then, setting a new default. However, 
I don't think the benefit here is worth the amount of code churn.

Shai.

[1] 
https://docs.oracle.com/cd/B28359_01/server.111/b28286/statements_10003.htm
>  - Anssi
> 
> On Mon, Dec 21, 2015 at 4:41 PM, Shai Berger  wrote:
> > While expensive operations may be unsuitable for general consumption,
> > this kind of validation may be useful in special cases and offline jobs,
> > notably in "data" migrations. So I think we should consider supporting
> > them via an argument to atomic().
> > 
> > Be that as it may, so long as we do not plan to change global behavior,
> > we should fix the documentation. The current text and example code
> > strongly imply that FKs are checked at the end of atomic blocks, even
> > though they manage to avoid saying it explicitly.
> > 
> > Shai.
> > 
> > On 21 בדצמבר 2015 14:17:53 GMT+02:00, Aymeric Augustin
> > 
> >  wrote:
> >> 2015-12-21 13:12 GMT+01:00 Anssi Kääriäinen :
> >>> The check_constraints operation is extremely expensive, and not
> >>> suitable for use at the end of savepoints.
> >> 
> >> Ah. Too bad!
> >> 
> >> --
> >> Aymeric.
> > 
> > --
> > Sent from my Android device with K-9 Mail. Please excuse my brevity.
> > 
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Django developers (Contributions to Django itself)" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to django-developers+unsubscr...@googlegroups.com.
> > To post to this group, send email to django-developers@googlegroups.com.
> > Visit this group at https://groups.google.com/group/django-developers.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/django-developers/F0A7E950-24FE-457C-87
> > 7D-905EA3E9AA65%40platonix.com.
> > 
> > For more options, visit https://groups.google.com/d/optout.


Re: [Question] MySQL Microseconds stripping

2015-12-21 Thread Josh Smeaton
I think this is a fairly big oversight that should be fixed in the most 
backwards compatible way, so users don't need to change their code, or only 
have to change it minimally. I'm with Aymeric here. Does Django have 
visibility of the field constraints at insert/select queryset time? Ideally 
Django would handle the differences transparently. If that's not possible 
then we should have a migration or script that'll do the conversion on 
behalf of users once off.

./manage.py mysql-upgrade-microseconds && ./manage.py migrate ?


On Monday, 21 December 2015 19:39:44 UTC+11, Aymeric Augustin wrote:
>
> 2015-12-20 22:57 GMT+01:00 Cristiano Coelho  >:
>
>> Thanks for the suggestion, I think that work around might just add too 
>> much code, so I'm probably going the way of converting every datetime 
>> column of every table to datetime(6) and afford the extra storage (and 
>> probably a little performance impact ?).
>> I think the documented change might need a little more of attention, and 
>> mention something about that any equality query will stop working if you 
>> either don't strip microseconds or update datetime columns to datetime(6) 
>> (and not even datetime(3) will work...)
>>
>
> If that's the solution we end up recommending -- because the horse has 
> left the barn months ago... -- then we must document it in detail.
>
> This is a large backwards incompatibility that may result in subtle bugs 
> and requires non-trivial steps to fix. It doesn't live up to Django's 
> standards.
>
> -- 
> Aymeric.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/126940e1-805f-4bf8-8f74-8cdccb83182c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: A community Poll

2015-12-21 Thread Marten Kenbeek

>
> I'm quite sure I was driven to that PR because importing ContentTypes 
> was causing lots of pain with AppState not being ready. 
>

Whenever I see this error message pop up, it's not because code directly 
imports ContentTypes, but because it imports some other models.py module, 
which in turn imports another models.py module before any class definition 
is reached. ContentTypes often seems to be the first module without any 
dependencies. However, it's easy enough to trace the error back to the 
models.py file that was imported first. Especially when used in a FK, 
changing it to a string reference won't help, because you're still 
importing the model with the FK before the app registry is ready. 

For what it's worth, I prefer to import models and use them directly, for 
no particular reason other than "I like it that way". 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/fede7124-b504-41fe-a43f-c12c6b118354%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: 1.8 shipping invalid .py files in the startapp template

2015-12-21 Thread Carl Johnson
FWIW, this also bit me. I have a deploy script that runs python -m 
compileall as part of its build process. It was pretty easy to work around 
by adding -x app_template, but why make everyone else work around Django? 
ISTM it would be better to call the files .py_template instead since they 
aren't valid Python.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e931889e-39f2-4067-91bb-a7c711cd346a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Deprecations vs. backwards-incompatible changes for tightening behavior that hides probable developer error

2015-12-21 Thread Carl Meyer
Hi Tim,

On 12/21/2015 08:09 AM, Tim Graham wrote:
> I'd like to ask for opinions about whether or not deprecations are more
> useful than making small backwards incompatible changes when they move
> Django in a direction toward unhiding probable developer error.
> 
> An example from a past release is the validation of fields in
> select_related() [1]. This was done as a backwards-incompatible change
> in 1.8: invalid fields immediately raised an error. Would you rather a
> change like that go through the normal deprecation cycle (pending
> deprecation, deprecation, error)? My reasoning against it is that it
> continues to hide probable errors in new and old code (especially in the
> first release since PendingDeprecationWarning isn't loud by default) and
> the cost for users of fixing existing code is likely low compared to the
> overhead of a deprecation cycle in Django. Did anyone find this was not
> the case in their own projects?
> 
> The question came up again with the behavior of the default error views
> silencing TemplateDoesNotExist when a custom template name is specified [2].

Obviously the first question is whether the behavior in question is
documented. If it is, then a deprecation path is definitely required. On
the other hand, if the current behavior contradicts the documentation,
then it's a bug and can just be fixed.

But the two examples you cite have an edge case with behavior that is
unspecified in the documentation. The next question I would ask in those
cases is "Can I imagine a scenario in which someone is intentionally
taking advantage of the current behavior - it's not just hiding an error
in their code? How likely is that scenario? How reasonable is their use
case? And if we conclude that there may be people out there with a
reasonable use case for the current behavior, how hard would it be for
them to change their code to continue working as-is, if Django's
behavior is changed?"

So to take the TemplateDoesNotExist case with the built-in error views:

The one scenario I can imagine is that someone has written their own set
of reusable error views wrapping the default ones with a different
default template naming convention, but they still want to use their
custom reusable error views in projects that may or may not choose to
actually provide the custom error templates.

I think this use case is reasonable and the scenario is possible, but
I'll concede that it may be unlikely. But if someone were doing this, I
would consider it a perfectly reasonable use of the existing API
contract ("nonexistent templates fall back to the default error view
content"), not abuse of a bug.

To adapt their code to the new behavior, they would need to add a
try/except for TemplateDoesNotExist in their wrappers. Not totally
trivial (and might require them to write some new tests), but not
unreasonable.

So for me, this case falls in a murky area. I tend to think it's better
for us to err on the side of being more backwards-compatible when the
current behavior isn't clearly a bug, and someone might be reasonably
relying on it. But I concede that it's in a grey area, and wouldn't
stand in the way of a decision to just treat it as a bugfix.

In cases where we really can't construct any realistic scenario for
intentional use of the current behavior, then I think we just make the
change right away and call it a bug fix. I don't think it's a bad thing
to cause code that was previously silently broken to start breaking loudly.

I think `select_related` with invalid field names is closer to this end
of the spectrum. I _can_ construct a scenario for intentional use of
that, but it's very unlikely and feels much more like abuse of a buggy
API. So if I did earlier suggest that the select_related change should
have gone through a deprecation path, I hereby retract that suggestion :-)

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/567848C6.5060504%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: A community Poll

2015-12-21 Thread Carl Meyer
Hi Curtis,

On 12/21/2015 06:52 AM, Curtis wrote:
> Whilst I'm awake... a pull request I made to a 3rd party project
> included replacing explicit model imports for relation fields with lazy
> binding -- ForeignKey('contenttypes.ContentType') in this case.
> 
> The author pointed out this didn't play well with their IDEs refactoring
> tool, and asked if there was a general discussion on why lazy references
> were better.
> 
> So aside from the obvious "preventing circular imports", what other
> benefits do people see?

I consider string FK references to be an ugly wart, to be used only when
using regular imports is impossible (that is, only in circular reference
situations).

> I'm quite sure I was driven to that PR because importing ContentTypes
> was causing lots of pain with AppState not being ready.

Hard to say exactly what would be causing that without seeing the
specific case, but it should be resolvable without resorting to string
references. "Fixing" app-registry-not-ready by using string FK
references would feel to me like papering over the problem, instead of
finding the root cause and fixing it.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/56783E8F.4010808%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: FK constraints are not checked at the end of nested atomic blocks

2015-12-21 Thread Anssi Kääriäinen
The only generic way to check the constraints is to run through all
tables and their constraints. The constraint check itself is a check
that a matching row in a foreign table exists for each row of the
checked table. This means that the check could take hours on large
databases. In addition the check isn't safe against concurrency
issues. I'm afraid that if we have a flag for this for atomic(), then
users are going to think it is a good idea to run the check just in
case.

We could add a documented API to check wanted models for foreign key
constraint violations (.check_constraints(Model1, Model2) would be a
likely API). Then lets just document that depending on the database
this check can be slow. On some databases we can get away with SET
CONSTRAINTS IMMEDIATE
[http://www.postgresql.org/docs/9.1/static/sql-set-constraints.html],
which should be fast.

I do think DEFERRABLE INITIALLY IMMEDIATE constraints would make a lot
of sense for what Django is doing with the constraints, that is we
want to keep the constraints in check except in special cases.
Unfortunately I can't see an easy way to switch constraints to
INITIALLY IMMEDIATE mode due to backwards compat and migrations
issues.

 - Anssi


On Mon, Dec 21, 2015 at 4:41 PM, Shai Berger  wrote:
> While expensive operations may be unsuitable for general consumption, this
> kind of validation may be useful in special cases and offline jobs, notably
> in "data" migrations. So I think we should consider supporting them via an
> argument to atomic().
>
> Be that as it may, so long as we do not plan to change global behavior, we
> should fix the documentation. The current text and example code strongly
> imply that FKs are checked at the end of atomic blocks, even though they
> manage to avoid saying it explicitly.
>
> Shai.
>
> On 21 בדצמבר 2015 14:17:53 GMT+02:00, Aymeric Augustin
>  wrote:
>>
>> 2015-12-21 13:12 GMT+01:00 Anssi Kääriäinen :
>>>
>>> The check_constraints operation is extremely expensive, and not suitable
>>> for use at the end of savepoints.
>>
>>
>> Ah. Too bad!
>>
>> --
>> Aymeric.
>
>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/F0A7E950-24FE-457C-877D-905EA3E9AA65%40platonix.com.
>
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1FENw4gKKYhN2PS2ZFxisUAt_kzDV9wKD%2B9fxmQo9h3cg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Add "view_decorators" to django.views.generic.View

2015-12-21 Thread Tim Graham
Could you summarize the pros and cons of this new technique versus the 
existing @method_decorator() recommendation [1]? Does it duplicate all the 
functionality provided by method_decorator such that we would no longer 
recommend that technique in the class-based view docs?

[1] 
https://docs.djangoproject.com/en/dev/topics/class-based-views/intro/#decorating-the-class

On Monday, December 21, 2015 at 12:03:15 PM UTC-5, Carl Johnson wrote:
>
> I would find that useful. In my experience, I often have a mixture of CBV 
> and non-CBV code, and it's annoying to shoehorn decorators into the as_view 
> method. It would be much more convenient to just add to 
> self.view_decorators.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/39d40372-9e63-4dfc-9378-b2187a95bd25%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Add "view_decorators" to django.views.generic.View

2015-12-21 Thread Carl Johnson
I would find that useful. In my experience, I often have a mixture of CBV 
and non-CBV code, and it's annoying to shoehorn decorators into the as_view 
method. It would be much more convenient to just add to 
self.view_decorators.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/dff976d3-e987-4a7f-87d0-05a64d740036%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Help needed with the MySQL max index length problem for Django 1.10

2015-12-21 Thread Tim Graham
I merged the often requested increase of User.username max_length to 254 
characters [1] a few weeks ago, however, the ticket was reopened pointing 
out this issue:


"This patch breaks on MySQL installations using the utf8mb4 charset, 
because it breaks the index length limit - it comes out at a maximum of 254 
* 4 = 1016 bytes whilst by default InnoDB can only index 767 bytes. I found 
this because I am using utf8mb4 in django-mysql's tests.

Django encourages using the utf8 charset (3 bytes per character - cannot 
store emojis), although there has been some discussion for moving to 
utf8mb4 in #18392 . One can 
index 254 character utf8mb4 fields in MySQL by using a couple settings as 
described in that ticket, Django could enforce those, or the field could be 
changed to just 191 characters instead which is the maximum indexable (767 
// 4)."

Do we have any MySQL enthusiasts willing to champion a patch (or at least a 
decision design about the best way to proceed) for #18392 [2] to resolve 
this? 

[1] https://code.djangoproject.com/ticket/20846
[2] https://code.djangoproject.com/ticket/18392

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ceb190b5-5218-446c-a6a0-0aea75e7fd6b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: CBV contrib.sitemaps

2015-12-21 Thread Carl Johnson


On Tuesday, December 15, 2015 at 6:35:06 PM UTC-5, Shai Berger wrote:
>
> Hi Carl, 
>
> On Tuesday 15 December 2015 18:17:36 Carl Johnson wrote: 
> > I was adding a sitemap to a project, and I more or less had to rewrite 
> the 
> > views from scratch because sitemap expects to receive a dictionary of 
> all 
> > possible pages at start up time, [...] 
>
> This seems to be a misrepresentation; it takes a dictionary of all site 
> sections, but each section may well change. E.g. the documentation[1] 
> mentions 
> a BlogSitemap class -- but surely, new blog posts may be added without 
> requiring system restart to update the site map... 
>

Right, but if you want to add a new section, that would require it to be 
baked in. My issue was I was skirting the 50,000 page per sitemap limit, so 
I needed to break my site map up, and the logical way to do that was to add 
one section for each show on our site, but the number of shows can change 
dynamically.
 

> > 
> > So my question is this, if I created a class-based version of sitemaps 
> > (django.contrib.sitemaps_cbv or whatever), would that pull request be 
> > something you would have interest in merging? Or do you consider the 
> > existing sitemaps are good enough, and you don't want to have support 
> > something new (for example, rewriting all the docs)? 
> > 
> The above notwithstanding, you may have some real improvements to suggest, 
> in 
> which case we may be interested. However, we would probably like to see 
> the 
> prospective new sitemaps used before we'd accept it, so your best path 
> towards 
> that is to develop your version as a 3rd-party app. We'd probably not like 
> to 
> keep both versions, though, so if you're trying to write a replacement, 
> you 
> need to consider issues of backwards compatibility and upgrade path. 
>

Yeah, that's really the sticking point. From my point of view, rewriting 
sitemaps for one use case isn't so complicated. Rewriting it from scratch 
as a CBV is also not so complicated. The complicated part would be to 
rewrite it as a CBV but keep backwards compatibility, especially because to 
me the point of the rewrite would be to create a new (hopefully easier to 
use!) interface for sitemaps. 

I'll think some more about it.

— Carl Johnson 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6649267a-4e22-479d-82fe-02712de6ab2a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Deprecations vs. backwards-incompatible changes for tightening behavior that hides probable developer error

2015-12-21 Thread Tim Graham
I'd like to ask for opinions about whether or not deprecations are more 
useful than making small backwards incompatible changes when they move 
Django in a direction toward unhiding probable developer error.

An example from a past release is the validation of fields in 
select_related() [1]. This was done as a backwards-incompatible change in 
1.8: invalid fields immediately raised an error. Would you rather a change 
like that go through the normal deprecation cycle (pending deprecation, 
deprecation, error)? My reasoning against it is that it continues to hide 
probable errors in new and old code (especially in the first release since 
PendingDeprecationWarning isn't loud by default) and the cost for users of 
fixing existing code is likely low compared to the overhead of a 
deprecation cycle in Django. Did anyone find this was not the case in their 
own projects?

The question came up again with the behavior of the default error views 
silencing TemplateDoesNotExist when a custom template name is specified [2].

[1] 
https://docs.djangoproject.com/en/dev/releases/1.8/#select-related-now-checks-given-fields
[2] https://code.djangoproject.com/ticket/25697

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3d60212a-1379-4a0e-ac6a-3b9b41e5c82a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: FK constraints are not checked at the end of nested atomic blocks

2015-12-21 Thread Shai Berger
While expensive operations may be unsuitable for general consumption, this kind 
of validation may be useful in special cases and offline jobs, notably in 
"data" migrations. So I think we should consider supporting them via an 
argument to atomic().

Be that as it may, so long as we do not plan to change global behavior, we 
should fix the documentation. The current text and example code strongly imply 
that FKs are checked at the end of atomic blocks, even though they manage to 
avoid saying it explicitly.

Shai. 

On 21 בדצמבר 2015 14:17:53 GMT+02:00, Aymeric Augustin 
 wrote:
>2015-12-21 13:12 GMT+01:00 Anssi Kääriäinen :
>
>> The check_constraints operation is extremely expensive, and not
>suitable
>> for use at the end of savepoints.
>
>
>Ah. Too bad!
>
>-- 
>Aymeric.
>
>-- 
>You received this message because you are subscribed to the Google
>Groups "Django developers  (Contributions to Django itself)" group.
>To unsubscribe from this group and stop receiving emails from it, send
>an email to django-developers+unsubscr...@googlegroups.com.
>To post to this group, send email to
>django-developers@googlegroups.com.
>Visit this group at https://groups.google.com/group/django-developers.
>To view this discussion on the web visit
>https://groups.google.com/d/msgid/django-developers/CANE-7mWP%3DB_WPg886fqr30LWN2B4parKEdSRvSKiFmSMO4rO2g%40mail.gmail.com.
>For more options, visit https://groups.google.com/d/optout.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/F0A7E950-24FE-457C-877D-905EA3E9AA65%40platonix.com.
For more options, visit https://groups.google.com/d/optout.


Re: A community Poll

2015-12-21 Thread Aymeric Augustin
Hello Curtis,

2015-12-21 14:52 GMT+01:00 Curtis :

> So aside from the obvious "preventing circular imports", what other
> benefits do people see?
>

I'm afraid this is an example of "worse is better".

String references to other models in FK definitions are one of the remnants
of "Django string magic". Such magic has steadily been removed from Django.
I believe the last significant step was to remove the ability to describe
views as strings in URLconfs.

I find it best to have a tree of models with as few circular relations as
possible. Only relations that "jump back up the tree" should be declared
with a string reference to make importing possible. They will usually be
nullable (otherwise you could never create such an instance) and also a
pain to manage ;-)

I found that this principle helps with the relational design.

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANE-7mUAZOiY7ktxCExAnVw1VD1CJ4Octdy7FKdnHu3fgCFcBw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


A community Poll

2015-12-21 Thread Curtis
Whilst I'm awake... a pull request I made to a 3rd party project 
included replacing explicit model imports for relation fields with lazy 
binding -- ForeignKey('contenttypes.ContentType') in this case.


The author pointed out this didn't play well with their IDEs refactoring 
tool, and asked if there was a general discussion on why lazy references 
were better.


So aside from the obvious "preventing circular imports", what other 
benefits do people see?


I'm quite sure I was driven to that PR because importing ContentTypes 
was causing lots of pain with AppState not being ready.


--
Curtis

--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/56780406.2030801%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.


Add "view_decorators" to django.views.generic.View

2015-12-21 Thread Curtis

Hey all,

Tim wanted me to open some discussion on this idea:

https://github.com/django/django/pull/5637/files

Essentially, it allows you to add an explicit list of decorators for a 
View to apply to its view function when someone calls as_view.


This allows the View author to dictate which decorators are needed, 
instead of relying on whomever includes it in their URL pattern to 
remember it.


--
Curtis

--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/56780351.7070401%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.


Re: FK constraints are not checked at the end of nested atomic blocks

2015-12-21 Thread Aymeric Augustin
2015-12-21 13:12 GMT+01:00 Anssi Kääriäinen :

> The check_constraints operation is extremely expensive, and not suitable
> for use at the end of savepoints.


Ah. Too bad!

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANE-7mWP%3DB_WPg886fqr30LWN2B4parKEdSRvSKiFmSMO4rO2g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: FK constraints are not checked at the end of nested atomic blocks

2015-12-21 Thread Aymeric Augustin
Hello,

When a question of "should Django do A or B here?" comes up, adding a
setting to let the user choose is the last resort, because it's really a
cop-out, not the default choice.

Django explicitly creates FKs on PostgreSQL as DEFERRABLE INITIALLY
DEFERRED in order not to check constraints until the end of a transaction.
Certainly it does that for a reason -- although I'm not sure which one,
perhaps ensuring fixture data can be loaded in arbitrary order?

The thought process here should be:

1. For what reason does Django do that? Does this reason also apply, or
still apply, to the situation where you disagree with Django's behavior?
2. What's going to break if we make this change? We aren't going to provide
an option that's incompatible with other documented features.
3. Is there a good reason to let the user pick one behavior or the other?
If so, how are we going to explain how to choose the adequate behavior for
a given use case?
4. Given the above, what should be implemented?

I never thought about deferred constraints check when I added support for
savepoints to all built-in backends. Previously one PostgreSQL had support
for savepoints and it was minimal.

My gut feeling is that it's a decent idea to check integrity constraints in
atomic(savepoint=True), assuming this doesn't have horrific consequences
for performance. Note that savepoint=True is the default for users, but
that Django uses savepoint=False internally whenever possible for
performance.

-- 
Aymeric.


2015-12-21 4:24 GMT+01:00 Артём Клименко :

> I opened this ticket https://code.djangoproject.com/ticket/25955
>
> The last proposal for the improvement
> https://code.djangoproject.com/ticket/25955#comment:5
> add parameter in OPTIONS
>
> DATABASES = {
> 'default': {
> 'ENGINE': 'django.db.backends.postgresql_psycopg2',
> # ...
> 'OPTIONS': {
> 'savepoint_check_constraints': True
> }
> },
> }
>
>
> https://code.djangoproject.com/attachment/ticket/25955/django_master_v2.patch
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/f1d474af-1191-4110-b664-2561d9477631%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANE-7mX02TM8E%3Dc%2BbUny2s%2BAT%3Dd26%2BumvgpDRd_%3D0CCODbY5vg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: FK constraints are not checked at the end of nested atomic blocks

2015-12-21 Thread Anssi Kääriäinen
The check_constraints operation is extremely expensive, and not suitable
for use at the end of savepoints.

The reason why we use deferred constraints by default is mostly historical.
We do need deferred constraints for fixture loading, and applications might
rely on current implementation, but we could perhaps get away with
deferrable initially immediate constraints if the user chooses those.
Allowing users to choose constraint style requires relatively large amount
of work, so nobody has tackled that problem yet.

 - Anssi

On Monday, December 21, 2015, Артём Клименко  wrote:

> I opened this ticket https://code.djangoproject.com/ticket/25955
>
> The last proposal for the improvement
> https://code.djangoproject.com/ticket/25955#comment:5
> add parameter in OPTIONS
>
> DATABASES = {
> 'default': {
> 'ENGINE': 'django.db.backends.postgresql_psycopg2',
> # ...
> 'OPTIONS': {
> 'savepoint_check_constraints': True
> }
> },
> }
>
>
> https://code.djangoproject.com/attachment/ticket/25955/django_master_v2.patch
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com
> 
> .
> To post to this group, send email to django-developers@googlegroups.com
> .
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/f1d474af-1191-4110-b664-2561d9477631%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1GjzZWEQxvZAGZuaSsjzVYhV8eNVLAgY8eqa5Chmw9NEg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: MOSS Award to Django

2015-12-21 Thread Andrew Godwin
On Mon, Dec 21, 2015 at 6:05 AM, Samuel Bishop  wrote:

> Hi Andrew,
>
> Thanks for the detailed response. I'm glad to hear you have tried a few
> different ways to implement this, and having read through your reply, and
> the channels docs again, I have more questions :-)
>
> Your point about asynchronous code introducing needless complexity is very
> well founded, its one of the reasons I like Actor Model Concurrency over
> other concurrency models, it scales well computationally, and *mentally*,
> as a programmer, I am trying break down my real world goals into discrete
> steps of computations, comparisons, iterations and by combining these
> steps, I construct my program. Actor Model Concurrency, is no more
> complicated than saying "This step can be done without affecting anything
> else thats going on", which makes the 'actor/task' much easier to introduce
> to existing synchronous code than most other concurrency methods.
>

I agree - I'm a big fan of actors in general and it's usually what I turn
to to write complex simulations when I'm mucking around with game backends.


>
> One thing that sticks with me as I think about the current channels
> design, relates to the idea I've raised in another discussion (
> https://groups.google.com/forum/#!topic/django-developers/-BOsuVdPFXc ).
> If I put my architect hat on and think about the future a bit, a 'more'
> actor based design to channels might simplify later additional async work.
> Since all the websocket stuff is new, I'll stick to the existing HTTP cycle
> in this example. In version 1 a 'vanilla' Django task/actor would be a full
> HTTP request response cycle, like how Django and WSGI currently work. Our
> Django task is effectively our WSGI task, much like Pulsar Pulse. If all of
> Django is contained inside our new task/worker/supervisor Actor layer, then
> it becomes fairly trivial to do things like create separate request
> handler, database query, and template rendering task/actors and chain their
> calls. Since in the previously linked to discussion thread, I'm talking
> about decoupling the ORM from the 'upper' layers of Django, its hard to not
> also think what opportunities that would open up for async efforts. How do
> you see the architecture of Django channels accommodating such improvements
> in future? At the moment your focused on keeping things compatible and
> avoiding the need for any of the existing synchronous Django code to
> change, is this introducing any design tradeoffs that you are aware of with
> regards to hypothetical future developments?
>

So, what you propose obviously isn't feasible in the short term - I'm doing
as much as I can to decouple Channels from "rewrite all of Django" - but
notably, channels is a subset of the actor model. In particular, you could
see consumers as stateless, single-action actors - but, this means that any
future abstraction would be able to build on that.


>
> Django Channels feels like it contains a possible asynchronous replacement
> for signals but is 'holding back', many cases where I can see myself using
> a channel, I'm using them with a signal handler, can you elaborate a bit on
> your thoughts here? I assume its a compatibility decision, but is there
> more to it than just that?
>

It's entirely a compatibility decision - Channels needs to ship in a way
where code written for 1.9 will work the same, barring a few edge cases
where people dip down to raw WSGI.

You can entirely replace signals with custom channels, and I want to ship
example code that does so (it's two lines of code, pretty much), but they
work in a different way - for example, right now the pre_save signal can
actually prevent a save from occurring, whereas if it were on a channel
we'd have to then link the save consumer up to be run as a result of
pre_save, which starts down a path of compatibility nightmares.

Channels should be flexible enough that if you want to pursue the actor
pattern you can - the underlying library basically just has send() and
receive_many([channel_name, ...]) methods, so you can see how that could be
used to make different task-oriented processes rather than the generic
worker model that's going in now - it's just a case of keeping a major
addition like this as simple and backwards-compatible as possible.
Rewriting everything from scratch would be lovely, but unfortunately part
of maintaining a framework is not doing that.

Part of me, though, would be very interested in such a project being
undertaken to see how far it could go, though I'm not sure you could then
call the end result Django - at least, not without getting there
incrementally over many releases.

Andrew


> On Thursday, 17 December 2015 19:08:09 UTC+8, Andrew Godwin wrote:
>>
>> Hi Samuel,
>>
>> You'll notice if you look carefully that I avoid saying "async" almost
>> anywhere in the Channels announcements or documentation, and when I do,
>> it's qualified very carefully. Channels makes Django 

FK constraints are not checked at the end of nested atomic blocks

2015-12-21 Thread Артём Клименко
I opened this ticket https://code.djangoproject.com/ticket/25955

The last proposal for the improvement 
https://code.djangoproject.com/ticket/25955#comment:5
add parameter in OPTIONS

DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
# ...
'OPTIONS': {
'savepoint_check_constraints': True
}
},
}

https://code.djangoproject.com/attachment/ticket/25955/django_master_v2.patch

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f1d474af-1191-4110-b664-2561d9477631%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [Question] MySQL Microseconds stripping

2015-12-21 Thread Aymeric Augustin
2015-12-20 22:57 GMT+01:00 Cristiano Coelho :

> Thanks for the suggestion, I think that work around might just add too
> much code, so I'm probably going the way of converting every datetime
> column of every table to datetime(6) and afford the extra storage (and
> probably a little performance impact ?).
> I think the documented change might need a little more of attention, and
> mention something about that any equality query will stop working if you
> either don't strip microseconds or update datetime columns to datetime(6)
> (and not even datetime(3) will work...)
>

If that's the solution we end up recommending -- because the horse has left
the barn months ago... -- then we must document it in detail.

This is a large backwards incompatibility that may result in subtle bugs
and requires non-trivial steps to fix. It doesn't live up to Django's
standards.

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANE-7mUbPJVxwspfK4i1J85rJfxNXMxFTjyYDWs69e6LEG21MA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.