Change executed raw mysql queries to return timezone aware datetimes.

[Bryce Drennan]

Request

Change executed raw mysql queries to return timezone aware datetimes.

Explanation

Django 1.9 changed so that raw queries executed in mysql do not return 
timezone aware datetimes. Removing this feature now requires that all 
custom SQL queries involving datetimes have special handling to convert the 
datetimes to timezone aware datetimes. 

The explanation for this was that the conversion was happening globally and 
having adverse affects on 3rd parties. There was also a desire to allow 
per-db timezone settings. 

While sqlite converters are registered globally, mysql converters appear to 
be registered per connection. I'm not sure why mysql was included in this 
change.  It should be possible instead to use the per db TIMEZONE setting 
at the connection level for mysql.  

I read through the code and all related to discussion and must confess I'm 
still not sure I understand the reasoning for this change happening in 
mysql.  I'm posting here hoping to understand better why the mysql 
connection datetime behavior doesn’t key off the per db TIMEZONE setting 
and global USE_TZ settings.

In our local codebase we plan to monkeypatch django to continue returning 
timezone aware datetimes for mysql, thus saving us from manually fixing 
many many queries.  Is this a bad plan? Is there a better way?

Thank you,

Bryce Drennan

Senior Engineer @ Circleup

References

   1. 
   
   https://code.djangoproject.com/ticket/19738 - "manage.py shell" on a 
   fresh project raises RuntimeWarning about naive datetime, if IPython is 
   installed
   2. 
   
   https://code.djangoproject.com/ticket/23820 - Make it possible to 
   override USE_TZ & TIME_ZONE on a per-database basis
   3. 
   
   https://github.com/django/django/pull/4601 - Refactor adaptation and 
   conversion of aware datetimes
   4. 
   
   
   
https://groups.google.com/forum/#!searchin/django-developers/datetimes%7Csort:date/django-developers/Ci_cs0KTagM/iAduocfXFOcJ
 
   - Discussion of removing timezone conversion
   5. http://mysql-python.sourceforge.net/MySQLdb.html

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c7305bb1-097b-439b-8027-cb91f85c15aa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Making Django more PaaS-friendly

[Mike Bryant]

On 11 April 2016 at 07:33, James Bennett  wrote:
> The main problem with this is that it's really not very elegant. But at the
> moment I can't think of anything better, and so I'd like to throw the floor
> open to ideas on nicer approaches to this. If one can't be found, I do think
> Django 1.10 should at least figure out how to handle database config from
> env since that's such a common use case nowadays, but ideally we'd be able
> to pin down a good API for generically pulling configuration from the
> environment.

Sorry for a rather belated response, but I'd like to share our approach to this.
We wrote django-autoconfig [1] to help with our settings. For some
background, it's main role is to allow reusable apps to define
settings that they must have to work effectively (e.g. additional
middleware),

Of relevance though is, to aid in our deployments into AWS we went
with pulling in environment variables named DJANGO_{setting_name} and
coercing them into valid settings.

I mention this here as it seems to be of relevance to the audience,
and I'd welcome any feedback. Apologies if this is just noise.

Mike

[1] https://github.com/mikebryant/django-autoconfig

-- 


Notice:  This email is confidential and may contain copyright material of 
members of the Ocado Group. Opinions and views expressed in this message 
may not necessarily reflect the opinions and views of the members of the 
Ocado Group. 

 

If you are not the intended recipient, please notify us immediately and 
delete all copies of this message. Please note that it is your 
responsibility to scan this message for viruses. 

 

Fetch and Sizzle are trading names of Speciality Stores Limited, a member 
of the Ocado Group.

 

References to the “Ocado Group” are to Ocado Group plc (registered in 
England and Wales with number 7098618) and its subsidiary undertakings (as 
that expression is defined in the Companies Act 2006) from time to time.  
The registered office of Ocado Group plc is Titan Court, 3 Bishops Square, 
Hatfield Business Park, Hatfield, Herts. AL10 9NE.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAPkekMtB-dq243jKHqXJ7GxUDq2bKjNdi4Ti_3MJw81y_SdzUg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: NumericListfilter or similar

[John Griebel]

I agree that we need a replacement; the current behaviour isn't all that
useful, IMHO. My vote would for choice #2, after all it seems more natural
(to me, at least), to think of numbers as a range, especially while
searching. Option 1 seems less intuitive for a situation like this.

+1 from me, especially if we were to go with option 2.

On Tue, Apr 19, 2016 at 10:46 AM, Omer Korner  wrote:

> This is a proposal for a new ListFilter subclass.
>
> Currently, if you add a numeric field to list_filter, Django admin outputs
> all possible values of that field using AllValuesFieldListFilter. This can
> get ugly very quickly.
> For example, in my case, I have an application that manages transactions
> (orders) from an ecommerce site, and saves the transaction value. I can't
> easily filter the values (hundreds of thousands of them).
>
> I have two suggestions for a replacement, instead of outputting all
> available values for a numeric field:
>
>
>1. Output something similar to what DateFieldListFilter does, eg. "10
>or less/more, 15 or less" and so on, depending on real values.
>
>
>
> 
>
>2. A numeric input. These an be two inputs for a range, or a single
>input for a specific value. Looking at the ListFilter code, this would be a
>much more complex change.
>
>
> Happy to hear your thoughts.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/9ef08349-f6a6-48ed-bc30-5b62072fb749%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAOK12WZy_B%2BaY%3Df%3DQ1SP_wJiQUvq%2BadFBV0QBRgVd0sv0H%3DODw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Enforcing a max size for form field values read into memory (review/determination of next steps needed)

[Tom Christie]

> If you are using django-rest-framework, how would the fields counter 
work?. It would be a shame if only multi part and urlencoded uploads would 
have the benefit of these checks, while still allowing json, xml and others 
still be "exploited".
Note I didn't really read the code changes completely so I'm talking with 
almost no knowledge on the proposed change.

They wouldn't be respected by anything other than multi-part or urlencoded 
requests.
Tim's correct in noting that accessing `request.body` or `request.stream` 
won't apply these checks (which is for example, what REST framework does).

Even so I think this is probably a reasonable approach. We could add 
support for respecting these settings in REST framework too, once they 
exist.(Although I think we'd have need to have a stricter consideration of 
backwards compat wrt. folks POSTing large amounts of JSON data)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b78c3cef-60ae-481d-9217-3b9b40eb176e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Enforcing a max size for form field values read into memory (review/determination of next steps needed)

[Tim Graham]

My understanding is that Django doesn't do any parsing of JSON, XML, etc 
but rather simply makes such content available as a raw bystring, 
request.body. Therefore I don't see how Django could offer protection for 
the cases you mentioned.

On Friday, April 15, 2016 at 6:43:27 PM UTC-4, Cristiano Coelho wrote:
>
> I have a small concern.
>
> The two new settings looks like will work on uploaded files count 
> (multipart encoding types) and number of fields sent (url encoded 
> encoding). What happens to other request types such as JSON, XML, plain 
> text etc... If you are using django-rest-framework, how would the fields 
> counter work?. It would be a shame if only multi part and urlencoded 
> uploads would have the benefit of these checks, while still allowing json, 
> xml and others still be "exploited".
> Note I didn't really read the code changes completely so I'm talking with 
> almost no knowledge on the proposed change.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/75df0b09-e5a3-48c6-8dd1-1072c3ead6e1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: JsonpResponse subclass of HttpResponse helps easily create JSONP-encoded responses.

[Joey Wilhelm]

Agreed on discouraging (or at least not actively encouraging) the use of
JSONP. Everything I've read on it has started with big "Do not try this at
home!" warnings.

I also very strongly agree on built-in support for CORS, especially
elements of DRF getting more tightly integrated into the core. I think CORS
is an absolutely essential component if Django is going to claim to be able
to support a REST API out of the box, primarily because of the trend of
using domain.com for an app, and api.domain.com for the backend.

-Joey

On Tue, Apr 19, 2016 at 6:59 AM, Tom Christie 
wrote:

> JSONP is essentially a browser hack.
> There have been security issues raised around it, and we I don't believe
> we should encourage its usage by including it in Django core.
> (If anything we *might?* want to consider CORS for built-in support at
> some point.)
> I'd agree with Florian here - I'd rather see this as either a third party
> package, a code snippet that's shared on a blog post, or similar.
>
>   - Tom
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/37f473d0-8654-4774-82aa-adaf23ea49ab%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADBkHd%2BAwWALAGg4Upy6vA31tZqd1Xt5s5_ESpvYx-koV4agbg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

NumericListfilter or similar

[Omer Korner]

This is a proposal for a new ListFilter subclass.

Currently, if you add a numeric field to list_filter, Django admin outputs 
all possible values of that field using AllValuesFieldListFilter. This can 
get ugly very quickly.
For example, in my case, I have an application that manages transactions 
(orders) from an ecommerce site, and saves the transaction value. I can't 
easily filter the values (hundreds of thousands of them).

I have two suggestions for a replacement, instead of outputting all 
available values for a numeric field:


   1. Output something similar to what DateFieldListFilter does, eg. "10 or 
   less/more, 15 or less" and so on, depending on real values. 
   
   
   

   
   2. A numeric input. These an be two inputs for a range, or a single 
   input for a specific value. Looking at the ListFilter code, this would be a 
   much more complex change.


Happy to hear your thoughts.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9ef08349-f6a6-48ed-bc30-5b62072fb749%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: JsonpResponse subclass of HttpResponse helps easily create JSONP-encoded responses.

[Tom Christie]

JSONP is essentially a browser hack.
There have been security issues raised around it, and we I don't believe we 
should encourage its usage by including it in Django core.
(If anything we *might?* want to consider CORS for built-in support at some 
point.)
I'd agree with Florian here - I'd rather see this as either a third party 
package, a code snippet that's shared on a blog post, or similar.

  - Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/37f473d0-8654-4774-82aa-adaf23ea49ab%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.