Re: [Django] #24076: Query may fail with pytz exception

[Django]

#24076: Query may fail with pytz exception
-+-
 Reporter:  lvella   |Owner:  Adam
 |  Zapletal
 Type:  Bug  |   Status:  closed
Component:  Documentation|  Version:  1.6
 Severity:  Normal   |   Resolution:  fixed
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Sarah Boyce <42296566+sarahboyce@…>):

 * resolution:   => fixed
 * status:  assigned => closed

Comment:

 In [changeset:"99273fd525129a973639044dfb12cfd732d8f1d6" 99273fd]:
 {{{#!CommitTicketReference repository=""
 revision="99273fd525129a973639044dfb12cfd732d8f1d6"
 Fixed #24076 -- Added warnings on usage of dates with DateTimeField and
 datetimes with DateField.
 }}}
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018fa4e85dcd-9bf40491-d1ed-4c1b-8caa-de011210a1e3-00%40eu-central-1.amazonses.com.

Re: [Django] #24076: Query may fail with pytz exception

[Django]

#24076: Query may fail with pytz exception
-+-
 Reporter:  lvella   |Owner:  Adam
 |  Zapletal
 Type:  Bug  |   Status:  assigned
Component:  Documentation|  Version:  1.6
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Sarah Boyce):

 * stage:  Accepted => Ready for checkin

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018fa49d78d1-e8cd59bc-a70f-40c5-9e28-dcd4621b62f7-00%40eu-central-1.amazonses.com.

Re: [Django] #24076: Query may fail with pytz exception

[Django]

#24076: Query may fail with pytz exception
---+-
 Reporter:  lvella |Owner:  Adam Zapletal
 Type:  Bug|   Status:  assigned
Component:  Documentation  |  Version:  1.6
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+-
Changes (by Adam Zapletal):

 * needs_better_patch:  1 => 0

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018ef30a20c7-f59e1364-3d78-40b8-a53b-46e9a2f7d417-00%40eu-central-1.amazonses.com.

Re: [Django] #24076: Query may fail with pytz exception

[Django]

#24076: Query may fail with pytz exception
---+-
 Reporter:  lvella |Owner:  Adam Zapletal
 Type:  Bug|   Status:  assigned
Component:  Documentation  |  Version:  1.6
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  1
Easy pickings:  0  |UI/UX:  0
---+-
Changes (by Sarah Boyce):

 * needs_better_patch:  0 => 1

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018ef1de1886-3830dfe0-0751-4fd5-9882-053179408b43-00%40eu-central-1.amazonses.com.

Re: [Django] #24076: Query may fail with pytz exception

[Django]

#24076: Query may fail with pytz exception
---+-
 Reporter:  lvella |Owner:  Adam Zapletal
 Type:  Bug|   Status:  assigned
Component:  Documentation  |  Version:  1.6
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+-
Changes (by Adam Zapletal):

 * has_patch:  0 => 1
 * owner:  nobody => Adam Zapletal
 * status:  new => assigned

Comment:

 Thanks! I opened a PR with an attempt at documenting this. Please let me
 know if the content or location needs to be improved.
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018e59d181da-68c9687d-2ffd-40de-bef9-08e2ca050830-00%40eu-central-1.amazonses.com.

Re: [Django] #24076: Query may fail with pytz exception

[Django]

#24076: Query may fail with pytz exception
---+
 Reporter:  lvella |Owner:  nobody
 Type:  Bug|   Status:  new
Component:  Documentation  |  Version:  1.6
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+
Comment (by Mariusz Felisiak):

 The following comment by Aymeric is crucial, and, as far as I'm aware,
 it's not fixed:

 > I'm going to requalify this as a documentation issue because comparing
 dates with datetimes is probably a common error and I don't think the docs
 warn against it. I don't know where the warning should be added, though.
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018e5586bdf5-3ed3362d-5995-46f4-bfd5-157fc47b8072-00%40eu-central-1.amazonses.com.

Re: [Django] #24076: Query may fail with pytz exception

[Django]

#24076: Query may fail with pytz exception
---+
 Reporter:  lvella |Owner:  nobody
 Type:  Bug|   Status:  new
Component:  Documentation  |  Version:  1.6
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+
Changes (by Adam Zapletal):

 * cc: Adam Zapletal (added)

Comment:

 I wonder if this issue can be closed with no documentation changes for the
 following reasons:

 1. The `__date__gte` lookup (mentioned in comment 1) and others for
 `DateTimeField` have been implemented for a long time.
 2. `pytz` has been removed as of Django 5.0.
 3. Support for time zones has now been core to Django for almost 12 years.
 4. The documentation for the lookup mentioned above seems sufficient to me
 to help future users wondering how to compare a `DateTimeField` with a
 `datetime.date`:
 https://docs.djangoproject.com/en/dev/ref/models/querysets/#date.

 However, I'd be happy to work on a documentation patch if someone can
 provide an idea of what is needed to close this ticket.
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018e42915eb7-ae49bf1a-ed09-4809-b035-b97bfba6ac13-00%40eu-central-1.amazonses.com.

Re: [Django] #24076: Query may fail with pytz exception

[Django]

#24076: Query may fail with pytz exception
---+
 Reporter:  lvella |Owner:  Akshat verma
 Type:  Bug|   Status:  assigned
Component:  Documentation  |  Version:  1.6
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+
Changes (by Akshat verma):

 * owner:  nobody => Akshat verma
 * status:  new => assigned


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018746cbdb39-804667df-c9b3-4bcb-b939-769d080e7724-00%40eu-central-1.amazonses.com.

Re: [Django] #24076: Query may fail with pytz exception

[Django]

#24076: Query may fail with pytz exception
---+
 Reporter:  lvella |Owner:  Akshat verma
 Type:  Bug|   Status:  assigned
Component:  Documentation  |  Version:  1.6
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  1
  Needs tests:  1  |  Patch needs improvement:  1
Easy pickings:  1  |UI/UX:  1
---+
Changes (by Akshat verma):

 * cc: Akshat verma (added)
 * needs_better_patch:  0 => 1
 * needs_tests:  0 => 1
 * easy:  0 => 1
 * needs_docs:  0 => 1
 * has_patch:  0 => 1
 * ui_ux:  0 => 1


Old description:

> If I give date object as input to a filter on a DateTimeField field, like
> this:
>
> {{{
> class Ticket(models.Model):
> register_time = models.DateTimeField(auto_now_add=True)
>
> day = datetime.strptime('2014-10-19', '%Y-%m-%d').date()
> Ticket.objects.filter(register_time__gte=day)
> }}}
>
> I may get a pytz.exceptions.NonExistentTimeError exception. The exact
> exception was:
>
> pytz.exceptions.NonExistentTimeError: 2014-10-19 00:00:00
>
> This date is the start of DST in my timezone:
>
> {{{
> TIME_ZONE = 'America/Sao_Paulo'
> }}}
>
> I believe this is a bug because Django tries to convert my input to a
> datetime object before building the query, and in this case, this
> conversion yields an invalid datetime.
>
> This bug seems very related to issue #9596, because in these cases,
> datetime should be converted to date, not the opposite.
>
> A minimal showcase is attached. Unzip and run ./manage.py test

New description:

 If I give date object as input to a filter on a DateTimeField field, like
 this:

 {{{
 class Ticket(models.Model):
 register_time = models.DateTimeField(auto_now_add=True)

 day = datetime.strptime('2014-10-19', '%Y-%m-%d').date()
 Ticket.objects.filter(register_time__gte=day)
 }}}

 I may get a pytz.exceptions.NonExistentTimeError exception. The exact
 exception was:

 pytz.exceptions.NonExistentTimeError: 2014-10-19 00:00:00

 This date is the start of DST in my timezone:

 {{{
 TIME_ZONE = 'America/Sao_Paulo'
 }}}

 I believe this is a bug because Django tries to convert my input to a
 datetime object before building the query, and in this case, this
 conversion yields an invalid datetime.

 This bug seems very related to issue #9596, because in these cases,
 datetime should be converted to date, not the opposite.

 A minimal showcase is attached. Unzip and run ./manage.py test


 # Changes by Akshat verma
 Traceback (most recent call last):
   File "./manage.py", line 10, in 
 execute_from_command_line(sys.argv)
   File "/Users/user/.virtualenvs/app/lib/python3.4/site-
 packages/django/core/management/__init__.py", line 385, in
 execute_from_command_line
 utility.execute()
   File "/Users/user/.virtualenvs/app/lib/python3.4/site-
 packages/django/core/management/__init__.py", line 354, in execute
 django.setup()
   File "/Users/user/.virtualenvs/app/lib/python3.4/site-
 packages/django/__init__.py", line 18, in setup
 from django.utils.log import configure_logging
   File "/Users/user/.virtualenvs/app/lib/python3.4/site-
 packages/django/utils/log.py", line 10, in 
 from django.views.debug import ExceptionReporter,
 get_exception_reporter_filter
   File "/Users/user/.virtualenvs/app/lib/python3.4/site-
 packages/django/views/debug.py", line 10, in 
 from django.http import (HttpResponse, HttpResponseServerError,
   File "/Users/user/.virtualenvs/app/lib/python3.4/site-
 packages/django/http/__init__.py", line 4, in 
 from django.http.response import (HttpResponse, StreamingHttpResponse,
   File "/Users/user/.virtualenvs/app/lib/python3.4/site-
 packages/django/http/response.py", line 13, in 
 from django.core.serializers.json import DjangoJSONEncoder
   File "/Users/user/.virtualenvs/app/lib/python3.4/site-
 packages/django/core/serializers/__init__.py", line 23, in 
 from django.core.serializers.base import SerializerDoesNotExist
   File "/Users/user/.virtualenvs/app/lib/python3.4/site-
 packages/django/core/serializers/base.py", line 6, in 
 from django.db import models
   File "/Users/user/.virtualenvs/app/lib/python3.4/site-
 packages/django/db/models/__init__.py", line 6, in 
 from django.db.models.query import Q, QuerySet, Prefetch  # NOQA
   File "/Users/user/.virtualenvs/app/lib/python3.4/site-
 packages/django/db/models/query.py", line 13, in 
 from django.db.models.fields import AutoField, Empty
   File "/Users/user/.virtualenvs/app/lib/python3.4/site-
 packages/django/db/models/fields/__init__.py", line 15, in 
 from django.db.models.lookups import default_lookups,
 RegisterLookupMixin
   File 

Re: [Django] #24076: Query may fail with pytz exception

[Django]

#24076: Query may fail with pytz exception
---+
 Reporter:  lvella |Owner:  Akshat verma
 Type:  Bug|   Status:  assigned
Component:  Documentation  |  Version:  1.6
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+
Changes (by Akshat verma):

 * owner:  nobody => Akshat verma
 * status:  new => assigned


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018746a1e8b8-4846296f-bdb5-4b6f-9423-8e3eb1dae8d0-00%40eu-central-1.amazonses.com.

Re: [Django] #24076: Query may fail with pytz exception

[Django]

#24076: Query may fail with pytz exception
---+
 Reporter:  lvella |Owner:  nobody
 Type:  Bug|   Status:  new
Component:  Documentation  |  Version:  1.6
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+
Changes (by aaugustin):

 * component:  Database layer (models, ORM) => Documentation
 * stage:  Unreviewed => Accepted


Comment:

 Thanks for the proposal. Unfortunately it won't work on databases other
 than PostgreSQL because they store datetimes in UTC. As a consequence
 they're vulnerable to the effect I described in question 3 in the
 [https://docs.djangoproject.com/en/1.7/topics/i18n/timezones/#troubleshooting
 timezone troubleshooting FAQ].

 I considered this issue carefully when I implemented support for time
 zones in Django 1.4. I was aware of the points you're making. I decided
 that forbidding mixing dates and datetimes outright was too extreme. It
 would require many changes in projects that work just fine because they
 never see datetimes during the DST switch.

 It's the left-hand-side of a lookup that determines the type, not the
 right-hand-side. Therefore the correct solution is to implement an
 explicit `__date` lookup i.e. #9596. That's the opposite of what you
 requested when you filed the ticket but that also what you recommended in
 your last comment:

 > the database type should be converted to date, dropping the time
 information in the process

 This solution requires timezone conversion of datetimes in the database
 but it shouldn't be an issue as it already exists to support the `dates()`
 and `datetimes()` queryset methods.

 I'm going to requalify this as a documentation issue because comparing
 dates with datetimes is probably a common error and I don't think the docs
 warn against it. I don't know where the warning should be added, though.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.b982b60950f5bd4f54ff74cf6851910e%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24076: Query may fail with pytz exception

[Django]

#24076: Query may fail with pytz exception
-+-
 Reporter:  lvella   |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  1.6
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by lvella):

 Replying to [comment:1 aaugustin]:
 > That's the intended behavior -- Django won't guess what you mean if you
 pass invalid inputs.

 But if using date to compare with datetime works 99% of the time, and
 there is no big warning in the documentation, saying that it must not be
 done, it clearly violates the principle of least astonishment.

 In such cases, either it should always work, or should always fail with an
 error. And it is not that hard to make it work, for instead of trying to
 add time information to the date before passing it to the database, the
 database type should be converted to date, dropping the time information
 in the process.

 Actually, my workaround in the case was to force Django to deliver my
 input in SQL as date, not as datetime:

 {{{
 q = Ticket.objects.filter(register_time__gte=day + (F('id') - F('id')))
 print(q.query)
 }}}

 yields:

 {{{
 SELECT "reproduce_ticket"."id", "reproduce_ticket"."register_time" FROM
 "reproduce_ticket" WHERE "reproduce_ticket"."register_time" >=  2014-10-19
 + ("reproduce_ticket"."id" - "reproduce_ticket"."id")
 }}}

 and it worked!
 Compare with the original query (with another date so it won't raise the
 exception):

 {{{
 q = Ticket.objects.filter(register_time__gte=day)
 print(q.query)

 SELECT "reproduce_ticket"."id", "reproduce_ticket"."register_time" FROM
 "reproduce_ticket" WHERE "reproduce_ticket"."register_time" >= 2014-11-19
 02:00:00
 }}}

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.b4ad88b5620a66e1ec77833f479e605f%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #24076: Query may fail with pytz exception

[Django]

#24076: Query may fail with pytz exception
-+-
 Reporter:  lvella   |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  1.6
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by aaugustin):

 * needs_better_patch:   => 0
 * needs_tests:   => 0
 * needs_docs:   => 0


Comment:

 That's the intended behavior -- Django won't guess what you mean if you
 pass invalid inputs.

 However, if #9596 gets fixed with the idea of comment 24, you will be able
 to run `Ticket.objects.filter(register_time__date__gte=day)`.

 Until then, you're stuck with comparing with an aware datetime object. See
 https://www.python.org/dev/peps/pep-0431/#ambiguous-times for why this is
 awfully complicated.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.e488b5a893df0606fa8a0fa927cb40e3%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.