#29323: HTTPRequest QueryDict wrongly decodes binary encoded values in python2
-------------------------------------------+------------------------
               Reporter:  Thomas Riccardi  |          Owner:  nobody
                   Type:  Uncategorized    |         Status:  new
              Component:  HTTP handling    |        Version:  1.11
               Severity:  Normal           |       Keywords:
           Triage Stage:  Unreviewed       |      Has patch:  0
    Needs documentation:  0                |    Needs tests:  0
Patch needs improvement:  0                |  Easy pickings:  0
                  UI/UX:  0                |
-------------------------------------------+------------------------
 `application/x-www-form-urlencoded` bodies allow binary values: just
 percent-escape (most of) the bytes.

 It doesn't work with python2 and django.

 In python2, QueryDict returns an `unicode` string for values, which makes
 it impossible to represent binary data.
 Worse: it returns an `unicode` string whose unicode code points are the
 raw expected bytes (caused by lax behavior of `str.decode('iso-8859-1')`)
 so it silently fails.

 Example:
 {{{#!python
 q = QueryDict('foo=%00%7f%80%ff')
 q['foo']
 # actual:
 # u'\x00\x7f\x80\xff'
 # expected:
 # '\x00\x7f\x80\xff'
 }}}

 Relevant code:
 
https://github.com/django/django/blob/f89b11b879b83aa505dc8231da5f06ca4b1b062e/django/http/request.py#L397-L401

 This was introduced by
 
https://github.com/django/django/commit/fa02120d360387bebbbe735e86686bb4c7c43db2
 while trying to accept broken user agents that send non-ascii as
 urlencoded raw values:
 - the python 3 version seems fine: it tries to decode '''before''' calling
 the query string parser (from `urllib`)
 - the python 2 version made a mistake: it decodes '''after''' calling the
 query string parser (and only on the value, not the key strangely)


 I'm not sure how to fix this, as there are many locations where the key
 and value are re-encoded (`appendlist` does it too..)


 I have not (yet) tested with python3.

-- 
Ticket URL: <https://code.djangoproject.com/ticket/29323>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/058.69043153da29f9a7b789884dd05fea9b%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to