#31973: TypeError loading data in JSONField if DB has native JSON support
-------------------------------------+-------------------------------------
Reporter: Adam Alton | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 3.1
(models, ORM) |
Severity: Normal | Resolution: invalid
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Garry Polley):
I'm not sure of the best spot to put this information. I upgraded our code
base from Django 2.2.x to 3.2.x . In doing that upgrade I got a similar
error seen above:
{{{
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line
412, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line
704, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/celery/app/autoretry.py",
line 50, in run
raise task.retry(exc=exc, **retry_kwargs)
File "/usr/local/lib/python3.8/site-packages/celery/app/task.py", line
706, in retry
raise_with_context(exc)
File "/usr/local/lib/python3.8/site-packages/celery/app/autoretry.py",
line 35, in run
return task._orig_run(*args, **kwargs)
OUR CODE
for preference in get_queryset_page(
File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py",
line 280, in __iter__
self._fetch_all()
File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py",
line 1324, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py",
line 68, in __iter__
for row in compiler.results_iter(results):
File "/usr/local/lib/python3.8/site-
packages/django/db/models/sql/compiler.py", line 1122, in apply_converters
value = converter(value, expression, connection)
File "/usr/local/lib/python3.8/site-
packages/django/db/models/fields/json.py", line 83, in from_db_value
return json.loads(value, cls=self.decoder)
File "/usr/local/lib/python3.8/json/__init__.py", line 341, in loads
raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not dict
}}}
Our database uses the `json` datatype in Postgres rather than `jsonb`. The
database is not managed by the Django application, so we cannot change the
datatype.
The odd part here is that it is a dictionary coming out of the postgres
backend. I think whenever this change went in
https://github.com/django/django/commit/0be51d2226fce030ac9ca840535a524f41e9832c
it changed how existing `json` types were working.
The solution I'm going with right now is to use the current `JSONField`
implementation and do check before the `json.loads` call to see if `value`
is a dictionary type.
{{{#!python
class JSONFieldJSONType(JSONField):
"""
Custom JSON field because our postgres uses json type and not jsonb.
Details on these changes within Django can be seen here:
* https://code.djangoproject.com/ticket/31973
* https://code.djangoproject.com/ticket/31956#comment:8
PR that changed behavior for regular json type:
https://github.com/django/django/commit/0be51d2226fce030ac9ca840535a524f41e9832c
"""
def from_db_value(self, value, expression, connection):
if value is None:
return value
# Some backends (SQLite at least) extract non-string values in
their
# SQL datatypes.
if isinstance(expression, KeyTransform) and not isinstance(value,
str):
return value
try:
# Custom implementation for how our data comes out of our
postgres
# connection.
if isinstance(value, dict):
data_value = self.get_prep_value(value)
return json.loads(data_value, cls=self.decoder)
except json.JSONDecodeError:
return value
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/31973#comment:22>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/067.a40ff4ea2509a5e08eb152d0dc1edbd0%40djangoproject.com.
