#31973: TypeError loading data in JSONField if DB has native JSON support
-------------------------------------+-------------------------------------
Reporter: Adam Alton | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 3.1
(models, ORM) |
Severity: Normal | Resolution: invalid
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Garry Polley):
I'm not sure of the best spot to put this information. I upgraded our code
base from Django 2.2.x to 3.2.x . In doing that upgrade I got a similar
error seen above:
{{{
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line
412, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line
704, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/celery/app/autoretry.py",
line 50, in run
raise task.retry(exc=exc, **retry_kwargs)
File "/usr/local/lib/python3.8/site-packages/celery/app/task.py", line
706, in retry
raise_with_context(exc)
File "/usr/local/lib/python3.8/site-packages/celery/app/autoretry.py",
line 35, in run
return task._orig_run(*args, **kwargs)
OUR CODE
for preference in get_queryset_page(
File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py",
line 280, in __iter__
self._fetch_all()
File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py",
line 1324, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py",
line 68, in __iter__
for row in compiler.results_iter(results):
File "/usr/local/lib/python3.8/site-
packages/django/db/models/sql/compiler.py", line 1122, in apply_converters
value = converter(value, expression, connection)
File "/usr/local/lib/python3.8/site-
packages/django/db/models/fields/json.py", line 83, in from_db_value
return json.loads(value, cls=self.decoder)
File "/usr/local/lib/python3.8/json/__init__.py", line 341, in loads
raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not dict
}}}
Our database uses the `json` datatype in Postgres rather than `jsonb`. The
database is not managed by the Django application, so we cannot change the
datatype.
The odd part here is that it is a dictionary coming out of the postgres
backend. I think whenever this change went in
https://github.com/django/django/commit/0be51d2226fce030ac9ca840535a524f41e9832c
it changed how existing `json` types were working.
The solution I'm going with right now is to use the current `JSONField`
implementation and do check before the `json.loads` call to see if `value`
is a dictionary type.
{{{#!python
class JSONFieldJSONType(JSONField):
"""
Custom JSON field because our postgres uses json type and not jsonb.
Details on these changes within Django can be seen here:
* https://code.djangoproject.com/ticket/31973
* https://code.djangoproject.com/ticket/31956#comment:8
PR that changed behavior for regular json type:
https://github.com/django/django/commit/0be51d2226fce030ac9ca840535a524f41e9832c
"""
def from_db_value(self, value, expression, connection):
if value is None:
return value
# Some backends (SQLite at least) extract non-string values in
their
# SQL datatypes.
if isinstance(expression, KeyTransform) and not isinstance(value,
str):
return value
try:
# Custom implementation for how our data comes out of our
postgres
# connection.
if isinstance(value, dict):
data_value = self.get_prep_value(value)
return json.loads(data_value, cls=self.decoder)
except json.JSONDecodeError:
return value
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/31973#comment:22>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/067.a40ff4ea2509a5e08eb152d0dc1edbd0%40djangoproject.com.
