#32111: Django 3.1.2 JSONField handling error with an Postgres Foreign Data
Wrapper-based model
-------------------------------------+-------------------------------------
Reporter: Shaheed | Owner: nobody
Haque |
Type: Bug | Status: new
Component: Database | Version: 3.1
layer (models, ORM) |
Severity: Normal | Keywords:
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Over at https://groups.google.com/g/django-users/c/Sr1TDVRpCLE, it was
suggested that I post an issue here...
I have a Django model working fine under Django 3.0 (i.e. "Django<3.1")
which looks like this:
{{{
class Job(models.Model):
id = models.CharField(max_length=36, primary_key=True)
queue = models.CharField(max_length=40)
args = JSONField()
kwargs = JSONField()
type = models.CharField(max_length=80)
...
class Meta:
managed = False # <------ The table is implemented as a
Postgres FDW wrapper.
db_table = 'jobs'
}}}
I am testing the update to Django 3.1.2 and hit an error in executing this
line:
jobs = list(models.Job.objects.filter(queue='celery',
state='scheduled'))
The error is as follows from pytest (i.e. stack trace with local variables
too):
{{{
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:287: in
__iter__
self._fetch_all()
self = <QuerySet []>
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:1308: in
_fetch_all
self._result_cache = list(self._iterable_class(self))
self = <QuerySet []>
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:70: in
__iter__
for row in compiler.results_iter(results):
annotation_col_map = {}
compiler = <django.db.models.sql.compiler.SQLCompiler object at
0x7f8685e49160>
db = 'fdw'
init_list = ['id', 'queue', 'args', 'kwargs', 'type', 'state',
...]
klass_info = {'model': <class 'paiyroll.models.batch.Job'>,
'select_fields': [0, 1, 2, 3, 4, 5, ...]}
known_related_objects = []
model_cls = <class 'paiyroll.models.batch.Job'>
model_fields_end = 9
model_fields_start = 0
queryset = <QuerySet []>
related_populators = []
results = [[('8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery',
['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {},
'paiyroll.tasks.function_run', 'scheduled', ...)]]
select = [(Col(jobs, paiyroll.Job.id), ('"jobs"."id"', []),
None), (Col(jobs, paiyroll.Job.queue), ('"jobs"."queue"', []), None...,
paiyroll.Job.type), ('"jobs"."type"', []), None), (Col(jobs,
paiyroll.Job.state), ('"jobs"."state"', []), None), ...]
select_fields = [0, 1, 2, 3, 4, 5, ...]
self = <django.db.models.query.ModelIterable object at
0x7f86836f3040>
/usr/local/lib/python3.8/dist-
packages/django/db/models/sql/compiler.py:1100: in apply_converters
value = converter(value, expression, connection)
connection = <django.db.backends.postgresql.base.DatabaseWrapper
object at 0x7f869a321670>
converter = <bound method JSONField.from_db_value of
<django.contrib.postgres.fields.jsonb.JSONField: args>>
converters = [(2, ([<bound method JSONField.from_db_value of
<django.contrib.postgres.fields.jsonb.JSONField: args>>], Col(jobs,
pa...NField.from_db_value of
<django.contrib.postgres.fields.jsonb.JSONField: details>>], Col(jobs,
paiyroll.Job.details)))]
convs = [<bound method JSONField.from_db_value of
<django.contrib.postgres.fields.jsonb.JSONField: args>>]
expression = Col(jobs, paiyroll.Job.args)
pos = 2
row = ['8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery',
['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {},
'paiyroll.tasks.function_run', 'scheduled', ...]
rows = <itertools.chain object at 0x7f8683ae7520>
self = <django.db.models.sql.compiler.SQLCompiler object at
0x7f8685e49160>
value = ['paiyroll.tasks', 'call_to_string', 'call_to_string',
[], {}]
/usr/local/lib/python3.8/dist-packages/django/db/models/fields/json.py:74:
in from_db_value
return json.loads(value, cls=self.decoder)
connection = <django.db.backends.postgresql.base.DatabaseWrapper
object at 0x7f869a321670>
expression = Col(jobs, paiyroll.Job.args)
self = <django.contrib.postgres.fields.jsonb.JSONField: args>
value = ['paiyroll.tasks', 'call_to_string', 'call_to_string',
[], {}]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _
s = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], cls =
None
object_hook = None, parse_float = None, parse_int = None, parse_constant =
None
object_pairs_hook = None, kw = {}
def loads(s, *, cls=None, object_hook=None, parse_float=None,
parse_int=None, parse_constant=None, object_pairs_hook=None,
**kw):
"""Deserialize ``s`` (a ``str``, ``bytes`` or ``bytearray``
instance
containing a JSON document) to a Python object.
``object_hook`` is an optional function that will be called with
the
result of any object literal decode (a ``dict``). The return value
of
``object_hook`` will be used instead of the ``dict``. This feature
can be used to implement custom decoders (e.g. JSON-RPC class
hinting).
``object_pairs_hook`` is an optional function that will be called
with the
result of any object literal decoded with an ordered list of pairs.
The
return value of ``object_pairs_hook`` will be used instead of the
``dict``.
This feature can be used to implement custom decoders. If
``object_hook``
is also defined, the ``object_pairs_hook`` takes priority.
``parse_float``, if specified, will be called with the string
of every JSON float to be decoded. By default this is equivalent to
float(num_str). This can be used to use another datatype or parser
for JSON floats (e.g. decimal.Decimal).
``parse_int``, if specified, will be called with the string
of every JSON int to be decoded. By default this is equivalent to
int(num_str). This can be used to use another datatype or parser
for JSON integers (e.g. float).
``parse_constant``, if specified, will be called with one of the
following strings: -Infinity, Infinity, NaN.
This can be used to raise an exception if invalid JSON numbers
are encountered.
To use a custom ``JSONDecoder`` subclass, specify it with the
``cls``
kwarg; otherwise ``JSONDecoder`` is used.
The ``encoding`` argument is ignored and deprecated since Python
3.1.
"""
if isinstance(s, str):
if s.startswith('\ufeff'):
raise JSONDecodeError("Unexpected UTF-8 BOM (decode using
utf-8-sig)",
s, 0)
else:
if not isinstance(s, (bytes, bytearray)):
> raise TypeError(f'the JSON object must be str, bytes or
bytearray, '
f'not {s.__class__.__name__}')
E TypeError: the JSON object must be str, bytes or
bytearray, not list
cls = None
kw = {}
object_hook = None
object_pairs_hook = None
parse_constant = None
parse_float = None
parse_int = None
s = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [],
{}]
}}}
As you can perhaps see from the section for /usr/local/lib/python3.8/dist-
packages/django/db/models/fields/json.py:74, the value being written into
the JSONField called "args" is a Python list, (shown as "s" on the last
line of the traceback-with-values). I am aware of documented changes
around serializers but did not think that affected me; however, given that
I am using an FDW to return the data, is it that I should now be
serialising returned data into a string?
Any pointers appreciated.
Thanks, Shaheed
--
Ticket URL: <https://code.djangoproject.com/ticket/32111>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/055.44740ef3f8286c3a882a54d8ec109578%40djangoproject.com.
