Thanks Jason, I filed https://code.djangoproject.com/ticket/32111#ticket.
On Thursday, 15 October 2020 at 12:51:22 UTC+1 Jason wrote: > this is pretty interesting, and may be something to report to the devs at > https://groups.google.com/g/django-developers, or open up a ticket at > https://code.djangoproject.com/ I don't see any tickets with the query > *jsonfield > foreign data* that are similar with your experience, so you may have > found an edge case. > > On Thursday, October 15, 2020 at 5:31:09 AM UTC-4 [email protected] > wrote: > >> >> Hi, >> >> I have a Django model working fine under Django 3.0 (i.e. "Django<3.1") >> which looks like this: >> >> =================== >> class Job(models.Model): >> id = models.CharField(max_length=36, primary_key=True) >> queue = models.CharField(max_length=40) >> args = JSONField() >> kwargs = JSONField() >> type = models.CharField(max_length=80) >> ... >> >> class Meta: >> managed = False # <------ The table is implemented as a >> Postgres FDW wrapper. >> db_table = 'jobs' >> =================== >> >> I am testing the update to Django 3.1.2 and hit an error in executing >> this line: >> >> jobs = list(models.Job.objects.filter(queue='celery', >> state='scheduled')) >> >> The error is as follows from pytest (i.e. stack trace with local >> variables too): >> >> ================== >> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >> _ _ _ >> /usr/local/lib/python3.8/dist-packages/django/db/models/query.py:287: in >> __iter__ >> self._fetch_all() >> self = <QuerySet []> >> /usr/local/lib/python3.8/dist-packages/django/db/models/query.py:1308: in >> _fetch_all >> self._result_cache = list(self._iterable_class(self)) >> self = <QuerySet []> >> /usr/local/lib/python3.8/dist-packages/django/db/models/query.py:70: in >> __iter__ >> for row in compiler.results_iter(results): >> annotation_col_map = {} >> compiler = <django.db.models.sql.compiler.SQLCompiler object at >> 0x7f8685e49160> >> db = 'fdw' >> init_list = ['id', 'queue', 'args', 'kwargs', 'type', 'state', >> ...] >> klass_info = {'model': <class 'paiyroll.models.batch.Job'>, >> 'select_fields': [0, 1, 2, 3, 4, 5, ...]} >> known_related_objects = [] >> model_cls = <class 'paiyroll.models.batch.Job'> >> model_fields_end = 9 >> model_fields_start = 0 >> queryset = <QuerySet []> >> related_populators = [] >> results = [[('8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery', >> ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {}, >> 'paiyroll.tasks.function_run', 'scheduled', ...)]] >> select = [(Col(jobs, paiyroll.Job.id), ('"jobs"."id"', []), >> None), (Col(jobs, paiyroll.Job.queue), ('"jobs"."queue"', []), None..., >> paiyroll.Job.type), ('"jobs"."type"', []), None), (Col(jobs, >> paiyroll.Job.state), ('"jobs"."state"', []), None), ...] >> select_fields = [0, 1, 2, 3, 4, 5, ...] >> self = <django.db.models.query.ModelIterable object at >> 0x7f86836f3040> >> /usr/local/lib/python3.8/dist-packages/django/db/models/sql/compiler.py:1100: >> >> in apply_converters >> value = converter(value, expression, connection) >> connection = <django.db.backends.postgresql.base.DatabaseWrapper >> object at 0x7f869a321670> >> converter = <bound method JSONField.from_db_value of >> <django.contrib.postgres.fields.jsonb.JSONField: args>> >> converters = [(2, ([<bound method JSONField.from_db_value of >> <django.contrib.postgres.fields.jsonb.JSONField: args>>], Col(jobs, >> pa...NField.from_db_value of >> <django.contrib.postgres.fields.jsonb.JSONField: details>>], Col(jobs, >> paiyroll.Job.details)))] >> convs = [<bound method JSONField.from_db_value of >> <django.contrib.postgres.fields.jsonb.JSONField: args>>] >> expression = Col(jobs, paiyroll.Job.args) >> pos = 2 >> row = ['8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery', >> ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {}, >> 'paiyroll.tasks.function_run', 'scheduled', ...] >> rows = <itertools.chain object at 0x7f8683ae7520> >> self = <django.db.models.sql.compiler.SQLCompiler object at >> 0x7f8685e49160> >> value = ['paiyroll.tasks', 'call_to_string', >> 'call_to_string', [], {}] >> >> >> >> >> >> */usr/local/lib/python3.8/dist-packages/django/db/models/fields/json.py:74: >> in from_db_value return json.loads(value, cls=self.decoder) >> connection = <django.db.backends.postgresql.base.DatabaseWrapper >> object at 0x7f869a321670> expression = Col(jobs, paiyroll.Job.args) >> self = <django.contrib.postgres.fields.jsonb.JSONField: args> >> value = ['paiyroll.tasks', 'call_to_string', 'call_to_string', >> [], {}] * >> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >> _ _ _ >> >> s = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], cls = >> None >> object_hook = None, parse_float = None, parse_int = None, parse_constant >> = None >> object_pairs_hook = None, kw = {} >> >> def loads(s, *, cls=None, object_hook=None, parse_float=None, >> parse_int=None, parse_constant=None, object_pairs_hook=None, >> **kw): >> """Deserialize ``s`` (a ``str``, ``bytes`` or ``bytearray`` >> instance >> containing a JSON document) to a Python object. >> >> ``object_hook`` is an optional function that will be called with >> the >> result of any object literal decode (a ``dict``). The return value >> of >> ``object_hook`` will be used instead of the ``dict``. This feature >> can be used to implement custom decoders (e.g. JSON-RPC class >> hinting). >> >> ``object_pairs_hook`` is an optional function that will be called >> with the >> result of any object literal decoded with an ordered list of >> pairs. The >> return value of ``object_pairs_hook`` will be used instead of the >> ``dict``. >> This feature can be used to implement custom decoders. If >> ``object_hook`` >> is also defined, the ``object_pairs_hook`` takes priority. >> >> ``parse_float``, if specified, will be called with the string >> of every JSON float to be decoded. By default this is equivalent >> to >> float(num_str). This can be used to use another datatype or parser >> for JSON floats (e.g. decimal.Decimal). >> >> ``parse_int``, if specified, will be called with the string >> of every JSON int to be decoded. By default this is equivalent to >> int(num_str). This can be used to use another datatype or parser >> for JSON integers (e.g. float). >> >> ``parse_constant``, if specified, will be called with one of the >> following strings: -Infinity, Infinity, NaN. >> This can be used to raise an exception if invalid JSON numbers >> are encountered. >> >> To use a custom ``JSONDecoder`` subclass, specify it with the >> ``cls`` >> kwarg; otherwise ``JSONDecoder`` is used. >> >> The ``encoding`` argument is ignored and deprecated since Python >> 3.1. >> """ >> if isinstance(s, str): >> if s.startswith('\ufeff'): >> raise JSONDecodeError("Unexpected UTF-8 BOM (decode using >> utf-8-sig)", >> s, 0) >> else: >> if not isinstance(s, (bytes, bytearray)): >> > raise TypeError(f'the JSON object must be str, bytes or >> bytearray, ' >> f'not {s.__class__.__name__}') >> E TypeError: the JSON object must be str, bytes or >> bytearray, not list >> >> cls = None >> kw = {} >> object_hook = None >> object_pairs_hook = None >> parse_constant = None >> parse_float = None >> parse_int = None >> s = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], >> {}] >> =============================== >> >> As you can perhaps see from the bolded part (for >> /usr/local/lib/python3.8/dist-packages/django/db/models/fields/json.py:74), >> the value being written into the JSONField called "args" is a Python list, >> (shown as "s" on the last line of the traceback-with-values). I am aware >> of documented changes around serializers but did not think that affected >> me; however, given that I am using an FDW to return the data, is it that I >> should now be serialising returned data into a string? >> >> Any pointers appreciated. >> >> Thanks, Shaheed >> > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/f1392121-70f6-4359-8d2c-10dbb157607bn%40googlegroups.com.
