#32542: Native postgres json field support is broken
-------------------------------------+-------------------------------------
Reporter: Artem | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 3.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Description changed by Artem:
Old description:
> In Django 3.1.1, there was a refactoring of encoders/decoders of DB
> fields which affected postgres json/jsonb field support.
>
> Before:
>
> Postgres **json** and **jsonb** field values are encoded/decoded into
> python native format by **psycopg2**. **JSONField** had it's own decoder
> but if was omitted if db has native json support.
>
> After:
>
> **Jsonb** field is no longer decoded by **psycopg2** (an empty stub is
> used instead of `json.joads`), all decoding stuff is delegated to
> `JSONField.from_db_value(...)`.
> But postgres **json** field is still decoded by **psycopg2**, which
> causes 'TypeError' while trying to decode it again by `JSONField`.
>
> Breaking commit:
> https://github.com/django/django/commit/0be51d2226fce030ac9ca840535a524f41e9832c
>
> Example:
>
> {{{
> class CustomJSONField(models.JSONField):
>
> def db_type(self, *args, **kwargs):
> # stored as-is, keys order is preserved
> return 'json'
>
> class TestModel(models.Model)
> json_data = CustomJSONField(default=dict)
>
> TestModel(json_data={"foo": "bar"}).save()
> TestModel.objects.last()
> }}}
>
> **Expected behavior:** code snippet works
>
> **Actual behavior:**
> {{{
> TypeError: the JSON object must be str, bytes or bytearray, not dict
> }}}
>
> The issue was mentioned in few tickets (#31973, #31956) but eventually
> their authors found workarounds (e.g. by switching to **jsonb** field) or
> still using Django <= 3.1. I think support of postgres json (not only
> jsonb) is important too.
>
> The possible fix is to add the decoder stub for **json** field too:
>
> Before
> (https://github.com/django/django/blob/main/django/db/backends/postgresql/base.py#L218):
> {{{
> psycopg2.extras.register_default_jsonb(conn_or_curs=connection,
> loads=lambda x: x)
> }}}
>
> After:
>
> {{{
> psycopg2.extras.register_default_json(conn_or_curs=connection,
> loads=lambda x: x)
> psycopg2.extras.register_default_jsonb(conn_or_curs=connection,
> loads=lambda x: x)
> }}}
New description:
In Django 3.1.1, there was a refactoring of encoders/decoders of DB fields
which affected postgres json/jsonb field support.
Before:
Postgres **json** and **jsonb** field values are encoded/decoded into
python native format by **psycopg2**. **JSONField** had it's own decoder
but if was omitted if db has native json support.
After:
**Jsonb** field is no longer decoded by **psycopg2** (an empty stub is
used instead of `json.joads`), all decoding stuff is delegated to
`JSONField.from_db_value(...)`.
But postgres **json** field is still decoded by **psycopg2**, which causes
'TypeError' while trying to decode it again by `JSONField`.
Breaking commit:
https://github.com/django/django/commit/0be51d2226fce030ac9ca840535a524f41e9832c
Example:
{{{
class CustomJSONField(models.JSONField):
def db_type(self, *args, **kwargs):
# stored as-is, keys order is preserved
return 'json'
class TestModel(models.Model)
json_data = CustomJSONField(default=dict)
TestModel(json_data={"foo": "bar"}).save()
TestModel.objects.last()
}}}
**Expected behavior:** code snippet works (as in Django 3.1).
**Actual behavior:**
{{{
TypeError: the JSON object must be str, bytes or bytearray, not dict
}}}
The issue was mentioned in few tickets (#31973, #31956) but eventually
their authors found workarounds (e.g. by switching to **jsonb** field) or
still using Django <= 3.1. I think support of postgres json (not only
jsonb) is important too.
The possible fix is to add the decoder stub for **json** field too:
Before
(https://github.com/django/django/blob/main/django/db/backends/postgresql/base.py#L218):
{{{
psycopg2.extras.register_default_jsonb(conn_or_curs=connection,
loads=lambda x: x)
}}}
After:
{{{
psycopg2.extras.register_default_json(conn_or_curs=connection,
loads=lambda x: x)
psycopg2.extras.register_default_jsonb(conn_or_curs=connection,
loads=lambda x: x)
}}}
--
--
Ticket URL: <https://code.djangoproject.com/ticket/32542#comment:1>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/063.169a2423047cfd71ab6b9d9caf906cfd%40djangoproject.com.
