#35525: Django error with parsing JSONField
-------------------------------------+-------------------------------------
Reporter: Nirjal | Owner: nobody
Paudel |
Type: New | Status: new
feature |
Component: Database | Version: dev
layer (models, ORM) | Keywords:
Severity: Normal | Models,fields,encoding,decoding
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Hello, I have come across with error in django whilst parsing JSON fields
in Django.
This happened in Django==5.2.
Database I used: Postgres 14
So I had a django JSONField table with column called logs which is a json
field [ Not jsonb field ]. By default it the error followed as
{{{
>>> Synclogs.objects.last().logs
Traceback (most recent call last):
File "<console>", line 1, in <module>
File
"/Users/nirjalpaudel/Programming/test/django_new/.venv/lib/python3.12
/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/Users/nirjalpaudel/Programming/test/django_new/.venv/lib/python3.12
/site-packages/django/db/models/query.py", line 1110, in last
for obj in queryset[:1]:
File
"/Users/nirjalpaudel/Programming/test/django_new/.venv/lib/python3.12
/site-packages/django/db/models/query.py", line 400, in __iter__
self._fetch_all()
File
"/Users/nirjalpaudel/Programming/test/django_new/.venv/lib/python3.12
/site-packages/django/db/models/query.py", line 1928, in _fetch_all
self._result_cache = list(self._iterable_class(self))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/Users/nirjalpaudel/Programming/test/django_new/.venv/lib/python3.12
/site-packages/django/db/models/query.py", line 123, in __iter__
for row in compiler.results_iter(results):
File
"/Users/nirjalpaudel/Programming/test/django_new/.venv/lib/python3.12
/site-packages/django/db/models/sql/compiler.py", line 1500, in
apply_converters
value = converter(value, expression, connection)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/Users/nirjalpaudel/Programming/test/django_new/.venv/lib/python3.12
/site-packages/django/db/models/fields/json.py", line 94, in from_db_value
return json.loads(value, cls=self.decoder)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/json/__init__.py",
line 339, in loads
raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not list
}}}
Now, I come to see in the manual and code of postgres, using some kind of
custom encoder and decoder will have helped the code, I followed using
this
{{{
class SyncLogField(models.JSONField):
description = "A custom made json field for parsing and storing
logs"
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
def from_db_value(self, value, expression, connection):
if value is None:
return value
## Here I was skipping the value.
return value
}}}
And then I tried psycopg2 to see how does it interact with the postgres
library and turns out that for JSONField, I print the original type of
query parsed by the psyopg2 and tried to emulate what JSONField in django
would do. Here is the code I tried to emulate django behaviour
{{{
import psycopg2
import json
# Database connection parameters
DB_NAME = "testname"
DB_USER = "testuser"
DB_PASSWORD = "testpassword"
DB_HOST = "localhost"
DB_PORT = "5437"
# Connect to the PostgreSQL database
try:
connection = psycopg2.connect(
dbname=DB_NAME,
user=DB_USER,
password=DB_PASSWORD,
host=DB_HOST,
port=DB_PORT
)
cursor = connection.cursor()
# Perform the query to get the last log entry in ascending order by id
query = "SELECT logs FROM synclogs ORDER BY id ASC LIMIT 1"
cursor.execute(query)
# Fetch the result
result = cursor.fetchone()
if result:
# Get the original log
log = result[0]
# Print the type of the original log
print("Type of original log:", type(log))
# Parse the JSON log if it's a string
if isinstance(log, str):
log_data = json.loads(log)
print("Last log entry:", log_data)
else:
print("Log is not a JSON string:", log)
else:
print("No logs found in the synclogs table.")
except Exception as e:
print("An error occurred while connecting to the database or
querying:", str(e))
finally:
if connection:
cursor.close()
connection.close()
}}]
The result I got is that the json content got from db by psycopg2 is not
string but is a list, so I had to skip the value entirely in my custom
serializer.
Instead of doing this, why don't we check if the content/value parsed from
db is list or dict, by default python types for json
If yes - Don't do anything
If no and is a string instance - then perform json.loads() to it
I think that if we do this changes in database, we will have a JSONField
that will work for both JSON and JSONB column in postgres
--
Ticket URL: <https://code.djangoproject.com/ticket/35525>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/01070190165c6ca5-dc0094bb-22ba-49d1-a95e-5bae85de3873-000000%40eu-central-1.amazonses.com.
