Re: Intermittent IntegrityError on Model Save with auto_now and auto_now_add Fields
Looks like this issue isn't affecting lots of folks, since nobody is piping up, so I'll just add that if anybody arrives here in the future, we'll be tracking this in a public issue here: https://github.com/freelawproject/courtlistener/issues/3359 My theory is it has something to do with race conditions/timings and async, since I *think* it started around the time we started switching to async, but others on the team tell me that's a red herring (it's not the cause). Anyhow, if anybody ever arrives here in the future, we fixed it <https://github.com/grossir/courtlistener/commit/91b38620fcc837be5f848f0f594d7ae968439a1f> by overriding the save method to catch IntegrityErrors. It's pretty lame to have code like this in our save method, but it seems to be working: try: # Without a transaction wrapper, a failure will invalidate outer transactions with transaction.atomic(): super(Docket, self).save( update_fields=update_fields, *args, **kwargs ) except IntegrityError: # Temporary patch while we solve #3359 # If the error is not related to `date_modified` it will raise again self.date_modified = timezone.now() super(Docket, self).save( update_fields=update_fields, *args, **kwargs ) Anyhow, thanks for reading, everybody. On Monday, March 18, 2024 at 7:08:22 AM UTC-7 Matthew Pava wrote: > Hi Bill, > > We ended up using a package called Django-audit-log: > https://pypi.org/project/django-audit-log/. > > It’s outdated now, but we used the source code for their > CreatingUserField, LastUserField, CreationDateTimeField, and > ModificationDateTimeField. More modern packages may have enhanced features > than these. > > > > In your case, I may just add a save method to the abstract model and add > timezone.now() to the corresponding fields. If there’s not a pk, then > populate the created date field. > > > > You are right, though. Perhaps the Django community can explain the issue > between auto_now and auto_now_add fields. > > > > *From:* 'William Palin' via Django developers (Contributions to Django > itself) > *Sent:* Sunday, March 17, 2024 9:59 AM > *To:* Django developers (Contributions to Django itself) < > [email protected]> > *Subject:* Intermittent IntegrityError on Model Save with auto_now and > auto_now_add Fields > > > > Hello Django community, > > > > We are reaching out after encountering a persistent and elusive issue that > manifests as an IntegrityError during the save operation of a Django model. > This problem has been sporadically occurring since last year and has > successfully stumped four seasoned Django developers on our team. The error > seems to involve the non-update of auto_now fields upon model save, leading > us to suspect a deeper issue within Django, though we are cautious about > drawing premature conclusions. Given the complex and intermittent nature of > this bug, we are seeking insights, advice, or any form of guidance from the > community. > > > > *Issue Overview:* > > > > Our model inherits date_created and date_modified fields from an abstract > base class designed to standardize these timestamps across our models. Here > is how the abstract base class is defined: > > > > > > class AbstractDateTimeModel(models.Model): > > """An abstract base class for most models""" > > > > date_created = models.DateTimeField( > > help_text="The moment when the item was created.", > > auto_now_add=True, > > db_index=True, > > ) > > date_modified = models.DateTimeField( > > help_text="The last moment when the item was modified. A value in > year" > > " 1750 indicates the value is unknown", > > auto_now=True, > > db_index=True, > > ) > > > > class Meta: > > abstract = True > > > > *The Problem:* > > > > Intermittently, the .save() method triggers an IntegrityError, apparently > because the auto_now field (date_modified) does not get created. A specific > instance of this error showed that while date_created is correctly > populated, date_modified remains null, which directly leads to the > IntegrityError upon attempting to insert the record: > > > > [datetime.datetime(2023, 10, 2, 14, 33, 49, 99833, > tzinfo=datetime.timezone.utc), None, ...] > > 'INSERT INTO "search_docket" ("date_created", "date_modified",... ] > > > > *What We've Tried
RE: Intermittent IntegrityError on Model Save with auto_now and auto_now_add Fields
Hi Bill,
We ended up using a package called Django-audit-log:
https://pypi.org/project/django-audit-log/.
It’s outdated now, but we used the source code for their CreatingUserField,
LastUserField, CreationDateTimeField, and ModificationDateTimeField. More
modern packages may have enhanced features than these.
In your case, I may just add a save method to the abstract model and add
timezone.now() to the corresponding fields. If there’s not a pk, then populate
the created date field.
You are right, though. Perhaps the Django community can explain the issue
between auto_now and auto_now_add fields.
From: 'William Palin' via Django developers (Contributions to Django itself)
Sent: Sunday, March 17, 2024 9:59 AM
To: Django developers (Contributions to Django itself)
Subject: Intermittent IntegrityError on Model Save with auto_now and
auto_now_add Fields
Hello Django community,
We are reaching out after encountering a persistent and elusive issue that
manifests as an IntegrityError during the save operation of a Django model.
This problem has been sporadically occurring since last year and has
successfully stumped four seasoned Django developers on our team. The error
seems to involve the non-update of auto_now fields upon model save, leading us
to suspect a deeper issue within Django, though we are cautious about drawing
premature conclusions. Given the complex and intermittent nature of this bug,
we are seeking insights, advice, or any form of guidance from the community.
Issue Overview:
Our model inherits date_created and date_modified fields from an abstract base
class designed to standardize these timestamps across our models. Here is how
the abstract base class is defined:
class AbstractDateTimeModel(models.Model):
"""An abstract base class for most models"""
date_created = models.DateTimeField(
help_text="The moment when the item was created.",
auto_now_add=True,
db_index=True,
)
date_modified = models.DateTimeField(
help_text="The last moment when the item was modified. A value in year"
" 1750 indicates the value is unknown",
auto_now=True,
db_index=True,
)
class Meta:
abstract = True
The Problem:
Intermittently, the .save() method triggers an IntegrityError, apparently
because the auto_now field (date_modified) does not get created. A specific
instance of this error showed that while date_created is correctly populated,
date_modified remains null, which directly leads to the IntegrityError upon
attempting to insert the record:
[datetime.datetime(2023, 10, 2, 14, 33, 49, 99833,
tzinfo=datetime.timezone.utc), None, ...]
'INSERT INTO "search_docket" ("date_created", "date_modified",... ]
What We've Tried:
* Investigated the possibility of an issue with update_fields being
non-null and excluding date_modified during .save(), but confirmed through
Sentry logs that update_fields was indeed None in all instances of the error.
* Attempted to reproduce the issue in a controlled environment without
success, leaving us without a clear direction for a solution.
Request for Help:
We're wondering if this could point to an undocumented edge case in Django's
auto_now and auto_now_add implementation or a specific database behavior under
certain conditions. Any advice on further debugging steps, experiences with
similar issues, or knowledge of potential Django nuances that we might not be
considering would be incredibly valuable.
We appreciate your time and any feedback or suggestions you can offer.
Thanks
Bill
django v5.0.2
python 3.12.2
db is postgres
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
[email protected]<mailto:[email protected]>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/a788c065-3a96-472c-9b41-7c2aebca6967n%40googlegroups.com<https://groups.google.com/d/msgid/django-developers/a788c065-3a96-472c-9b41-7c2aebca6967n%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/DS7PR15MB5910838A1CE2563A49DADC61822D2%40DS7PR15MB5910.namprd15.prod.outlook.com.
Intermittent IntegrityError on Model Save with auto_now and auto_now_add Fields
Hello Django community,
We are reaching out after encountering a persistent and elusive issue that
manifests as an IntegrityError during the save operation of a Django model.
This problem has been sporadically occurring since last year and has
successfully stumped four seasoned Django developers on our team. The error
seems to involve the non-update of auto_now fields upon model save, leading
us to suspect a deeper issue within Django, though we are cautious about
drawing premature conclusions. Given the complex and intermittent nature of
this bug, we are seeking insights, advice, or any form of guidance from the
community.
*Issue Overview:*
Our model inherits date_created and date_modified fields from an abstract
base class designed to standardize these timestamps across our models. Here
is how the abstract base class is defined:
class AbstractDateTimeModel(models.Model):
"""An abstract base class for most models"""
date_created = models.DateTimeField(
help_text="The moment when the item was created.",
auto_now_add=True,
db_index=True,
)
date_modified = models.DateTimeField(
help_text="The last moment when the item was modified. A value in
year"
" 1750 indicates the value is unknown",
auto_now=True,
db_index=True,
)
class Meta:
abstract = True
*The Problem:*
Intermittently, the .save() method triggers an IntegrityError, apparently
because the auto_now field (date_modified) does not get created. A specific
instance of this error showed that while date_created is correctly
populated, date_modified remains null, which directly leads to the
IntegrityError upon attempting to insert the record:
[datetime.datetime(2023, 10, 2, 14, 33, 49, 99833,
tzinfo=datetime.timezone.utc), None, ...]
'INSERT INTO "search_docket" ("date_created", "date_modified",... ]
*What We've Tried:*
- Investigated the possibility of an issue with update_fields being
non-null and excluding date_modified during .save(), but confirmed through
Sentry logs that update_fields was indeed None in all instances of the
error.
- Attempted to reproduce the issue in a controlled environment without
success, leaving us without a clear direction for a solution.
*Request for Help:*
We're wondering if this could point to an undocumented edge case in
Django's auto_now and auto_now_add implementation or a specific database
behavior under certain conditions. Any advice on further debugging steps,
experiences with similar issues, or knowledge of potential Django nuances
that we might not be considering would be incredibly valuable.
We appreciate your time and any feedback or suggestions you can offer.
Thanks
Bill
django v5.0.2
python 3.12.2
db is postgres
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/a788c065-3a96-472c-9b41-7c2aebca6967n%40googlegroups.com.
