#32708: Django cron file lock breaks with django 3.2
-------------------------------------+-------------------------------------
Reporter: daillouf | Owner: nobody
Type: Bug | Status: new
Component: Core (Other) | Version: 3.2
Severity: Normal | Resolution:
Keywords: file lock, too many | Triage Stage:
connections, django_cron | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Description changed by daillouf:
Old description:
> Hello everyone !
>
> After I upgraded django to 3.2, I noticed that I had the following error
> popping up
>
> `django.db.utils.OperationalError: FATAL: sorry, too many clients
> already`
>
> I then inspected the postgresql connections, and my processes, and it
> turned out, that was my django cron jobs that were running multiple times
> at the same time, thus opening more connections than necessary.
>
>
> [Django cron has a lock mecanism to avoid this,](https://github.com/Tivix
> /django-cron/blob/v0.5.1/django_cron/backends/lock/file.py) that uses the
> lock function from django :
> ` from django.core.files import locks`
>
> But in django3.2 with latest django_cron version, that check totally
> fails and we can run a cron multiple times. I looked up a bit the code
> to see what's the issue.
>
> In django 3.2 [the code for that lock function is
> ](https://github.com/django/django/blob/main/django/core/files/locks.py)
>
>
> {{{
> def lock(f, flags):
> try:
> fcntl.flock(_fd(f), flags)
> return True
> except BlockingIOError:
> return False
>
> def unlock(f):
> fcntl.flock(_fd(f), fcntl.LOCK_UN)
> return True
> }}}
>
> So BlockingIOError are catched, and the function then returns false.
>
> [This changes the behavior from 3.1.7 :
> ](https://github.com/django/django/blob/stable/3.1.x/django/core/files/locks.py)
>
> {{{
>
> def lock(f, flags):
> ret = fcntl.flock(_fd(f), flags)
> return ret == 0
>
> def unlock(f):
> ret = fcntl.flock(_fd(f), fcntl.LOCK_UN)
> return ret == 0
>
> }}}
>
> in previous behaviour, checking for «ret==0» was indeed useless, because
> when flock fails, it raises a IOError,
>
> But in 3.1, the lock function didn't catch the error, and django_cron
> made good use of that behaviour.
>
> I believe that lock function is not documented though, so it's up to you
> to decide,
>
> whether it's Django or Django_cron that needs fixing.
>
> have a good day !
New description:
Hello everyone !
After I upgraded django to 3.2, I noticed that I had the following error
popping up
`django.db.utils.OperationalError: FATAL: sorry, too many clients
already`
I then inspected the postgresql connections, and my processes, and it
turned out, that was my django cron jobs that were running multiple times
at the same time, thus opening more connections than necessary.
[Django cron has a lock mecanism to avoid this,](https://github.com/Tivix
/django-cron/blob/v0.5.1/django_cron/backends/lock/file.py) that uses the
lock function from django :
` from django.core.files import locks`
But in django3.2 with latest django_cron version, that check totally fails
and we can run a cron multiple times. I looked up a bit the code to see
what's the issue.
In django 3.2 [the code for that lock function is
](https://github.com/django/django/blob/main/django/core/files/locks.py)
{{{
def lock(f, flags):
try:
fcntl.flock(_fd(f), flags)
return True
except BlockingIOError:
return False
def unlock(f):
fcntl.flock(_fd(f), fcntl.LOCK_UN)
return True
}}}
So BlockingIOError are catched, and the function then returns false.
[This changes the behavior from 3.1.7 :
](https://github.com/django/django/blob/stable/3.1.x/django/core/files/locks.py)
{{{
def lock(f, flags):
ret = fcntl.flock(_fd(f), flags)
return ret == 0
def unlock(f):
ret = fcntl.flock(_fd(f), fcntl.LOCK_UN)
return ret == 0
}}}
in previous behaviour, checking for «ret==0» was indeed useless, because
when flock fails, it raises a IOError,
But in 3.1, the lock function didn't catch the error, and django_cron made
good use of that behaviour.
Edit: that problem was spotted at code review but ignored :
[ https://github.com/django/django/pull/13410#discussion_r624988346]
have a good day !
--
--
Ticket URL: <https://code.djangoproject.com/ticket/32708#comment:1>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/066.5783ca2c48c0d26d8549d59128e9bfa0%40djangoproject.com.
