On 20/11/2013 4:33 PM, Brian May wrote:
On 18 October 2013 13:13, Rasjid Wilcox <[email protected]
<mailto:[email protected]>> wrote:
The insert into get_data_guard should be completely atomic - it
will only ever succeed for one caller, and so it should eliminate
the race condition.
Hmmm. Wonder how this will work with transactions...
Just tried running the following test code:
I can't really comment a lot, since I've avoided django after looking at
it a number of years ago and deciding that its ORM was too limited for
my needs. You may need to drop down to raw sql, rather than relying on
the django ORM. At the very least you will want to look at exactly what
sql is being sent to the server. See
https://docs.djangoproject.com/en/dev/faq/models/#how-can-i-see-the-raw-sql-queries-django-is-running
for info on that, although I don't know if that will tell you when each
transaction is starting and stopping.
However, looking at your code below, I'm guessing that the issue is you
are wrapping the entire process up in a single transaction.
I think you would be better off with two steps or at least two transactions:
My original post suggested having a new and completely separate table
just to do the guarding, and that may be the simpler way.
If you don't want to do that, I would suggest using two transactions.
In transaction number one you attempt to insert your key (start + stop +
machine?) into your table, with a flag saying that the data is being
generated. This should either succeed (if you are the first) or fail
(for subsequent attempts) almost instantly.
If the key insert is successful, you generate the data (outside a
database transaction). You then (in transaction two) store the data
into the table and clear the 'being generated' flag.
If the key insert fails, then someone else has already or is currently
generating the data. You read the table - if the 'being generated' flag
is clear, your data should already be there. If the flag is true, you
wait a bit and try again. (With Postgresql you could use NOTIFY and
LISTEN rather than polling.)
With either approach, you want a transaction just around the insert, not
the whole data generation process. If this is not easy to code using
Django's ORM, then I would suggest going the separate 'guard table'
route and using raw sql (just for that table).
Cheers,
Rasjid.
--- cut ---
#!/usr/bin/env python
import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'kgadmin.conf.settings'
import datetime
import time
import django.db.transaction
from karaage.machines.models import Machine
from karaage.cache.models import MachineCache
@django.db.transaction.commit_on_success
def meow():
start=datetime.date(year=2003,month=12,day=25)
end=datetime.date(year=2003,month=12,day=25)
print "meowing"
machine = Machine.objects.get(pk=1)
print "barking"
MachineCache.objects.create(machine=machine, start=start, end=end,
cpu_hours=10, no_jobs=10)
print "sleeping"
time.sleep(10)
print "eating"
raise RuntimeError("oops")
meow()
--- cut ---
If I run one copy, it works fine:
meowing
barking
sleeping
eating
Traceback (most recent call last):
File "./test", line 24, in <module>
meow()
File "/usr/lib/python2.7/dist-packages/django/db/transaction.py",
line 223, in inner
return func(*args, **kwargs)
File "./test", line 22, in meow
raise RuntimeError("oops")
RuntimeError: oops
If I run another copy while the first one is sleeping, the 2nd copy
gets locked out until the first one finishes:
meowing
barking
[ hangs here ]
sleeping
eating
Traceback (most recent call last):
File "./test", line 24, in <module>
meow()
File "/usr/lib/python2.7/dist-packages/django/db/transaction.py",
line 223, in inner
return func(*args, **kwargs)
File "./test", line 22, in meow
raise RuntimeError("oops")
RuntimeError: oops
I wasn't aware a transaction could lock out other processes like this.
Something to be careful of when doing http requests that should return
quickly.
Would be interested to know just what is happening here, and what the
scope is of this "lock".
(also note this is Django 1.5, apparently transactions support is
different in 1.6; I don't really understand what has changed however).
Will continue looking at this tomorrow.
--
Brian May <[email protected]
<mailto:[email protected]>>
_______________________________________________
melbourne-pug mailing list
[email protected]
https://mail.python.org/mailman/listinfo/melbourne-pug
_______________________________________________
melbourne-pug mailing list
[email protected]
https://mail.python.org/mailman/listinfo/melbourne-pug
