Also, thank you Tom, Christian, Christophe, and Anssi for helping me track down this problem. Users are now receiving notifications in a timely manner because of you.
On Tuesday, May 28, 2013 6:07:22 PM UTC-4, Chris Conover wrote: > > Adding commit_unless_managed() before the get() seems to fix the problem. > Looking at it locally, the Gearman worker starts a transactions and just > calls commit when the processing is done, over and over, never starting a > new transaction. That combined with REPEATABLE-READ seems to be the culprit. > > My next step was going to be to set the isolation level to READ-COMMITTED > which would probably have fixed it as well. Thought I'm not sure how that > would have affected performance. > > Will the changes to transactions in 1.6 possibly address this issue so I > can remove the commit_unless_managed() call? It looks like the major change > is using the underlying database's autocommit mode rather than emulating it > in the ORM. I suppose I'll have to test... > > On Tuesday, May 28, 2013 5:43:31 PM UTC-4, Chris Conover wrote: >> >> I bribed the DBA to turn on the general query log briefly (which >> generated 327k logs in 2 minutes and 2 seconds). Looking at the logs, I see >> the transaction block for the INSERT. The SELECT is definitely coming after >> the COMMIT in a different thread. I'll try the commit_unless_managed fix >> quickly and if that doesn't work, I'll post the full query trace. >> >> On Tuesday, May 28, 2013 5:27:36 PM UTC-4, akaariai wrote: >>> >>> On 28 touko, 22:20, Chris Conover <[email protected]> wrote: >>> > Well, you can inspect the object and see it's primary key. Surely that >>> > means the INSERT is completed? So the workflow goes like this: >>> > >>> > foo = Foo() >>> > foo.save() >>> > foo.pk # new primary key is available >>> > submit_gearman_task(foo.pk) >>> > >>> > Then in the Gearman worker: >>> > >>> > foo = Foo.objects.get(pk=foo_pk) # this causes a Foo.DoesNotExist >>> exception >>> > >>> > That's what I don't understand. The primary key is assigned and >>> available >>> > in the application layer in one place but then the lookup using that >>> > primary key fails in another place. Maybe it has something to do with >>> the >>> > fact that the database is set to REPEATABLE-READ isolation level >>> versus >>> > READ-COMMITTED. Still investigating... >>> >>> Could it be that the Gearman worker is in a transaction? When you >>> issue Foo.objects.get(), if the transaction Gearman is in has started >>> before the save() you will not see the new object (as MySQL uses >>> REPEATABLE READ transaction). >>> >>> Note that any read query will start a transaction implicitly, and ORM >>> writes commit implicitly started transaction. You can also use >>> connection.commit_unless_managed() to commit implicitly started >>> transaction (not part of public API!). So, even if you haven't >>> explicitly started any transactions in Gearman worker an earlier read >>> query might have started one for you. (If you find this behavior >>> confusing this is one of the reasons why transaction handling will be >>> changed in Django 1.6). >>> >>> In short: try if connection.commit_unless_managed() before the get() >>> changes anything. If so, investigate why there is an ongoing >>> transaction in the worker. >>> >>> - Anssi >>> >> -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/django-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
