subject:"\[Django\] #25432\: Django ORM race condition"

Re: [Django] #25432: Django ORM race condition

2015-09-20 Thread Django

#25432: Django ORM race condition
-+-
 Reporter:  yuvadm   |Owner:  nobody
 Type:  Bug  |   Status:  closed
Component:  Database layer   |  Version:  1.8
  (models, ORM)  |
 Severity:  Normal   |   Resolution:  invalid
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by aaugustin):

 * status:  new => closed
 * resolution:   => invalid


Comment:

 > This is happening on Heroku Postgres.

 You should ask the Heroku Postgres support.

 I'm going to close this ticket because there is no evidence of a bug in
 Django.

 You can't reproduce with a local Postgres; you can reproduce with Heroku
 Postgres; so start there :-)

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.d30548ec355dd3bff892757de24a7ea3%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #25432: Django ORM race condition

2015-09-19 Thread Django

#25432: Django ORM race condition
-+-
 Reporter:  yuvadm   |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  1.8
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by yuvadm):

 Replying to [comment:8 shaib]:

 > Can you elaborate on that a  little? In  particular, is the test case
 using Django's test-client for these calls? If it is, there's one thread
 on the server doing everything, and it all happens in the same
 transaction; I'd ask you to check again about caching and replication and
 that sort of thing.

 This isn't Django test client, I'm calling from any external process
 (specifically I'm testing with curl from command line).

 > If it isn't the test-client (or even the Django test framework), I'd ask
 you to verify again that the second request is only sent after full
 processing of the first request has completed.

 Yep, I'm using {{{curl http://example.com/foo && curl
 http://example.com/foo}}}

 > Either way, please also verify that your PostgreSQL configuration is
 sane. I've seen recommendations to make testing faster by turning off its
 fsync etc, and that could be the cause of such behavior.

 This is happening on Heroku Postgres. Any recommendations on how to tinker
 with the config there? Is it known to be problematic?

 > On a side note: I'm pretty sure get_or_create ''only'' works reliably
 under serializable transactions; under the default (read-committed)
 isolation level there are failure scenarios, whether you try to get first
 or create first.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.14311a260c4e2ba2610f1b2e2634f250%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #25432: Django ORM race condition

2015-09-19 Thread Django

Comment (by shaib):

> I have seen this happen in a test case where I make two subsequent API
calls that do exactly this and got a ~5% failure rate.

Can you elaborate on that a little? In particular, is the test case
using Django's test-client for these calls? If it is, there's one thread
on the server doing everything, and it all happens in the same
transaction; I'd ask you to check again about caching and replication and
that sort of thing.

If it isn't the test-client (or even the Django test framework), I'd ask
you to verify again that the second request is only sent after full
processing of the first request has completed.

Either way, please also verify that your PostgreSQL configuration is sane.
I've seen recommendations to make testing faster by turning off its fsync
etc, and that could be the cause of such behavior.

On a side note: I'm pretty sure get_or_create ''only'' works reliably
under serializable transactions; under the default (read-committed)
isolation level there are failure scenarios, whether you try to get first
or create first.

--
Ticket URL:
Django
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/064.dd0782c6e9b97a64b4f2beee527de683%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #25432: Django ORM race condition

2015-09-19 Thread Django

Comment (by aaugustin):

Actually making both requests from the same thread doesn't matter because
they may be handled by different processes on the server.

Even at the SERIALIZABLE isolation level, the database is allowed to
serialize transactions made on different connections in any consistent
order.

I think this is part of the consistency model of multi-processes databases
servers, which are distributed systems even if they run on one server

I don't think there's anything specific to Django here. You should be able
to reproduce this behavior with plain WSGI & pyscopg2.

--
Ticket URL:
Django
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/064.b258d7604fa3996eaec8e9ced7dcc9b0%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #25432: Django ORM race condition

2015-09-19 Thread Django

#25432: Django ORM race condition
-+-
 Reporter:  yuvadm   |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  1.8
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by aaugustin):

 The SERIALIZABLE isolation level is the answer to question 1. at the
 bottom of your bug report, though.

 Regarding question 2. we don't want to document it because too many people
 will think it's a good idea without understanding the consequences and
 because it breaks some APIs e.g. get_or_create doesn't work anymore (I
 think).

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.84892d198abbdecb04e34c44aeafb8ff%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #25432: Django ORM race condition

2015-09-19 Thread Django

#25432: Django ORM race condition
-+-
 Reporter:  yuvadm   |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  1.8
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by yuvadm):

 Replying to [comment:4 aaugustin]:
 > Just to be clear, the scenario is the following:
 >

 Yes, exactly.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.bf77fa468f288fbaec80ce55ec642697%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #25432: Django ORM race condition

2015-09-19 Thread Django

#25432: Django ORM race condition
-+-
 Reporter:  yuvadm   |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  1.8
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by aaugustin):

 Just to be clear, the scenario is the following:

 - you make the first API request
 - the server processes this request, performs some database operations,
 commits the database transaction, sends back the response
 - you receive and process the response to the first request
 - you make the second API request from the same thread that made the first
 request
 - the server cannot see the data written by the first request

 You question almost looks like the answer should involve isolation levels,
 specifically the SERIALIZABLE level, but if my description above is
 correct that cannot explain the behavior you're describing.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.db49f363c386d198b3eda2fb4775227c%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #25432: Django ORM race condition

2015-09-19 Thread Django

#25432: Django ORM race condition
-+-
 Reporter:  yuvadm   |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  1.8
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by yuvadm):

 Replying to [comment:2 aaugustin]:
 > Is it possible that you're writing to the primary and reading from a
 replica? In that case all you're seeing is the replication delay. That
 would explain why you can't reproduce locally.

 Nope, there's only a single Postgres database at play.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.a048d154d5fd1ed9a0ee643b3230f8fc%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #25432: Django ORM race condition

2015-09-19 Thread Django

#25432: Django ORM race condition
-+-
 Reporter:  yuvadm   |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  1.8
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by aaugustin):

 Is it possible that you're writing to the primary and reading from a
 replica? In that case all you're seeing is the replication delay. That
 would explain why you can't reproduce locally.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.b382acc71b68f208023ff8723d9ff2ef%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #25432: Django ORM race condition

2015-09-19 Thread Django

#25432: Django ORM race condition
-+-
 Reporter:  yuvadm   |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  1.8
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by yuvadm):

 * needs_better_patch:   => 0
 * needs_tests:   => 0
 * needs_docs:   => 0


Comment:

 This has also been referenced in
 https://code.djangoproject.com/ticket/20429#comment:22

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.1495927529427c7502caff1bff565433%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

[Django] #25432: Django ORM race condition

2015-09-19 Thread Django

#25432: Django ORM race condition
--+
 Reporter:  yuvadm|  Owner:  nobody
 Type:  Bug   | Status:  new
Component:  Database layer (models, ORM)  |Version:  1.8
 Severity:  Normal|   Keywords:
 Triage Stage:  Unreviewed|  Has patch:  0
Easy pickings:  0 |  UI/UX:  0
--+
 I've hit an interesting problem that isn't covered by the current Django
 documentation, and might even be a bug that Django can handle better. It
 started off as a SO question at http://stackoverflow.com/q/32661885/24545
 but here's the gist of it.

 After creating a new object {{{MyModel.objects.create(foo=goo)}}} and
 inserting it into the database, it is possible that immediate subsequent
 calls to fetch that object might fail (i.e.
 {{{MyModel.objects.get(foo=goo)}}} will throw {{{DoesNotExist}}}). I have
 seen this happen in a test case where I make two subsequent API calls that
 do exactly this and got a ~5% failure rate.

 In most cases this might not be a problem, but I am using this query to
 make sure I'm not creating two duplicate objects. This is essentially an
 UPSERT problem. In my case, my solution was to set {{{unique=True}}} on my
 {{{foo}}} field and attempt to create the object in any case, which will
 naturally fail on a duplicate, then I just catch the {{{IntegrityError}}}
 and fail gracefully. In this we we use DB semantics which guarantees no
 duplicates.

 The relevant settings for this test case are: PostgreSQL, default Django
 transaction settings and no specific caching.

 So 2 questions here:

 1. What happens if my application '''requires''' that any two transactions
 {{{a}}} and {{{b}}} behave such that {{{b}}} always sees fresh data that
 was written in {{{a}}}? How can I enforce this in Django?
 2. How do we document this behavior in a better way? If this is possible
 or impossible, Django must be clearer on how such transactions are
 handled.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/049.95dfb6ef23113cb64a09695e864f72d9%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #25432: Django ORM race condition

Re: [Django] #25432: Django ORM race condition

Re: [Django] #25432: Django ORM race condition

Re: [Django] #25432: Django ORM race condition

Re: [Django] #25432: Django ORM race condition

Re: [Django] #25432: Django ORM race condition

Re: [Django] #25432: Django ORM race condition

Re: [Django] #25432: Django ORM race condition

Re: [Django] #25432: Django ORM race condition

Re: [Django] #25432: Django ORM race condition

[Django] #25432: Django ORM race condition

11 matches

Site Navigation

Mail list logo

Footer information