Re: Understanding Django transactions

[Michael]

Hi Aymeric,

That's awesome, really appreciate you clarifying all of that! Look forward 
to watching the video of your talk, the slides look great.

Cheers!


On Tuesday, 21 May 2013 17:43:00 UTC+1, Aymeric Augustin wrote:
>
> Hi Michael,
>  
>>
>> *Django 1.5:*
>>
>
>>- The database-level auto-commit is turned OFF.
>>
>> Yes. 
>
>>
>>- SQL standards dictate that every query opens a transaction if one 
>>does not already exist. This happens regardless of the database's 
>>auto-commit setting.
>>
>> Yes, this is required by PEP 249 and taken care of by the database 
> adapter. 
>
>>
>>- After each Django ORM operation (read or write), Django commits 
>>this open transaction. This emulates the database-level auto-commit.
>>
>> No, transactions are only committed after write operations, so it isn't 
> exactly like database-level autocommit. 
>
>>
>>- Transaction decorators cannot be nested. When one is opened, any 
>>previous transaction will be committed.
>>
>>  Not necessarily. It's complicated. It depends on the previous state 
> (auto / managed), the current state (auto / managed) and the dirty flag.
>
> I never understood the logic. Your best chance is to read the source if 
> you really need to know how it works.
>
>>
>>- When executing raw SQL (with a database cursor), Django marks the 
>>current transaction as dirty but does not issue a commit. If data changes 
>>were made then they need manually committing. *Why do the 
>> **docs*
>>* say that you only need to commit the change if data was changed? If 
>>the transaction is marked as dirty regardless of a read or a write, would 
>>it not always need committing or rolling back to ensure the transaction 
>> is 
>>properly closed by the end of the connection?*
>>
>> In Django 1.2 and earlier, the connection wasn't marked dirty on raw SQL 
> operations. If you didn't commit and you didn't perform any other writes 
> until the end of the query, Django would issue a rollback.
>
> In Django 1.3-1.5, I'm not sure what happens exactly. I believe the 
> transaction management will always issue a commit or a rollback before the 
> connection is aborted and closed by the request_finished signal, but I 
> could be wrong.
>
> Honestly, when I rewrote transaction management, no one understood how it 
> interacted with the dirty flag. So I can't answer this question and I don't 
> think anyone else can. Once again, read the source if you need to know how 
> it works in a specific case.
>
>>
>>- Setting TRANSACTIONS_MANAGED to True stops Django from sending any 
>>commits after ORM operations. The database-level auto-commit is still 
>>disabled.* With this setting, using any Django ORM read or write 
>>operation (all wrap**ped in `transaction.commit_on_success`) fails 
>>with TransactionManagementError('This code isn't under transaction 
>>management'). Is this expected?*
>>
>> I have no idea.
>
>>
>>
>> *Django 1.6:*
>>
>>- The database-level auto-commit is turned ON. 
>>- Every database operation via the ORM will be committed using the 
>>database's auto-commit, including any custom SQL executed with the 
>> database 
>>cursor. 
>>- The `atomic` decorator / context manager either starts a new 
>>transaction, or creates a new savepoint if it's nested within an existing 
>>transaction. They are committed as long as no exception is raised. 
>>- If ATOMIC_REQUESTS is specified in the database config, all views 
>>are wrapped in `atomic`, unless it's wrapped in `non_atomic_requests`.
>>
>> Yes, these four sentences are correct. 
>
>>
>>- If you set AUTOCOMMIT to False in a database configuration, this 
>>will disable the database-level auto-commit. All DB reads and writes will 
>>need manually wrapping in transactions. *The 
>> **docs*
>>* say that with this disabled, Django won't perform any commits. Does 
>>this mean that the `atomic` decorator won't work properly and you have to 
>>use the underlying database library to handle transactions? In 1.6 it 
>> would 
>>appear that Django never performs a commit outside of `atomic`, so I'm 
>>confused by this comment!* 
>>
>> First, it's extremely unlikely that you will ever need to set AUTOCOMMIT 
> to False. This is only useful if you want to implement your own transaction 
> management, and I don't know anyone who's done that, nor a use case for 
> doing that. I'd be surprised if this discussion was relevant to a single 
> person on this mailing list (but I've been surprised before!). Also, your 
> link goes to the 1.5 docs, not the dev docs. This may have confused you.
>
> Anyway, in this scenario, `atomic` will work as expected. When autocommit 
> is off, you're always in a transaction, and as a 

Re: Understanding Django transactions

[Christophe Pettus]

On May 21, 2013, at 12:43 PM, Aymeric Augustin wrote:
> Anyway, in this scenario, `atomic` will work as expected. When autocommit is 
> off, you're always in a transaction, and as a consequence `atomic` uses 
> savepoints to guarantee atomicity; it'll never commit. You have to call 
> transaction.commit() at some point to save changes.

Just to clarify, `atomic` will commit in this scenario:

with atomic:
   my_model_object.save()

You don't have to explicitly call transaction.commit() to issue a commit after 
the .save(), correct?

--
-- Christophe Pettus
   x...@thebuild.com

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Understanding Django transactions

[Aymeric Augustin]

Hi Michael,
 
>
> *Django 1.5:*
>

>- The database-level auto-commit is turned OFF.
>
> Yes. 

>
>- SQL standards dictate that every query opens a transaction if one 
>does not already exist. This happens regardless of the database's 
>auto-commit setting.
>
> Yes, this is required by PEP 249 and taken care of by the database 
adapter. 

>
>- After each Django ORM operation (read or write), Django commits this 
>open transaction. This emulates the database-level auto-commit.
>
> No, transactions are only committed after write operations, so it isn't 
exactly like database-level autocommit. 

>
>- Transaction decorators cannot be nested. When one is opened, any 
>previous transaction will be committed.
>
>  Not necessarily. It's complicated. It depends on the previous state (auto 
/ managed), the current state (auto / managed) and the dirty flag.

I never understood the logic. Your best chance is to read the source if you 
really need to know how it works.

>
>- When executing raw SQL (with a database cursor), Django marks the 
>current transaction as dirty but does not issue a commit. If data changes 
>were made then they need manually committing. *Why do the 
> **docs*
>* say that you only need to commit the change if data was changed? If 
>the transaction is marked as dirty regardless of a read or a write, would 
>it not always need committing or rolling back to ensure the transaction is 
>properly closed by the end of the connection?*
>
> In Django 1.2 and earlier, the connection wasn't marked dirty on raw SQL 
operations. If you didn't commit and you didn't perform any other writes 
until the end of the query, Django would issue a rollback.

In Django 1.3-1.5, I'm not sure what happens exactly. I believe the 
transaction management will always issue a commit or a rollback before the 
connection is aborted and closed by the request_finished signal, but I 
could be wrong.

Honestly, when I rewrote transaction management, no one understood how it 
interacted with the dirty flag. So I can't answer this question and I don't 
think anyone else can. Once again, read the source if you need to know how 
it works in a specific case.

>
>- Setting TRANSACTIONS_MANAGED to True stops Django from sending any 
>commits after ORM operations. The database-level auto-commit is still 
>disabled.* With this setting, using any Django ORM read or write 
>operation (all wrap**ped in `transaction.commit_on_success`) fails 
>with TransactionManagementError('This code isn't under transaction 
>management'). Is this expected?*
>
> I have no idea.

>
>
> *Django 1.6:*
>
>- The database-level auto-commit is turned ON. 
>- Every database operation via the ORM will be committed using the 
>database's auto-commit, including any custom SQL executed with the 
> database 
>cursor. 
>- The `atomic` decorator / context manager either starts a new 
>transaction, or creates a new savepoint if it's nested within an existing 
>transaction. They are committed as long as no exception is raised. 
>- If ATOMIC_REQUESTS is specified in the database config, all views 
>are wrapped in `atomic`, unless it's wrapped in `non_atomic_requests`.
>
> Yes, these four sentences are correct. 

>
>- If you set AUTOCOMMIT to False in a database configuration, this 
>will disable the database-level auto-commit. All DB reads and writes will 
>need manually wrapping in transactions. *The 
> **docs*
>* say that with this disabled, Django won't perform any commits. Does 
>this mean that the `atomic` decorator won't work properly and you have to 
>use the underlying database library to handle transactions? In 1.6 it 
> would 
>appear that Django never performs a commit outside of `atomic`, so I'm 
>confused by this comment!* 
>
> First, it's extremely unlikely that you will ever need to set AUTOCOMMIT 
to False. This is only useful if you want to implement your own transaction 
management, and I don't know anyone who's done that, nor a use case for 
doing that. I'd be surprised if this discussion was relevant to a single 
person on this mailing list (but I've been surprised before!). Also, your 
link goes to the 1.5 docs, not the dev docs. This may have confused you.

Anyway, in this scenario, `atomic` will work as expected. When autocommit 
is off, you're always in a transaction, and as a consequence `atomic` uses 
savepoints to guarantee atomicity; it'll never commit. You have to call 
transaction.commit() at some point to save changes.

I hope that helps,

-- 
Aymeric.

PS: If you weren't at DjangoCon Europe, you might want to check my talk on 
this topic. Slides are at 
http://static.myks.org/data/20130517-DjangoCon-Transactions.pdf and the 

Re: Understanding Django transactions

[John DeRosa]

Regardless of whatever comments and corrections may come, I thank you for 
summarizing what you think the changes will be! I didn't know about these 
impending changes.

John

On May 21, 2013, at 4:23 AM, Michael  wrote:

> 
> I've been reading up on how transactions work in Django 1.5 and how they will 
> work in 1.6. I think I have a grasp of things, but I wanted to reach out on 
> here to just check that my understanding of things is correct, and ask a 
> question or two.
> 
> Django 1.5:
> 
> The database-level auto-commit is turned OFF.
> SQL standards dictate that every query opens a transaction if one does not 
> already exist. This happens regardless of the database's auto-commit setting.
> After each Django ORM operation (read or write), Django commits this open 
> transaction. This emulates the database-level auto-commit.
> Transaction decorators cannot be nested. When one is opened, any previous 
> transaction will be committed.
> When executing raw SQL (with a database cursor), Django marks the current 
> transaction as dirty but does not issue a commit. If data changes were made 
> then they need manually committing. Why do the docs say that you only need to 
> commit the change if data was changed? If the transaction is marked as dirty 
> regardless of a read or a write, would it not always need committing or 
> rolling back to ensure the transaction is properly closed by the end of the 
> connection?
> Setting TRANSACTIONS_MANAGED to True stops Django from sending any commits 
> after ORM operations. The database-level auto-commit is still disabled. With 
> this setting, using any Django ORM read or write operation (all wrapped in 
> `transaction.commit_on_success`) fails with TransactionManagementError('This 
> code isn't under transaction management'). Is this expected?
> Django 1.6:
> 
> The database-level auto-commit is turned ON.
> Every database operation via the ORM will be committed using the database's 
> auto-commit, including any custom SQL executed with the database cursor.
> The `atomic` decorator / context manager either starts a new transaction, or 
> creates a new savepoint if it's nested within an existing transaction. They 
> are committed as long as no exception is raised.
> If ATOMIC_REQUESTS is specified in the database config, all views are wrapped 
> in `atomic`, unless it's wrapped in `non_atomic_requests`.
> If you set AUTOCOMMIT to False in a database configuration, this will disable 
> the database-level auto-commit. All DB reads and writes will need manually 
> wrapping in transactions. The docs say that with this disabled, Django won't 
> perform any commits. Does this mean that the `atomic` decorator won't work 
> properly and you have to use the underlying database library to handle 
> transactions? In 1.6 it would appear that Django never performs a commit 
> outside of `atomic`, so I'm confused by this comment!
> I'd really appreciate any information if some of what I understand to be true 
> is not accurate.
> 
> Many thanks!
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Understanding Django transactions

[Michael]

 

I've been reading up on how transactions work in Django 1.5 and how they 
will work in 1.6. I think I have a grasp of things, but I wanted to reach 
out on here to just check that my understanding of things is correct, and 
ask a question or two.

*Django 1.5:*

   - The database-level auto-commit is turned OFF. 
   - SQL standards dictate that every query opens a transaction if one does 
   not already exist. This happens regardless of the database's auto-commit 
   setting. 
   - After each Django ORM operation (read or write), Django commits this 
   open transaction. This emulates the database-level auto-commit. 
   - Transaction decorators cannot be nested. When one is opened, any 
   previous transaction will be committed. 
   - When executing raw SQL (with a database cursor), Django marks the 
   current transaction as dirty but does not issue a commit. If data changes 
   were made then they need manually committing. *Why do the 
**docs*
   * say that you only need to commit the change if data was changed? If 
   the transaction is marked as dirty regardless of a read or a write, would 
   it not always need committing or rolling back to ensure the transaction is 
   properly closed by the end of the connection?* 
   - Setting TRANSACTIONS_MANAGED to True stops Django from sending any 
   commits after ORM operations. The database-level auto-commit is still 
   disabled.* With this setting, using any Django ORM read or write 
   operation (all wrapped in `transaction.commit_on_success`) fails 
   with TransactionManagementError('This code isn't under transaction 
   management'). Is this expected?* 

*Django 1.6:*

   - The database-level auto-commit is turned ON. 
   - Every database operation via the ORM will be committed using the 
   database's auto-commit, including any custom SQL executed with the database 
   cursor. 
   - The `atomic` decorator / context manager either starts a new 
   transaction, or creates a new savepoint if it's nested within an existing 
   transaction. They are committed as long as no exception is raised. 
   - If ATOMIC_REQUESTS is specified in the database config, all views are 
   wrapped in `atomic`, unless it's wrapped in `non_atomic_requests`. 
   - If you set AUTOCOMMIT to False in a database configuration, this will 
   disable the database-level auto-commit. All DB reads and writes will need 
   manually wrapping in transactions. *The 
**docs*
   * say that with this disabled, Django won't perform any commits. Does 
   this mean that the `atomic` decorator won't work properly and you have to 
   use the underlying database library to handle transactions? In 1.6 it would 
   appear that Django never performs a commit outside of `atomic`, so I'm 
   confused by this comment!* 

I'd really appreciate any information if some of what I understand to be 
true is not accurate.

Many thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.