Re: [Zope] Re: database conflict errors
Florent Guillaume wrote at 2006-6-30 02:46 +0200: ... that's interesting. I did a test once to see what effect it would have to add objects with a completely random id to a BTree folder (OOBTree in that case) instead of using the object's type nam and add a number at the end - and the result was the opposite in term of read performance. Looking up keys was much faster if the ids followed a pattern like: - something-1 - something-2 ... Sure, in single-threaded mode this will decrease performance because the keys are spread randomly among all the buckets so many more buckets get written. But in multi-threaded mode, this very spreading leads to better conflict resolution behavior. But still can reduced performance... That's why the catalog uses the initial id randomly but then assigns id's (within a single thread) sequentially. -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: database conflict errors
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jonathan wrote: During recent load testing of a new application 3.1% to 7.6% of all http requests resulted in conflict errors (3.1% with 10 simultaneous users; 7.6% with 50 simultaneous users). The conflict error occurs when the application attempts to write a small image object into a temporary folder, and each conflict error generates the same traceback: Traceback (innermost last): Module Zope2.App.startup, line 173, in zpublisher_exception_hook Module ZPublisher.Publish, line 121, in publish Module Zope2.App.startup, line 240, in commit Module transaction._manager, line 96, in commit Module transaction._transaction, line 380, in commit Module transaction._transaction, line 378, in commit Module transaction._transaction, line 433, in _commitResources Module ZODB.Connection, line 484, in commit Module ZODB.Connection, line 518, in _commit ConflictError: database conflict error (oid 0x07, class Products.TemporaryFolder.TemporaryFolder.SimpleTemporaryContainer) I am running Zope 2.9.2 on CentOS 4.3 (linux). Does anyone have any ideas as to what I could do to reduce/eliminate these conflict errors? First, make sure that your application is not trying to overwrite the *same* image in each request. If it needs to do that, then the conflicts are unavoidable. Next, making simultaneous writes into any naive container is going to cause conflicts. You need to think carefully about how you want those writes to work, and plan to minimize conflicts: - Use a BTreeFolder, rather than a normal OFS.Folder (or derivative). - Arrange for the IDs of your images to be substantially different, typically by mangling in a random number (or, for instance, the microseconds value of the current timestamp). Tres. - -- === Tres Seaver +1 202-558-7113 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEo+/U+gerLs4ltQ4RAhsUAKCaZIJKLKOcO/X6wkAUo8x83vx3eQCfY5hW /2SHvahGxJ3DSpg4qMlL4D4= =17k2 -END PGP SIGNATURE- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: database conflict errors
- Original Message - From: Tres Seaver [EMAIL PROTECTED] Jonathan wrote: During recent load testing of a new application 3.1% to 7.6% of all http requests resulted in conflict errors (3.1% with 10 simultaneous users; 7.6% with 50 simultaneous users). The conflict error occurs when the application attempts to write a small image object into a temporary folder, and each conflict error generates the same traceback: ConflictError: database conflict error (oid 0x07, class Products.TemporaryFolder.TemporaryFolder.SimpleTemporaryContainer) I am running Zope 2.9.2 on CentOS 4.3 (linux). Does anyone have any ideas as to what I could do to reduce/eliminate these conflict errors? First, make sure that your application is not trying to overwrite the *same* image in each request. If it needs to do that, then the conflicts are unavoidable. Next, making simultaneous writes into any naive container is going to cause conflicts. You need to think carefully about how you want those writes to work, and plan to minimize conflicts: - Use a BTreeFolder, rather than a normal OFS.Folder (or derivative). - Arrange for the IDs of your images to be substantially different, typically by mangling in a random number (or, for instance, the microseconds value of the current timestamp). The ids are assigned sequentially, so they never collide, but should id be 'substantially different' to improve BTreeFolder2 performance? As per your suggestion I added a BTreeFolder2 folder inside of the temporary folder and used that for image storage and ran some more load tests. There was a significant improvement: with 25 simultaneous users the error conflict error rate dropped from 4.4% to 0.6% and there was an unexpected side effect - a 20% improvement in server performance!!! Thanks for the most excellent suggestion Tres! Jonathan ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: database conflict errors
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jonathan wrote: - Original Message - From: Tres Seaver [EMAIL PROTECTED] Jonathan wrote: During recent load testing of a new application 3.1% to 7.6% of all http requests resulted in conflict errors (3.1% with 10 simultaneous users; 7.6% with 50 simultaneous users). The conflict error occurs when the application attempts to write a small image object into a temporary folder, and each conflict error generates the same traceback: ConflictError: database conflict error (oid 0x07, class Products.TemporaryFolder.TemporaryFolder.SimpleTemporaryContainer) I am running Zope 2.9.2 on CentOS 4.3 (linux). Does anyone have any ideas as to what I could do to reduce/eliminate these conflict errors? First, make sure that your application is not trying to overwrite the *same* image in each request. If it needs to do that, then the conflicts are unavoidable. Next, making simultaneous writes into any naive container is going to cause conflicts. You need to think carefully about how you want those writes to work, and plan to minimize conflicts: - Use a BTreeFolder, rather than a normal OFS.Folder (or derivative). - Arrange for the IDs of your images to be substantially different, typically by mangling in a random number (or, for instance, the microseconds value of the current timestamp). The ids are assigned sequentially, so they never collide, but should id be 'substantially different' to improve BTreeFolder2 performance? Sequential IDs can be problematic: they lead to many more bucket splits (and therefore conflicts) in the BTree. Adding some entropy (like a random number or the milliseconds of the time) *earlier* in the key than any sequence number, will help keep splits down to a minimum. As per your suggestion I added a BTreeFolder2 folder inside of the temporary folder and used that for image storage and ran some more load tests. There was a significant improvement: with 25 simultaneous users the error conflict error rate dropped from 4.4% to 0.6% and there was an unexpected side effect - a 20% improvement in server performance!!! Thanks for the most excellent suggestion Tres! You're welcome. Tres. - -- === Tres Seaver +1 202-558-7113 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEo/9H+gerLs4ltQ4RAqgRAJ9zWJMtCLWT1rkyoOM3GIPpovlccwCeOlnl 5yioYSdimjt7FyXWRODDA6A= =tNlC -END PGP SIGNATURE- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: database conflict errors
Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jonathan wrote: - Original Message - From: Tres Seaver [EMAIL PROTECTED] Jonathan wrote: During recent load testing of a new application 3.1% to 7.6% of all http requests resulted in conflict errors (3.1% with 10 simultaneous users; 7.6% with 50 simultaneous users). The conflict error occurs when the application attempts to write a small image object into a temporary folder, and each conflict error generates the same traceback: ConflictError: database conflict error (oid 0x07, class Products.TemporaryFolder.TemporaryFolder.SimpleTemporaryContainer) I am running Zope 2.9.2 on CentOS 4.3 (linux). Does anyone have any ideas as to what I could do to reduce/eliminate these conflict errors? First, make sure that your application is not trying to overwrite the *same* image in each request. If it needs to do that, then the conflicts are unavoidable. Next, making simultaneous writes into any naive container is going to cause conflicts. You need to think carefully about how you want those writes to work, and plan to minimize conflicts: - Use a BTreeFolder, rather than a normal OFS.Folder (or derivative). - Arrange for the IDs of your images to be substantially different, typically by mangling in a random number (or, for instance, the microseconds value of the current timestamp). The ids are assigned sequentially, so they never collide, but should id be 'substantially different' to improve BTreeFolder2 performance? Sequential IDs can be problematic: they lead to many more bucket splits (and therefore conflicts) in the BTree. Adding some entropy (like a random number or the milliseconds of the time) *earlier* in the key than any sequence number, will help keep splits down to a minimum. that's interesting. I did a test once to see what effect it would have to add objects with a completely random id to a BTree folder (OOBTree in that case) instead of using the object's type nam and add a number at the end - and the result was the opposite in term of read performance. Looking up keys was much faster if the ids followed a pattern like: - something-1 - something-2 ... I will try again. /JM ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: database conflict errors
Jean-Marc Orliaguet wrote: Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jonathan wrote: - Original Message - From: Tres Seaver [EMAIL PROTECTED] Jonathan wrote: During recent load testing of a new application 3.1% to 7.6% of all http requests resulted in conflict errors (3.1% with 10 simultaneous users; 7.6% with 50 simultaneous users). The conflict error occurs when the application attempts to write a small image object into a temporary folder, and each conflict error generates the same traceback: ConflictError: database conflict error (oid 0x07, class Products.TemporaryFolder.TemporaryFolder.SimpleTemporaryContainer) I am running Zope 2.9.2 on CentOS 4.3 (linux). Does anyone have any ideas as to what I could do to reduce/eliminate these conflict errors? First, make sure that your application is not trying to overwrite the *same* image in each request. If it needs to do that, then the conflicts are unavoidable. Next, making simultaneous writes into any naive container is going to cause conflicts. You need to think carefully about how you want those writes to work, and plan to minimize conflicts: - Use a BTreeFolder, rather than a normal OFS.Folder (or derivative). - Arrange for the IDs of your images to be substantially different, typically by mangling in a random number (or, for instance, the microseconds value of the current timestamp). The ids are assigned sequentially, so they never collide, but should id be 'substantially different' to improve BTreeFolder2 performance? Sequential IDs can be problematic: they lead to many more bucket splits (and therefore conflicts) in the BTree. Adding some entropy (like a random number or the milliseconds of the time) *earlier* in the key than any sequence number, will help keep splits down to a minimum. that's interesting. I did a test once to see what effect it would have to add objects with a completely random id to a BTree folder (OOBTree in that case) instead of using the object's type nam and add a number at the end - and the result was the opposite in term of read performance. Looking up keys was much faster if the ids followed a pattern like: - something-1 - something-2 ... Sure, in single-threaded mode this will decrease performance because the keys are spread randomly among all the buckets so many more buckets get written. But in multi-threaded mode, this very spreading leads to better conflict resolution behavior. Florent -- Florent Guillaume, Nuxeo (Paris, France) Director of RD +33 1 40 33 71 59 http://nuxeo.com [EMAIL PROTECTED] ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
