Re: [Zope-dev] [ZODB-Dev] [announce] NEO 1.0 - scalable and redundant storage for ZODB
On Tue, Aug 28, 2012 at 06:31:05PM +0200, Vincent Pelletier wrote: On Tue, 28 Aug 2012 16:31:20 +0200, Martijn Pieters m...@zopatista.com wrote : Anything else different? Did you make any performance comparisons between RelStorage and NEO? I believe the main difference compared to all other ZODB Storage implementation is the finer-grained locking scheme: in all storage implementations I know, there is a database-level lock during the entire second phase of 2PC, whereas in NEO transactions are serialised only when they alter a common set of objects. This could be a compelling point. I've seen deadlocks in an app that tried to use both ZEO and PostgreSQL via the Storm ORM. (The thread holding the ZEO commit lock was blocked waiting for the PostgreSQL commit to finish, while the PostgreSQL server was waiting for some other transaction to either commit or abort -- and that other transaction couldn't proceed because it was waiting for the ZEO lock.) Marius Gedminas -- People who think, Oh this is a one-off, need to be offed, or perhaps politely removed from the project. -- George Neville-Neil signature.asc Description: Digital signature ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] [ZODB-Dev] [announce] NEO 1.0 - scalable and redundant storage for ZODB
On Wed, Aug 29, 2012 at 2:29 AM, Marius Gedminas mar...@gedmin.as wrote: On Tue, Aug 28, 2012 at 06:31:05PM +0200, Vincent Pelletier wrote: On Tue, 28 Aug 2012 16:31:20 +0200, Martijn Pieters m...@zopatista.com wrote : Anything else different? Did you make any performance comparisons between RelStorage and NEO? I believe the main difference compared to all other ZODB Storage implementation is the finer-grained locking scheme: in all storage implementations I know, there is a database-level lock during the entire second phase of 2PC, whereas in NEO transactions are serialised only when they alter a common set of objects. This could be a compelling point. I've seen deadlocks in an app that tried to use both ZEO and PostgreSQL via the Storm ORM. (The thread holding the ZEO commit lock was blocked waiting for the PostgreSQL commit to finish, while the PostgreSQL server was waiting for some other transaction to either commit or abort -- and that other transaction couldn't proceed because it was waiting for the ZEO lock.) This sounds like an application/transaction configuration problem. To avoid this sort of deadlock, you need to always commit in a a consistent order. You also need to configure ZEO (or NEO) to time-out transactions that take too long to finish the second phase. I don't think NEO's locking strategy mitigates the deadlock problem much, if at all. The strategy should provide greater transaction throughput and reduce latency. It's a strategy I'd like to implement for ZEO at some point. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] [ZODB-Dev] [announce] NEO 1.0 - scalable and redundant storage for ZODB
On Tue, Aug 28, 2012 at 12:31 PM, Vincent Pelletier vinc...@nexedi.com wrote: ... I forgot in the original mail to mention that NEO does all conflict resolutions on client side rather than server side. The same happens in relStorage, but this is different from ZEO. That's good. I'd like to move ZEO in this direction. I'd also like to stop hanging conflict-resolution on classes and have some kind of registry, so that people can set CR policies independent of class implementation. I didn't realize that relstorage did client side CR, but thinking about it, it has to work that way, since there's no relstorage server. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope-tests - FAILED: 24, OK: 17, UNKNOWN: 2
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/28/2012 09:00 PM, Zope tests summarizer wrote: [1]UNKNOWN UNKNOWN : Zope-trunk Python-2.6.8 : Linux https://mail.zope.org/pipermail/zope-tests/2012-August/067337.html [2]UNKNOWN UNKNOWN : Zope-trunk Python-2.7.3 : Linux https://mail.zope.org/pipermail/zope-tests/2012-August/067338.html This appears to be due to a change in ZConfig: - % --- Test-module import failures: Module: Zope2.Startup.tests.testStarter Traceback (most recent call last): File /home/stefan/autotest/temp/python27-zope214all/src/Zope2/Startup/tests/testStarter.py, line 53, in module class ZopeStarterTestCase(test_logger.LoggingTestBase): AttributeError: 'module' object has no attribute 'LoggingTestBase' Module: zLOG.tests.test_logging Traceback (most recent call last): File /home/stefan/autotest/temp/python27-zope214all/eggs/zLOG-2.11.1-py2.7.egg/zLOG/tests/test_logging.py, line 19, in module from ZConfig.components.logger.tests.test_logger import LoggingTestBase ImportError: cannot import name LoggingTestBase - % --- That base class has been gone since ZConfig 2.9.2. I don't think the Zope2 trunk has pinned / unpinned ZConfig in a long time, so I'm not sure why it would just now break (ZConfig 2.9.2 was released in February, and 2.9.3 in June). skipping the z3c.* failures: I don't know enough to diagnose them skipping the zc.buildout failures: project has moved outside my purview [26] FAILED winbot / zope.app.twisted_py_265_32 https://mail.zope.org/pipermail/zope-tests/2012-August/067324.html The first failure here seems to be the same as the ZConfig-bsed error breaking Zope2. Some of the others seem to break on the following: File c:\eggs\zodb3-3.10.5-py2.6-win32.egg\ZEO\zrpc\server.py, line 21, in module s = socket.socket(socket.AF_INET6, socket.SOCK_STREAM) Maybe the Windows bot doesn't know from IPv6? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlA+GEkACgkQ+gerLs4ltQ7dGACgiysr8ewQgSbX8WqRvK9yhaBc NYQAniTVBMovxd0h9Cuep4JFllf46W6X =0SxW -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope-tests - FAILED: 24, OK: 17, UNKNOWN: 2
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/29/2012 09:25 AM, Tres Seaver wrote: That base class has been gone since ZConfig 2.9.2. I don't think the Zope2 trunk has pinned / unpinned ZConfig in a long time, so I'm not sure why it would just now break (ZConfig 2.9.2 was released in February, and 2.9.3 in June). I just pushed a change to the Zope2 trunk to use the new speling. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlA+HMwACgkQ+gerLs4ltQ59RgCgv3fZhhIjw+LxpE35cH5O/KII ODwAoMY1ECTmuNoPKVdo185A1do76Wnp =cK7D -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope-tests - FAILED: 24, OK: 17, UNKNOWN: 2
On Wed, Aug 29, 2012 at 3:25 PM, Tres Seaver tsea...@palladion.com wrote: That base class has been gone since ZConfig 2.9.2. I don't think the Zope2 trunk has pinned / unpinned ZConfig in a long time, so I'm not sure why it would just now break (ZConfig 2.9.2 was released in February, and 2.9.3 in June). I updated the ZTK trunk to some new versions recently. Zope2 trunk likely tracks ZTK trunk and thus broke. Hanno ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] zope-tests - FAILED: 21, OK: 19, UNKNOWN: 3
This is the summary for test reports received on the zope-tests list between 2012-08-28 00:00:00 UTC and 2012-08-29 00:00:00 UTC: See the footnotes for test reports of unsuccessful builds. An up-to date view of the builders is also available in our buildbot documentation: http://docs.zope.org/zopetoolkit/process/buildbots.html#the-nightly-builds Reports received [1]UNKNOWN : Zope-trunk Python-2.6.8 : Linux [2]UNKNOWN : Zope-trunk Python-2.7.3 : Linux [3]UNKNOWN : winbot / zc_buildout_dev py_270_win32 1.6.x Zope-2.10 Python-2.4.6 : Linux Zope-2.11 Python-2.4.6 : Linux Zope-2.12 Python-2.6.8 : Linux Zope-2.13 Python-2.6.8 : Linux Zope-2.13 Python-2.7.3 : Linux [4]winbot / ZODB_dev py_265_win32 winbot / ZODB_dev py_265_win32 winbot / ZODB_dev py_265_win64 winbot / ZODB_dev py_265_win64 winbot / ZODB_dev py_270_win32 winbot / ZODB_dev py_270_win32 [5]winbot / ZODB_dev py_270_win64 winbot / ZODB_dev py_270_win64 [6]winbot / z3c.contents_py_265_32 [7]winbot / z3c.coverage_py_265_32 [8]winbot / z3c.datagenerator_py_265_32 [9]winbot / z3c.layer.pagelet_py_265_32 [10] winbot / z3c.layer.ready2go_py_265_32 [11] winbot / z3c.menu.ready2go_py_265_32 [12] winbot / z3c.recipe.paster_py_265_32 [13] winbot / z3c.rml_py_265_32 [14] winbot / zc_buildout_dev py_254_win32 1.6.x [15] winbot / zc_buildout_dev py_265_win32 1.6.x [16] winbot / zc_buildout_dev py_265_win32 1.6.x [17] winbot / zc_buildout_dev py_265_win32 master [18] winbot / zc_buildout_dev py_265_win64 1.6.x [19] winbot / zc_buildout_dev py_265_win64 master [20] winbot / zc_buildout_dev py_270_win32 1.6.x [21] winbot / zc_buildout_dev py_270_win32 master [22] winbot / zc_buildout_dev py_270_win64 1.6.x [23] winbot / zc_buildout_dev py_270_win64 master [24] winbot / zope.app.twisted_py_265_32 winbot / ztk_10 py_254_win32 winbot / ztk_10 py_265_win32 winbot / ztk_10 py_265_win64 winbot / ztk_11 py_254_win32 winbot / ztk_11 py_265_win32 winbot / ztk_11 py_265_win64 winbot / ztk_11 py_270_win32 winbot / ztk_11 py_270_win64 Non-OK results -- [1]UNKNOWN UNKNOWN : Zope-trunk Python-2.6.8 : Linux https://mail.zope.org/pipermail/zope-tests/2012-August/067377.html [2]UNKNOWN UNKNOWN : Zope-trunk Python-2.7.3 : Linux https://mail.zope.org/pipermail/zope-tests/2012-August/067378.html [3]UNKNOWN UNKNOWN : winbot / zc_buildout_dev py_270_win32 1.6.x https://mail.zope.org/pipermail/zope-tests/2012-August/067379.html [4]FAILED winbot / ZODB_dev py_265_win32 https://mail.zope.org/pipermail/zope-tests/2012-August/067402.html [5]FAILED winbot / ZODB_dev py_270_win64 https://mail.zope.org/pipermail/zope-tests/2012-August/067380.html [6]FAILED winbot / z3c.contents_py_265_32 https://mail.zope.org/pipermail/zope-tests/2012-August/067365.html [7]FAILED winbot / z3c.coverage_py_265_32 https://mail.zope.org/pipermail/zope-tests/2012-August/067369.html [8]FAILED winbot / z3c.datagenerator_py_265_32 https://mail.zope.org/pipermail/zope-tests/2012-August/067370.html [9]FAILED winbot / z3c.layer.pagelet_py_265_32 https://mail.zope.org/pipermail/zope-tests/2012-August/067371.html [10] FAILED winbot / z3c.layer.ready2go_py_265_32 https://mail.zope.org/pipermail/zope-tests/2012-August/067368.html [11] FAILED winbot / z3c.menu.ready2go_py_265_32 https://mail.zope.org/pipermail/zope-tests/2012-August/067363.html [12] FAILED winbot / z3c.recipe.paster_py_265_32 https://mail.zope.org/pipermail/zope-tests/2012-August/067366.html [13] FAILED winbot / z3c.rml_py_265_32 https://mail.zope.org/pipermail/zope-tests/2012-August/067364.html [14] FAILED winbot / zc_buildout_dev py_254_win32 1.6.x https://mail.zope.org/pipermail/zope-tests/2012-August/067397.html [15] FAILED winbot / zc_buildout_dev py_265_win32 1.6.x https://mail.zope.org/pipermail/zope-tests/2012-August/067398.html [16] FAILED winbot / zc_buildout_dev py_265_win32 1.6.x https://mail.zope.org/pipermail/zope-tests/2012-August/067382.html [17] FAILED winbot / zc_buildout_dev py_265_win32 master https://mail.zope.org/pipermail/zope-tests/2012-August/067393.html [18] FAILED winbot / zc_buildout_dev py_265_win64 1.6.x https://mail.zope.org/pipermail/zope-tests/2012-August/067399.html [19] FAILED winbot / zc_buildout_dev py_265_win64 master https://mail.zope.org/pipermail/zope-tests/2012-August/067394.html [20] FAILED winbot / zc_buildout_dev py_270_win32 1.6.x https://mail.zope.org/pipermail/zope-tests/2012-August/067400.html [21] FAILED winbot / zc_buildout_dev py_270_win32 master https://mail.zope.org/pipermail/zope-tests/2012-August/067395.html
