Re: [Zope-dev] connection commit ordering
On 18 June 2010 01:24, Leonardo Rochael Almeida leoroch...@gmail.com wrote: By the way, this issue is completely separate from the two-phase-commit discussion that we had recently, since all the connectors involved here are fully transactional. As you can see here: http://zope3.pov.lt/trac/browser/Zope/trunk/src/Shared/DC/ZRDB/TM.py def tpc_vote(self, *ignored): self._finalize = 1 def tpc_finish(self, *ignored): if self._finalize: try: self._finish() finally: self._registered=0 The transaction manager is only doing one phase commit. It sorts first as it commits in the second phase. If you change the sort order, you lose the guarantee of transactional integrity. Perhaps a better way to solve this would be to include the zope transaction id in the table, then in the background thread only reindex the queued items with a tid = the current tid of the connection. Laurence ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Zope Tests: 28 OK, 10 Failed
Summary of messages to the zope-tests list. Period Thu Jun 17 12:00:00 2010 UTC to Fri Jun 18 12:00:00 2010 UTC. There were 38 messages: 6 from Zope Tests, 11 from ccomb at free.fr, 5 from ct at gocept.com, 16 from jdriessen at thehealthagency.com. Test failures - Subject: FAILED : Total languishing bugs: 1 From: ct at gocept.com Date: Thu Jun 17 20:30:17 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015413.html Subject: FAILED : Total languishing bugs: 103 From: ct at gocept.com Date: Thu Jun 17 20:36:17 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015414.html Subject: FAILED : Total languishing bugs: 178 From: ct at gocept.com Date: Thu Jun 17 20:41:08 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015415.html Subject: FAILED : Total languishing bugs: 75 From: ct at gocept.com Date: Thu Jun 17 20:45:29 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015416.html Subject: FAILED: Repository policy check found errors in 453 projects From: ct at gocept.com Date: Thu Jun 17 21:10:26 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015417.html Subject: FAILED : BlueBream template / Python2.4.6 32bit linux From: ccomb at free.fr Date: Thu Jun 17 22:00:58 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015423.html Subject: FAILED : BlueBream template / Python2.5.2 32bit linux From: ccomb at free.fr Date: Thu Jun 17 22:01:01 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015424.html Subject: FAILED : BlueBream template / Python2.6.4 32bit linux From: ccomb at free.fr Date: Thu Jun 17 22:01:01 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015425.html Subject: FAILED : Zope 3.4.1 KGS / Python2.4.6 32bit linux From: ccomb at free.fr Date: Fri Jun 18 00:28:50 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015430.html Subject: FAILED : Zope Buildbot / ztk_win slave-win From: jdriessen at thehealthagency.com Date: Fri Jun 18 07:22:59 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015442.html Tests passed OK --- Subject: OK : Zope Buildbot / zope2.12 slave-osx From: jdriessen at thehealthagency.com Date: Thu Jun 17 08:52:11 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015407.html Subject: OK : Zope Buildbot / zope2 slave-ubuntu64 From: jdriessen at thehealthagency.com Date: Thu Jun 17 08:54:45 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015408.html Subject: OK : Zope Buildbot / zope2 slave-ubuntu32 From: jdriessen at thehealthagency.com Date: Thu Jun 17 08:54:51 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015409.html Subject: OK : Zope Buildbot / zope2 slave-osx From: jdriessen at thehealthagency.com Date: Thu Jun 17 08:59:43 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015410.html Subject: OK : Zope Buildbot / ztk slave-osx From: jdriessen at thehealthagency.com Date: Thu Jun 17 09:29:03 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015411.html Subject: OK : Zope Buildbot / zope2 slave-osx From: jdriessen at thehealthagency.com Date: Thu Jun 17 09:36:38 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015412.html Subject: OK : Zope-2.10 Python-2.4.6 : Linux From: Zope Tests Date: Thu Jun 17 21:52:28 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015418.html Subject: OK : Zope-2.11 Python-2.4.6 : Linux From: Zope Tests Date: Thu Jun 17 21:54:39 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015419.html Subject: OK : Zope-2.12 Python-2.6.5 : Linux From: Zope Tests Date: Thu Jun 17 21:56:44 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015420.html Subject: OK : Zope-2.12-alltests Python-2.6.5 : Linux From: Zope Tests Date: Thu Jun 17 21:58:54 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015421.html Subject: OK : Zope-trunk Python-2.6.5 : Linux From: Zope Tests Date: Thu Jun 17 22:00:54 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015422.html Subject: OK : Zope-trunk-alltests Python-2.6.5 : Linux From: Zope Tests Date: Thu Jun 17 22:02:54 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015426.html Subject: OK : ZTK 1.0dev / Python2.4.6 Linux 32bit From: ccomb at free.fr Date: Fri Jun 18 00:07:44 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015427.html Subject: OK : ZTK 1.0dev / Python2.5.2 Linux 32bit From: ccomb at free.fr Date: Fri Jun 18 00:09:00 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015428.html Subject: OK : ZTK 1.0dev / Python2.6.4 Linux 32bit From: ccomb at free.fr Date: Fri Jun 18 00:09:05 EDT 2010 URL: http://mail.zope.org/pipermail/zope-tests/2010-June/015429.html Subject: OK : Zope 3.4.1 KGS / Python2.5.2 32bit linux From: ccomb at free.fr Date: Fri Jun 18 00:51:46 EDT 2010 URL:
Re: [Zope-dev] connection commit ordering
Hi Laurence On Fri, Jun 18, 2010 at 08:06, Laurence Rowe l...@lrowe.co.uk wrote: On 18 June 2010 01:24, Leonardo Rochael Almeida leoroch...@gmail.com wrote: By the way, this issue is completely separate from the two-phase-commit discussion that we had recently, since all the connectors involved here are fully transactional. As you can see here: http://zope3.pov.lt/trac/browser/Zope/trunk/src/Shared/DC/ZRDB/TM.py def tpc_vote(self, *ignored): self._finalize = 1 def tpc_finish(self, *ignored): if self._finalize: try: self._finish() finally: self._registered=0 The transaction manager is only doing one phase commit. It sorts first as it commits in the second phase. If you change the sort order, you lose the guarantee of transactional integrity. For me this means that TM subclasses need to override tpc_vote and implement a proper commit preparation [1] [2] to assure they are correctly participating in the TPC dance. And if that is not the case, but you have, for example, more than one MySQL connector, you are already in a situation where you can't guarantee transactional integrity, so this discussion is actually orthogonal to the sortOrder one. Perhaps a better way to solve this would be to include the zope transaction id in the table, then in the background thread only reindex the queued items with a tid = the current tid of the connection. Possibly, but is there a way to know the id of a transaction that hasn't been committed yet, to store it on MySQL? Besides, when working with multiple mount points, you might have to store multiple TIDs, for all storages involved, or else there should be a global transaction ID that should be recorded everywhere, and I don't see the 'transaction' package providing one. In any case, does anyone oppose the existence of a .setSortKey() on the TM class? Cheers, Leo [1] http://www.postgresql.org/docs/current/static/sql-prepare-transaction.html [2] http://dev.mysql.com/doc/refman/5.0/en/xa.html ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] connection commit ordering
On Fri, Jun 18, 2010 at 10:55, Hanno Schlichting ha...@hannosch.eu wrote: On Fri, Jun 18, 2010 at 3:32 PM, Leonardo Rochael Almeida leoroch...@gmail.com wrote: In any case, does anyone oppose the existence of a .setSortKey() on the TM class? Your branch looks simple enough. Feel free to merge it to Zope trunk but don't forget to add a change entry in doc/CHANGES.rst Thanks. Can I commit it to Zope 2.12 as well? It is backward compatible and helps us address a bug we see on production. Cheers, ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] connection commit ordering
On Fri, Jun 18, 2010 at 4:29 PM, Leonardo Rochael Almeida leoroch...@gmail.com wrote: Thanks. Can I commit it to Zope 2.12 as well? It is backward compatible and helps us address a bug we see on production. Go ahead. It really is trivial :) Hanno ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Bug day report
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Roman Joost wrote: On Thu, Jun 17, 2010 at 07:59:18AM -0400, Tres Seaver wrote: Roman, I landed my version of the fix in 'zope.testrunner', but didn't touch it in 'zope.testing', since you were working on it. I was rushing a little because I wanted to release 'zope.testrunner', but wanted to check with you first. The patch is here: http://svn.zope.org/zope.testrunner/trunk/src/zope/testrunner/find.py?rev=113540view=diffr1=113540r2=113539p1=zope.testrunner/trunk/src/zope/testrunner/find.pyp2=/zope.testrunner/trunk/src/zope/testrunner/find.py I'm fine with letting you land it in zope.testing, or even tweak it on the zope.testruner side, if you see a better fix. Thank you. I actually had troubles running all tests in zope.testrunner. I need to check why to be sure, to avoid commiting something broken. Marius fixed the buildout for that. I usually prefer to let someone more experienced review my code, although it seems to be not worth it for this little fix. So to clarify - can I commit the patch to zope.testrunner after I verified that the tests are working fine on my local sandbox? You should be able to check in just fine -- do you mean zope.testing, or zope.testrunner? I checked in my changes to zope.testrunner already. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkwbi6MACgkQ+gerLs4ltQ5mQACg0BejhsCAi8jw696URNPJ/KKh /6cAn0eFmusq+6EIlvSeudv+TdvpcG33 =6IHS -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] connection commit ordering
On 18 June 2010 14:32, Leonardo Rochael Almeida leoroch...@gmail.com wrote: Hi Laurence On Fri, Jun 18, 2010 at 08:06, Laurence Rowe l...@lrowe.co.uk wrote: On 18 June 2010 01:24, Leonardo Rochael Almeida leoroch...@gmail.com wrote: By the way, this issue is completely separate from the two-phase-commit discussion that we had recently, since all the connectors involved here are fully transactional. As you can see here: http://zope3.pov.lt/trac/browser/Zope/trunk/src/Shared/DC/ZRDB/TM.py def tpc_vote(self, *ignored): self._finalize = 1 def tpc_finish(self, *ignored): if self._finalize: try: self._finish() finally: self._registered=0 The transaction manager is only doing one phase commit. It sorts first as it commits in the second phase. If you change the sort order, you lose the guarantee of transactional integrity. For me this means that TM subclasses need to override tpc_vote and implement a proper commit preparation [1] [2] to assure they are correctly participating in the TPC dance. zope.sqlalchemy does this, but that brings a whole orm into the equation and does away with ZRDB legacy. And if that is not the case, but you have, for example, more than one MySQL connector, you are already in a situation where you can't guarantee transactional integrity, so this discussion is actually orthogonal to the sortOrder one. That's true, but don't you see this problem even with only a single ZODB and a single ZRDB connection? Perhaps a better way to solve this would be to include the zope transaction id in the table, then in the background thread only reindex the queued items with a tid = the current tid of the connection. Possibly, but is there a way to know the id of a transaction that hasn't been committed yet, to store it on MySQL? Besides, when working with multiple mount points, you might have to store multiple TIDs, for all storages involved, or else there should be a global transaction ID that should be recorded everywhere, and I don't see the 'transaction' package providing one. The ZODB storage's transaction id is set in tpc_begin, so you should be able to get it in tcp_vote or tpc_finish of the ZRDB data manager. Though doing so probably horribly complicates the ZSQLCatalog code. In any case, does anyone oppose the existence of a .setSortKey() on the TM class? I don't oppose it, but I also don't see how this will fix the problem unless you set the sort key to be greater than the ZODB's sort key. This strikes me as a very bad idea for a TM that is designed to tpc_finish before anything else. Laurence ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
