Re: [Zope-dev] Conflict errors on startup, revisited
On Thu, Feb 24, 2005 at 04:11:35PM -0500, Tim Peters wrote: > [Tim Peters] > > ... > > Do you use a persistent ZEO client cache? If so, one thing to try is > > to physically remove all the cache files -- maybe they've gotten into > > an insane state. > > Off list, Paul said "bingo!" to that one -- deleting his ZEO client > cache files apparently made the problem go away. Yep, meant to mail the list, hit the wrong button. Thanks again Tim! > I'm not really happy with that, because I have no theory for how ZEO > client cache files _get_ out of synch. I suspect they do, though, > more often than stray gamma rays and pilot error could account for. > > Alas, I've got no time to give to that these days. If someone else > wants to solve that, I'll pass on the cigar Paul offered to me . Oh-sure-pretend-you-didn't-want-the-bag-of-ham-ly y'rs, -- Paul Winkler http://www.slinkp.com ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Conflict errors on startup, revisited
[Tim Peters] > ... > Do you use a persistent ZEO client cache? If so, one thing to try is > to physically remove all the cache files -- maybe they've gotten into > an insane state. Off list, Paul said "bingo!" to that one -- deleting his ZEO client cache files apparently made the problem go away. I'm not really happy with that, because I have no theory for how ZEO client cache files _get_ out of synch. I suspect they do, though, more often than stray gamma rays and pilot error could account for. Alas, I've got no time to give to that these days. If someone else wants to solve that, I'll pass on the cigar Paul offered to me . ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Conflict errors on startup, revisited
On Thu, Feb 24, 2005 at 03:16:15PM -0500, Paul Winkler wrote: > But - duh - my traceback clearly shows the ConflictError > happens before that, in the call to get_transaction().commit(). > SO I moved the call to app._p_jar.sync() to right before hte call > to get_transaction().commit(). > Aaaand... no more ConflictError. Never mind, this breaks existing unit tests. Duh, it prevents products from getting initialized at all :-\ now checking out tim's suggestion... -- Paul Winkler http://www.slinkp.com ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Conflict errors on startup, revisited
[Paul Winkler] > ... > But - duh - my traceback clearly shows the ConflictError > happens before that, in the call to get_transaction().commit(). > SO I moved the call to app._p_jar.sync() to right before hte call > to get_transaction().commit(). > Aaaand... no more ConflictError. Well, that's certain: the first thing sync() does is abort the current transaction. Then of course commit() won't raise an exception: sync() threw away all the changes. > ... > If nobody objects, I wouldnt' mind checking the fix in anyway :-) It would be much clearer instead to change the get_transaction().commit(). to get_transaction().abort() That would get rid of the ConflictError too, and for the same reason. It has the advantage of being obviously insane, though . ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope and python2.3.5
Andreas Jung wrote at 2005-2-21 21:33 +0100: > ... >> Maybe zope configure shouldnt stop on newer, but from the same branch, >> version of python? > >Things are as they are and it is exhausting to discuss about advantages and >disadvantages. >Python 2.3.5 will be officially recognized by Zope 2.7.5...nothing more to >be added from my >side (as released manager). But the proposal seems valuable in case there is interest in improving Zope ... -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Conflict errors on startup, revisited
[Paul Winkler] > ... > ZODB.POSException.ConflictError: database conflict error ( > serial this txn started with 0x035b25f36751f988 2005-02-10 18:59:24.215675, > serial currently committed 0x035b743c6d186822 2005-02-24 17:00:25.569220) No time here for thought, but those timestamps are two weeks apart. Yikes! Just thought I'd mention that . A write conflict (which this is) happens when (and in this time order): 1. A transaction reads an object O. 2. A different transaction modifies O and commits its new state. 3. The original transaction also modifies O, and tries to commit its new state. #3 fails, because the original transaction has, in general, no way to know whether the changes it tried to make to O's state are consistent with the changes the other transaction made to its state in step #2. For your specific exception above, step #1 started by reading the state of the object as it existed on 2005-02-10. In step #2, someone else modified the same object, and committed its state around 2005-02-24 17:00:25. The original transaction then failed in step #3. Do you use a persistent ZEO client cache? If so, one thing to try is to physically remove all the cache files -- maybe they've gotten into an insane state. It's not impossible that you "should" have a 2-week difference in timestamps here, but it does look unusual. It's certainly possible (even expected) if two transactions try to modify the same object, and the last time that object was changed was in fact two weeks ago. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Conflict errors on startup, revisited
On Thu, Feb 24, 2005 at 02:58:01PM -0500, Chris McDonough wrote: > On Thu, 2005-02-24 at 14:57, Paul Winkler wrote: > > On Thu, Feb 24, 2005 at 02:52:45PM -0500, Chris McDonough wrote: > > > Two words: *hurt system*. Please, let me kill it. > > > > Interesting. How on earth did you determine that? > > Womanly intuition. > > > > > You think it has to do with registering help for > > PluginIndexes? I'm gonna go test that theory - > > i'll hack registerHelp() to no-op. > > That's a good verifier. Well, no joy. I hacked ProductContext.register* to return from the first line, so even registerClass() now does nothing. Barf, I still get the ConflictError. Twiddling around with some old suggestions from Dieter Maurer (see http://aspn.activestate.com/ASPN/Mail/Message/Zope-CMF/1978842 ), I added a _p_jar.sync() where he suggested - right before the final "except:" in initialize_product(). No joy. But - duh - my traceback clearly shows the ConflictError happens before that, in the call to get_transaction().commit(). SO I moved the call to app._p_jar.sync() to right before hte call to get_transaction().commit(). Aaaand... no more ConflictError. Weird weird weird. I'd be happy to check this in to cvs / svc, except that A) I have no clue how to write a test that exercises the bug, and B) so far nobody seems to understand what's really happening. If nobody objects, I wouldnt' mind checking the fix in anyway :-) -- Paul Winkler http://www.slinkp.com ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Conflict errors on startup, revisited
On Thu, 2005-02-24 at 14:57, Paul Winkler wrote: > On Thu, Feb 24, 2005 at 02:52:45PM -0500, Chris McDonough wrote: > > Two words: *hurt system*. Please, let me kill it. > > Interesting. How on earth did you determine that? Womanly intuition. > > You think it has to do with registering help for > PluginIndexes? I'm gonna go test that theory - > i'll hack registerHelp() to no-op. That's a good verifier. - C > ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Conflict errors on startup, revisited
On Thu, Feb 24, 2005 at 02:52:45PM -0500, Chris McDonough wrote: > Two words: *hurt system*. Please, let me kill it. Interesting. How on earth did you determine that? You think it has to do with registering help for PluginIndexes? I'm gonna go test that theory - i'll hack registerHelp() to no-op. -- Paul Winkler http://www.slinkp.com ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Conflict errors on startup, revisited
On Thu, Feb 24, 2005 at 02:29:50PM -0500, Paul Winkler wrote: > * Is the PIL stuff in the traceback a red herring or what? Note that PIL is not referred to by anything in my Products directory. (Heck, *nothing* is referred to by anything in Products.) That's what's so baffling about it appearing in the traceback. -- Paul Winkler http://www.slinkp.com ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Conflict errors on startup, revisited
Two words: *hurt system*. Please, let me kill it. On Thu, 2005-02-24 at 14:29, Paul Winkler wrote: > This is a topic that comes up occasionally, but since the symptom > seems to be rare and hard to provoke, it never gets resolved. > Well it's biting me again and I'm determined to fix it, but > would like some input from the list. > > Related collector issues: > http://www.zope.org/Collectors/Zope/300 > http://www.zope.org/Collectors/Zope/1201 > > The issue: > On rare occasions I have seen ConflictErrors raised during > product installation. > > Today, I can finally provoke them at will with a pretty minimal setup. > > The setup: > Zope 2.7.3, using ZEO, with DirectoryStorage on the ZEO server. > Hardware: dual-processor PIII 800, some kinda scsi disk, ext3, > some old redhat flavor of linux. > > To provoke it I need merely do > ./bin/zopectl run foo.py > > ... where foo.py is empty. > Products/ now contains only one dummy product, > which I've named Conflictor, which contains only an empty __init__.py. > I get a ConflictError during registration of Conflictor. > > Interestingly I can only provoke this ConflictError in this particular > INSTANCE_HOME. I have several other instances and am not seeing the > symptom there. > > I started trivially hacking OFS/Application.py to add some debugging output > (a diff against 2.7.3 is attached), and here's what I get: > > > [EMAIL PROTECTED] PaulMiniInstanceHome]$ ls Products/ > [EMAIL PROTECTED] PaulMiniInstanceHome]$ mkdir Products/Conflicter > [EMAIL PROTECTED] PaulMiniInstanceHome]$ touch Products/Conflicter/__in > it__.py > [EMAIL PROTECTED] PaulMiniInstanceHome]$ touch foo.py > [EMAIL PROTECTED] PaulMiniInstanceHome]$ ./bin/zopectl run foo.py > installing product PluginIndexes...OK! > installing product Conflicter... BZZT > Traceback (most recent call last): > File "", line 1, in ? > File "/usr/local/lib/python2.3/site-packages/PIL/__init__.py", line 51, in > app > > > File "/usr/local/lib/python2.3/site-packages/PIL/__init__.py", line 47, in > startup > > File "/zope/Zope-2.7.3-SoftwareHome/lib/python/Zope/App/startup.py", line > 96, > in startup > File "/zope/ZopeSoftwareHome/lib/python/OFS/Application.py", line 279, in > init > ialize > initializer.initialize() > File "/zope/ZopeSoftwareHome/lib/python/OFS/Application.py", line 306, in > init > ialize > self.install_products() > File "/zope/ZopeSoftwareHome/lib/python/OFS/Application.py", line 561, in > inst > all_products > return install_products(app) > File "/zope/ZopeSoftwareHome/lib/python/OFS/Application.py", line 597, in > inst > all_products > folder_permissions, raise_exc=debug_mode) > File "/zope/ZopeSoftwareHome/lib/python/OFS/Application.py", line 803, in > inst > all_product > get_transaction().commit() > File "/zope/Zope-2.7.3-SoftwareHome/lib/python/ZODB/Transaction.py", line > 252, > in commit > File "/zope/Zope-2.7.3-SoftwareHome/lib/python/ZODB/Connection.py", line > 738, > in tpc_vote > File "/zope/Zope-2.7.3-SoftwareHome/lib/python/ZEO/ClientStorage.py", line > 841 > , in tpc_vote > File "/zope/Zope-2.7.3-SoftwareHome/lib/python/ZEO/ClientStorage.py", line > 825 > , in _check_serials > ZODB.POSException.ConflictError: database conflict error (serial this txn > starte > d with 0x035b25f36751f988 2005-02-10 18:59:24.215675, serial currently > committed > 0x035b743c6d186822 2005-02-24 17:00:25.569220) > > > I guess what I'm asking, before I hack more on OFS/Application.py, > is this: > > * any ideas re. what on earth could be causing this ConflictError? > the product doesn't DO anything fer chrissakes. > (Maybe registration is finishing too fast??) > > * Is the PIL stuff in the traceback a red herring or what? > ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Conflict errors on startup, revisited
This is a topic that comes up occasionally, but since the symptom seems to be rare and hard to provoke, it never gets resolved. Well it's biting me again and I'm determined to fix it, but would like some input from the list. Related collector issues: http://www.zope.org/Collectors/Zope/300 http://www.zope.org/Collectors/Zope/1201 The issue: On rare occasions I have seen ConflictErrors raised during product installation. Today, I can finally provoke them at will with a pretty minimal setup. The setup: Zope 2.7.3, using ZEO, with DirectoryStorage on the ZEO server. Hardware: dual-processor PIII 800, some kinda scsi disk, ext3, some old redhat flavor of linux. To provoke it I need merely do ./bin/zopectl run foo.py ... where foo.py is empty. Products/ now contains only one dummy product, which I've named Conflictor, which contains only an empty __init__.py. I get a ConflictError during registration of Conflictor. Interestingly I can only provoke this ConflictError in this particular INSTANCE_HOME. I have several other instances and am not seeing the symptom there. I started trivially hacking OFS/Application.py to add some debugging output (a diff against 2.7.3 is attached), and here's what I get: [EMAIL PROTECTED] PaulMiniInstanceHome]$ ls Products/ [EMAIL PROTECTED] PaulMiniInstanceHome]$ mkdir Products/Conflicter [EMAIL PROTECTED] PaulMiniInstanceHome]$ touch Products/Conflicter/__in it__.py [EMAIL PROTECTED] PaulMiniInstanceHome]$ touch foo.py [EMAIL PROTECTED] PaulMiniInstanceHome]$ ./bin/zopectl run foo.py installing product PluginIndexes...OK! installing product Conflicter... BZZT Traceback (most recent call last): File "", line 1, in ? File "/usr/local/lib/python2.3/site-packages/PIL/__init__.py", line 51, in app File "/usr/local/lib/python2.3/site-packages/PIL/__init__.py", line 47, in startup File "/zope/Zope-2.7.3-SoftwareHome/lib/python/Zope/App/startup.py", line 96, in startup File "/zope/ZopeSoftwareHome/lib/python/OFS/Application.py", line 279, in init ialize initializer.initialize() File "/zope/ZopeSoftwareHome/lib/python/OFS/Application.py", line 306, in init ialize self.install_products() File "/zope/ZopeSoftwareHome/lib/python/OFS/Application.py", line 561, in inst all_products return install_products(app) File "/zope/ZopeSoftwareHome/lib/python/OFS/Application.py", line 597, in inst all_products folder_permissions, raise_exc=debug_mode) File "/zope/ZopeSoftwareHome/lib/python/OFS/Application.py", line 803, in inst all_product get_transaction().commit() File "/zope/Zope-2.7.3-SoftwareHome/lib/python/ZODB/Transaction.py", line 252, in commit File "/zope/Zope-2.7.3-SoftwareHome/lib/python/ZODB/Connection.py", line 738, in tpc_vote File "/zope/Zope-2.7.3-SoftwareHome/lib/python/ZEO/ClientStorage.py", line 841 , in tpc_vote File "/zope/Zope-2.7.3-SoftwareHome/lib/python/ZEO/ClientStorage.py", line 825 , in _check_serials ZODB.POSException.ConflictError: database conflict error (serial this txn starte d with 0x035b25f36751f988 2005-02-10 18:59:24.215675, serial currently committed 0x035b743c6d186822 2005-02-24 17:00:25.569220) I guess what I'm asking, before I hack more on OFS/Application.py, is this: * any ideas re. what on earth could be causing this ConflictError? the product doesn't DO anything fer chrissakes. (Maybe registration is finishing too fast??) * Is the PIL stuff in the traceback a red herring or what? -- Paul Winkler http://www.slinkp.com *** Application.py.ORIG Thu Feb 24 14:27:23 2005 --- Application.py Thu Feb 24 14:32:24 2005 *** *** 34,39 --- 34,40 from Acquisition import aq_base from App.Product import doInstall from App.config import getConfiguration + from ZODB.POSException import ConflictError class Application(Globals.ApplicationDefaultPermissions, ZDOM.Root, Folder.Folder, *** *** 587,594 if done.has_key(product_name): continue done[product_name]=1 ! install_product(app, product_dir, product_name, meta_types, ! folder_permissions, raise_exc=debug_mode) Products.meta_types=Products.meta_types+tuple(meta_types) Globals.default__class_init__(Folder.Folder) --- 588,606 if done.has_key(product_name): continue done[product_name]=1 ! # collector issue 300: handle ConflictErrors during installation ! sys.stderr.write('installing product %s' % product_name) ! try: ! import time; time.sleep(0.1) ! app._p_jar.sync() # DM's suggestion ! install_product(app, product_dir, product_name, meta_types, ! folder_permissions, raise_exc=debug_mode) ! sys.stderr.write('...OK!\n') ! except ConflictError: ! # somehow we should retry it. XXX look at zpublisher ! sys.stderr.write('... BZZT\n') !