Re: [ZODB-Dev] mvcc related error?
Dieter Maurer wrote: Chris Withers wrote at 2007-3-16 08:45 +: ... Is there any way an object could be invalidated without there being a non-current revision to read? Sure (through a call to ZODB.DB.DB.invalidate), although usually it is done only after the object changed. OK. I'm still not clear on whether what you describe is a bug or simply what might happen now? We are using it to cause objects to be flushed from all ZODB connection caches (because we know that their _v_ might contain an inconsistent state). Fortunately, we did not yet met the loadBefore problem you have hit. Oh, so it's just me who's lucky? ;-) What are the possible causes of invalidation? A change of the object or some explicit call for invalidation. I don't think any explicit calls are involved here... cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] mvcc related error?
On 3/20/07, Chris Withers [EMAIL PROTECTED] wrote: Dieter Maurer wrote: Chris Withers wrote at 2007-3-16 08:45 +: ... Is there any way an object could be invalidated without there being a non-current revision to read? Sure (through a call to ZODB.DB.DB.invalidate), although usually it is done only after the object changed. OK. I'm still not clear on whether what you describe is a bug or simply what might happen now? The MVCC implementation might not be prepared to cope with an explicit call to invalidate. Yould also achieve this by calling some method on the object, right? _p_invalidate()? I don't remember the details. In either case, I suspect that is a bug in the implementation, but it's not clear if it explains your problem. Jeremy We are using it to cause objects to be flushed from all ZODB connection caches (because we know that their _v_ might contain an inconsistent state). Fortunately, we did not yet met the loadBefore problem you have hit. Oh, so it's just me who's lucky? ;-) What are the possible causes of invalidation? A change of the object or some explicit call for invalidation. I don't think any explicit calls are involved here... cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] mvcc related error?
Jeremy Hylton wrote: transaction end committed. If end is None, it implies that the revision returned by loadBefore() is the current revision. There is an assert here, because the _setstate_noncurrent() is only called if the object is in the invalidated set, which implies that there is a non-current revision to read. Is there any way an object could be invalidated without there being a non-current revision to read? What are the possible causes of invalidation? If I had to guess, I'd say it was a bug in loadBefore(). It looks like the only ways for loadBefore() to return None for end are - The very first record for the object has a transaction id less than the tid argument. If so, end_tid is never set. Not sure this is compatible with the object being in the invalidated set. Since I'm not using versions, this is where I'd point the finger. Perhaps the reasoning about invalidated sets and transaction ids is wrong in the presence of versions. Well, there are no versions here, so I can't help with that ;-) cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] mvcc related error?
Dieter wrote: Unfortunately, neither of these means anything to me ;-) That is because you did not look at the code :-) Much as I wish I had time to read and learn the whole zodb code base, I don't. It wasn't clear what that code did and what those assertions really meant... Jim wrote: I'm glad you brought that up. I'd like to set up a project in Launchpad. https://bugs.launchpad.net/zodb/+bug/92507 cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] mvcc related error?
On 3/15/07, Chris Withers [EMAIL PROTECTED] wrote: Dieter wrote: Unfortunately, neither of these means anything to me ;-) That is because you did not look at the code :-) Much as I wish I had time to read and learn the whole zodb code base, I don't. It wasn't clear what that code did and what those assertions really meant... The code in question has some docstrings that explain the basic idea. You certainly don't need to read the whole codebase. _setstate_noncurrent(obj) attempts to load the state of obj that was current before the transaction started (technically, before _txn_time). loadBefore() returns a 3-tuple including the transaction ids the delimit the lifetime of this particular revision of the object. It was written by transaction start and was current until transaction end committed. If end is None, it implies that the revision returned by loadBefore() is the current revision. There is an assert here, because the _setstate_noncurrent() is only called if the object is in the invalidated set, which implies that there is a non-current revision to read. If I had to guess, I'd say it was a bug in loadBefore(). It looks like the only ways for loadBefore() to return None for end are - The very first record for the object has a transaction id less than the tid argument. If so, end_tid is never set. Not sure this is compatible with the object being in the invalidated set. - Something is happening with versions. Are you using versions? It seems likely that there are bugs here. - There's a bug in the code that reads the data record from the storage where it reads None for a transaction id. That seems very unlikely. Perhaps the reasoning about invalidated sets and transaction ids is wrong in the presence of versions. MVCC should not work with versions, but I don't see code that will abort the loadBefore() call if the Connection has a version. You aren't using versions, are you? Jeremy Jeremy Jim wrote: I'm glad you brought that up. I'd like to set up a project in Launchpad. https://bugs.launchpad.net/zodb/+bug/92507 cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] mvcc related error?
Dieter Maurer wrote: Yes, it looks like an error: Apparently, assert end is not None failed. Apparently storage.loadBefore returned a wrong value. Unfortunately, neither of these means anything to me ;-) I guess I should file a bug report? Why collector? cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] mvcc related error?
Chris Withers wrote at 2007-3-14 10:18 +: Dieter Maurer wrote: Yes, it looks like an error: Apparently, assert end is not None failed. Apparently storage.loadBefore returned a wrong value. Unfortunately, neither of these means anything to me ;-) That is because you did not look at the code :-) I guess I should file a bug report? Yes. Why collector? Formerly, the Zope collector was the right one -- with topic database. Not sure, whether this has changed recently. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] mvcc related error?
On Mar 14, 2007, at 1:43 PM, Dieter Maurer wrote: Chris Withers wrote at 2007-3-14 10:18 +: Dieter Maurer wrote: Yes, it looks like an error: Apparently, assert end is not None failed. Apparently storage.loadBefore returned a wrong value. Unfortunately, neither of these means anything to me ;-) That is because you did not look at the code :-) I guess I should file a bug report? Yes. Why collector? Formerly, the Zope collector was the right one -- with topic database. Not sure, whether this has changed recently. I'm glad you brought that up. I'd like to set up a project in Launchpad. I'll do that and write back here. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] mvcc related error?
(trying again to send to the right list) Hi All, One of the users on one of my projects saw this error under high load: Module Products.QueueCatalog.QueueCatalog, line 458, in reindexObject Module Products.QueueCatalog.QueueCatalog, line 341, in catalog_object Module Products.QueueCatalog.QueueCatalog, line 284, in _update Module ZODB.Connection, line 732, in setstate Module ZODB.Connection, line 765, in _setstate Module ZODB.Connection, line 791, in _load_before_or_conflict Module ZODB.Connection, line 814, in _setstate_noncurrent AssertionError This is on Zope 2.9.4. Any ideas? cheers, Chris ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] mvcc related error?
Chris Withers wrote at 2007-3-13 11:34 +: One of the users on one of my projects saw this error under high load: Module Products.QueueCatalog.QueueCatalog, line 458, in reindexObject Module Products.QueueCatalog.QueueCatalog, line 341, in catalog_object Module Products.QueueCatalog.QueueCatalog, line 284, in _update Module ZODB.Connection, line 732, in setstate Module ZODB.Connection, line 765, in _setstate Module ZODB.Connection, line 791, in _load_before_or_conflict Module ZODB.Connection, line 814, in _setstate_noncurrent AssertionError Yes, it looks like an error: Apparently, assert end is not None failed. Apparently storage.loadBefore returned a wrong value. I have no idea how this could happen. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
