[ZODB-Dev] Re: [Zope-dev] Re: [Bug] ZODB invalidation processing
one very important finding: tryToResolveConflict fails in the resolve function resolve built-in method _p_resolveConflict of BTrees._IOBTree.IOBucket object at 0xb1ab82b4 by raising an exception, when I call it again from the debugger I get. (Pdb) resolved = resolve(old, committed, newstate) *** BTreesConflictError: BTrees conflict error at -1/47/47: Conflicting inserts So that is the real error. -- Gruß Joachim ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Bug] ZODB invalidation processing
Tres Seaver schrieb: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Not if the sessions being used are from 'faster' -- it doesn't use IOBTree. The major application use of that module is in the catalog. you correct see below: Try dumping out the contents of the bucket: for k, v in bucket.items(): print k, type(v) resolve built-in method _p_resolveConflict of BTrees._IOBTree.IOBucket object at 0xb1ab82b4 print root._p_jar[p64(0xb1ab82b4)] *** POSKeyError: ZODB.POSException.POSKeyError instance at 0xa8f9e6cc with the recepies here http://www.zopelabs.com/cookbook/1114086617 I was able to get the information about the oid, which is passed to tryToResolveConflict, here is the result BTrees._IOBTree.IOBTree object at 0xb562fadc DateIndex at created Products.ZCatalog.Catalog.Catalog object at 0xb12d622c CatalogTool at portal_catalog CPSDefaultSite at uniben Application at {'Application': Application at , 'ZGlobals': BTrees._OOBTree.OOBTree object at 0xb2739224} What does this tell us ? -- Gruß Joachim ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Concurrent transactions
Counting in the ZODB is more or less a no-go. You will get write conflicts, and your ZODB will grow (too) quickly. That said, you may want to look at BTrees.Length for a counter with built-in conflict resolution. Stefan On 1. Jun 2007, at 09:47, Kai Diefenbach wrote: After reading some documentation about transactions, concurrency and atomicity, I don't think that the following issue is a problem, but I'd be nice to get confirmation from some exports. Or - of course - that it doesn't behave in the way I think. Within an adapter I increase a counter (e.g. for every download of an file) by doing: self.counter[0] += 1 self.counter is a PersistentList stored as annotation. Could there be a problem with concurrent access to the counter? With other words: -- Anything that, in happening, causes something else to happen, causes something else to happen. --Douglas Adams ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: Concurrent transactions
Christian Theune [EMAIL PROTECTED] wrote: Am Freitag, den 01.06.2007, 09:47 +0200 schrieb Kai Diefenbach: A gets counter = 5 B gets counter (before A writes the increased counter) = 5 A increases and writes the counter = 6 B increases and writes the counter = 6 (*) *) In my understanding, here a ConflictError is raised and the whole transaction will be repeated, that means it starts with the counter of 6. Is this right? Yes, in Zope. That's nothing ZODB does. ZODB only raises a conflict error and aborts the transaction. Ah, thanks! And you can make it perform better by implementing application level conflict resolution so your counter knows that if two transactions moved from 5 to 6 concurrently the real value should be 7. This is *the* example for when application level conflict resolution is helpful. But: Your ALC code must be available on the ZEO server if needed. Okay, sounds interesting. I check this out. Thanks! Kai -- Kai Diefenbach - http://diefenba.ch iqplusplus - http://iqpp.de ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: Concurrent transactions
Stefan H. Holek [EMAIL PROTECTED] wrote: Counting in the ZODB is more or less a no-go. You will get write conflicts, and your ZODB will grow (too) quickly. But what's an alternative? In general I don't like the thought to use a RDBMS in parallel to Zope's ZODB approach. And more than ever for a counter. That said, you may want to look at BTrees.Length for a counter with built-in conflict resolution. Looks good. Thanks for this hint. -- Kai Diefenbach - http://diefenba.ch iqplusplus - http://iqpp.de ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Todo: Zope support for Slots and Decorators?
On May 31, 2007, at 6:13 PM, Chris Spencer wrote: A couple questions. Does ZODB support classes using __slots__? Much to my surprise, it seems to. I just tried it and it worked just fine. :) Also, are there any problems with using decorators with ZODB? Not that I'm aware of. There is one surprise. If you use a descriptor, that you might install via a decorator, to moderate writes, a setattr will be viewed by a base class as as causing a write, even if the descriptor elects not to modify state. I'd have to think hard about whether this might be considered a bug (or perhaps even a missfeature). For instance, a simple memoization decorator like http://paste.plone.org/5416 I can't see this. appears to work in simple tests, but are there any potential issues I should be aware of? Other than what I mention above, I'm not aware of any. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Bug] ZODB invalidation processing
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Joachim Schmitz wrote: Tres Seaver schrieb: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Not if the sessions being used are from 'faster' -- it doesn't use IOBTree. The major application use of that module is in the catalog. you correct see below: Try dumping out the contents of the bucket: for k, v in bucket.items(): print k, type(v) resolve built-in method _p_resolveConflict of BTrees._IOBTree.IOBucket object at 0xb1ab82b4 print root._p_jar[p64(0xb1ab82b4)] *** POSKeyError: ZODB.POSException.POSKeyError instance at 0xa8f9e6cc with the recepies here http://www.zopelabs.com/cookbook/1114086617 I was able to get the information about the oid, which is passed to tryToResolveConflict, here is the result BTrees._IOBTree.IOBTree object at 0xb562fadc DateIndex at created Products.ZCatalog.Catalog.Catalog object at 0xb12d622c CatalogTool at portal_catalog CPSDefaultSite at uniben Application at {'Application': Application at , 'ZGlobals': BTrees._OOBTree.OOBTree object at 0xb2739224} What does this tell us ? That is a real conflict: both transactions have inserted values into the 'created' date index's '_index' under the same key, which can't be resolved. Retrying the transaction is the only choice here. Because DateIndexes convert the indexed value to an integer with precision of one minute, a date index on 'created' is fairly likely to generate such conflicts when two parties both create content at the same time. Ideally, one would examine the two values being inserted, note that they were both IITreeSet instances containing one int apiece, and exploit our knowledge of the application semantics to merge them, removing the conflict; however, *because* they are IITreeSets, and therefore separate persistent objects, their state is not available to the bucket's '_p_resolveConflict', which must therefore lose. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGYBDQ+gerLs4ltQ4RAoRNAKCb86Bjhp5fuk7bp9OV2IMUXDKm7ACeO/aH hVfzx/U0rXsM3iNT2fOl2As= =egtx -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Concurrent transactions
On Jun 1, 2007, at 6:07 AM, Kai Diefenbach wrote: Stefan H. Holek [EMAIL PROTECTED] wrote: Counting in the ZODB is more or less a no-go. You will get write conflicts, and your ZODB will grow (too) quickly. I kind of doubt that updating a little counter will make the database grow too quickly, but that might depend on how often the counter was updated and how you defined too. :) An issue with current ZODB storage implementations is that they require explicit packing to remove old revisions and, in the case of the file storage implementation, packing is rather expensive. (I think the FileStorage pack implementation could be improved quite a bit in this regard.) If you have a very busy counter, you might have to pack more often then you otherwise would. But what's an alternative? That depends on the underlying use case. If you want to track traffic, there's always log analysis. :) In general I don't like the thought to use a RDBMS in parallel to Zope's ZODB approach. And more than ever for a counter. If the counter if updated a lot, then transactional databases, including most RDBMSs and ZODB are likely to be too slow, again, for some definition of too. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: [Zope-dev] Re: [Bug] ZODB invalidation processing
some more findings: 1. The conflict error really happens on the Portalcatalog 2. It is a BTreesConflictError: BTrees conflict error at -1/47/47: Conflicting inserts which disguised as ConflictError, through the various try: excepts: 3. It happens on DateIndexes like created and modified. 4. I put for a test: try: resolved = resolve(old, committed, newstate) except: return invalidated into tryToResolveConflict and into the store-method of FileStorage: if serial != cached_tid: rdata = self.tryToResolveConflict(oid, cached_tid,serial, data) if rdata is None: raise POSException.ConflictError(oid=oid, serials=(cached_tid, serial), data=data) elif rdata == invalidated: #new for test old = False # pass # With these hacks I get rid of the conflictErrors. But when I delete the created Items I get: 2007-06-01 13:20:33 ERROR Zope.UnIndex DateIndex: unindex_object tried to retrieve set 1075379718 from index created but couldn't. This should not happen. 2007-06-01 13:20:33 ERROR Zope.UnIndex DateIndex: unindex_object tried to retrieve set 1075379838 from index Date but couldn't. This should not happen. 2007-06-01 13:20:33 ERROR Zope.UnIndex DateIndex: unindex_object tried to retrieve set 1075379719 from index modified but couldn't. This should not happen. So obviously I just fix the symptom with this hack. I think the real cause: lies in these lines: if old: cached_tid = self._get_cached_tid(oid) if cached_tid is None: h = self._read_data_header(old, oid) if h.version: if h.version != version: raise VersionLockError(oid, h.version) pnv = h.pnv cached_tid = h.tid if serial != cached_tid: rdata = self.tryToResolveConflict(oid, cached_tid,serial, data) Which is also supported by the fact, that on our live system which is running on 4 zeo-clients we get this traceback: 2007-06-01T12:38:59 ERROR Zope.SiteErrorLog http://uniben.waeup.org/campus/students/F541700/study_course/create_level Traceback (innermost last): Module Zope2.App.startup, line 173, in zpublisher_exception_hook Module ZPublisher.Publish, line 121, in publish Module Zope2.App.startup, line 240, in commit Module transaction._manager, line 96, in commit Module Products.CPSCompat.PatchZODBTransaction, line 175, in commit Module transaction._transaction, line 436, in _commitResources Module ZODB.Connection, line 665, in tpc_vote Module ZEO.ClientStorage, line 893, in tpc_vote Module ZEO.ClientStorage, line 877, in _check_serials ConflictError: database conflict error (oid 0x8fe833, class BTrees._IOBTree.IOBucket, serial this txn started with 0x036e0418ddcccf55 2007-06-01 11:36:51.984384, serial currently committed 0x036e041a3a33e311 2007-06-01 11:38:13.641254) Any suggestion for a temporary fix would be very welcome, since we get about 6000 conflict errors per day now about 15 % unresolved, and they are really killing our portal. -- Gruß Joachim ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Bug] ZODB invalidation processing
Tres Seaver schrieb: BTrees._IOBTree.IOBTree object at 0xb562fadc DateIndex at created Products.ZCatalog.Catalog.Catalog object at 0xb12d622c CatalogTool at portal_catalog CPSDefaultSite at uniben Application at {'Application': Application at , 'ZGlobals': BTrees._OOBTree.OOBTree object at 0xb2739224} What does this tell us ? That is a real conflict: both transactions have inserted values into the 'created' date index's '_index' under the same key, which can't be resolved. Retrying the transaction is the only choice here. Because DateIndexes convert the indexed value to an integer with precision of one minute, really one minute or do you mean second. a date index on 'created' is fairly likely to generate such conflicts when two parties both create content at the same time. But then I wonder how a CMF site could ever work, this error shows up already on my lokal system when to users add something. On our live system it is killing our portal ? I would consider this a bug. Ideally, one would examine the two values being inserted, note that they were both IITreeSet instances containing one int apiece, and exploit our knowledge of the application semantics to merge them, removing the conflict; however, *because* they are IITreeSets, and therefore separate persistent objects, their state is not available to the bucket's '_p_resolveConflict', which must therefore lose. -- Gruß Joachim ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Zope-dev] Re: [Bug] ZODB invalidation processing
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Joachim Schmitz wrote: some more findings: 1. The conflict error really happens on the Portalcatalog 2. It is a BTreesConflictError: BTrees conflict error at -1/47/47: Conflicting inserts which disguised as ConflictError, through the various try: excepts: 3. It happens on DateIndexes like created and modified. 4. I put for a test: try: resolved = resolve(old, committed, newstate) except: return invalidated into tryToResolveConflict and into the store-method of FileStorage: if serial != cached_tid: rdata = self.tryToResolveConflict(oid, cached_tid,serial, data) if rdata is None: raise POSException.ConflictError(oid=oid, serials=(cached_tid, serial), data=data) elif rdata == invalidated: #new for test old = False # pass # With these hacks I get rid of the conflictErrors. But when I delete the created Items I get: 2007-06-01 13:20:33 ERROR Zope.UnIndex DateIndex: unindex_object tried to retrieve set 1075379718 from index created but couldn't. This should not happen. 2007-06-01 13:20:33 ERROR Zope.UnIndex DateIndex: unindex_object tried to retrieve set 1075379838 from index Date but couldn't. This should not happen. 2007-06-01 13:20:33 ERROR Zope.UnIndex DateIndex: unindex_object tried to retrieve set 1075379719 from index modified but couldn't. This should not happen. So obviously I just fix the symptom with this hack. I think the real cause: lies in these lines: if old: cached_tid = self._get_cached_tid(oid) if cached_tid is None: h = self._read_data_header(old, oid) if h.version: if h.version != version: raise VersionLockError(oid, h.version) pnv = h.pnv cached_tid = h.tid if serial != cached_tid: rdata = self.tryToResolveConflict(oid, cached_tid,serial, data) Which is also supported by the fact, that on our live system which is running on 4 zeo-clients we get this traceback: 2007-06-01T12:38:59 ERROR Zope.SiteErrorLog http://uniben.waeup.org/campus/students/F541700/study_course/create_level Traceback (innermost last): Module Zope2.App.startup, line 173, in zpublisher_exception_hook Module ZPublisher.Publish, line 121, in publish Module Zope2.App.startup, line 240, in commit Module transaction._manager, line 96, in commit Module Products.CPSCompat.PatchZODBTransaction, line 175, in commit Module transaction._transaction, line 436, in _commitResources Module ZODB.Connection, line 665, in tpc_vote Module ZEO.ClientStorage, line 893, in tpc_vote Module ZEO.ClientStorage, line 877, in _check_serials ConflictError: database conflict error (oid 0x8fe833, class BTrees._IOBTree.IOBucket, serial this txn started with 0x036e0418ddcccf55 2007-06-01 11:36:51.984384, serial currently committed 0x036e041a3a33e311 2007-06-01 11:38:13.641254) Any suggestion for a temporary fix would be very welcome, since we get about 6000 conflict errors per day now about 15 % unresolved, and they are really killing our portal. A workaround might be to replace the DateIndexes for 'created' and 'modified' with FieldIndexes: you will see a big jump in the number of keys in the index, but (perhaps) a reduction in conflicts (altheough there will be more bucket splits, which can also cause conflicts). A longer term solution might be to come up with a derived index type which used non-persistent objects (rather than the IITreeSet used by UnIndex) for the set of RIDs stored under each 'minute': you could then inspect the state and do more aggressive conflict resolution than the stock implementation allows (see my other post). Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGYB4w+gerLs4ltQ4RAhsLAJ9/TR4unlzPMQBY7gfpT6WmQZovywCfQrAF EnkttfiHM7gk1ENrm+ineA4= =oJqE -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Concurrent transactions
Kai Diefenbach wrote: Within an adapter I increase a counter (e.g. for every download of an file) by doing: self.counter[0] += 1 self.counter is a PersistentList stored as annotation. Note that if you use a PersistentList, then the entire list must be written to the database each time a non-Persistent element changes (or an item is added or removed from the list). If the list is of any appreciably size, that will be bad. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Bug] ZODB invalidation processing
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Joachim Schmitz wrote: Tres Seaver schrieb: BTrees._IOBTree.IOBTree object at 0xb562fadc DateIndex at created Products.ZCatalog.Catalog.Catalog object at 0xb12d622c CatalogTool at portal_catalog CPSDefaultSite at uniben Application at {'Application': Application at , 'ZGlobals': BTrees._OOBTree.OOBTree object at 0xb2739224} What does this tell us ? That is a real conflict: both transactions have inserted values into the 'created' date index's '_index' under the same key, which can't be resolved. Retrying the transaction is the only choice here. Because DateIndexes convert the indexed value to an integer with precision of one minute, really one minute or do you mean second. Yes, really one minute. a date index on 'created' is fairly likely to generate such conflicts when two parties both create content at the same time. But then I wonder how a CMF site could ever work, You only see conflicts on this index when two people try to insert content at the same time (i.e., they both start from the same base transaction ID). this error shows up already on my lokal system when to users add something. On our live system it is killing our portal ? I would consider this a bug. Perhaps the QueueCatalog product would help you: that would allow you to defer updates to the conflict-prone indexes (created, modified, plus any {ZC,}TextIndexes) for handling by a separate batch process: http://svn.zope.org/Products.QueueCatalog It looks as though you may need the trunk version for compatibility with Zope = 2.9. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGYCDR+gerLs4ltQ4RAuI8AJ9eRg+ZIdh3Cdqog3adXQuSIdByHgCeJN0M 8Qb5I8saP9W0eIy+5/OlAmg= =e5qN -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: Concurrent transactions
Jim Fulton [EMAIL PROTECTED] wrote: On Jun 1, 2007, at 6:07 AM, Kai Diefenbach wrote: Stefan H. Holek [EMAIL PROTECTED] wrote: Counting in the ZODB is more or less a no-go. You will get write conflicts, and your ZODB will grow (too) quickly. I kind of doubt that updating a little counter will make the database grow too quickly, but that might depend on how often the counter was updated and how you defined too. :) I didn't want to answer to Stefan before I tested this again. The first time I tested I remember that I didn't consider the growing of the db as to much. An issue with current ZODB storage implementations is that they require explicit packing to remove old revisions and, in the case of the file storage implementation, packing is rather expensive. (I think the FileStorage pack implementation could be improved quite a bit in this regard.) If you have a very busy counter, you might have to pack more often then you otherwise would. I see. But what's an alternative? That depends on the underlying use case. If you want to track traffic, there's always log analysis. :) As I think about that, this could be true for more uses cases as one might believe :-) Thanks Kai -- Kai Diefenbach - http://diefenba.ch iqplusplus - http://iqpp.de ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Todo: Zope support for Slots and Decorators?
[Chris Spencer] A couple questions. Does ZODB support classes using __slots__? [Jim Fulton] Much to my surprise, it seems to. I just tried it and it worked just fine. :) FYI, this is intentional :-), and was part of the changes to make ZODB play nicely with Python's new-style classes. It wasn't tested heavily, but there are a few very simple __slot__ doctests in persistent/tests/persistent.txt. Chris, you should add more tests ;-) ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Bug] ZODB invalidation processing
We also run into this kind of problems. The only save way to solve it was to serialize our index updates. We do this by using one zope instance which runs only a lovely.remotetask task service. This service contains an indexing task. When indexing is needed we just create a new indexing job which is stored in the job list of the remote task service. This solves two problems : - conflicts - low speed because of complex index value calculations Btw the setup for this application contains 18 zope's each zope running one thread. 17 zope's are used to handle browser requests and one is used to handle the remote tasks (not only indexing tasks). Jürgen Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Joachim Schmitz wrote: Any suggestion for a temporary fix would be very welcome, since we get about 6000 conflict errors per day now about 15 % unresolved, and they are really killing our portal. A workaround might be to replace the DateIndexes for 'created' and 'modified' with FieldIndexes: you will see a big jump in the number of keys in the index, but (perhaps) a reduction in conflicts (altheough there will be more bucket splits, which can also cause conflicts). Unfortunately that does not work the FieldIndex also gives an conflict-error FieldIndex at Date Products.ZCatalog.Catalog.Catalog object at 0xb250a6ec CatalogTool at portal_catalog CPSDefaultSite at uniben Application at {'Application': Application at , 'ZGlobals': BTrees._OOBTree.OOBTree object at 0xb242989c} 2007-06-01 20:23:41 INFO ZPublisher.Conflict ConflictError at /uniben/campus/students/A923157/study_course/create_level: database conflict error (oid 0x36c429, class BTrees._OOBTree.OOBTree, serial this txn started with 0x036e0590d58661ee 2007-06-01 17:52:50.044906, serial currently committed 0x036e059346ef95dd 2007-06-01 17:55:16.625597) (1 conflicts (0 unresolved) since startup at Fri Jun 1 19:54:43 2007) A longer term solution might be to come up with a derived index type which used non-persistent objects (rather than the IITreeSet used by UnIndex) for the set of RIDs stored under each 'minute': you could then inspect the state and do more aggressive conflict resolution than the stock implementation allows (see my other post). any other shortterm suggestions. QueueCatalog? Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGYGZU+gerLs4ltQ4RAk4BAJ46DKm1vLlygIqee1OxUjSPYF40pwCfSLvy mS+9UyTtv0OuNWuotzqk5Tg= =Fevy -END PGP SIGNATURE- ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Bug] ZODB invalidation processing
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jürgen Kartnaller wrote: We also run into this kind of problems. The only save way to solve it was to serialize our index updates. We do this by using one zope instance which runs only a lovely.remotetask task service. This service contains an indexing task. When indexing is needed we just create a new indexing job which is stored in the job list of the remote task service. That is exactly how QueueCatalog functions: it batches expensive / conflict-prone indexing operations into jobs which are handled later, in a serialized way. This solves two problems : - conflicts - low speed because of complex index value calculations Btw the setup for this application contains 18 zope's each zope running one thread. 17 zope's are used to handle browser requests and one is used to handle the remote tasks (not only indexing tasks). How many of the front-end servers do writes? Or are all your visitors potential writers? Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGYOWr+gerLs4ltQ4RAmpyAJwPxdx+ob46RgvasStZwoPWQQTa8gCdG/+t 05J+ocnTgZVOnf7MvLw3oGw= =kyUn -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Bug] ZODB invalidation processing
Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jürgen Kartnaller wrote: We also run into this kind of problems. The only save way to solve it was to serialize our index updates. We do this by using one zope instance which runs only a lovely.remotetask task service. This service contains an indexing task. When indexing is needed we just create a new indexing job which is stored in the job list of the remote task service. That is exactly how QueueCatalog functions: it batches expensive / conflict-prone indexing operations into jobs which are handled later, in a serialized way. This solves two problems : - conflicts - low speed because of complex index value calculations Btw the setup for this application contains 18 zope's each zope running one thread. 17 zope's are used to handle browser requests and one is used to handle the remote tasks (not only indexing tasks). How many of the front-end servers do writes? Or are all your visitors potential writers? All servers do writes. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGYOWr+gerLs4ltQ4RAmpyAJwPxdx+ob46RgvasStZwoPWQQTa8gCdG/+t 05J+ocnTgZVOnf7MvLw3oGw= =kyUn -END PGP SIGNATURE- ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev